Julian LaNeve is the Chief Technical Officer (CTO) at Astronomer, the driving pressure behind Apache Airflow and fashionable knowledge orchestration to energy every thing from AI to normal analytics.
Julian does product and engineering at Astronomer the place he focuses on developer expertise, knowledge observability, and AI. He’s additionally the creator of Cosmos, an Airflow supplier for working dbt Core tasks as Airflow DAGs.
He’s captivated with all issues knowledge and open supply as he spends his spare time doing hackathons, prototyping new tasks, and exploring the most recent in knowledge.
May you share your private story of the way you turned concerned with software program engineering, and labored your means as much as being CTO of Astronomer?
I’ve been coding since I used to be in center faculty. For me, engineering has at all times been a terrific artistic outlet: I can provide you with an concept and use no matter know-how’s needed to construct in direction of a imaginative and prescient. After spending a while in engineering, although, I wished to do extra. I wished to know how companies are run, how merchandise are bought and the way groups are constructed –– and I wished to be taught rapidly.
I spent just a few years working in administration consulting at BCG, the place I labored on all kinds of tasks in numerous industries. I realized a ton, however in the end missed constructing merchandise and dealing in direction of a longer-term imaginative and prescient. I made a decision to affix Astronomer’s product administration workforce, the place I might nonetheless work with prospects and construct methods (the issues I loved from consulting), however might additionally get very arms on constructing out the precise product and dealing with know-how.
For some time, I acted as a hybrid PM/engineer –– I’d work with prospects to know the challenges they had been dealing with and design merchandise and options as a PM. Then, I’d take the product necessities and work with the engineering workforce to truly construct out the product or function. Over time, I did this with a bigger set of merchandise at Astronomer, which in the end led to the CTO position I’m now in.
For customers who’re unfamiliar with Airflow, are you able to clarify what makes it the perfect platform to programmatically creator, schedule and monitor workflows?
Apache Airflow is an open-source platform for creating, scheduling, and monitoring batch-oriented workflows. Airflow supplies the workflow administration capabilities which might be integral to fashionable cloud-native knowledge platforms. It automates the execution of jobs, coordinates dependencies between duties, and provides organizations a central level of management for monitoring and managing workflows.
Knowledge platform architects leverage Airflow to automate the motion and processing of knowledge by and throughout numerous programs, managing complicated knowledge flows and offering versatile scheduling, monitoring, and alerting. All of those options are extraordinarily useful for contemporary knowledge groups, however what makes Airflow the perfect platform is that it’s an open-source venture –– which means there’s a group of Airflow customers and contributors who’re continually working to additional develop the platform, resolve issues and share greatest practices.
Airflow additionally has many knowledge integrations with widespread databases, purposes, and instruments, in addition to dozens of cloud providers — and extra are added each month.
How does Astronomer use Airflow for inside processes?
We use Airflow a ton! Naturally, we now have our personal knowledge workforce that makes use of Airflow to ship knowledge to the enterprise and our prospects. They’ve some fairly refined tooling they’ve constructed round Airflow that we’ve used as inspiration for function growth on the broader platform.
We additionally use Airflow for some fairly untraditional use instances, however it performs very properly. For instance, our CRE workforce makes use of Airflow to observe the a whole lot of Kubernetes clusters and 1000’s of Airflow deployments we run on behalf of our prospects. Their pipelines run continually to verify for points, and if we discover any, we’ll open proactive assist tickets on behalf of our prospects.
I’ve even used Airflow for private use instances. My favourite (up to now) was after I was transferring to New York Metropolis. In case you’ve ever lived right here, you’ll know the rental market is loopy. Residences get rented out inside hours of them being listed. My roommates and I had an inventory of standards all of us agreed upon (location, variety of bedrooms, loos, and so on), and I constructed an Airflow DAG that ran each couple of minutes, pulled new listings from varied house itemizing websites, and texted me (thanks Twilio!) each time there was one thing new that matched our standards. The house I’m now residing in was discovered due to Airflow!
Astronomer designed Astro, a contemporary knowledge orchestration platform, powered by Airflow. Are you able to share with us how this software allows corporations to simply place Airflow on the core of their knowledge operations?
Astro allows organizations and extra particularly, knowledge engineers, knowledge scientists, and knowledge analysts, to construct, run, and develop their mission-critical knowledge pipelines on a single platform for all of their knowledge flows. It’s the solely managed Airflow service that gives excessive ranges of knowledge safety and safety and helps corporations scale their deployments and unencumber sources to give attention to their overarching enterprise targets.
One among our prospects, Anastasia, a cutting-edge know-how firm, selected Astro to handle Airflow as a result of they didn’t have sufficient time or sources to keep up Airflow on their very own. Astro works on the again finish so groups can give attention to core enterprise actions, fairly than spending time on undifferentiated actions like managing Airflow.
One of many core elements of Astro is elastic scalability, might you outline what that is and why it’s vital for cloud computing environments?
For us, this simply means our skill to fulfill the compute calls for of our prospects with out working a ton of infrastructure on a regular basis. Our prospects use our platform for all kinds of use instances, nearly all of which have excessive compute necessities (coaching machine studying fashions, processing huge knowledge, and so on). One of many core worth propositions of Astronomer is that, as a buyer, you don’t have to consider the machines working your pipelines. You deploy your pipelines to Astro, and might count on that they work. We’ve constructed a set of options and programs that assist scale our infrastructure to fulfill the altering calls for of our prospects, and it’s one thing we’re excited to maintain constructing upon sooner or later.
You had been accountable for the Astronomer workforce constructing Ask-Astro, the LLM-powered chatbot for Apache Airflow. Are you able to share with us particulars on what’s Ask-Astro and the LLMs that energy it?
Our workforce at Astronomer has among the most educated Airflow group members and we wished to make it simpler to share their information. To do this, we created a reference implementation of Andreessen Horowitz’s Rising Architectures for LLM Functions, which exhibits the commonest programs, instruments, and design patterns they’ve seen utilized by AI startups and complicated tech corporations. We began with some knowledgeable opinions about this reference implementation and Apache Airflow additionally performs a central position within the structure. Ask Astro is a real-life reference to indicate how one can glue all the assorted items collectively.
Ask Astro is extra than simply one other chatbot. The Astronomer workforce selected to develop the appliance within the open and frequently publish about challenges, concepts, and options as a way to develop institutional information on behalf of the group. What had been among the largest challenges that the workforce confronted?
The most important problem was the dearth of clear greatest practices locally. As a result of “state-of-the-art” was redefined each week, it was powerful to know how one can method sure issues (doc ingestion, mannequin choice, output accuracy measurement, and so on). This was a key driver for us to construct Ask Astro within the open. We wished to ascertain a set of practices for LLM orchestration that work properly for varied use instances so our prospects and group might really feel well-prepared to undertake LLMs and generative AI applied sciences.
It’s confirmed to be a terrific alternative –– the software itself will get a ton of utilization, we’ve given a number of public talks on how one can construct LLM purposes, and we’ve even began working with a choose group of shoppers to roll out inside variations of Ask Astro!
What’s your private imaginative and prescient for the way forward for Airflow and Astronomer?
I’m actually enthusiastic about the way forward for each Airflow and Astronomer. The Airflow group continues to develop and at Astronomer, we’re dedicated to fostering its growth, assist and connection throughout groups and people.
With rising demand for data-driven insights and an inflow of knowledge sources, knowledge engineers have a difficult job. We wish to lighten the load for these people and groups by empowering them to combine and handle complicated knowledge at scale. At the moment, this additionally means supporting AI adoption and implementation. In 2023, like many different corporations, we centered on how we are able to speed up AI use for our prospects. Our platform, Astro, accelerates AI deployment, streamlines ML growth, and supplies the sturdy compute energy wanted for next-gen purposes. AI will proceed to catch the attention of us this yr and we’ll assist our prospects as new applied sciences and frameworks emerge.
As well as, Astronomer’s a terrific place to work and develop a profession. As the information panorama continues evolving, working right here will get an increasing number of thrilling. We’re constructing a terrific workforce right here and have numerous technical challenges to unravel. We additionally lately moved our headquarters to New York Metropolis the place we are able to grow to be an excellent higher a part of the tech group that exists there and we’ll be higher outfitted to draw the perfect, most expert expertise within the trade. In case you’re interested by becoming a member of the workforce to assist us ship the world’s knowledge on time, attain out!
Thanks for the nice interview, readers who want to be taught extra ought to go to Astronomer.