AI Engineer- The next big tech role!
Bridging the gap between AI research and engineering
In numbers, there’s probably going to be significantly more AI Engineers than there are ML engineers / LLM engineers. One can be quite successful in this role without ever training anything. — Andrej Karpathy
More than $1 billion in revenue from startups alone, with early signs of success for Gen AI, every forward-thinking tech company is racing to infuse their products, customer support bots, and marketing with Gen AI capabilities. AI, as a technology, is at a junction similar to where the internet was during the late 90s, if not the same.
Increasing demand and builders in AI
To spot the trend, I looked at YCombinator’s portfolio of companies. For those of you who don’t know, YCombinator is a prestigious startup accelerator based in the US. They have backed many startups that have now become tech giants such as Airbnb, Dropbox, Stripe, and Reddit.
And here’s what I got, look at the number of companies building with AI from 2017 to 2023. There has been a significant rise in the number of companies building with AI since 2021 and then it shot up after the launch of ChatGPT in 2022.
This gives some confidence that there are going to be more and more companies building with AI in the near future which as a result will lead to increased demand for AI Engineers.
And, for any programmer, this is the best time to start building and learning.
Why NOW?!
The AI development space has evolved enough with open source LLMs, frameworks, and readily available APIs to get a quick start and the community has grown enough to get the required support.
AI startups, AI accelerator programs, open-source repositories, SDKs, packages, cloud platforms — one common theme — build, solve, and ship, as quickly as you can.
What once required a dedicated research team and years of intensive work can now be efficiently managed with API documentation and a few days of focused effort.
So, how can a builder (one who enjoys building products) or an engineer look to delve into the AI space today?
To answer that question, you should first understand what all is going in the field.
The AI Development Layers
I capture the major areas of development in AI today into 3 different layers.
- Application Development — This layer includes developing applications (interfaces) on top of readily available LLM APIs using some AI Engineering framework like Langchain, LlamaIndex, Autogen, etc., and then monitoring and evaluating your application.
This is the most active and buzzing layer. This is where the money is. The more real the use case, the greater the value. To develop these AI applications, you need expertise in a special set of engineering skills which I will explain later in this article. - Model Development — Going a layer deeper, here we work on everything that can deliver a more optimized model. Engineering the datasets, distributed training, evaluations and benchmarking, and inference serving with a variety of toolings.
This layer requires deep expertise in deep learning, distributed systems, dataset curation, and engineering. - Infrastructure — Underpinning everything is the infrastructure layer encompassing the hardware, cloud service providers, and GPUs, where these large models are trained.
This layer requires deep expertise in computing(OS, networks, security), distributed systems, and of course AI model development.
Besides these, there is another layer that peeks into research towards AI risk and safety alignment to safeguard against rogue AI. Here’s a $10M Superalignment grants program launched by OpenAI earlier this year.
Given the highest traction in the application layer, it has led to an increasing demand for a special kind of engineer who knows how to build on top of AI. While there is no standard term for these engineers most companies are calling them AI Engineers.
While I was writing this essay, Chip Huyen published an amazingly detailed post on 900 most popular open-source AI tools which further reinforces my observations and discovery about the rising need for AI Engineers:
With readily available models, anyone can develop applications on top of them. This layer has seen the most actions in the last 2 years and is still rapidly evolving. This layer is also known as AI engineering. — Chip Huyen
So, how do we define the role of an AI Engineer? Do they need to be experts in AI or Deep Learning?
An AI Engineer is a specialized programmer skilled at leveraging AI technologies to develop comprehensive form-agnostic applications.
“Form-agnostic” refers to the versatility in application type, ranging from simple chat interfaces to complex full-stack applications, Chrome extensions, Python packages, or SDKs.
Unlike AI researchers who delve deep into algorithmic foundations, AI Engineers focus on applying existing AI models to create user-centric products.
But again, the question arises, don’t I need to be an expert in AI to become an AI Engineer?
The short answer is No.
This role does not require exhaustive expertise in AI principles such as understanding the inner workings of a Transformer model, similar to how learning to swim doesn’t necessitate a deep dive into the physics of buoyancy.
While a profound knowledge of Deep Learning and Machine Learning can be advantageous, providing a distinct edge, the current industry demand leans more towards practical application than theoretical research.
So, how do we draw the line between an AI Engineer and an AI Researcher?
AI Engineer vs AI Researcher
The following diagram plots engineering skills like working with APIs against AI research skills like designing model architectures or learning how a transformer works.
An AI Engineer excels in creating AI-powered applications, focusing on maximizing model capabilities and optimizing workflows for large language models (LLMs).
With this diagram, I also think that a more engineering-native profile will shine brighter in this role as compared to an ML-native profile but do share your thoughts in the comments.
You must be wondering if AI Researchers are people who are good with engineering and have deep expertise in AI, why don’t companies hire them over AI Engineers?
The short answer is scarcity which in turn leads to increased cost.
The Next Big Tech role — AI Engineers?
Here are a few interesting insights into how this ecosystem is rapidly evolving with “Models as a service”:
- Demand and Supply Dynamics: All the top LLM researchers are already picked up by giants like Google, OpenAI, Microsoft, and Meta and this scarcity of LLM researchers signals a critical need for AI Engineers. This class of professionals serves as a bridge between cutting-edge research and practical application, ensuring wider accessibility and implementation of AI technologies.
- Rapid Prototyping and Agility: Unlike traditional ML approaches that require a lot of research into whether we need ML for a problem, AI Engineers can quickly prototype and iterate on AI products using readily available model APIs.
- Innovation made easier and faster: Foundation Models demonstrate remarkable adaptability across various tasks with minimal input, making them invaluable for AI Engineers who leverage these capabilities to create innovative solutions beyond the original scope envisioned by researchers.
- Inference optimization to deal with compute constraints: The escalating demand for GPUs and the formation of exclusive compute clusters underscore the importance of AI Engineers who optimize model performance and innovation within these constraints.
While traditional ML problems like recommender systems, fraud detection, and anomaly detection will continue to improve, we have a whole new range of AI Applications to cater to.
Clem Delangue, co-founder of HuggingFace, said:
AI is the new paradigm to build all technology
Thus, we NEED more and more AI Engineers!
Look at this generative AI market map from Sequoia. The application layer is filled with use cases and companies in almost every domain:
Conclusion
To put this all together, we have:
- Callouts from industry leaders and deep experts in AI like Andrej Karpathy, Chip Huyen, and Clem amongst many others.
- Big incubators like YCombinator, VC firms, and Investors have been investing in AI companies and they are going long in AI proving that this is the next big paradigm to build all technology.
- The gap between AI research and engineering needs to be bridged which will be facilitated by AI Engineers.
- The rapidly growing ecosystem for AI-powered applications — new and improved developer tools, readily available APIs, libraries and cloud platforms launching every week. On top of that, we have a growing community to provide the required support.
Hence, this is the time to start building with AI, work on those skills, and get yourself ready for the next big tech role.
Check out the video version of this article on my YouTube channel here:
What’s next?!
In the next post, to be published on April 27, 2024, I will be sharing a comprehensive roadmap to becoming an AI engineer along with the learning resources to develop the required skills.
Follow me here and don’t forget to subscribe if you’d like the next series of AI Engineering tutorials to be delivered right in your inbox.
References:
- Andrej’s tweet: https://twitter.com/karpathy/status/1674873002314563584?lang=en
- Chip’s article on “What I learned from looking at 900 most popular open source AI tools”
- Sequoia’s blog on Generative AI Act two — https://www.sequoiacap.com/article/generative-ai-act-two/
- a16z’s how are consumers using AI — https://a16z.com/how-are-consumers-using-generative-ai/
