Roadmap to Become an AI Engineer
Skills, learning resources, and project ideas to become an AI Engineer in 2024
AI is the new paradigm to build all technology.
After establishing how AI Engineer is turning out to be the next big tech role, it’s time to learn how to become one.
This post explains:
- the roadmap one can follow to go from being a Software Engineer to AI Engineer.
- required skills to become an AI Engineer.
- sample tools that you should learn to use while developing those skills.
- the best way to develop those skills.
Let’s unveil the roadmap first!
Roadmap to Become an AI Engineer 🧭
First things first!
Who is it for?
Programmers / SDEs / Analysts / Data Scientists planning to upskill for AI Engineering.
Since this is a core engineering skill, following are the pre-requisites:
- intermediate level understanding of Python / JS programming.
- you must ideally have experience of coding at least 2–3 moderately complex applications like a blogging web app in Flask or Rails or Node.js.
- you are at least comfortable reading the documentation to build.
- you are comfortable coding in an IDE like VS Code.
- working with git and GitHub, although important, can be learnt while working on a project.
Breaking down the roadmap
I have broken down the entire AI Engineering track into 3 stages, as shown, you go from left to right in this roadmap i.e. from beginner to intermediate to advanced.
Here’s what each stage represents:
- Beginner ( <= 1 month) — Build basic application to learn to consume LLM APIs, carefully engineering prompts for your applications and working with open source LLMs.
- Intermediate ( ~ 2 months) — Go deep into building more context aware advanced applications using Retreival Augmented Generation (RAG). Learn about vector databases and how to work with one. Learn to build agents with LLMs and tools.
- Advanced (~ 3 months) — After mastering application building, learn to deploy, optimize, and manage LLM-powered apps in production with LLMOps. Learn to fine-tune pre-trained models to adapt to downstream applications efficiently and at low cost.
Beginner-level skills are as follows:
- understand the basics of LLM — you should just know how ChatGPT works at a high level.
- learn Prompt Engineering for Developers. How to write prompts to improve the response of an LLM.
- learn to consume data from APIs, learn to work with JSON data.
- learn to call closed and open-source LLM models, function calling, passing prompts, and parsing response.
- learn to manage context space from the conversation.
- learn to create and automate a sequence of operations — Chains using langchain.
- basic app development using Gradio or Streamlit for POCs and demos.
- deploy your apps to make them accessible — basic deployment on HuggingFace Space or Streamlit cloud.
- multimodal generation — code, images, audio using HuggingFace
transformerlibrary.
Intermediate-level projects require:
- understanding vector embedding and vector databases.
- learning how to use vector databases for your application.
- Building retrieval-augmented Generation (RAG) — chat with your knowledge base.
- developing advanced RAG pipelines like a sub-question query engine that can provide responses after going through multiple data sources.
- building agents — iterative workflows to finish off a big task.
- building multi-agent applications where more than one agent work together to provide better solution.
- automation with multiple Agents — Autogen and Crew AI
- evaluating RAGs — RAGAs framework.
- managing databases, retreival, deploying complete applications, versioning, logging and monitoring model behavior.
Advanced projects will need you to:
- fine-tuning pre-trained LLMs for domain-specific knowledge and tailored responses like medical research, financial research, legal analysis.
- curate datasets and engineer (ETL pipelining) pipelines for model finetuning.
- evaluate and benchmark the model performance
- LLMOps — build complete e2e pipelines with model registry, observability and automated testing.
- build multi-modal applications — hybrid semantic search with text and image
- build SDKs, packages and custom solutions to enable other developers
- secure your AI applications using techniques like prompt hacking and incorporating defensive measures by checking against vulnerabilities and potential risks.
Here’s how your skills should progress with enough time spent building apps and tuning models:
Now, how to develop these skills?
Learning resources, references and projects 📚
AI is buzzing with learning resources and tutorials.
There are many good resources to learn all these concept and there are many good folks who are pushing out very deep and detailed material on each of these topics.
It will be very hard to capture all the resources here, so I’ve created a separate resource list here.
The best way to learn — BUILD!
We have learnt what we are supposed to do, let’s talk about the best way to do it.
Work on projects.
The best way to not just learn but really grok these concepts. It will further sharpen the skill to think about cutting edge use cases.
You can find project ideas in the same repository.
Besides that, I will keep sharing projects and tutorials on my YouTube channel.
For first steps, I’d recommend checking out this project-based course that I’ve built where you build LLM-powered applications including RAG pipelines that are served as WhatsApp chatbots.
Video version of this post:
Don’t forget to join my Discord community!
Besides learning, there are a lot of announcements and developments being made on a daily basis in the AI Community, many people are finding interesting use cases and are working on interesting products that might interest you.
If you want to be part of this discussion, do join our community on Discord. It’s called Wiplane.
What’s Next!
I’ll follow the same roadmap and debunk the process, challenges and learnings along the way.
Make sure you’re subscribed to receive those exceptionally detailed posts.
