- Introduction
- The Anatomy of a Machine Learning System
- Types of Machine Learning
- 3.1. Supervised Learning
- 3.2. Unsupervised Learning
- 3.3. Self-Supervised Learning
- 3.4. Reinforcement Learning
- 3.5. In-Context Learning
- Data Sources
- 4.1. Tabular Data
- 4.2. Event Data
- 4.3. Graph Data
- 4.4. Unstructured Data
- 4.5. API-Scraped Data
- Mutable Data
- A Brief History of Machine Learning Systems
- MLOps and LLMOps
- 7.1. Testing in ML Systems
- 7.2. Core MLOps Principles
- A Unified Architecture for AI Systems: Feature, Training, and Inference (FTI) Pipelines
- ML Frameworks and ML Infrastructure Used in This Book
- Summary
In plain English: Building a one-time prediction model is like cooking a single meal. Building an ML system is like setting up a restaurant kitchen that can serve meals repeatedly with consistent quality.
In technical terms: ML systems automate the entire workflow from data ingestion to predictions, enabling models to generate value continuously rather than making one-off predictions on static datasets.
Why it matters: Organizations need models that deliver ongoing value. A model that predicts once is limited; a system that predicts continuously transforms how businesses operate.
Imagine you've been tasked with producing a financial forecast for the upcoming year. You decide to use machine learning (ML), as there's abundant data available, but the data is scattered across spreadsheets and various tables in the data warehouse. You've done this task for several years at your organization. Every year, the final output has been a PowerPoint presentation showing financial projections. Each year, you trained a new model, it made one prediction, and you were done. Each year, you started from scratch—finding data sources again, requesting access, and digging out last year's Jupyter notebook to update it.
This year, however, you realize it may be worth investing time in building the scaffolding for this project to reduce next year's work. Instead of delivering a PowerPoint, you build a dashboard. Instead of requesting one-off data access, you build feature pipelines that extract historical data from sources and compute the features (and labels) for your model.
💡 Insight
Feature pipelines serve a dual purpose: they compute both historical features for training models and fresh features for making predictions. This architectural pattern eliminates duplicate code and ensures consistency between training and inference.
You have an insight: feature pipelines can compute both historical features (for training) and fresh features (for predictions). After training your model, you connect it to the feature pipelines to make predictions that power your dashboard. Next year, you only need to tweak the ML system by adding, updating, or removing features and retraining the model. You update forecast frequency to quarterly with no extra work. The time saved from grunt work on data sourcing, cleaning, and feature engineering now goes into investigating new ML frameworks and model architectures, resulting in a much-improved financial model.
This example illustrates the difference between:
One-off Prediction ML System
──────────────── ──────────
Static dataset → Automated data reading
Manual feature work → Feature pipelines
Single prediction → Continuous predictions
PowerPoint output → Live dashboard
The key distinction: if you want a model to generate repeated value, it must make predictions more than once. You're not finished when you've evaluated performance on a static test set. Instead, you must build ML pipelines—programs that transform raw data into features, enable easy model retraining, and feed new features to models for continuous value generation.
Progressive Journey: This book takes you from training models on static datasets to building complete ML systems:
- Simple: Decision trees on curated datasets
- Intermediate: Deep learning with dynamic data
- Advanced: LLM-powered agents with real-time data
The most important part is working with dynamic data—moving from static (hand-curated datasets like Kaggle competitions) to batch data (updated hourly, daily, weekly, yearly) to real-time data needed for intelligent interactive applications.
In plain English: ML systems are categorized by how they handle incoming data. Think of it like food delivery: batch systems prepare meals on a schedule (meal prep services), while real-time systems cook on demand (restaurant orders).
In technical terms: ML systems process new prediction data either on a schedule (batch) or in response to real-time user requests (online/interactive).
Why it matters: The processing pattern determines architecture choices, infrastructure requirements, and what kinds of applications you can build.
One main challenge in building ML systems is managing data for both training models and making predictions. We categorize ML systems by how they process new prediction data:
- Batch systems: Make predictions on a schedule
- Real-time systems: Make predictions in response to user requests
Simple - Batch ML System: Spotify's Discovery Weekly is a batch ML system—a recommendation engine that once per week predicts songs you might enjoy and adds them to your playlist. The ML system:
- Reads a batch of data (all 575M+ users)
- Makes predictions using the trained recommender model
- Takes input features (listening frequency, genres)
- Predicts the 30 "best" songs for each user
- Stores predictions in a database (Cassandra)
- Serves recommendations when users log in
Intermediate - Real-Time ML System: TikTok's recommendation engine adapts recommendations in near real-time as you click and watch videos. It predicts which videos to show as you scroll. Andrej Karpathy, ex-head of AI at Tesla, said TikTok's recommendation engine "is scary good. It's digital crack." TikTok was the first online video platform with real-time recommendations, giving them a competitive advantage that helped build the world's second most popular online video platform.
Advanced - Agentic AI System: Lovable is a coding assistant for building web applications from a chat window. It reached $100 million in revenue in just eight months—the fastest-growing software company to this milestone. Lovable is an agentic AI system that takes instructions and uses an LLM to create and run web applications as TypeScript code with CSS styling and optional integrated database.
💡 Insight
Agentic systems have natural language interfaces. You give them high-level goals, and they work with autonomy to achieve them. They can be interactive or batch systems, but the key differentiator is their autonomous decision-making capability.
System Type Flow:
Batch ML System
───────────────
Schedule trigger → Batch data → Model → Predictions → Database → User
Real-Time ML System
───────────────────
User request → Real-time features → Model → Instant prediction → User
Agentic AI System
─────────────────
User goal → LLM reasoning → Tool selection → Context retrieval → Action → Result
This book provides a unified architecture based on ML pipelines for building all three types: batch, real-time, and LLM applications. The architecture specifically addresses data challenges—most ML systems need to process different data types from different sources for both training and inference.
Example - TikTok's Data Challenge: When TikTok recommends videos, it uses:
- Recent viewing behavior (clicks, swipes, likes)
- Historical viewing patterns and preferences
- Aggregated information (trending videos for users like you, near you)
Processing all this data in ML pipelines at scale is a significant engineering challenge covered throughout this book.
In plain English: Machine learning algorithms learn patterns from data in different ways, like how humans learn from textbooks (supervised), by exploration (unsupervised), or by trial and error (reinforcement).
In technical terms: ML algorithms differ in what information they learn from—labeled data, unlabeled data, self-generated labels, rewards, or contextual examples.
Why it matters: Understanding learning types helps you choose the right approach for your problem and data availability.
The main types of machine learning used in ML systems are:
In plain English: Teaching by example with answer keys. Like learning math with a textbook that shows problems and solutions.
In technical terms: Training models with datasets containing features and labels, where algorithms learn relationships between input features and target outcomes.
Why it matters: Supervised learning is the most common ML approach and powers most production systems.
In supervised learning, you train a model with data containing features and labels. Each row in a training dataset contains input feature values and a label (the outcome for those inputs). Supervised ML algorithms learn relationships between labels (target variable) and input features.
Use cases:
- Classification problems - Answer yes/no questions (Is there a hotdog in this photo?) or multiclass questions (What type of hotdog is this?)
- Regression problems - Predict numeric values (estimate apartment price given area, condition, location)
- Fine-tuning chatbots - Train with questions (features) and answers (labels) from a domain (e.g., legal profession) so the chatbot talks like a domain expert
In plain English: Finding patterns without answer keys. Like sorting your closet by color without instructions—you find natural groupings.
In technical terms: Algorithms learn from input features without labels, discovering patterns, clusters, or anomalies in data.
Why it matters: Useful when labeled data is expensive or unavailable, particularly for anomaly detection and pattern discovery.
Unsupervised learning algorithms learn from input features without any labels.
Example: Train an anomaly detection system with credit card transactions. When an anomalous transaction arrives, flag it as suspected fraud.
In plain English: Creating your own practice problems from the data. Like hiding words in a sentence and teaching yourself to guess them.
In technical terms: Generating labeled datasets from unlabeled data through masking, enabling supervised learning without manual labeling.
Why it matters: Enables training powerful models without expensive manual labeling, foundational for modern LLMs.
Self-supervised learning generates labeled datasets from fully unlabeled ones. The main method is masking.
For Natural Language Processing (NLP):
- Masked language modeling - Provide text, mask individual words, train model to predict missing word
- Next sentence prediction - Teach model to understand longer-term dependencies across sentences
- Example: BERT uses both techniques
For Image Classification:
- Mask small random parts of each image
- Train model to reproduce original image with high fidelity
💡 Insight
Self-supervised learning bridges the gap between unsupervised and supervised learning. It creates supervision signals from the data itself, enabling models to learn rich representations without human-labeled data.
In plain English: Learning by trial and error with rewards. Like training a dog with treats for good behavior.
In technical terms: Learning optimal decision-making by interacting with an environment and receiving rewards or penalties.
Why it matters: Enables AI to learn complex decision-making tasks where explicit examples are unavailable.
Reinforcement learning (RL) is concerned with learning how to make optimal decisions. (Note: Not covered in depth in this book)
In plain English: Learning from examples in the moment, then forgetting them. Like a chef adapting to a new recipe by looking at it, but not memorizing it.
In technical terms: LLMs' ability to learn new tasks by providing examples in the prompt, without updating model weights.
Why it matters: Enables flexible adaptation to new tasks without retraining, but requires careful context engineering.
Supervised ML, unsupervised ML, and reinforcement learning only learn from training data. However, sufficiently large LLMs exhibit in-context learning—the ability to learn new tasks by providing context (examples) in the prompt.
Key properties:
- LLMs exhibit this even though trained only for next token prediction
- Agents build on in-context learning but require context engineering
- The newly learned skill is forgotten after the context window is emptied
- No model weights are updated (unlike training)
Example - ChatGPT: ChatGPT combines multiple ML types:
- Self-supervised learning - Pretrain the LLM
- Supervised learning - Fine-tune foundation model for specific tasks (chatbot)
- Reinforcement learning - Align model with human values (remove bias, vulgarity)
- In-context learning - Learn from data in the input prompt
ML Types Comparison:
Supervised Unsupervised Self-Supervised In-Context
────────── ───────────── ─────────────── ──────────
Features + Labels → Features only → Create own labels → Examples in prompt
Learn patterns → Find clusters → Mask & predict → Temporary learning
Fixed knowledge → Fixed knowledge → Fixed knowledge → Ephemeral knowledge
In plain English: Data for ML systems comes from many places, like ingredients for a restaurant come from different suppliers. Each source has its own format and access method.
In technical terms: ML systems integrate data from various sources including databases, event streams, graph stores, file systems, and APIs, each optimized for different access patterns.
Why it matters: Understanding data sources and their characteristics enables efficient data integration and feature engineering.
Data for ML systems can come from any available source. Some sources and formats are more popular in enterprise computing—the information storage and processing platforms businesses use for operations, analytics, and data science.
In plain English: Data organized in spreadsheet-like tables with rows and columns. Like an Excel file but stored in a database.
In technical terms: Data stored as tables with columns and rows in databases, either row-oriented (optimized for row operations) or column-oriented (optimized for column operations).
Why it matters: Tabular data is the most common format in enterprises and the primary input for most ML models.
Tabular data is stored as tables containing columns and rows, typically in databases. Two main types:
- Examples: Relational databases (MySQL, Postgres), NoSQL databases (Cassandra, RocksDB, MongoDB)
- Optimized for: Reading and writing individual rows
- APIs: SQL, ORMs, key-value APIs, JSON store APIs
- Use case: Operational applications storing user transactions, profiles, orders
- Examples: Data warehouses and lakehouses (Snowflake, BigQuery, Databricks)
- Optimized for: Reading and processing columns (min/max/average/sum operations)
- APIs: SQL and DataFrame APIs (Spark, Pandas, Polars)
- Use case: Analytics and ML training data
💡 Insight
In enterprises, operational data flows from many row-oriented stores into centralized column-oriented stores via data pipelines. This analytical data becomes the most common source for AI systems, as it provides historical patterns across the entire business.
Enterprise Data Flow:
Row-Oriented Stores Column-Oriented Store ML System
──────────────────── ───────────────────── ─────────
App Database 1 ┐ ┌──────────┐
App Database 2 ├─→ Data Pipelines → │Data │ → Feature Pipelines → Model
App Database 3 ┘ │Warehouse │
└──────────┘
In plain English: Continuous streams of actions or occurrences, like a log of everything happening in real-time. Think of it as a timeline of events.
In technical terms: Records of discrete occurrences at specific points in time (clicks, sensor readings) stored in event streaming platforms.
Why it matters: Event data enables real-time ML systems that react within seconds of user actions.
Event data contains records of discrete occurrences or actions at specific points in time—clicks on a website, sensor readings, user interactions.
Event Streaming Platform (e.g., Apache Kafka):
- Collects and temporarily stores event data
- Enables downstream consumers to process events
- Consumer examples:
- Columnar stores (for historical analysis)
- Stream processing programs (for real-time ML systems reacting within a second)
In plain English: Data that shows how things connect to each other, like a social network showing who knows whom.
In technical terms: Data represented as nodes (entities) and edges (relationships), stored in graph databases optimized for interconnected data retrieval.
Why it matters: Graph structure enables ML models to leverage relationship patterns for tasks like fraud detection and recommendation.
Graph data is represented as nodes (entities) and edges (relationships). Graph databases support efficient storage and retrieval of complex, interconnected data.
ML use cases:
- Link prediction (suggesting connections)
- Fraud detection (identifying suspicious patterns)
- LLM knowledge sources (structured information for reasoning and QA)
In plain English: Data without a predefined structure—documents, images, videos. Like the difference between a filled form (structured) and a handwritten letter (unstructured).
In technical terms: Data lacking a defined schema, including text (PDFs, HTML, markdown), images, video, and audio, typically stored in file systems or object stores.
Why it matters: Unstructured data represents the majority of enterprise data and requires specialized processing techniques (deep learning).
Data with a schema (SQL table, JSON object, graph data) is structured. All other types are unstructured:
- Text: PDFs, docs, HTML, markdown
- Media: Images, video, audio
Storage: Files (sometimes very large, GBs+) in file systems or object stores (Amazon S3)
Processing: Deep learning has made huge strides with unstructured data, powering:
- Image tagging services
- Self-driving cars
- Voice transcription systems
- Many other AI systems trained on vast amounts of manually labeled unstructured data
In plain English: Data pulled from online services and websites, like copying information from a website into your system automatically.
In technical terms: Data retrieved from SaaS platforms via public APIs or web scraping, increasingly important as data moves to cloud services.
Why it matters: Much valuable data now lives in SaaS platforms; accessing it requires API integration or scraping capabilities.
More data is being stored in software-as-a-service (SaaS) systems, making it crucial to retrieve data using public APIs or web scraping.
Low-code solutions: Systems like Airbyte know APIs to popular platforms (Salesforce, HubSpot) and can pull data into data warehouses.
Custom scraping: Sometimes external APIs or websites lack integration support, requiring custom scraping.
Example (Chapter 3): We'll build an air quality prediction ML system that scrapes data from the closest public air quality sensor (tens of thousands available globally—probably one closer to you than you imagine).
Data Source Comparison:
Type Storage Access Pattern ML Use Case
────────── ──────────── ────────────── ───────────
Tabular Databases SQL/DataFrame Training data, features
Event Kafka/Streams Stream processing Real-time features
Graph Graph DB Graph queries Relationship features
Unstructured Object store File reading Deep learning inputs
API-scraped External SaaS API calls External features
In plain English: Most ML courses use frozen datasets that never change, like studying from a textbook. Production systems use living data that constantly updates, like studying from live news.
In technical terms: Production ML systems work with mutable data that supports inserts, updates, and deletions, unlike the immutable CSV files used in ML courses.
Why it matters: Real-world ML systems must handle data changes, privacy regulations (GDPR/CCPA), and dynamic training set selection.
While data work is often the majority of work in ML systems, existing ML courses typically use the simplest form: immutable datasets. Smaller datasets (few GBs) are stored in CSV files, while larger datasets (GBs to TBs) use compressible formats like Apache Parquet.
The well-known Titanic passenger dataset consists of:
train.csv- Training set for model trainingtest.csv- Test set for model evaluation
The data is static. Your job: train an ML model to predict passenger survival. First task: feature engineering—fill missing values (imputation), remove columns with no predictive power.
The Titanic dataset is popular for learning:
- Data cleaning basics
- Transforming data into features
- Fitting models to data
The Problem: There are no new passengers arriving for the Titanic. You don't have to worry about:
- Adding new passengers as they arrive
- Removing passengers due to GDPR requests
- Selecting training data subsets (you can't or don't want to use all data)
- Recreating training and test sets from selected rows
💡 Insight
Immutable files are unsuitable as the data layer in enterprises where GDPR (EU's General Data Protection Regulation) and CCPA (California Consumer Privacy Act) require that users can have their data deleted, updated, and its usage tracked.
Production ML systems work with mutable data stored in row-oriented or column-oriented stores that support:
- Efficient inserts
- Appends
- Updates
- Deletions
Challenge for data scientists: Previously required learning SQL and working directly with databases. Now you can also read/write mutable data using Python and DataFrame APIs—the main focus of this book.
Data transformations fall into two categories:
Before storing (in databases/feature stores):
- Aggregations
- Binning
- Dimensionality reduction
After reading (from feature store):
- Encoding categorical strings to numerical representation
- Normalizing numerical variables
- (Parameterized by training data - unknown until data is selected)
Feature Engineering Pipeline:
Mutable Data → Pre-storage transformations → Feature Store → Post-read transformations → Model
(aggregations, binning) (encoding, normalization)
Key Decision: Which transformations happen before vs. after storage?
Chapter 2: Introduces a taxonomy of data transformations helping identify where transformations should occur.
Chapters 6-7: Deep dive into data transformations for batch, real-time, and LLM ML systems.
In plain English: ML systems evolved from simple batch jobs to intelligent real-time systems, like how cooking evolved from preparing meals in advance to having on-demand restaurants.
In technical terms: ML systems progressed from monolithic batch programs to distributed architectures with separate training and inference pipelines, then to stateful systems with feature stores, and finally to agentic AI systems.
Why it matters: Understanding this evolution helps you choose the right architecture for your use case and avoid repeating historical mistakes.
In the mid-2010s, revolutionary ML systems appeared in consumer internet applications—image tagging in Facebook, Google Translate. The first generation comprised:
- Batch ML systems (predictions on a schedule)
- Interactive online ML systems (predictions in response to user actions)
┌─────────────────────────────────────────────────────────┐
│ Monolithic Batch ML System │
│ │
│ Mode 1: Training │
│ ┌──────────────┐ ┌────────────┐ ┌──────────────┐ │
│ │ Historical │───▶│ Create │───▶│ Train │ │
│ │ Data │ │ Features │ │ Model │ │
│ └──────────────┘ └────────────┘ └──────┬───────┘ │
│ │ │
│ ▼ │
│ ┌──────────────┐ │
│ │ Model │ │
│ │ Registry │ │
│ └──────────────┘ │
│ │
│ Mode 2: Inference │
│ ┌──────────────┐ ┌────────────┐ ┌──────────────┐ │
│ │ New Data │───▶│ Create │───▶│ Batch │ │
│ │ │ │ Features │ │ Predictions │ │
│ └──────────────┘ └────────────┘ └──────────────┘ │
└─────────────────────────────────────────────────────────┘
Challenge: Ensure consistent features between training and inference.
Solution: Monolithic architecture—same "Create Features" code runs in both training and inference modes.
Offline Training Pipeline:
┌──────────────┐ ┌────────────┐ ┌──────────────┐ ┌──────────────┐
│ Historical │───▶│ Create │───▶│ Train │───▶│ Model │
│ Data │ │ Features │ │ Model │ │ Registry │
└──────────────┘ └────────────┘ └──────────────┘ └──────────────┘
Online Inference Pipeline (Stateless):
┌──────────────┐ ┌────────────┐ ┌──────────────┐ ┌──────────────┐
│ Client │───▶│ Create │───▶│ Model │───▶│ Prediction │
│ Request │ │ Features │ │ Inference │ │ Response │
└──────────────┘ └────────────┘ └──────────────┘ └──────────────┘
Challenge: Two separate systems—can't ensure consistency with single monolithic program.
Early solution: Version feature creation source code; ensure training and serving use same version.
Use cases for stateless systems:
- Image tagging - Photo input → Model predicts bounding boxes and labels
- Early chatbots - User input appended to system prompt → LLM responds
Limitations of stateless systems:
- Image classifier: Only identifies objects from training data labels
- Chatbot: Cannot answer questions about post-training events
💡 Insight
The key limitation of stateless systems is their inability to incorporate recent context or history. A recommender can't personalize without knowing your recent activity; a chatbot can't discuss current events without being told about them.
Overcoming limitations: Provide history and context as input features.
Example: Online recommender takes recent products you viewed/liked as input to predict recommendations. Recent history as input features allows predictions with fresh data without retraining.
For LLMs: Retrieve and add context to prompt so LLMs can answer questions about post-training events.
Example: LLM trained in 2024 can answer "Who won the 2025 NBA finals?" if the prompt includes the Wikipedia article about the 2025 NBA finals.
The Core Question: Where does context and history come from?
Options:
- Client passes it as parameters (limited)
- Client's application database stores it (common)
The Problem: Source data for features stored in client's database, not available from client application.
The Solution: Feature stores—a separate stateful system that creates and stores features that online models can read when prediction requests arrive.
Stateful Real-Time ML System with Feature Store:
┌────────────────┐ ┌─────────────────────────────────┐
│ Data Sources │────────▶│ Feature Pipeline │
└────────────────┘ │ (Batch or Streaming) │
└────────────┬────────────────────┘
│
▼
┌─────────────────────────────────┐
│ Feature Store │
│ • Online (low latency) │
│ • Offline (historical) │
└────┬────────────────────────┬───┘
│ │
┌──────────▼─────────┐ ┌────────▼──────────┐
│ Training Pipeline │ │ Inference Pipeline│
│ │ │ │
│ Creates training │ │ Retrieves │
│ data from │ │ precomputed │
│ historical features│ │ features │
└──────────┬─────────┘ └────────┬──────────┘
│ │
▼ ▼
┌──────────────────┐ ┌──────────────────────┐
│ Model Registry │ │ Prediction Consumer │
└──────────────────┘ └──────────────────────┘
Feature stores (introduced by Uber in 2017 with Michelangelo platform):
- Manage transformation and storage of context and history as features
- Enable online models to access personalized features at low latency
- Provide historical features for training data creation
Feature pipeline properties:
- Reads historical or new data from one or more sources
- Transforms data into features
- Stores feature data in feature store
- Can be batch programs (scheduled) or stream processing programs (for very recent events)
Coverage in this book:
- Chapters 4-7: Feature stores and data transformations
- Chapters 8-9: Batch and streaming feature pipelines
- ML pipeline: Collective term for feature pipelines, training pipelines, and inference pipelines
In plain English: Early chatbots faced the same problem as early ML systems—they needed current information but only knew what they learned during training. The solution: retrieve relevant information at request time.
In technical terms: Retrieval-augmented generation (RAG) retrieves context data at request time and adds it to the LLM prompt, overcoming training data limitations.
Why it matters: RAG enables LLMs to work with current events and private data not in their training sets.
Stateless LLM applications (early chatbots) faced similar challenges:
- Needed relevant, timely context as input
- Required information about post-training events
- Needed access to private data not in training
Solution: Include context data retrieved at request time in system prompts.
First widely adopted approach: Retrieval-augmented generation (RAG) using a vector database.
RAG Architecture:
┌──────────────────┐ ┌─────────────────────────┐
│ Source Data │────────▶│ Vector Embedding │
│ (Documents, │ │ Pipeline │
│ Knowledge Base) │ │ 1. Chunk text │
└──────────────────┘ │ 2. Generate embeddings │
│ 3. Write to vector DB │
└────────────┬────────────┘
│
▼
┌─────────────────────────┐
│ Vector Database │
│ (Vector Index + ANN) │
└────────────┬────────────┘
│
┌──────────────────┐ │
│ User Input │──────────────────────┤
└──────────────────┘ │
▼
┌─────────────────────────┐
│ Retrieval Step │
│ Query vector DB with │
│ user input, get │
│ similar text chunks │
└────────────┬────────────┘
│
▼
┌─────────────────────────┐
│ Prompt Template │
│ Instructions + │
│ Retrieved Context + │
│ User Input │
└────────────┬────────────┘
│
▼
┌─────────────────────────┐
│ LLM │
│ Generates response │
└────────────┬────────────┘
│
▼
┌─────────────────────────┐
│ Response │
└─────────────────────────┘
How RAG works:
- User input as string queries vector database
- Vector database returns text chunks similar to input using approximate nearest neighbor (ANN) search
- Context information must first be written to vector database
- Vector embedding pipeline keeps data up to date:
- Transforms source data into text chunks
- Transforms chunks into vector embeddings using embedding model
- Writes vector embeddings to vector index
- System prompt is now dynamic—a prompt template with:
- Instructions (static)
- Empty slots filled with retrieved text (dynamic)
- Context window stores both input and output (few KBs to few MBs)
Context engineering: The challenge of preparing and retrieving context data for LLMs. Goal: construct a prompt that maximizes LLM output performance.
In plain English: As LLM applications tackled more complex tasks, they needed more independence in deciding what information to find and what actions to take—like upgrading from a assistant who follows instructions to one who can figure out how to achieve goals.
In technical terms: Agents are LLM applications with autonomy in querying diverse data sources and planning/executing tasks to achieve user-defined goals.
Why it matters: Agents represent a paradigm shift from human-machine interaction to machine-machine interaction, enabling AI systems to handle complex, multi-step tasks independently.
First LLM applications were tightly focused assistants (coding help, medical questions, cooking recipes). As tasks grew complex, they needed more autonomy in:
- What data to query
- What tasks to execute
Agents: A class of LLM application with autonomy in:
- How to query diverse data sources (vector indexes, search engines, feature stores)
- How to plan and execute tasks to achieve goals
Anthropic's definition: "Systems where LLMs dynamically direct their own processes and tool usage, maintaining control over how they accomplish tasks."
Paradigm shift: Human-machine interaction → Machine-machine interaction
- Users set high-level goals
- Developers provide tools and context
- Agents determine how to achieve goals
Evolution from LLM RAG to Agentic AI:
RAG Application Agentic AI System
─────────────── ─────────────────
┌──────────────┐ ┌──────────────────┐
│ User Input │ │ User Goal │
└──────┬───────┘ └────────┬─────────┘
│ │
▼ ▼
┌──────────────┐ ┌──────────────────────────┐
│ Fixed Query │ │ Agent Program │
│ to Vector DB │ │ • Analyzes goal │
└──────┬───────┘ │ • Plans approach │
│ │ • Selects tools │
▼ │ • Chooses data sources │
┌──────────────┐ └────────┬─────────────────┘
│ Retrieved │ │
│ Context │ ▼
└──────┬───────┘ ┌────────────────────────────────────┐
│ │ Model Context Protocol (MCP) │
▼ │ Unified standard for: │
┌──────────────┐ │ • RAG data retrieval │
│ Static │ │ • Vector indexes │
│ Prompt │ │ • Row/column stores │
│ Template │ │ • External APIs (tools) │
└──────┬───────┘ └────────┬───────────────────────────┘
│ │
▼ ▼
┌──────────────┐ ┌──────────────────────────┐
│ LLM Response │ │ Dynamic Actions │
└──────────────┘ │ • Query multiple sources│
│ • Execute tools │
│ • Iterate as needed │
└────────┬─────────────────┘
│
▼
┌──────────────────────────┐
│ Goal Achievement │
└──────────────────────────┘
Model Context Protocol (MCP): Unified standard for:
- Retrieving RAG data from various sources
- Using internal and external APIs as tools
In both LLM and agent architectures:
- Training LLMs is optional
- Can fine-tune foundation LLMs using instruction data from feature store (Chapter 10)
- Same engineering challenges: precomputing context data and vector embeddings
Coverage in this book:
- Chapters 5-6: Vector embedding pipelines
- Chapter 12: RAG and context engineering
💡 Insight
ML systems and AI systems are related but distinct. An ML system learns from data through algorithms and statistical models. AI is broader, covering search, memory, and agent techniques. We use: batch ML system, real-time ML system, and agentic AI system. The general term "AI system" applies except when referring to specific ML system classes.
In plain English: MLOps is like quality control and assembly line optimization for ML systems—it's about building reliable systems that can be quickly updated and improved with minimal manual work.
In technical terms: MLOps is a set of practices for building reliable, scalable ML systems that can be quickly and incrementally developed, tested, and deployed to production using automation.
Why it matters: Most ML work isn't training models—it's data management and infrastructure. MLOps practices help you build better systems faster.
The evolution of ML system architectures didn't happen in isolation. It occurred within machine learning operations (MLOps), a field dating back to 2015 when Google published "Hidden Technical Debt in Machine Learning Systems."
Key adage from the paper: Only a small percentage of ML work is training models. Most work is:
- Data management
- Building ML system infrastructure
- Operating ML systems
MLOps inspiration: DevOps movement in software engineering (integrates development and IT operations to improve and shorten the development lifecycle).
MLOps goals:
- Build reliable and scalable ML systems
- Enable quick, incremental development
- Test thoroughly
- Roll out to production using automation
- Tighten the development loop: make changes → test changes → deploy changes
For data scientists: Don't be intimidated by systems focus on automation, testing, and operations. DevOps' North Star is to get to a minimal viable product (MVP) as fast as possible, then iteratively improve.
Chapter 2: Our process for building an MVP.
In plain English: ML systems need more types of tests than regular software because bugs in data or code can cause incorrect predictions. It's like quality control not just for the product, but also for the ingredients and the recipe.
In technical terms: ML systems require testing at all stages—feature engineering, model training, and deployment—covering code, data, models, and integrations.
Why it matters: Small bugs in input data or code easily cause incorrect predictions. Comprehensive testing ensures high-quality, unbiased predictions.
ML systems require more testing levels than traditional software systems:
Development-time tests:
- Unit tests - Validate feature logic (changes can pollute training data)
- Integration tests - Validate ML pipelines, catch Python code errors
- Model validation tests - Check performance and bias
- Evals - Safety, reliability, and performance of LLM applications and agents
Production tests and monitoring:
- Data validation tests - Prevent bad data from entering system
- Model performance monitoring - Most models degrade over time
- Feature drift detection - Check if inference data differs statistically from training data
- A/B tests - Test new model versions before full rollout
- Guardrails - Prevent harmful LLM inputs and outputs
ML Testing Hierarchy:
Development Production
──────────── ──────────
Unit Tests ─→ Data Validation
Integration Tests ─→ Model Monitoring
Model Validation ─→ Drift Detection
LLM Evals ─→ Guardrails
A/B Tests
In plain English: MLOps principles are like best practices for running a efficient, reliable ML factory—automate what you can, version everything, monitor constantly, and make it easy to fix problems.
In technical terms: MLOps principles include minimal-friction testing, automated deployment, data quality assurance, versioning for rollback capability, thorough evaluation, and comprehensive monitoring.
Why it matters: These principles distinguish hobbyist ML projects from production-grade systems that deliver reliable value.
Testing should have minimal impact on development speed. Automate test execution to improve productivity.
Popular CI platforms:
- GitHub Actions
- Jenkins
- Azure DevOps
Getting started:
- Start with unit tests for functions
- Add model performance and bias testing in training pipelines
- Add integration tests for all ML pipelines
- Automate tests with CI when you push code
- Add after validating MVP is worth maintaining
💡 Insight
CI is not a prerequisite to start building ML systems. If you have a data science background, comprehensive testing may be new. It's OK to incrementally add testing to both your skills and your ML systems.
MLOps folks love when pushing code automatically deploys ML artifacts/systems.
Development environments:
- Dev - Development environment
- Preprod - Preproduction testing environment
- Prod - Production environment
Flow: Build in dev → Test in preprod → Test again → Deploy to prod
Continuous deployment (CD): Automated deployment process (human may sign off on prod deployment)
Philosophy in this book: Build, test, and run entire ML system in dev, preprod, or prod environments. Data access may depend on environment.
Chapter 13: Detailed coverage of CD.
Database community maxim: "Garbage in, garbage out."
The problem: Many ML systems use data with few quality guarantees. Blindly ingesting garbage data leads to well-trained models that predict garbage.
Chapter 6: Design and write data validation tests for feature pipelines. Detail mitigating actions for incorrect, missing, or corrupt data.
- Big green button - Upgrade the system
- Big red button - Roll back problematic upgrade
Prerequisites:
- Versioning of features and models
- Enables A/B testing
- Enables upgrade/downgrade without downtime
- Quick rollback to working earlier versions
Don't like surprises when new LLM or agent versions introduce unexpected behavior (like Amazon Q coding agent that could wipe users' filesystems!).
Chapter 13: Design and run evals to evaluate changes to LLM applications and agents before production.
Production AI systems should collect metrics for dashboards and alerts:
- Model quality - Monitor predictions against business KPIs
- Data drift - Monitor new data for statistical drift
- Platform performance - Measure throughput and latency of:
- Model serving
- Feature store
- Vector index
- LLMs
- ML pipelines
Operational service logs enable debugging and improvement.
Techniques:
- LLMs - Eyeballing model logs for error analysis (Chapter 14)
- Classical ML - Logs for debugging errors and understanding model performance
💡 Insight
This book takes a nontraditional approach to MLOps. You won't learn Terraform, Dockerfiles, or Kubernetes. Instead, you'll learn to test, version, operate, and monitor ML pipelines that power AI systems.
MLOps Best Practices Summary:
Practice Tool/Approach Benefit
──────── ───────────── ───────
Automated Testing → CI platforms → Fast feedback
Automated Deployment → CD pipelines → Reliable releases
Data Quality → Validation tests → Prevent garbage
Versioning → Feature/model versions → Easy rollback
Evaluation → Evals & A/B tests → Prevent surprises
Monitoring → Metrics & dashboards → Know performance
Logging → Operational logs → Debug & improve
In plain English: Just as web applications are built with separate presentation, business logic, and database layers, AI systems can be built with separate feature, training, and inference pipelines that connect through shared data storage.
In technical terms: A unified architecture decomposes AI systems into independent feature pipelines, training pipelines, and inference pipelines, connected via a feature store and model registry.
Why it matters: Modularity enables independent development, testing, and operation of components, resulting in higher-quality AI systems built faster.
Modularity in software: Decomposing a system into smaller, manageable modules that can be independently developed and composed into a complete system.
Benefits:
- Higher-quality, more reliable software
- Independent module testing
- Easier to understand and document
- Reuse of functionality
- Clear separation of work between teams
- Better team communication through shared understanding
Earlier we presented five different AI system architectures: batch, stateless real-time, stateful real-time, RAG LLM, and agentic AI. These are useful architectural patterns, but they're very different—challenging for developers to transfer knowledge between them.
We can do better. There's a unified architecture for developing all AI systems following a natural decomposition into feature creation, model training, and inference pipelines.
Real-world validation: At KTH, students built different AI systems in teams as projects. Despite building different systems, they could easily:
- Divide work in building systems
- Communicate system architecture using this decomposition
In enterprises:
- Data engineers - Help with feature creation
- Data scientists - Model training realm
- IT operations - Inference involvement
- ML engineers - Contribute to all three pipeline types
Clear inputs and outputs, developed/tested/operated independently:
-
Feature Pipeline
- Input: Data
- Output: Reusable feature data
-
Training Pipeline
- Input: Feature data
- Output: Trained model
-
Inference Pipeline
- Input: Feature data + model
- Output: Predictions + prediction logs
💡 Insight
Modularity only helps if modules compose easily into functioning systems. Web applications succeeded for 30 years with presentation, business logic, and database modules. Microservices can suffer from too many services increasing operational complexity. Our AI system decomposition composes naturally via a shared data layer.
Shared data layer components:
- Feature store:
- Real-time data (row-oriented store, low latency for online inference and agents)
- Historical data (columnar store for training and batch inference)
- Vector embeddings (vector index for inference pipelines and agents)
- Model registry: Stores trained models
Unified FTI Architecture:
┌────────────────────────────────────────────────────────────────────────┐
│ AI System │
│ │
│ ┌──────────────┐ ┌─────────────────────────────┐ │
│ │ Data Sources │───────▶│ Feature Pipeline │ │
│ │ │ │ (Batch or Streaming) │ │
│ └──────────────┘ └────────────┬────────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────────────┐ │
│ │ Feature Store │ │
│ │ • Online (row-oriented) │ │
│ │ • Offline (columnar) │ │
│ │ • Vectors (vector index) │ │
│ └─────┬──────────────────┬────┘ │
│ │ │ │
│ ┌──────────▼─────┐ ┌───────▼──────────┐ │
│ │ Training │ │ Inference │ │
│ │ Pipeline │ │ Pipeline │ │
│ │ │ │ (Batch/Online/ │ │
│ │ • Read features│ │ Agent) │ │
│ │ • Train model │ │ │ │
│ │ • Validate │ │ • Read features │ │
│ └────────┬────────┘ │ • Load model │ │
│ │ │ • Predict │ │
│ ▼ └────────┬─────────┘ │
│ ┌──────────────────┐ │ │
│ │ Model Registry │◀─────────┘ │
│ └──────────────────┘ │
│ │
│ ┌────────────────────────┐ │
│ │ Operational Logs │ │
│ │ (Monitoring/Debugging)│ │
│ └────────────────────────┘ │
└────────────────────────────────────────────────────────────────────────┘
Definition: An AI system is a set of independent feature pipelines, training pipelines, and inference pipelines connected via a feature store and model registry.
Feature pipelines:
- Ingest backfill and production data
- Compute feature data stored as tabular data in feature store
- Can be batch programs or stream processing programs
Training pipelines:
- Read training data from feature store
- Store trained models in model registry
Inference pipelines:
- Output predictions using model (downloaded from registry or via API)
- Use new feature data (precomputed from feature store and/or computed at request time)
ML pipelines can run on any compute engine:
Batch compute engines:
- SQL in data warehouses
- Spark
- Pandas
- Polars
- DuckDB
Stream processing engines:
- Flink
- Spark Streaming
- Feldera
Training and inference:
- Most commonly Python
- Batch inference: PySpark, Pandas, Polars
In plain English: AI systems are categorized by how they compute predictions, not by what application uses them. Like restaurants are categorized by cooking style (fast food, fine dining) not by who eats there.
In technical terms: AI systems with a feature store are categorized by their inference pattern: real-time (interactive), agentic workflows, batch, or stream processing.
Why it matters: Understanding system classes helps you choose the right architecture for your requirements.
An AI system is defined by how it computes predictions, not by the consuming application type.
- Make predictions in response to user requests
- Compute features on demand from request parameters
- Read precomputed features from feature store or external systems
- Often use stream processing to precompute fresh features
- React faster to user actions than batch feature pipelines
- User-guided AI systems
- Achieve goals with some autonomy
- Use LLMs and tools to:
- Execute actions on external systems
- Acquire context from data sources (vector index, row/column stores, external APIs)
- Feature pipelines, vector embedding pipelines, and real-time feature engineering create context data
- Run batch inference programs on a schedule
- Take new feature data and model
- Output predictions to inference store (downstream database)
- Consumed by ML-enabled applications
- Use embedded model to make predictions on streaming data
- No user input required
- Often machine-to-machine
- Example: Network intrusion detection
- Extract features from network traffic via stream processing
- Use model to predict network intrusion
💡 Insight
Real-time ML systems and agentic workflows are both interactive systems providing prediction request APIs and handling concurrent requests. The distinction: real-time ML systems have custom-trained models (decision trees, deep learning) hosted internally with simple inference pipelines. Agentic workflows have complex inference programs (agents) using tools and LLMs via external APIs.
AI System Classes:
Real-Time ML Agentic Workflow Batch ML Stream ML
──────────── ──────────────── ──────── ─────────
User request → User goal → Schedule → Stream data
Feature lookup → Agent reasoning → Batch read → Feature extract
Model predict → Tool selection → Model batch → Model predict
Respond → LLM + tools → Store results → Store/alert
→ Achieve goal →
In plain English: Some ML systems run on devices without internet connections, processing everything locally—like a smart camera that identifies defects without sending data to the cloud.
In technical terms: Real-time ML systems running on resource-constrained network-detached devices, using embedded models and computing features from input data without precomputed features.
Why it matters: Edge ML enables AI in environments where data cannot leave premises due to connectivity, privacy, or latency constraints.
A popular type not covered in this book: embedded or edge ML systems.
Characteristics:
- Use embedded model
- Compute features from input data
- No precomputed features from feature store
- Run on resource-constrained network-detached devices
- All data processing at network edge
Example - Tetra Pak:
- Makes paper packaging
- Uses image classifier to identify carton anomalies on factory floor
- No data leaves factory floor
- All processing at network edge
Chapter 3: Air Quality Prediction Dashboard
- Shows air quality forecasts for a location near you
- Features: Observations from public sensors + weather data
- Trains model to predict air quality using weather forecasts
Chapters 4+: Credit Card Fraud Detection
- Takes credit card transaction
- Retrieves precomputed features about recent card use from feature store
- Builds feature vector for decision tree model
- Predicts if transaction is suspected fraud
Chapter 15: Video Recommender System (TikTok-like)
- Based on retrieval and ranking architecture
- Stream processing creates features from user actions (clicks, swipes)
- Two-tower embedding model for retrieval
- XGBoost model for ranking
- Add LLM capabilities to air quality prediction system
- Add LLM capabilities to TikTok recommender system
- Examples using agents in LlamaIndex
Book Project Progression:
Simple Intermediate Advanced
────── ──────────── ────────
Air Quality Credit Card Fraud → Video Recommender
(Batch) → (Real-time) → (Real-time + Streaming)
→
+ Agentic AI
(LLM Integration)
In plain English: We'll use Python for everything because it's accessible and widely used, along with popular open-source tools that have free tiers so you can build without cost.
In technical terms: The book uses Python with open-source frameworks (Pandas, Polars, Scikit-Learn, PyTorch), serverless platforms (Modal, GitHub Actions), and Hopsworks for ML infrastructure, all accessible via free tiers.
Why it matters: The technology choices let you focus on building AI systems without infrastructure costs or operational overhead.
We'll build AI systems using Python programs. Given our goal to build AI systems (not ML infrastructure), we must make well-motivated platform choices within space constraints.
Python chosen because:
- Accessible to developers
- Dominant language of data science
- Increasingly important in data engineering
Open source frameworks:
- Feature engineering: Pandas, Polars
- ML: Scikit-Learn, PyTorch
- Model serving: KServe
- LLMs: Pretrained open source foundation models
Python capabilities: Everything from creating features from raw data, to model training, to developing user interfaces.
Enterprise alternatives (when appropriate):
- Scalable data processing: Spark, dbt/SQL
- Real-time ML: Stream processing frameworks
Prerequisite: Only Python knowledge required for all example AI systems.
Running Python programs as pipelines in the cloud:
- Modal - Requires credit card registration, free tier available
- GitHub Actions - Free tier available
- Hopsworks - Can run ML pipelines if you have dedicated cluster
- Other platforms - Examples should work on any Python job platform
Exploratory analysis and model training:
- Open source Jupyter Notebooks
Serverless user interfaces:
- Streamlit - Free cloud tier
- Alternative: Hugging Face Spaces and Gradio
Hopsworks as serverless ML infrastructure:
- Feature store
- Model registry
- Model serving platform
- Open source (first open source and enterprise feature store)
- Free tier for serverless platform
- No cost deployment and operation
Why Hopsworks: I'm a developer, so I can provide deeper insights into its inner workings as a representative ML infrastructure platform.
Alternatives: All examples in common open source Python frameworks, easily modified to use:
- Feature store: Feast
- Model registry: MLflow
- Model serving: MLflow, KServe
- Cloud platforms: Databricks, GCP Vertex, AWS SageMaker
💡 Insight
The infrastructure choices prioritize accessibility over vendor lock-in. Using standard Python frameworks and open APIs means you can start with free tiers and migrate to enterprise platforms without rewriting code.
Technology Stack:
Layer Primary Choice Alternatives
───── ────────────── ────────────
Language Python -
Feature Eng. Pandas, Polars Spark, dbt/SQL
ML Training Scikit-Learn, PyTorch -
Model Serving KServe MLflow
Pipelines Modal, GitHub Actions Hopsworks, custom
UI Streamlit Gradio, HF Spaces
Feature Store Hopsworks Feast, Databricks
Model Registry Hopsworks MLflow, cloud platforms
This chapter introduced the foundational concepts of modern AI systems:
-
ML Systems vs. One-off Models: We explored the difference between training models for single predictions and building systems that deliver continuous value through automated pipelines.
-
System Anatomy: ML systems are categorized by their inference patterns—batch systems make predictions on schedules (Spotify's Discovery Weekly), real-time systems respond to user requests (TikTok recommendations), and agentic systems achieve goals autonomously (Lovable coding assistant).
-
Machine Learning Types: We covered supervised learning (training with labels), unsupervised learning (finding patterns without labels), self-supervised learning (creating labels through masking), reinforcement learning (learning through rewards), and in-context learning (LLMs learning from prompt examples).
-
Data Sources: Enterprise AI systems draw from tabular data (row and column-oriented stores), event streams (Kafka), graph databases (interconnected relationships), unstructured data (images, text, video), and API-scraped data (SaaS platforms).
-
Mutable Data Challenges: Production systems work with mutable data requiring inserts, updates, and deletions—unlike static Kaggle datasets. This introduces feature engineering challenges around when to apply transformations.
-
Historical Evolution: ML systems evolved from monolithic batch programs to stateless interactive systems, then to stateful systems with feature stores (Uber 2017), RAG-powered LLMs (vector databases for context), and finally agentic AI systems (autonomous decision-making).
-
MLOps Principles: Modern ML development follows principles of minimal-friction testing, automated deployment, data quality assurance, versioning for rollbacks, comprehensive evaluation, performance monitoring, and operational logging.
-
Unified FTI Architecture: All AI systems can be built using a decomposition into Feature pipelines (create reusable features), Training pipelines (produce models), and Inference pipelines (generate predictions), connected via a feature store and model registry.
-
Technology Stack: This book uses Python with open-source frameworks (Pandas, Polars, Scikit-Learn, PyTorch), serverless platforms (Modal, GitHub Actions, Streamlit), and Hopsworks ML infrastructure—all accessible via free tiers.
The next chapter examines the FTI pipeline architecture in detail, showing how to build AI systems faster and more reliably as connected pipelines.
Previous: [None] | Next: Chapter 2 - Machine Learning Pipelines