if-statements wins.Learn AI Series):We've spent 116 episodes learning to build models. Time for an uncomfortable truth, the kind you don't hear at conferences: building the model is usually the easy part. Getting that model to solve a real business problem, at scale, reliably, day after boring day -- that is where the overwhelming majority of ML projects quietly die. Not with a crash. With a shrug, six months in, when someone finally asks "wait, is this thing actually doing anything?"
So today we don't train a single network. Instead we learn to think in systems rather than models. Because when a stakeholder walks up and says "we want to use AI for X", the very first question is NOT "which model?". It's "should we even use ML for this at all?" And if the honest answer turns out to be yes, the next question still isn't about accuracy -- it's about architecture. Let's get into it ;-)
Every ML project starts with a translation: a vague, hand-wavy business goal gets turned into a precise ML formulation. This is the step where experience earns its keep, and it's also the step everybody skips because it feels like paperwork. It is not paperwork. It's the difference between shipping and flailing.
from dataclasses import dataclass
from typing import Optional
@dataclass
class MLProblemFrame:
"""A structure for framing a business problem as an ML problem."""
business_goal: str
ml_objective: str
input_data: str
output: str
success_metric: str
baseline: str
latency_requirement: Optional[str] = None
update_frequency: Optional[str] = None
# Example: "We want to reduce customer churn"
churn_frame = MLProblemFrame(
business_goal="Reduce monthly churn from 5% to 3%",
ml_objective="Binary classification: will this customer churn in the next 30 days?",
input_data="Activity logs, billing history, support tickets",
output="Churn probability per customer",
success_metric="Precision@top-10% (catch the high-risk few worth a retention offer)",
baseline="Rule: flag anyone with no login in 14 days",
latency_requirement="Batch -- daily predictions are perfectly fine",
update_frequency="Retrain weekly as fresh churn labels arrive",
)
# Example: "We need content moderation"
moderation_frame = MLProblemFrame(
business_goal="Remove harmful content within 5 minutes of posting",
ml_objective="Multi-label classification: toxicity, spam, NSFW, etc.",
input_data="Text content, user metadata, posting patterns",
output="Per-category probability scores",
success_metric="Recall > 95% for severe content, precision > 80% for auto-removal",
baseline="Keyword blocklist plus a fistful of regexes",
latency_requirement="Real-time -- under 200ms per item",
update_frequency="Daily, on newly labeled examples from human review",
)
Notice the baseline field, because it is the most important one in that whole dataclass. Every ML system needs a non-ML baseline. If your fancy deep-learning model can't beat a list of keywords and a handful of if-statements, then congratulations -- you don't have an ML problem, you have an engineering problem, and you've just saved yourself half a year. We first met this mindset all the way back in episode #1, where we drew the line between problems ML is for and problems it merely makes fashionable.
And look at the two success_metric lines. Neither says "accuracy". Churn cares about precision at the top slice (you only have budget to phone so many at-risk customers). Moderation cares about recall on the severe stuff (missing one bad item is far worse than over-flagging ten harmless ones). We hammered this in episode #13: the metric IS the problem statement in disguise, and picking the wrong one bakes failure in from day one.
When you design a complete system -- not a model, a system -- you work through these layers, in this order:
class MLSystemDesign:
"""The five layers of ML system design, in the order you should tackle them."""
layers = [
"1. Problem Definition", # What are we optimizing, and for whom?
"2. Data Pipeline", # Where does data come from? How fresh must it be?
"3. Feature Engineering", # Raw data -> model inputs, at scale
"4. Model & Training", # Architecture, training, honest evaluation
"5. Serving & Monitoring", # Deploy, observe, iterate, repeat forever
]
Here's the pattern I've watched play out more times than I can count: most engineers sprint straight to layer 4, because layer 4 is the fun one -- the one with the GPUs and the papers and the leaderboards. The engineers who actually ship spend a solid 60% of their effort on layers 1 to 3, and treat layer 4 as almost an afterthought. Having said that, layer 5 is the one nobody plans for and everybody eventually bleeds on. We'll spend the next stretch of episodes crawling through those outer layers in detail -- the whole unglamorous machinery that keeps a model alive once the notebook is closed.
This single architectural fork shapes everything downstream -- your infra bill, your on-call pager, your Sunday evenings. Get it right early:
@dataclass
class PredictionArchitecture:
pattern: str
latency: str
freshness: str
cost: str
complexity: str
use_when: str
architectures = {
'batch': PredictionArchitecture(
pattern="Precompute predictions for every entity on a schedule",
latency="N/A -- pre-computed, the lookup is instant",
freshness="Stale by hours or days",
cost="Low (run once, serve from a cache or DB)",
complexity="Low",
use_when="Predictions change slowly and the entity set is bounded",
),
'online': PredictionArchitecture(
pattern="Compute the prediction at request time",
latency="10ms-500ms per request",
freshness="Real-time -- uses the very latest features",
cost="High (CPU/GPU burned per request)",
complexity="High (model serving, autoscaling, fallbacks, the works)",
use_when="Features change per-request, cold-start items, unbounded inputs",
),
'near_online': PredictionArchitecture(
pattern="Stream features in, recompute predictions on change",
latency="Minutes behind real-time",
freshness="Near-real-time",
cost="Medium",
complexity="Medium (you now own streaming infrastructure)",
use_when="Features update often, but not on every single request",
),
}
A lot of newcomers assume "online = better, obviously, it's real-time!" -- and then get flattened by the bill. Batch is gorgeous when it fits: you run one big job overnight, stuff the results in a table, and every serving request becomes a trivial key lookup. No GPU on the hot path, no autoscaling drama, no 3am pages. The churn model above? Pure batch. Nobody needs a churn score recomputed the instant someone reloads their billing page.
The recommendation systems we built back in episode #27 are the classic hybrid, and it's a genuinely clever trick: a batch stage generates candidate items offline (millions of items squeezed down to a few hundred per user), then a lightweight online stage re-ranks those few hundred using live signals -- current session behaviour, time of day, what you clicked ninety seconds ago. You get the freshness of online with the cost profile of batch. Best of both, and it's not an accident -- it's a deliberate architecture.
If you go online, "under 200ms" is not a vibe, it's a budget -- and like any budget it gets spent whether you track it or not. So track it:
def latency_budget(total_ms, stages):
"""Split a latency budget across the stages of an online request.
'stages' maps stage name -> fraction of the budget it's allowed."""
assert abs(sum(stages.values()) - 1.0) < 1e-6, "fractions must sum to 1.0"
budget = {name: round(total_ms * frac, 1) for name, frac in stages.items()}
spent = sum(budget.values())
budget["_headroom_ms"] = round(total_ms - spent, 1)
return budget
# A 200ms end-to-end budget for one ranking request:
plan = latency_budget(200, {
"network_in": 0.10, # 20ms -- request arrives
"feature_fetch": 0.35, # 70ms -- pull features from the store
"model_inference": 0.30, # 60ms -- the actual forward pass
"post_processing": 0.15, # 30ms -- business rules, formatting
"network_out": 0.10, # 20ms -- response leaves
})
print(plan)
Run that and the punchline jumps out: the model's forward pass -- the thing we spent a hundred episodes obsessing over -- gets a measly 60ms, less than a third of the budget. Feature fetching eats more than inference does. This surprises people every time, and it's why "just make the model faster" is so often the wrong lever. Your bottleneck is usually the plumbing, not the pump.
Right, the single most important concept in production ML, and it isn't a model architecture at all. It's the data flywheel:
Better model -> Better product -> More users -> More data -> Better model -> ...
Companies with a spinning flywheel have a compounding, borderline-unfair advantage. Every prediction the deployed model makes -- right OR wrong -- becomes a training example for the next version. Explicit signals (clicks, purchases, star ratings) are the obvious fuel. But the implicit signals (dwell time, scroll depth, the silent abandonment of a cart) are frequently more valuable, precisely because they're abundant and honest -- nobody lies with their scroll wheel.
import time
class DataFlywheel:
"""A sketch of how production ML quietly improves itself over time."""
def __init__(self, model):
self.model = model
self.model_version = 0
self.pending = [] # served predictions, still awaiting an outcome label
def serve(self, features):
"""Serve a prediction AND log the input for future training."""
prediction = self.model.predict(features)
self.log_for_labeling(features, prediction)
return prediction
def log_for_labeling(self, features, prediction):
"""Every served prediction is a future training example.
The label arrives later, from downstream user behaviour."""
self.pending.append({
"features": features,
"prediction": prediction,
"timestamp": time.time(),
"label": None, # filled in once the outcome is observed
})
def attach_outcome(self, index, label):
"""Real life eventually tells you what actually happened."""
self.pending[index]["label"] = label
def retrain(self):
"""Periodically retrain on whatever has been labeled by reality."""
labeled = [d for d in self.pending if d["label"] is not None]
if not labeled:
return
self.model.fit(labeled)
self.model_version += 1
The flywheel is why first-mover advantage is so brutal in ML products. A competitor can copy your model architecture in an afternoon -- the papers are public, the weights are half the time on Hugging Face (episode #74). What they cannot copy is the millions of real, labeled interactions your deployed system has been quietly hoarding since launch. That data is the moat. The model is just the bucket you carry it in.
Nota bene: the flywheel has an evil twin. If your model's own predictions start shaping the data it later trains on -- a recommender that only ever shows what it already thinks you'll like -- you get feedback loops that slowly wall the model inside its own opinions. We flagged this danger back in episode #35 on bias, and it's worth keeping a wary eye on. A flywheel that spins the wrong way still spins.
Back in episode #15 we engineered features on a single tidy dataset that fit comfortably in memory. Production features are a different animal entirely. They come from multiple sources, get computed at wildly different time scales, and -- this is the part that bites -- they must be identical between training and serving.
class FeatureStore:
"""A simplified feature-store concept -- the heart of scaled features."""
def __init__(self):
self.batch_features = {} # recomputed hourly / daily
self.stream_features = {} # updated per event, live
self.static_features = {} # rarely change (e.g. account age bucket)
def register(self, name, source, freshness):
"""Register a feature with its source and how fresh it stays."""
return {"name": name, "source": source, "freshness": freshness}
def get_features(self, entity_id, feature_names):
"""Fetch features for one entity -- the SAME path for training and serving."""
out = {}
for name in feature_names:
if name in self.stream_features:
out[name] = self.stream_features[name].get(entity_id)
elif name in self.batch_features:
out[name] = self.batch_features[name].get(entity_id)
elif name in self.static_features:
out[name] = self.static_features[name].get(entity_id)
return out
# The crux: ONE function computes features for both training (on
# historical data) and serving (on live data). When those two paths
# drift apart, you get "training-serving skew" -- one of the top
# causes of production ML systems silently rotting.
Training-serving skew is subtle and quietly devastating, and I want you to really feel the shape of it. Your model was trained on features computed with a specific aggregation window, a specific join order, a specific way of handling nulls. If the serving pipeline computes even slightly different features -- a 7-day window here where training used 30, a NaN filled with zero instead of the median -- the model's predictions degrade. And the horror is that you may not notice for weeks, because the metric doesn't crash, it just sags, gently, like a slow leak. This is exactly why feature stores exist: not for tidiness, but so that a single code path feeds both worlds and simply cannot drift out of sync with itself.
Not every problem deserves ML. In fact, a genuinely uncomfortable fraction of "AI projects" would be better, cheaper, and more maintainable as fifty lines of plain Python. Here's the honest framework, the one that would have quietly killed a good chunk of the doomed projects I've watched:
def should_use_ml(scores):
"""A blunt decision aid: ML, simple ML, or just rules?
Score each factor 0-3, higher meaning 'more ML-worthy'."""
# pattern_complexity: are the rules hard to write by hand?
# spam -> high (adversarial, always mutating)
# 'is this a valid email format?' -> zero, a regex owns this
# data_availability: no labeled data == no ML. Full stop.
# "we'll collect data after we launch" almost never survives contact.
# value_of_accuracy: does 85% vs 95% actually change a decision?
# 80 -> 90 is cheap. 90 -> 95 is dear. 95 -> 99 costs more than all the rest combined.
# maintenance_budget: ML systems rot -- models drift, data shifts.
# two engineers can maintain 1-2 ML systems well, or 10 systems terribly.
total = sum(scores.values())
if total < 6:
return "Use rules or heuristics -- ML is not earning its keep here"
elif total < 10:
return "Start simple -- logistic regression or gradient boosting (episodes #12, #19)"
else:
return "Deep learning is justified -- proceed, carefully"
print(should_use_ml({
"pattern_complexity": 3,
"data_availability": 3,
"value_of_accuracy": 2,
"maintenance_budget": 2,
}))
The pattern I've seen kill projects goes like this: a team spends six months lovingly hand-crafting a deep-learning system that performs a heroic 2% better than the gradient-boosting model they could have shipped in week two -- and by the time they finally deploy, the business requirements have shifted under their feet and the whole thing is stale. That 2% was real. It was also completely, catastrophically not worth it. Boring wins more often than anyone building their portfolio wants to admit ;-)
Enough theory -- let's sketch a real one, end to end. A content recommendation engine, the sort of thing that quietly powers half the internet:
system_design = """
CONTENT RECOMMENDATION SYSTEM
1. Problem Definition
- Goal: increase engagement (clicks, honest read-time)
- Metric: NDCG@10 for relevance, plus a diversity term for coverage
- Constraints: under 100ms latency, ~10K requests/sec at peak
2. Data Pipeline
- Interactions: clicks, reads, bookmarks (streamed live)
- Content metadata: tags, author, length (batch ETL)
- User profiles: history, preferences (batch)
- Labels: click/no-click; read_time > 30s counts as a positive
3. Feature Engineering
- User: avg_read_time, topic_preferences (embedding), recency
- Content: age, popularity_score, topic_vector
- Cross: user_topic_affinity, author_user_history
- Feature store: batch (hourly) + streaming (per-click)
4. Model
- Stage 1 (candidate generation): two-tower model, batch
10M items -> top 1000 candidates per user
- Stage 2 (ranking): cross-attention model, online
1000 candidates -> a ranked top 10
- Stage 3 (re-ranking): plain business rules (diversity, freshness)
5. Serving
- Candidate vectors pre-computed, stored for fast nearest-neighbour search (episode #63)
- Ranking model behind a serving layer on accelerated instances
- Fallback: popularity-based ranking if the model times out
- A/B testing: 5% of traffic to each new model version
6. Monitoring
- Online metrics: CTR, read_time, bounce_rate
- Model metrics: prediction-distribution drift
- Feature metrics: null rates, value distributions
- Alerting: a >10% CTR drop trips an investigation
"""
print(system_design)
Sit with that for a second, because there's a humbling lesson hiding in it. The model -- the thing we spent 100-plus episodes learning to build from first principles -- is one box. Box 4 of six. Everything around it (where the data comes from, how features stay consistent, how you serve under a latency budget, how you notice when it all goes sideways) is the other five-sixths of the work, and it's the part that decides whether the system survives contact with reality.
You cannot fix what you cannot see, and an ML model in production is mostly invisible -- it fails silently, with no exception and no stack trace, just numbers drifting quietly south. So the cheapest, most valuable thing you can bolt on is a simple watch on your input distributions:
import numpy as np
def population_stability_index(expected, actual, bins=10):
"""PSI: how far has a feature's live distribution drifted from training?
Rule of thumb: <0.1 stable, 0.1-0.25 shifting, >0.25 alarm bells."""
edges = np.percentile(expected, np.linspace(0, 100, bins + 1))
edges[0], edges[-1] = -np.inf, np.inf
e = np.histogram(expected, edges)[0] / len(expected) + 1e-6
a = np.histogram(actual, edges)[0] / len(actual) + 1e-6
return float(np.sum((a - e) * np.log(a / e)))
train_ages = np.random.normal(35, 8, 10_000) # feature at training time
live_ages = np.random.normal(42, 8, 2_000) # same feature, months later
print("PSI:", round(population_stability_index(train_ages, live_ages), 3))
That handful of lines is an early-warning system: when the world your model lives in stops looking like the world it was trained on, PSI climbs and you find out before your business metric craters, in stead of a month after. It's a taste of a much bigger topic we'll dig into properly very soon -- watching a living system, not just launching a dead one.
Three tasks to chew on before the next episode. As always, wrestle with them yourself first -- I'll walk through full solutions next time.
Frame three problems. Pick three real-ish scenarios (say: "detect fraudulent transactions", "auto-tag support tickets", "predict tomorrow's server load") and fill in an MLProblemFrame for each. For every one, force yourself to write a genuine non-ML baseline and a success_metric that is NOT accuracy. Then, using should_use_ml, score each and defend the verdict in a sentence.
Spend a latency budget. Extend latency_budget so it flags any single stage that exceeds a max_fraction cap (say 0.4) and prints a warning. Then model a two-stage recommender (candidate generation + ranking) under a 100ms budget and decide, with numbers, which stage you'd push to run in batch to buy yourself headroom.
Catch the skew. Write a tiny simulation: compute a feature two ways -- a "training" version using a 30-day mean and a "serving" version that accidentically uses a 7-day mean -- on the same synthetic data, then run population_stability_index between the two. Watch the PSI climb, and write one line on how you'd have caught this in production before a human noticed the metric sagging.
@scipio