pip install fastapi uvicorn covers the web bits);Learn AI Series):Last episode we spent the whole time making a model fast. And at the very end I grumbled that a fast model sitting on your laptop is still just being fast at nobody. Today we fix that. We build the front door -- the thing that takes a request from the outside world, hands it to your model, and hands an answer back before the user gives up and closes the tab.
This is the bridge between "impressive notebook" and "working product", and I'll be honest with you: it's where an astonishing number of ML projects quietly go to die. Not because the model was bad. Because nobody ever wired it into something a real user could actually call. The model was 95% accurate and served exactly zero people ;-)
Here's the mental shift that has to happen, and it's a big one. Model serving is software engineering, not data science. Once you're serving, the model is just one component in a larger system, and the questions that keep you up at night change completely. It's no longer "what's my F1 score" -- it's latency budgets, request routing, what happens when the model process crashes at 3 AM, and how you roll out v2 without taking down v1. Different world. Same you, hopefully a little wiser by the end of this post.
Before we reach for heavyweight infrastructure with scary names, let's build a serving layer from scratch. You learn what the big servers are doing for you by first doing it badly yourself -- that's been the whole philosophy of this series, and I'm not about to abandon it now.
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
import torch
import torch.nn as nn
import numpy as np
import time
from contextlib import asynccontextmanager
# The model lives here, loaded exactly once at startup
model = None
@asynccontextmanager
async def lifespan(app: FastAPI):
global model
model = load_model("model.pt")
model.eval() # inference mode: no dropout, no batchnorm updates
yield
del model # clean shutdown
app = FastAPI(lifespan=lifespan)
class PredictionRequest(BaseModel):
features: list[float]
class PredictionResponse(BaseModel):
prediction: int
confidence: float
latency_ms: float
model_version: str
def load_model(path):
model = nn.Sequential(
nn.Linear(784, 256), nn.ReLU(),
nn.Linear(256, 10),
)
model.load_state_dict(torch.load(path, weights_only=True))
return model
@app.post("/predict", response_model=PredictionResponse)
async def predict(request: PredictionRequest):
start = time.perf_counter()
if len(request.features) != 784:
raise HTTPException(400, "Expected 784 features")
tensor = torch.FloatTensor(request.features).unsqueeze(0)
with torch.no_grad():
logits = model(tensor)
probs = torch.softmax(logits, dim=1)
pred = probs.argmax(dim=1).item()
conf = probs.max().item()
latency = (time.perf_counter() - start) * 1000
return PredictionResponse(
prediction=pred,
confidence=conf,
latency_ms=round(latency, 2),
model_version="v1.0.0",
)
@app.get("/health")
async def health():
return {"status": "healthy", "model_loaded": model is not None}
Notice a few things that already matter. The model loads once at startup (in lifespan), not per request -- loading a model is expensive and doing it every call would be self-sabotage. There's a /health endpoint, because every orchestrator on earth wants to poke your service and ask "you alive?". And the response carries a model_version, which sounds trivial until you're debugging a weird prediction in production and desperately need to know which model actually answered.
This little server is genuinely fine for prototypes and low-traffic internal tools. But it has cracks that widen fast under load: a single Python process handles concurrent requests poorly, the model is frozen at startup with no way to hot-swap it, there's no batching to amortize GPU overhead, and if the model throws, the whole request just dies with a 500. Every one of those cracks is something the rest of this episode patches.
REST (HTTP plus JSON) is the default for web APIs, and for good reason -- it's universal, human-readable, and you can debug it with curl while half asleep. gRPC (Protocol Buffers over HTTP/2) is the choice when you're doing high-performance service-to-service calls and every millisecond of serialization counts.
# REST: simple, universal, human-readable
# curl -X POST http://localhost:8000/predict \
# -H "Content-Type: application/json" \
# -d '{"features": [0.1, 0.2, ...]}'
# gRPC: faster serialization, native streaming, generated client code.
# You define the contract in a .proto file first:
#
# service PredictionService {
# rpc Predict(PredictionRequest) returns (PredictionResponse);
# rpc PredictStream(stream PredictionRequest) returns (stream PredictionResponse);
# }
# Rough real-world comparison:
comparison = {
'serialization': {'REST/JSON': '1-5 ms', 'gRPC/protobuf': '0.1-0.5 ms'},
'payload_size': {'REST/JSON': '~2x bigger', 'gRPC/protobuf': 'baseline'},
'streaming': {'REST/JSON': 'SSE/WebSocket workaround', 'gRPC': 'native'},
'browser_support': {'REST/JSON': 'works everywhere', 'gRPC': 'needs grpc-web proxy'},
'debugging': {'REST/JSON': 'curl, any browser', 'gRPC': 'grpcurl, reflection'},
}
The rule of thumb I keep in my head: REST for anything a browser or a third party touches, gRPC for internal plumbing where latency and streaming matter. JSON is text -- readable, sure, but that readability costs you parsing time and roughly double the bytes on the wire. Protocol Buffers are a compact binary format with a schema, so both ends know the exact shape ahead of time and skip all the guessing.
There's one place gRPC stops being optional and becomes basically mandatory: serving LLMs. When you generate text token by token (episode #71), you do not want to make the user stare at a spinner until the entire paragraph is finished. You want to stream each token the instant it's produced, the way every chatbot you've used does. gRPC streaming (or Server-Sent Events on the REST side, if you must) is how that gets built. A part from that specific case, honestly, most teams start with REST and only introduce gRPC once they've actually measured a serialization bottleneck. Premature protocol optimization is a real way to waste a week ;-)
Here's a question that saves people enormous amounts of money and infrastructure, and almost nobody asks it early enough: does this prediction actually need to happen the moment the user asks?
class BatchPredictor:
"""Precompute predictions for known entities, store the answers."""
def __init__(self, model, feature_store, output_store):
self.model = model
self.feature_store = feature_store
self.output_store = output_store
def run(self, entity_ids):
self.model.eval()
results = {}
# Chunk it so we never blow up memory on a huge entity list
chunk_size = 1024
for i in range(0, len(entity_ids), chunk_size):
chunk_ids = entity_ids[i:i + chunk_size]
features = self.feature_store.get_batch(chunk_ids)
tensor = torch.FloatTensor(features)
with torch.no_grad():
probs = torch.softmax(self.model(tensor), dim=1)
for eid, prob in zip(chunk_ids, probs):
results[eid] = {
'prediction': prob.argmax().item(),
'scores': prob.tolist(),
}
self.output_store.write(results)
return len(results)
# Run it on a schedule (say, nightly at 2 AM).
# At serve time you just do a database lookup -- zero model inference.
Look at what just happened there. If your predictions don't need to reflect the last few seconds of reality, you can compute them all in one big overnight job and serve them as database lookups. No GPU on the hot path. No model server to keep alive. No latency spikes when traffic surges, because "serving" is now a key-value read. Wowzers -- the cheapest, most reliable inference is often the inference you did hours ago.
An enormous slice of production ML runs exactly this way: recommendation feeds, credit and risk scores, content classification, churn predictions. They all tolerate being a few hours stale, so they get precomputed in bulk. You reach for real-time inference only when freshness genuinely matters -- fraud detection on a card swipe happening right now, a search query nobody has ever typed before, a live chat response. When you don't need fresh, don't pay for fresh.
Alright, say you do need real-time. Here's the uncomfortable truth about GPUs: feeding them one request at a time is like driving a 50-seat bus to pick up a single passenger. The hardware is built for parallel throughput, and a batch of one wastes almost all of it. Dynamic batching is the fix -- collect requests arriving within a tiny time window, run them through the model together, then hand each caller back their own slice of the result.
import asyncio
from collections import deque
class DynamicBatcher:
"""Merge individual requests into batches for efficient GPU usage."""
def __init__(self, model, max_batch_size=32, max_wait_ms=10):
self.model = model
self.max_batch_size = max_batch_size
self.max_wait_ms = max_wait_ms
self.queue = deque()
self.processing = False
async def predict(self, features):
"""One request in, one batched result out -- caller never notices."""
future = asyncio.Future()
self.queue.append((features, future))
if not self.processing:
self.processing = True
asyncio.create_task(self._process_batch())
return await future
async def _process_batch(self):
# Wait a hair for more requests to pile in, THEN run the batch
await asyncio.sleep(self.max_wait_ms / 1000)
batch_items = []
while self.queue and len(batch_items) < self.max_batch_size:
batch_items.append(self.queue.popleft())
if not batch_items:
self.processing = False
return
features_batch = torch.FloatTensor([item[0] for item in batch_items])
with torch.no_grad():
results = self.model(features_batch)
# Hand each waiting caller its own row of the output
for (_, future), result in zip(batch_items, results):
future.set_result(result.tolist())
# Drain anything that arrived while we were busy
if self.queue:
asyncio.create_task(self._process_batch())
else:
self.processing = False
The trade here is beautiful in its honesty: you pay a small, bounded latency cost (the wait window, typically 5-20ms) in exchange for a throughput explosion. A GPU chewing through one request at a time might manage 100 requests per second. Batch 32 of them together and the same card handles 2000-plus. You didn't buy better hardware -- you just stopped wasting the hardware you had.
This is not a party trick I invented, by the way. It is precisely how Triton Inference Server and vLLM hit the throughput numbers they brag about. The max_wait_ms knob is where the tuning lives: too short and you barely batch anything, too long and low-traffic requests sit around waiting for friends who never arrive. There's no universal right value -- it depends on your traffic shape, and you find it by measuring (as always in this series).
You can hand-roll all of the above. For production you usually shouldn't, because the good model servers already did it, tested it under fire, and threw in a dozen features you'd otherwise reinvent badly.
# TorchServe -- PyTorch's official model server
#
# 1. Package the model into a .mar archive
# $ torch-model-archiver --model-name classifier \
# --version 1.0 --model-file model.py \
# --serialized-file model.pt --handler handler.py \
# --export-path model_store
#
# 2. Start serving
# $ torchserve --start --model-store model_store \
# --models classifier=classifier.mar
#
# 3. Predict
# $ curl http://localhost:8080/predictions/classifier -T input.json
# Triton Inference Server -- NVIDIA's multi-framework beast.
# Speaks PyTorch, TensorFlow, ONNX, TensorRT, and custom backends.
# You describe the model with a config file:
triton_config = """
name: "classifier"
platform: "onnxruntime_onnx"
max_batch_size: 64
input [{ name: "input", data_type: TYPE_FP32, dims: [784] }]
output [{ name: "output", data_type: TYPE_FP32, dims: [10] }]
dynamic_batching {
preferred_batch_size: [16, 32]
max_queue_delay_microseconds: 10000
}
instance_group [{ count: 2, kind: KIND_GPU }]
"""
Read that Triton config again, because it's doing a lot quietly. dynamic_batching -- the whole DynamicBatcher class we just wrote by hand, now a four-line config block. instance_group with count: 2 -- run two copies of the model on the GPU concurrently, so one request waiting on memory doesn't stall the next. On top of that you get model versioning, health metrics, multi-GPU spreading, and hot model reloads without downtime. That's a mountain of undifferentiated plumbing you get to not write.
For LLMs specifically, vLLM (which we met back in episode #70) is the standard bearer. Its PagedAttention trick manages the GPU's memory the way an operating system manages RAM with paging -- far less waste than a general-purpose server -- and that's a chunk of why it serves large language models so much more efficiently than the alternatives. The lesson generalises: match the server to the workload. Triton for classic models, vLLM for generative text.
If the exact same input shows up more than once, recomputing the prediction is just setting money on fire. Cache it.
import hashlib
class PredictionCache:
"""Skip the model entirely when we've seen this input before."""
def __init__(self, model, cache_size=10000):
self.model = model
self.cache = {}
self.cache_size = cache_size
self.hits = 0
self.misses = 0
def predict(self, features):
key = self._hash_features(features)
if key in self.cache:
self.hits += 1
return self.cache[key]
self.misses += 1
result = self._model_predict(features)
self.cache[key] = result
# Crude size cap: evict the oldest entry when we overflow
if len(self.cache) > self.cache_size:
oldest = next(iter(self.cache))
del self.cache[oldest]
return result
def _hash_features(self, features):
return hashlib.md5(np.array(features).tobytes()).hexdigest()
def _model_predict(self, features):
tensor = torch.FloatTensor(features).unsqueeze(0)
with torch.no_grad():
return self.model(tensor).squeeze().tolist()
@property
def hit_rate(self):
total = self.hits + self.misses
return self.hits / total if total > 0 else 0
Whether caching is worth the trouble depends entirely on how repetitive your inputs are, and the spread is enormous. A search-ranking service might see 30-60% cache hits, because loads of people search for the same popular things. A fraud detector sees close to 0%, because every transaction is a unique fingerprint of amount, time, and location -- nothing ever repeats. My blunt rule: if your hit rate clears 10%, caching earns its keep; below that, you've added a moving part for nothing. And do measure the real hit rate before you fall in love with the idea. Caching is one of those features that feels clever and sometimes does absolutely nothing.
Traffic is never flat. It breathes -- quiet at 4 AM, hammering at lunchtime, spiking when someone posts your product on a big forum. Running enough servers to survive your worst spike means paying for idle machines all night; running enough for the average means falling over exactly when people show up. Autoscaling is how you stop guessing and let demand drive the number of replicas.
class AutoScaler:
"""Add replicas when we're swamped, remove them when it's quiet."""
def __init__(self, min_replicas=2, max_replicas=20, target_latency_ms=100):
self.min_replicas = min_replicas
self.max_replicas = max_replicas
self.target_latency_ms = target_latency_ms
self.replicas = min_replicas
def decide(self, current_p95_ms, current_replicas):
"""Scale on observed P95 latency vs the target we promised."""
ratio = current_p95_ms / self.target_latency_ms
if ratio > 1.2: # we're 20%+ over budget -> scale OUT
desired = min(self.max_replicas, current_replicas + 1)
elif ratio < 0.5: # comfortably under budget -> scale IN
desired = max(self.min_replicas, current_replicas - 1)
else:
desired = current_replicas # inside the deadband, leave it
return desired
# In the real world Kubernetes runs this loop for you via an HPA:
#
# apiVersion: autoscaling/v2
# kind: HorizontalPodAutoscaler
# spec:
# minReplicas: 2
# maxReplicas: 20
# metrics:
# - type: Pods
# pods:
# metric: { name: inference_p95_latency_ms }
# target: { type: AverageValue, averageValue: "100" }
Two details make or break this in practice. First, that deadband in the middle (the "leave it alone" zone) is not laziness -- without it, the scaler oscillates, adding and removing replicas every few seconds as latency jitters across the threshold. That flapping is worse than doing nothing. Second, and this is the one that bites GPU teams hard: scaling out is not instant. Spinning up a new replica means scheduling a pod, pulling a multi-gigabyte image, and loading the model into GPU memory -- easily 30 to 90 seconds. So you scale on a leading signal and keep a small buffer of headroom. If you wait until you're already drowning, the reinforcements arrive a minute after the battle is lost. Nota bene: a min_replicas of at least 2 is not optional in production -- one replica means one crash equals total outage.
A serious serving layer plans for the model to fail, because eventually it will -- a bad input, an out-of-memory blip, a corrupted checkpoint. The question is never if, it's what happens then. The wrong answer is a 500 and a shrug. The right answer is a sensible fallback.
def predict_with_fallback(features, model, cache, default_response):
"""Try the real model; degrade gracefully instead of exploding."""
try:
return {"prediction": model(features), "source": "model"}
except Exception:
# 1. A slightly stale cached answer beats no answer at all
cached = cache.get(features)
if cached is not None:
return {"prediction": cached, "source": "cache_fallback"}
# 2. Last resort: a safe default (e.g. the majority class)
return {"prediction": default_response, "source": "default_fallback"}
The whole philosophy in one line: a slightly worse answer, served, beats a perfect answer that never arrives. A recommendation service that can't run its fancy model should fall back to "here are this week's most popular items" -- not throw an error page. The user gets something reasonable and never learns the clever model hiccuped. Notice we also tag every response with its source, so when you're staring at your dashboards you can see how often you're limping along on fallbacks instead of the real thing. A rising fallback rate is an early smoke alarm, and you want to hear it before the fire.
Final piece, and it's the one that saves your reputation. You have a shiny new v2 model. Every instinct screams to deploy it everywhere at once. Please don't. You have offline metrics, sure, but offline metrics have lied to all of us at least once -- the real world always has a surprise waiting.
import random
class ModelRouter:
"""Split live traffic across model versions for A/B tests and canaries."""
def __init__(self):
self.models = {}
self.traffic_split = {} # name -> percentage of traffic
def register_model(self, name, model, traffic_pct):
self.models[name] = model
self.traffic_split[name] = traffic_pct
def route(self, request):
roll = random.random() * 100
cumulative = 0
for name, pct in self.traffic_split.items():
cumulative += pct
if roll < cumulative:
return {
'prediction': self.models[name](request),
'model_version': name,
}
# Safety net: fall through to the first registered model
name = next(iter(self.models))
return {'prediction': self.models[name](request), 'model_version': name}
# Canary: 95% stay on the trusted model, 5% try the new one
router = ModelRouter()
router.register_model("v1.0_stable", model_v1, traffic_pct=95)
router.register_model("v1.1_canary", model_v2, traffic_pct=5)
# Watch the canary for a day or two. Good? 5% -> 25% -> 50% -> 100%.
# Bad? Flip it back to 100% stable in one move.
A canary deployment limits the blast radius. If v2 turns out to be broken, only 5% of your users ever felt it, and you roll back in seconds. The metrics to actually watch during a canary are a tidy little trio: latency (did it get slower?), error rate (does it fail more often?), and -- the sneaky one -- business metrics (do users behave differently? are they clicking, buying, staying?). That last one catches the nastiest failure of all: a model that's technically faster and lower-error while quietly making worse predictions that nobody notices until revenue dips. And whatever you build, wire in a one-click rollback before you need it. The middle of an incident is the worst possible time to go writing your escape hatch.
Three things to wrestle with before the next episode. Fight them yourself first -- as always, I'll walk through full solutions next time.
Feel the batching win. Take the DynamicBatcher above and wrap it around any small PyTorch model. Fire 200 concurrent requests at it using asyncio.gather, and measure total throughput (requests/second) with max_batch_size=1 versus max_batch_size=32. Then report how median latency changed too -- you should see throughput soar while individual latency ticks up slightly. Write one sentence on whether that trade is worth it for a service promising a 100ms P95.
Batch or real-time? Pick three concrete products -- say, a Netflix-style "recommended for you" row, a credit-card fraud check, and an autocomplete-as-you-type search box. For each, decide whether you'd serve it with batch prediction or real-time inference, and justify it in two sentences per product using the freshness question from this episode. There's a defensible answer for each; I want your reasoning, not just the label.
Build a canary. Using the ModelRouter, register two "models" that are just functions returning different constants (pretend one is v1, one is v2). Route 10,000 requests through a 90/10 split and confirm the actual traffic distribution lands near 90/10. Bonus: add a per-model counter for a fake "error" (have v2 randomly raise 3% of the time) and print each model's observed error rate -- the exact signal you'd watch to decide whether to promote or roll back the canary.
That's the front door built. The model now takes requests from the outside world and answers them -- batched, cached, autoscaled, and safe to roll out. But notice we've assumed the whole time that the model lives on a fat server with a GPU nearby. What happens when it has to run somewhere with no server at all -- on the phone in your pocket, offline, on a battery that hates you? That's a genuinely different set of constraints, and it's exactly where we're headed next.
@scipio