pip install torch) -- the snippets here are illustrative, nothing needs a GPU;Learn AI Series):I closed last episode with a deliberate teaser -- a "modality" that is oddly special. It is simultaneously human language and rigid machine logic. It comes with instant, brutal feedback: it either runs or it does not, no committee required. And the models that learn to speak it start to amplify the very people who build everything else in this series. That modality is code, and today we point all the AI machinery we've been assembling for 139 episodes straight at it.
Here is what makes code such a fascinating target. Natural language is squishy -- "correct" is subjective, and two fluent sentences can both be fine. Code is not squishy at all. It has an OBJECTIVE test built in: does it compile, does it run, does it produce the right output for the input you gave it? That single property -- verifiable ground truth -- turns out to change everything about how these systems are trained, evaluated, and made to improve themselves. Let's dive right in ;-)
As always, we open by settling last time's homework. Episode #138 was multimodal fusion -- early, late, and cross-attention -- so the exercises were all about tracing tensors through those architectures.
Exercise 1 -- Trace the shapes, then add a third modality. The EarlyFusionEncoder concatenates text tokens and image patches into one sequence. With 15 text tokens and 64 image patches (an 8x8 grid), the output length on paper is simply the SUM: 15 + 64 = 79. Adding a second image as a third modality means a third modality-embedding id and another 64 patches, giving 15 + 64 + 64 = 143.
# Solution 1: EarlyFusionEncoder from #138, extended to a SECOND image stream.
import torch
import torch.nn as nn
class EarlyFusionThreeWay(nn.Module):
"""Text + image A + image B, all poured into one shared sequence."""
def __init__(self, text_vocab, img_patch_dim, hidden_dim, n_heads=8, n_layers=4):
super().__init__()
self.text_embed = nn.Embedding(text_vocab, hidden_dim)
self.img_proj = nn.Linear(img_patch_dim, hidden_dim)
self.modality_embed = nn.Embedding(3, hidden_dim) # 0=text, 1=img A, 2=img B
layer = nn.TransformerEncoderLayer(
hidden_dim, n_heads, dim_feedforward=hidden_dim * 4, batch_first=True
)
self.transformer = nn.TransformerEncoder(layer, n_layers)
def _tag(self, x, mod_id):
ids = torch.full(x.shape[:2], mod_id, dtype=torch.long, device=x.device)
return x + self.modality_embed(ids)
def forward(self, text_ids, img_a, img_b):
t = self._tag(self.text_embed(text_ids), 0)
a = self._tag(self.img_proj(img_a), 1)
b = self._tag(self.img_proj(img_b), 2)
combined = torch.cat([t, a, b], dim=1)
return self.transformer(combined)
model = EarlyFusionThreeWay(text_vocab=10000, img_patch_dim=768, hidden_dim=256)
text = torch.randint(0, 10000, (2, 15)) # 15 text tokens
img_a = torch.randn(2, 64, 768) # 8x8 = 64 patches
img_b = torch.randn(2, 64, 768) # a second image, another 64
out = model(text, img_a, img_b)
print(f"sequence length: {out.shape[1]}") # 15 + 64 + 64 = 143
Why should that worry you? Because attention is quadratic in sequence length. Going from 79 to 143 tokens does not add a bit of cost -- it multiplies the attention matrix from roughly 6,200 entries to over 20,000. Every extra modality you bolt onto early fusion inflates the ONE shared sequence, and the bill grows with the square. That is precisely the pressure that pushes real systems toward cross-attention, where each modality keeps its own (shorter) sequence.
Exercise 2 -- Late vs cross-attention, by hand. Count the parameters and you feel the difference immediately.
# Solution 2: parameter counts of the two fusion styles from #138.
class LateFusion(nn.Module):
def __init__(self, text_dim, img_dim, hidden_dim, n_classes):
super().__init__()
self.text_encoder = nn.Sequential(
nn.Linear(text_dim, hidden_dim), nn.ReLU(), nn.Linear(hidden_dim, hidden_dim))
self.img_encoder = nn.Sequential(
nn.Linear(img_dim, hidden_dim), nn.ReLU(), nn.Linear(hidden_dim, hidden_dim))
self.classifier = nn.Linear(hidden_dim * 2, n_classes)
class CrossAttentionFusion(nn.Module):
def __init__(self, dim, n_heads=8):
super().__init__()
self.text_self = nn.MultiheadAttention(dim, n_heads, batch_first=True)
self.img_self = nn.MultiheadAttention(dim, n_heads, batch_first=True)
self.text_cross = nn.MultiheadAttention(dim, n_heads, batch_first=True)
self.img_cross = nn.MultiheadAttention(dim, n_heads, batch_first=True)
self.norms = nn.ModuleList([nn.LayerNorm(dim) for _ in range(4)])
def count(m):
return sum(p.numel() for p in m.parameters())
late = LateFusion(text_dim=256, img_dim=256, hidden_dim=256, n_classes=10)
cross = CrossAttentionFusion(dim=256, n_heads=8)
print(f"late fusion: {count(late):,}")
print(f"cross-attn: {count(cross):,}")
The extra parameters in the cross-attention version live almost entirely in those four MultiheadAttention blocks -- two self-attention, two cross-attention. What are you buying with them? Interaction DURING encoding. In late fusion the two encoders are strangers until the final cat; they never once look at each other's intermediate representations. Cross-attention lets text tokens query image tokens (and vice versa) while both are still being shaped. You pay in parameters and you pay in fiddliness, and in return the model can discover subtle cross-modal relationships that late fusion structurally cannot.
Exercise 3 -- Design a real pipeline. I picked receipt extraction, because it is beautifully multimodal and genuinely useful.
# Solution 3: a receipt-to-JSON pipeline, sketched as a chain of specialists.
# Step 1: OCR (episode #82) pulls raw text + bounding boxes from the image.
# Step 2: a multimodal model reads image + OCR text together -- layout matters
# ("total" is usually bottom-right, bold, larger font).
# Step 3: an LLM structures the messy result into {store, date, items[], total}.
# Step 4: VALIDATION in plain Python -- do the line items sum to the total?
#
# Where the human check goes:
# - total above some threshold (an expensive mistake is worse than a slow one)
# - the sum-check in step 4 disagrees with the printed total
# - OCR confidence is low on the total field
# Everything else auto-approves. The failure modes from #138 -- fine-detail
# misreads, confident hallucination -- are exactly what step 4 guards against.
def needs_human(receipt, ocr_conf, threshold=250.0):
line_sum = round(sum(i["price"] for i in receipt["items"]), 2)
mismatch = abs(line_sum - receipt["total"]) > 0.01
return receipt["total"] > threshold or mismatch or ocr_conf < 0.85
The architecture reasoning is the point: a relay team of specialists, with a cheap deterministic check catching the model's most likely mistakes before a human ever gets involved. Right -- homework settled. On to code.
At their core, code models are language models. The same transformer architecture from #52-53, the same next-token objective from #57. What differs is the DIET.
The data. Code models eat enormous corpora of source code. StarCoder trained on The Stack -- permissively licensed GitHub code, over a trillion tokens across 80-plus languages. CodeLlama chewed through hundreds of billions of code-heavy tokens. DeepSeek-Coder went with roughly 2 trillion, about 87% code and 13% natural language.
That natural-language slice is not filler -- it is load-bearing. Code never lives alone. It arrives wrapped in comments, docstrings, commit messages, issue threads, and long-suffering StackOverflow arguments. This paired code-and-prose data is what teaches the model the mapping between what a programmer WANTS (in English) and how you express that (in Python). Strip the prose and you get a model that autocompletes syntax but has no idea what you're asking for.
The objective. Same as any language model -- predict the next token. Given def fibonacci(n):\n if n <= 1:\n return, the model should predict n. Sounds almost insultingly simple. But do that across billions of tokens and millions of files, and the model quietly absorbs syntax, algorithms, design patterns, API conventions, and debugging habits. All of it, from pure next-token prediction. I still find that a little bit magical, honestly.
Fill-in-the-middle (FIM). Here is the one augmentation you cannot skip for code. Plain left-to-right modeling only ever trains the model to continue text forward. But when you're coding in an editor, your cursor is almost never at the end of the file -- it's smack in the MIDDLE, with code above it AND below it. FIM training fixes this by randomly slicing a file into prefix, middle, suffix, and asking the model to reconstruct the middle given the two ends.
# Fill-in-the-middle: reshuffle a file so the model learns to predict the middle.
def create_fim_example(code: str):
"""Split into prefix/middle/suffix and reorder for FIM training."""
n = len(code)
split_start = int(n * 0.3)
split_end = int(n * 0.7)
prefix = code[:split_start]
middle = code[split_start:split_end]
suffix = code[split_end:]
# FIM format the model is trained on: prefix suffix middle
fim_input = f"{prefix}{suffix}"
return fim_input, middle
sample = ("def merge_sort(arr):\n"
" if len(arr) <= 1:\n"
" return arr\n"
" mid = len(arr) // 2\n"
" left = merge_sort(arr[:mid])\n"
" right = merge_sort(arr[mid:])\n"
" return merge(left, right)")
fim_input, target = create_fim_example(sample)
print("model sees prefix + suffix, must produce the middle")
print(f"target length: {len(target)} characters")
Nota bene: the special tokens <PRE>, <SUF>, <MID> are literally added to the tokenizer vocabulary. The model learns that when it sees <MID>, its job flips from "continue forwards" to "bridge the gap". That single trick is why editor autocomplete feels like it understands context on BOTH sides of your cursor.
Code generation is not one task -- it is a spectrum, and reliability drops off a cliff as you climb it.
Line completion. The humblest form: finish the current line. This is what IDE autocomplete does, and it works remarkably well because the model only needs local context -- the variables in scope, the function's evident purpose, the surrounding style.
Function generation. Give it a signature and a docstring, get back a body. Now the model must understand INTENT (what does the docstring describe?) and produce a correct IMPLEMENTATION. Much harder, but often genuinely useful for standard patterns.
Multi-file generation. Give it a project and requirements, get back several files that must cooperate. This is where today's models start to sweat -- keeping imports consistent, respecting architectural decisions, reasoning about a large context all at once. We are not there yet, not reliably.
And because code has that objective test I keep banging on about, we can EVALUATE generation properly -- by running it.
# Evaluate generated code by actually executing it against test cases.
import ast
def evaluate_generated_code(code: str, test_cases: list) -> dict:
results = {"syntactically_valid": False, "tests_passed": 0,
"tests_total": len(test_cases), "errors": []}
try:
ast.parse(code)
results["syntactically_valid"] = True
except SyntaxError as e:
results["errors"].append(f"SyntaxError: {e}")
return results
namespace = {}
try:
exec(code, namespace) # illustrative only -- never exec untrusted code raw
except Exception as e:
results["errors"].append(f"Execution error: {e}")
return results
func = None
for name, obj in namespace.items():
if callable(obj) and not name.startswith("_"):
func = obj
if func is None:
results["errors"].append("no callable function found")
return results
for inputs, expected in test_cases:
try:
actual = func(*inputs)
if actual == expected:
results["tests_passed"] += 1
else:
results["errors"].append(f"{inputs}: expected {expected}, got {actual}")
except Exception as e:
results["errors"].append(f"{inputs}: {e}")
return results
generated = ("def fibonacci(n):\n"
" if n <= 0: return 0\n"
" if n == 1: return 1\n"
" a, b = 0, 1\n"
" for _ in range(2, n + 1):\n"
" a, b = b, a + b\n"
" return b\n")
tests = [((0,), 0), ((1,), 1), ((5,), 5), ((10,), 55), ((20,), 6765)]
r = evaluate_generated_code(generated, tests)
print(f"syntax ok: {r['syntactically_valid']}, tests: {r['tests_passed']}/{r['tests_total']}")
NB: I left a comment on that exec line for a reason -- running model-generated code straight into your own interpreter is how you get a very bad afternoon. Sandboxes exist. Use them.
This "run it and count passes" idea is exactly how the field measures itself. The HumanEval benchmark (from OpenAI, alongside Codex) is 164 hand-written Python problems, each with a signature, docstring, and hidden tests. The metric, pass@k, asks how often at least one of k sampled solutions passes ALL the tests. Modern models clear 90%-plus on pass@1 for HumanEval -- which sounds triumphant until you notice the benchmark is basically saturated. The honest measure now is SWE-bench, which hands the model a real GitHub issue and a whole repository and asks for a patch that resolves it. Scores there are dramatically lower, and THAT gap -- toy problems versus real repositories -- is the whole story of AI for code in one number.
Generation grabs the headlines, but understanding is arguably the bigger prize for daily work. Most of a programmer's life is spent reading code, not writing it.
Explanation. Point the model at a function, get plain-English of what it does. Brilliant for onboarding onto an unfamiliar codebase, or for figuring out what a gnarly bit ACTUALLY does versus what its (lying) comment claims.
Review. Flag bugs, security holes, performance traps, style slips. This goes beyond a linter -- a model can spot logical errors, suggest a better algorithm, and notice code that works today but is fragile.
Dependency analysis. Map which functions call which, where a change ripples, what's dead. This is the sweet spot for combining a precise AST parser with a reasoning model -- structure from the parser, meaning from the model.
# AST-based structure extraction: precise facts to hand a model as context.
import ast
class CodeAnalyzer(ast.NodeVisitor):
def __init__(self):
self.functions, self.classes, self.imports = [], [], []
def visit_FunctionDef(self, node):
self.functions.append({
"name": node.name,
"args": [a.arg for a in node.args.args],
"docstring": ast.get_docstring(node),
"complexity": self._complexity(node),
})
self.generic_visit(node)
def visit_ClassDef(self, node):
methods = [n.name for n in ast.walk(node) if isinstance(n, ast.FunctionDef)]
self.classes.append({"name": node.name, "methods": methods})
self.generic_visit(node)
def visit_Import(self, node):
for alias in node.names:
self.imports.append(alias.name)
def _complexity(self, node):
"""Rough cyclomatic complexity: count the branch points."""
branches = sum(1 for n in ast.walk(node)
if isinstance(n, (ast.If, ast.For, ast.While,
ast.ExceptHandler, ast.With)))
return branches + 1
code = ("import torch\n"
"class Encoder:\n"
" def forward(self, x):\n"
" if x is None:\n"
" raise ValueError('no input')\n"
" for _ in range(3):\n"
" x = x * 2\n"
" return x\n")
a = CodeAnalyzer()
a.visit(ast.parse(code))
print(f"classes: {[c['name'] for c in a.classes]}")
print(f"imports: {a.imports}")
for f in a.functions:
print(f" {f['name']}: complexity={f['complexity']}, args={f['args']}")
The parser gives you facts that are TRUE by construction -- it does not hallucinate a function that isn't there. The model gives you meaning -- intent, correctness, smell. Feed the parser's precise output into the model as grounding (this should ring a bell from RAG, #64) and you get something more trustworthy than either alone. That pattern -- deterministic tool for facts, model for judgement -- is one of the most useful things in this entire episode. Remember it.
If you want the single most immediately practical use of code AI, it's testing. Writing tests is tedious, so developers skip it, so codebases rot. A model that generates meaningful tests from existing code saves real hours and catches real bugs.
Test generation. Hand it a function, get tests that hit edge cases, boundaries, and typical usage. Because the model trained on millions of code-plus-test pairs, it has soaked up an implicit sense of what a decent test suite looks like.
# Prompt template that steers a model toward COMPREHENSIVE tests.
def test_prompt(func_code: str) -> str:
return f"""Write pytest unit tests for the function below. Cover:
- normal cases
- edge cases (empty, zero, None)
- boundary conditions
- expected exceptions
--- function under test ---
{func_code}
--- end function ---
Return only the test functions."""
func = ("def calculate_discount(price, pct):\n"
" if price < 0:\n"
" raise ValueError('price cannot be negative')\n"
" if not 0 <= pct <= 100:\n"
" raise ValueError('pct must be 0..100')\n"
" return round(price * (1 - pct / 100), 2)")
print(test_prompt(func)[:120], "...")
Notice how much of the quality is in the PROMPT, not the model -- spelling out the categories you want covered is prompt engineering (#62) applied to a code task. Vague prompt, vague tests.
Bug localisation and fixing. Given a failing test and a repo, which file and function is guilty? And then: produce a patch that turns the test green without breaking the other fifty. This is SWE-bench again, and current agents solve somewhere in the 30-50% range depending on model and scaffolding -- impressive, and nowhere near "fire the QA team".
Here is where code's objective feedback becomes a genuine superpower: the model can CHECK ITSELF and try again. This self-repair loop is the quiet engine behind most serious coding agents (#67-68).
# The execution-feedback loop: generate, run, read the error, retry.
def self_repair(spec, tests, generate_fn, max_attempts=3):
"""generate_fn(spec, feedback) -> candidate code string (a model call in real life)."""
feedback = None
for attempt in range(1, max_attempts + 1):
code = generate_fn(spec, feedback)
result = evaluate_generated_code(code, tests)
if result["tests_passed"] == result["tests_total"]:
return code, attempt # success -- objective proof, not vibes
feedback = result["errors"] # feed the REAL failure back in
return None, max_attempts
That loop is only possible because code answers back. You cannot do this with an essay -- there is no essay.run() that returns a stack trace. Code gives the model a teacher that never gets tired, and that is why coding models improve faster than almost any other application of LLMs.
This is the frontier that genuinely excites me. Formal verification proves, mathematically, that code meets its specification -- not for the handful of test cases you happened to write, but for ALL possible inputs. Historically it has been brutally labour-intensive and reserved for aerospace, chip design, and cryptography.
AI is prying that door open from two sides. First, models can draft formal specifications -- preconditions, postconditions, invariants -- from an informal English description, lowering the barrier to even getting started. Second, models can SEARCH for proofs, navigating the enormous tree of possible proof steps in systems like Lean 4, Coq, or Isabelle -- a search problem that looks a lot like the game-playing we did back in #112.
The payoff, if this matures, is code with correctness GUARANTEES -- mathematical proof rather than statistical confidence. For safety-critical systems that is not a nice-to-have, it's the whole ballgame. Early days, but keep an eye on it.
Let me be blunt about what actually works, because the hype is exhausting.
Works well: single-line and short-block autocomplete; boilerplate (CRUD, config, scaffolding); explaining unfamiliar code (often faster than the docs); generating tests from existing functions; translating straightforward code between languages; writing SQL from a plain description.
Works sometimes: whole functions from a description (heavily dependent on how specific you were); review for non-obvious bugs (catches some, misses others); refactoring suggestions (good ideas, occasional regressions); debugging from an error message (fine for common patterns, lost on the subtle ones).
Does NOT work reliably: multi-file changes needing real architectural understanding; performance optimisation (models write READABLE code, not necessarily FAST code); security-sensitive code (they reproduce common patterns -- including the insecure ones); anything depending on a specific library version or undocumented behaviour; genuinely novel algorithms (models interpolate from training data, they do not invent).
The fundamental point, and I want you to sit with it: code models are pattern matchers at colossal scale. They have seen so much code that they can reproduce almost any COMMON pattern -- but they do not "understand" code the way you do. They cannot reason about an edge case they've never seen, and they cannot guarantee correctness. They are power tools for programmers, not replacements for programming knowledge.
The ratio is shifting -- from "writing code from scratch" toward "reviewing, editing, and directing generated code". This does not make programming EASIER. It relocates the difficulty. Writing a function: easier. Judging whether that function is correct for YOUR exact context: harder. Maintaining a codebase you didn't fully write and don't fully understand: harder still.
And here's the part that matters for everyone reading a tutorial series: the people who benefit most from code AI are the ones who already understand the fundamentals cold. They can read a generated function and instantly smell when it's a plausible-but-wrong imitation. If you've worked through this series from #1, you have that foundation for ML code -- and the same principle generalises to every domain of programming. AI does not remove the need to understand code. It AMPLIFIES the gap between those who do and those who don't. That is the honest, slightly uncomfortable truth, and I'd rather you heard it from me ;-)
Before the next episode, get your hands dirty. Three tasks, climbing in difficulty:
Break the evaluator. Take evaluate_generated_code and feed it a generated string that is syntactically valid but WRONG (say, an off-by-one in the loop range for fibonacci). Confirm the function reports which specific inputs failed. Then feed it code that raises an exception at runtime, and one with a syntax error -- verify each of the three failure branches fires. Write one sentence on why "it parses" is a much weaker guarantee than "it passes the tests".
Ground a review in an AST. Extend CodeAnalyzer to also flag any function whose complexity exceeds 5, and to record the line number (node.lineno) of each function. Run it on a file of your own. Then write, in a comment, the exact prompt you would send a model -- including the AST facts you extracted -- to get a focused review of ONLY the high-complexity functions. The point is the grounding, not the model.
Design a self-repair budget. Look again at the self_repair loop. Right now it retries a fixed number of times. Sketch (in code or comments) a smarter stopping rule: when should it give up early, when should it try a DIFFERENT approach rather than tweaking the same one, and where would you put a hard limit so an agent can't loop forever burning tokens? Argue your choices in a short paragraph -- this is the reasoning that separates a toy from something you'd actually deploy.
We'll open the next episode with full solutions, as always.
And here is the thread for next time. We have watched AI learn to write the LANGUAGE of machines -- a domain with instant, binary feedback. But what happens when you point the same machinery at domains where the "ground truth" is not a passing test but a physical law, a chemical structure, a mathematical theorem -- where being right means matching the universe itself, and getting it wrong wastes a real laboratory's real month? That is where we head next ;-)
@scipio