pip install torch) -- every snippet here is illustrative, nothing needs a GPU or a lab;Learn AI Series):Last episode I left you with a deliberate cliffhanger. We had spent #139 watching AI learn to write the language of machines -- a domain with instant, binary feedback (it runs or it does not). And I asked: what happens when you point that exact same machinery at domains where "correct" is not a passing test but a physical law, a chemical structure, a mathematical theorem? Where being right means matching the universe itself, and being wrong wastes a real laboratory's real month?
That is today. This is the episode where AI stops being a clever autocomplete and starts sitting alongside the telescope, the microscope and the particle accelerator as an instrument of discovery. And I mean that literally -- not as marketing. Let's dive right in ;-)
As always, we settle last time's homework before anything new. Episode #139 was AI for code, so the exercises were all about the machinery that makes code special: objective feedback.
Exercise 1 -- Break the evaluator. The point was to prove to yourself that "it parses" is a MUCH weaker guarantee than "it passes". Here I feed the evaluator three deliberately broken strings and watch each of the three failure branches fire.
# Solution 1: exercising all three failure branches of evaluate_generated_code (#139).
import ast
def evaluate_generated_code(code, test_cases):
results = {"syntactically_valid": False, "tests_passed": 0,
"tests_total": len(test_cases), "errors": []}
try:
ast.parse(code)
results["syntactically_valid"] = True
except SyntaxError as e:
results["errors"].append(f"SyntaxError: {e}")
return results
ns = {}
try:
exec(code, ns) # illustrative only -- sandbox in real life
except Exception as e:
results["errors"].append(f"Execution error: {e}")
return results
func = next((o for n, o in ns.items()
if callable(o) and not n.startswith("_")), None)
for inputs, expected in test_cases:
try:
if func(*inputs) == expected:
results["tests_passed"] += 1
else:
results["errors"].append(f"{inputs}: expected {expected}, got {func(*inputs)}")
except Exception as e:
results["errors"].append(f"{inputs}: {e}")
return results
tests = [((5,), 5), ((10,), 55)]
# (a) parses fine, but WRONG -- off-by-one in the range
wrong = ("def fib(n):\n a, b = 0, 1\n"
" for _ in range(1, n):\n a, b = b, a + b\n return a")
# (b) parses fine, but blows up at runtime
crashes = "def fib(n):\n return undefined_name + n"
# (c) does not even parse
broken = "def fib(n)\n return n"
for label, c in [("wrong", wrong), ("crashes", crashes), ("broken", broken)]:
r = evaluate_generated_code(c, tests)
print(label, "-> valid:", r["syntactically_valid"],
"passed:", r["tests_passed"], "| first error:", r["errors"][0])
The key insight in one sentence: a parser only checks GRAMMAR, so wrong sails through ast.parse and still fails the tests, crashes reaches exec and dies there, and only broken trips the syntax branch -- three different guarantees, three different failure points, and only the test run tells you the code is actually correct.
Exercise 2 -- Ground a review in an AST. Extend the analyzer to flag high-complexity functions and record their line numbers, then hand those precise facts to a model. The grounding is the whole point (this should ring a bell from RAG, #64).
# Solution 2: complexity-flagging analyzer that produces facts, not guesses.
import ast
class ReviewAnalyzer(ast.NodeVisitor):
def __init__(self, threshold=5):
self.threshold = threshold
self.functions = []
def visit_FunctionDef(self, node):
branches = sum(1 for n in ast.walk(node)
if isinstance(n, (ast.If, ast.For, ast.While,
ast.ExceptHandler, ast.With)))
complexity = branches + 1
self.functions.append({
"name": node.name, "line": node.lineno,
"complexity": complexity, "hot": complexity > self.threshold,
})
self.generic_visit(node)
def review_prompt(facts):
hot = [f for f in facts if f["hot"]]
listing = "\n".join(f"- {f['name']} (line {f['line']}, complexity {f['complexity']})"
for f in hot)
return (f"Review ONLY these high-complexity functions for bugs and simplification:\n"
f"{listing}\nGround your review in the fact that each exceeds our "
f"complexity budget of 5. Do not comment on anything else.")
src = ("def tiny(x):\n return x + 1\n"
"def gnarly(x):\n"
" if x < 0:\n return 0\n"
" if x > 0:\n for i in range(x):\n"
" while i > 0:\n if i % 2: i -= 1\n"
" else: i -= 2\n return x\n")
a = ReviewAnalyzer()
a.visit(ast.parse(src))
for f in a.functions:
print(f"{f['name']}: line {f['line']}, complexity {f['complexity']}, hot={f['hot']}")
print("---\n" + review_prompt(a.functions))
Notice what happened: the AST supplies facts that are TRUE by construction (it cannot invent a function that isn't there), and the model only ever sees the functions worth its attention. Facts from the parser, judgement from the model -- the pattern I told you to remember.
Exercise 3 -- Design a self-repair budget. The naive loop retries a fixed number of times. A deployable one knows when to give up, when to change strategy instead of tweaking, and where the hard ceiling lives so an agent can't burn tokens till the heat death of the universe.
# Solution 3: a self-repair loop with a real stopping policy.
def smart_self_repair(spec, tests, generate_fn, max_attempts=5, hard_token_cap=8000):
feedback, tokens_spent, last_pass = None, 0, -1
for attempt in range(1, max_attempts + 1):
code, cost = generate_fn(spec, feedback) # real call returns (code, tokens)
tokens_spent += cost
result = evaluate_generated_code(code, tests)
passed = result["tests_passed"]
if passed == result["tests_total"]:
return {"code": code, "attempt": attempt, "tokens": tokens_spent}
# Give up early if we are making ZERO progress (same or fewer passes).
if passed <= last_pass:
feedback = {"errors": result["errors"], "hint": "previous approach stalled; "
"try a DIFFERENT algorithm, not a patch"}
else:
feedback = {"errors": result["errors"]} # progress -> keep refining
last_pass = max(last_pass, passed)
# Hard ceiling so an agent cannot loop forever.
if tokens_spent > hard_token_cap:
return {"code": None, "reason": "token budget exhausted", "tokens": tokens_spent}
return {"code": None, "reason": "max attempts", "tokens": tokens_spent}
My reasoning: refining works when you are CLOSER than last time (more tests green), so I keep the feedback narrow then. When passes stall or regress, tweaking the same broken idea is a trap -- so I switch the hint to "try a different approach entirely". And regardless of cleverness, a hard token cap sits underneath everything, because an autonomous agent with no ceiling is a bill with no ceiling. That gap between a toy loop and this one IS the gap between a demo and something you would actually deploy.
Right -- homework settled. On to the universe ;-)
In December 2020, DeepMind announced that AlphaFold had essentially solved protein folding -- a grand challenge biologists had chewed on for over 50 years. Within two years it had predicted the structure of essentially every known protein: over 200 MILLION structures, released free to the world. For context, all of experimental science, over half a century of X-ray crystallography and cryo-EM, had produced roughly 170,000.
Here is the problem in one breath. A protein arrives as a sequence of amino acids -- a string over a 20-letter alphabet. That string folds itself into a precise 3D shape, and the SHAPE determines the FUNCTION (enzyme, receptor, structural strut). Get the structure and you unlock drug design, disease understanding, bioengineering. Before AlphaFold, getting one structure took months to years and cost thousands to millions.
What makes this so satisfying to teach is that AlphaFold is not alien technology. It is OUR technology, the exact machinery we've been assembling since episode #1:
Input as evolutionary context. The target sequence is bundled with its evolutionary relatives into a Multiple Sequence Alignment (MSA). Amino acids that mutated together across millions of years of evolution tend to be physically close in the folded shape -- co-evolution leaks structural information, and the MSA is how you feed it in.
Attention, but biological. AlphaFold's "Evoformer" runs attention across TWO axes: along the MSA (different relatives) and across residue pairs (relationships between positions). It is cross-attention (#51, #138) wearing a lab coat.
Structure plus its own uncertainty. The final module predicts 3D coordinates AND a per-residue confidence that correlates startlingly well with real accuracy -- the model knows when it does not know.
import torch
import torch.nn as nn
class SimplifiedEvoformer(nn.Module):
"""A toy slice of AlphaFold's Evoformer.
The real thing has row attention, column attention, triangular
updates and outer-product means. This keeps only the core idea:
attention over sequence relationships, feeding a pair representation.
"""
def __init__(self, seq_dim=256, pair_dim=128, n_heads=8):
super().__init__()
self.row_attn = nn.MultiheadAttention(seq_dim, n_heads, batch_first=True)
self.pair_proj = nn.Sequential(
nn.Linear(pair_dim, pair_dim), nn.ReLU(),
nn.Linear(pair_dim, pair_dim),
)
self.norm1 = nn.LayerNorm(seq_dim)
self.norm2 = nn.LayerNorm(pair_dim)
def forward(self, msa_repr, pair_repr):
batch, n_seq, seq_len, dim = msa_repr.shape
flat = msa_repr.reshape(batch * n_seq, seq_len, dim)
attn_out, _ = self.row_attn(flat, flat, flat)
msa_repr = self.norm1(msa_repr + attn_out.reshape(batch, n_seq, seq_len, dim))
pair_repr = self.norm2(pair_repr + self.pair_proj(pair_repr))
return msa_repr, pair_repr
evoformer = SimplifiedEvoformer()
msa = torch.randn(1, 4, 50, 256) # 1 batch, 4 relatives, 50 residues, 256-dim
pair = torch.randn(1, 50, 50, 128) # pairwise features over 50x50 residues
msa_out, pair_out = evoformer(msa, pair)
print(f"MSA: {tuple(msa_out.shape)}, Pair: {tuple(pair_out.shape)}")
The follow-up, AlphaFold 3, pushed into protein-ligand, protein-DNA and protein-RNA complexes -- the interactions that underlie basically all of biology. Nota bene: this is the single result that shifted the mood of an entire field. After AlphaFold, "AI for science" stopped sounding like a pitch deck.
Structure prediction is one rung. The full drug pipeline traditionally runs 10-15 years and north of a billion dollars per approved drug. AI is squeezing several stages of that -- though, as you'll see, not the stage you might hope.
Target identification. Which protein, nudged, would treat the disease? Models chew through gene-expression data, protein-interaction networks (#131, graph neural networks) and mountains of literature to nominate candidates.
Molecular generation. Given a target, design a molecule that binds it. A molecule IS a graph -- atoms are nodes, bonds are edges -- so this is GNN territory. Predict properties (binding affinity, toxicity, solubility) from the graph, then generate novel graphs optimised for the properties you want.
# Molecular property prediction with a bare-bones message-passing GNN.
# In production you'd reach for PyTorch Geometric or DGL and real chemistry.
class MoleculeGNN(nn.Module):
"""Predict a molecular property from atoms + bond connectivity."""
def __init__(self, atom_features=32, hidden=64, output=1):
super().__init__()
self.atom_embed = nn.Linear(atom_features, hidden)
self.conv1 = nn.Linear(hidden, hidden)
self.conv2 = nn.Linear(hidden, hidden)
self.readout = nn.Linear(hidden, output)
def forward(self, atom_feats, adj):
h = torch.relu(self.atom_embed(atom_feats))
h = torch.relu(self.conv1(adj @ h)) # pull in neighbour info
h = torch.relu(self.conv2(adj @ h)) # two hops of message passing
graph_repr = h.mean(dim=0, keepdim=True) # readout over the whole molecule
return self.readout(graph_repr)
n_atoms = 9 # ethanol-ish: 2C + 6H + 1O
atom_feats = torch.randn(n_atoms, 32)
adj = torch.zeros(n_atoms, n_atoms)
adj[0, 1] = adj[1, 0] = 1 # C-C
adj[1, 8] = adj[8, 1] = 1 # C-O
pred = MoleculeGNN()(atom_feats, adj)
print(f"predicted property: {pred.item():.4f}")
Virtual screening. Score millions of candidate molecules and rank the promising ones BEFORE a single test tube gets touched. Classical screening simulates the physics of docking; learned representations do it roughly 1000x faster at comparable accuracy, shrinking the search space by orders of magnitude.
ADMET prediction. Absorption, Distribution, Metabolism, Excretion, Toxicity -- the unglamorous properties that kill most candidates. Models trained on historical ADMET data filter the doomed molecules out before expensive animal work.
Now the honest part, because I promised you honesty. AI has genuinely accelerated EARLY drug discovery. But as of early 2026, no AI-designed drug has completed clinical trials and reached patients. The bottleneck did not vanish -- it MOVED. It slid from molecular design (fast now) to clinical validation (still slow, because you are testing on actual humans and biology refuses to be rushed). AI compresses years of lab work into months. It cannot compress the years of trials that follow. Keep that distinction sharp -- it is exactly where the hype gets sloppy.
Maths feels like the LAST place a statistical model should thrive -- it is the summit of formal reasoning, and our models are, at heart, glorified next-token predictors. And yet.
Theorem proving. Formal provers (Lean 4, Isabelle, Coq) verify proofs with absolute certainty. The hard bit is WRITING them, which needs creativity and taste. Models are learning to generate proof steps -- DeepMind's AlphaProof solved several International Mathematical Olympiad problems by pairing a language model with an RL loop that searched proof strategies (a search that rhymes with the game-playing back in #112).
Conjecture generation. Before you prove something you must first GUESS it. Models have surfaced genuinely new relationships by spotting patterns in mathematical data -- Google's collaborations with mathematicians turned up publishable conjectures in knot theory and representation theory.
Symbolic computation. Transformers trained on expressions can integrate, solve differential equations and simplify -- treating manipulation as sequence-to-sequence translation, the same seq2seq idea from #50.
# Symbolic integration as seq2seq translation -- the conceptual shape.
integration_examples = [
("x^2", "x^3/3"),
("sin(x)", "-cos(x)"),
("e^x", "e^x"),
("1/x", "ln(x)"),
("x*e^x", "e^x*(x - 1)"),
]
# The model learns integration as a token transformation:
# input tokens: ['x', '^', '2']
# output tokens: ['x', '^', '3', '/', '3']
# Lample & Charton (2019) trained a transformer on millions of such pairs and
# beat Mathematica on some classes of symbolic integration. Yes, really.
print("integration as translation:")
for expr, integral in integration_examples:
print(f" integral({expr}) dx = {integral} + C")
The philosophical needle to thread: is the model "doing maths"? It does not understand a proof the way a human mathematician does. But its output is VERIFIABLE -- once a formal prover checks a generated proof, that proof is as valid as any human's, full stop. The process differs; the result is identical. I find that a weirdly comforting kind of honesty -- the machine does not need to understand, because the proof checker does the trusting for us.
For decades, forecasting meant Numerical Weather Prediction (NWP) -- gigantic simulations solving the equations of atmospheric physics on a 3D grid of the planet, running for hours on some of the largest supercomputers we own. In 2022-2023 that quietly got upended. GraphCast (DeepMind) and Pangu-Weather (Huawei) posted LOWER forecast error than the ECMWF gold standard, while running roughly 1000x faster -- on a single GPU.
GraphCast treats the atmosphere as a graph (of course it does). Grid points are nodes, nearby points share edges, and each node carries temperature, humidity, pressure and wind at many altitudes. The task: given the current state (plus the previous one, for momentum), predict the state 6 hours out. Chain those steps and you get a ten-day forecast.
# The conceptual skeleton of a neural weather model.
# Real GraphCast uses ~37 atmospheric levels and about a million grid points.
class SimpleWeatherPredictor(nn.Module):
"""Predict the next atmospheric state from current + previous."""
def __init__(self, n_features=6, hidden=128, n_layers=4):
super().__init__()
self.encoder = nn.Linear(n_features * 2, hidden) # current ++ previous
self.layers = nn.ModuleList([
nn.Sequential(nn.Linear(hidden, hidden), nn.ReLU(), nn.LayerNorm(hidden))
for _ in range(n_layers)
])
self.decoder = nn.Linear(hidden, n_features)
def forward(self, current, previous):
x = torch.cat([current, previous], dim=-1)
h = torch.relu(self.encoder(x))
for layer in self.layers:
h = h + layer(h) # residual stacks (#52-53 habit)
delta = self.decoder(h) # predict the CHANGE, not the absolute
return current + delta # next = current + predicted change
n_points, n_features = 1000, 6 # real: ~1 million points
current = torch.randn(1, n_points, n_features)
previous = torch.randn(1, n_points, n_features)
next_state = SimpleWeatherPredictor()(current, previous)
print(f"predicted next state: {tuple(next_state.shape)}")
WHY does this even work? The physics models solve the actual equations from first principles -- they KNOW Navier-Stokes. The neural models know nothing of the sort. They have simply seen so many "this state led to that state six hours later" examples that they learned the dynamics empirically. Decades of historical weather, distilled into a network that never once read a physics textbook.
The upside is speed, and speed changes what is POSSIBLE: a ten-day global forecast in under a minute means you can run hundreds of them with jittered initial conditions (ensemble forecasting) to estimate uncertainty. The downside is the classic one -- these models learned from the PAST. Unprecedented, climate-driven events sit outside their training distribution, and there the physics models, which encode the real laws, may prove more robust. The grown-up future is almost certainly HYBRID: neural nets for fast routine forecasting, physics for the extreme and the never-before-seen.
Discovering materials with desired properties -- stronger, lighter, more conductive, cheaper -- has forever meant expensive trial-and-error. AI is making it systematic.
Crystal structure prediction. Given a chemical composition, predict the most stable crystal arrangement (protein folding's inorganic cousin). GNoME (Google DeepMind, 2023) predicted 2.2 MILLION new stable crystals -- an order of magnitude more than humanity had found in its entire history. Hundreds have since been synthesised for real, confirming the calls.
Property prediction. Structure in, properties out: band gap for semiconductors, thermal conductivity, mechanical strength, melting point. GNNs on crystal graphs (atoms as nodes, bonds as edges) are the workhorse -- the SAME architecture you just saw predict a drug molecule's binding, pointed at a lattice instead.
Inverse design. Run it backwards -- desired properties in, a structure that achieves them out. This is generative modelling (#55, #84-85) applied to matter, the direct analogue of generating a drug molecule to hit a target.
# One architecture, two sciences: a crystal graph feeds the SAME GNN as a molecule.
# atoms -> nodes, bonds -> edges, readout -> a bulk material property.
crystal_atoms = torch.randn(12, 32) # 12 atoms in the unit cell
crystal_adj = torch.eye(12) # placeholder lattice connectivity
crystal_adj[0, 1] = crystal_adj[1, 0] = 1
band_gap = MoleculeGNN(atom_features=32)(crystal_atoms, crystal_adj)
print(f"predicted bulk property (e.g. band gap): {band_gap.item():.4f}")
That reuse is not a coincidence, and it is the whole lesson of this episode -- but let me make it explicit in a second.
There is a distinction here worth nailing down, because sloppy thinking about it produces both breathless hype and lazy dismissal.
AI as tool. The scientist frames the problem, designs the experiment, interprets the result. AI accelerates specific steps -- predict a structure, screen candidates, crunch data -- but the scientific JUDGEMENT stays human. This is where we live today, and it is already transformative.
AI as scientist. The system forms hypotheses, designs experiments to test them, reads the results and iterates -- the full autonomous-discovery loop. Pieces exist (automated labs running AI-suggested experiments), but the complete loop, especially the interpretation, is still beyond us.
The gap between those two is ENORMOUS, and here is the crux. Current AI can predict that a molecule binds a protein. It cannot tell you WHY in a way that advances fundamental understanding. AlphaFold predicts structures with astonishing accuracy -- yet it has not handed us new physics of WHY proteins fold. The prediction is superhuman; the understanding remains stubbornly human. That is not a criticism (a tool this powerful needs no apology), but be precise about what "AI for science" means right now: these are magnificent prediction machines, not yet discovery machines in the deepest sense.
Step back and squint across the whole landscape -- folding, drugs, maths, weather, materials -- and the same handful of techniques from THIS series keeps reappearing:
The architectures do not change. The training principles do not change. What changes is the DATA and the domain expertise needed to frame the problem correctly. This is precisely why understanding the fundamentals -- the thing this series has been drilling since episode #1 -- matters so much. The techniques transfer. The domain is the variable. Master the toolkit and every science becomes a place you can point it.
Before the next episode, get your hands into it. Three tasks, climbing:
Watch confidence and shape interact. Extend SimplifiedEvoformer so forward also returns a per-residue confidence vector of length seq_len (a small nn.Linear(seq_dim, 1) head on the MSA output, averaged over the relatives, squashed with sigmoid). Print its shape and confirm it matches the number of residues. Then write one sentence on why a model reporting its OWN uncertainty is more useful to a scientist than one that only reports an answer.
One GNN, two sciences. Take MoleculeGNN and feed it (a) a small molecule graph and (b) a crystal-cell graph of a different atom count, WITHOUT changing the model. Confirm both produce a single scalar. Then, in a comment, explain what has to be true about the readout step for the same architecture to swallow graphs of different sizes -- and connect it back to why "atoms as nodes" generalises across chemistry and materials.
Draw the tool/scientist line yourself. Pick any ONE system from this episode (AlphaFold, a drug-screening model, AlphaProof, GraphCast, GNoME). In a short paragraph, argue exactly where it sits on the tool-versus-scientist spectrum: what human judgement does it still depend on, and what would have to change for it to cross into autonomous discovery? Be specific -- "it would need to understand" is not an argument until you say what understanding would let it DO.
We'll open the next episode with full solutions, as always.
And here is the thread for next time. We've watched AI reason about molecules, proofs and weather -- but always as a mind in a box, reading numbers and emitting numbers. What happens when you rip it OUT of the box and bolt it into something with motors, sensors and a body that has to survive contact with the messy physical world -- where a wrong prediction does not waste a lab month but tips over a very expensive robot? That is where we head next ;-)
@scipio