pip install torch) -- everything here is illustrative and runs on a CPU in seconds, nothing needs a GPU;Learn AI Series):I left you last time with a thread hanging on purpose. Everything in #142 assumed a model that is already trained and now merely thinks HARDER at inference time -- chain-of-thought, tree-of-thought, MCTS, all of it happens with the weights frozen solid. And I asked the awkward question: what would it take for a model to keep LEARNING after deployment, the way you pick up a fact today without erasing what you knew yesterday? Because our models, bluntly, cannot do that. Freeze the weights and they are frozen. Retrain them on something new and they tend to forget the old. That failure has a name, it has a cause, and it has a small pile of half-solutions -- and that pile is today's subject. Let's dive right in ;-)
House rules first, same as always: we settle last time's homework before we open anything new. Episode #142 was about reasoning, and all three tasks were about feeling a mechanism in your own hands rather than nodding along to it.
Exercise 1 -- Watch adaptive computation actually adapt. Take the IterativeReasoner and count how many steps each input runs before its cumulative_halt crosses 0.99. Feed it an "easy" batch and a "hard" batch and report the averages.
import torch
import torch.nn as nn
class ReasoningStep(nn.Module):
def __init__(self, hidden_dim=256):
super().__init__()
self.reason = nn.Sequential(
nn.Linear(hidden_dim, hidden_dim), nn.GELU(),
nn.Linear(hidden_dim, hidden_dim),
)
self.norm = nn.LayerNorm(hidden_dim)
def forward(self, state):
return self.norm(state + self.reason(state))
class CountingReasoner(nn.Module):
"""The #142 reasoner, now reporting steps-used per input."""
def __init__(self, input_dim=64, hidden_dim=256, max_steps=8):
super().__init__()
self.encoder = nn.Linear(input_dim, hidden_dim)
self.step = ReasoningStep(hidden_dim)
self.halt_predictor = nn.Linear(hidden_dim, 1)
self.max_steps = max_steps
def forward(self, x):
state = self.encoder(x)
cumulative = torch.zeros(x.size(0), 1)
steps_used = torch.zeros(x.size(0))
done = torch.zeros(x.size(0), dtype=torch.bool)
for t in range(self.max_steps):
state = self.step(state)
halt = torch.sigmoid(self.halt_predictor(state))
cumulative = cumulative + halt
newly = (cumulative.squeeze(1) > 0.99) & (~done)
steps_used[newly] = t + 1
done = done | newly
steps_used[steps_used == 0] = self.max_steps # never halted -> maxed out
return steps_used
model = CountingReasoner()
easy = torch.randn(16, 64) * 0.1
hard = torch.randn(16, 64) * 5.0
print("easy avg steps:", round(model(easy).mean().item(), 2))
print("hard avg steps:", round(model(hard).mean().item(), 2))
Here's the punchline, and it is a trap I set on purpose: the two averages come out roughly the SAME, and they wobble every time you re-run it. That's the whole lesson. The halt_predictor is untrained -- random weights -- so it fires "am I done?" essentially at random, with zero relationship to how hard the input actually is. To make the halting meaningful you would train it against a target that combines task loss with a "ponder cost" (a penalty per extra step), so the model is rewarded for stopping early on inputs it has already solved and pushed to keep grinding on the ones it hasn't. That ponder-cost recipe is exactly Graves' Adaptive Computation Time. Adaptive depth is not free -- you have to teach the model WHEN to spend it.
Exercise 2 -- Give tree-of-thought a real evaluator. Instantiate TreeOfThought with a toy propose_fn that appends random digits and an evaluate_fn that scores a path by how close its digits sum to a target.
import random
class TreeOfThought:
def __init__(self, propose_fn, evaluate_fn, n_branches=3, max_depth=5):
self.propose = propose_fn
self.evaluate = evaluate_fn
self.n_branches = n_branches
self.max_depth = max_depth
def search(self, problem, beam_width=3):
frontier = [(problem, 0.0)]
for _ in range(self.max_depth):
candidates = []
for state, _ in frontier:
for step in self.propose(state, self.n_branches):
new_state = state + step
candidates.append((new_state, self.evaluate(new_state)))
candidates.sort(key=lambda x: x[1], reverse=True)
frontier = candidates[:beam_width]
return frontier[0]
random.seed(0)
TARGET = 20
def propose(state, k):
return [str(random.randint(0, 9)) for _ in range(k)]
def evaluate(path):
digits = [int(c) for c in path if c.isdigit()]
return -abs(sum(digits) - TARGET) # higher (less negative) = closer to target
tot = TreeOfThought(propose, evaluate)
best_path, best_score = tot.search("")
digits = [int(c) for c in best_path if c.isdigit()]
print("winning digits:", digits, "-> sum", sum(digits), "(target", TARGET, ")")
The winning path lands its digit-sum right on or next to 20, exactly as the design promises. And the one-sentence argument the exercise asked for: swap that evaluate for a function that returns random.random() and the sort becomes meaningless, so pruning keeps arbitrary branches and the whole search collapses into blind random sampling. Tree-of-thought is only ever as good as its evaluator -- which is precisely why #142 spent a whole section on process reward models, and precisely why continual learning matters here too, because a deployed evaluator that cannot keep learning goes stale the moment the world moves.
Exercise 3 -- Prove A stays optimal, then break it.* Build a small grid with a few blocked cells, run a_star with the Manhattan heuristic, confirm the shortest path -- then multiply the heuristic by 5 to make it inadmissible and watch it degrade.
import heapq
def a_star(start, goal, neighbors, heuristic):
frontier = [(0.0, start)]
cost_so_far = {start: 0.0}
came_from = {start: None}
while frontier:
_, current = heapq.heappop(frontier)
if current == goal:
break
for nxt, step_cost in neighbors(current):
new_cost = cost_so_far[current] + step_cost
if nxt not in cost_so_far or new_cost < cost_so_far[nxt]:
cost_so_far[nxt] = new_cost
priority = new_cost + heuristic(nxt, goal)
heapq.heappush(frontier, (priority, nxt))
came_from[nxt] = current
return came_from
GRID, blocked = 5, {(1, 1), (1, 2), (1, 3), (3, 2)}
def neighbors(cell):
r, c = cell
out = []
for dr, dc in [(-1, 0), (1, 0), (0, -1), (0, 1)]:
nr, nc = r + dr, c + dc
if 0 <= nr < GRID and 0 <= nc < GRID and (nr, nc) not in blocked:
out.append(((nr, nc), 1.0))
return out
def length(came_from, start, goal):
node, n = goal, 0
while node != start:
node = came_from[node]; n += 1
return n
start, goal = (0, 0), (4, 4)
manhattan = lambda a, b: abs(a[0] - b[0]) + abs(a[1] - b[1])
print("admissible path length:", length(a_star(start, goal, neighbors, manhattan), start, goal))
inflated = lambda a, b: 5 * manhattan(a, b)
print("inflated path length:", length(a_star(start, goal, neighbors, inflated), start, goal))
The Manhattan run returns the true shortest path (length 8 on this grid). Multiply the heuristic by 5 and A* starts charging greedily at the goal, overestimating the cost-to-go so badly that it commits to a longer route and never reconsiders. The sentence connecting the two: an ADMISSIBLE heuristic (one that never overestimates the true remaining cost) is exactly the condition A* needs to guarantee an optimal path -- break admissibility and you break the guarantee. Right, homework settled. Now let's talk about a model that forgets ;-)
Picture this. You train a network to classify cats and dogs. It nails it -- 97% accuracy, you are a happy camper. A week later you want it to ALSO recognise birds, so you fine-tune the same network on a pile of bird photos. Birds now work great. You go back and check cats and dogs... and accuracy has cratered to 60%. The model learned birds by TROMPLING the cat-and-dog knowledge on its way there.
This is catastrophic forgetting, and it is one of the most fundamental limitations in how neural networks learn. It feels absurd from a human standpoint -- learning to ride a bike does not make you forget how to walk. But a neural network, by default, is catastrophically bad at learning things one after another. Everything we've done across 142 episodes has been BATCH learning: gather all the data, shuffle it together, train on the lot at once. The real world does not hand you data that way. Data arrives over time. Tasks appear and change. Yesterday's distribution quietly drifts into something else (we saw exactly that drift monster back in the production-monitoring episode, #123). Continual learning -- also called lifelong or incremental learning -- is the whole research programme of building models that keep learning without wrecking what they already know.
Let me make the forgetting concrete before we fix it. Two absurdly simple "tasks" over the same inputs: task A says "output +1", task B says "output -1". Train A, then train B, then look back at A.
import torch
import torch.nn as nn
torch.manual_seed(0)
net = nn.Sequential(nn.Linear(4, 32), nn.ReLU(), nn.Linear(32, 1))
opt = torch.optim.Adam(net.parameters(), lr=1e-2)
X = torch.randn(64, 4)
task_a_y = torch.ones(64, 1) # task A: always +1
task_b_y = -torch.ones(64, 1) # task B: always -1
def train(target, steps=200):
for _ in range(steps):
loss = nn.functional.mse_loss(net(X), target)
opt.zero_grad(); loss.backward(); opt.step()
def err(target):
with torch.no_grad():
return round(nn.functional.mse_loss(net(X), target).item(), 4)
train(task_a_y)
print("after A -> error on A:", err(task_a_y)) # tiny, it learned A
train(task_b_y)
print("after B -> error on A:", err(task_a_y)) # blown up, A is gone
print("after B -> error on B:", err(task_b_y)) # tiny, it learned B
Run it and the story is stark: after task A the error on A is near zero, and after task B the error on A has exploded while B sits pretty. The network did not "blend" the two tasks -- it OVERWROTE the first with the second. That is catastrophic forgetting in eight lines, no hand-waving.
The root cause is parameter sharing. In a neural network every task leans on overlapping sets of weights. When you train on task B, the gradient updates that improve B do not know -- and could not care less -- about task A. They simply shove the weights toward a good solution for B. If that solution sits somewhere incompatible with the task A solution, task A gets bulldozed on the way.
Think of it as the loss landscape we spent episodes #40-41 crawling around in. Training on task A settles the weights into a minimum of A's loss surface. Training on task B then drags those same weights off toward a minimum of B's surface. Those two basins usually live in very different regions of weight space, and there is no rule saying you can occupy both at once -- moving into one means climbing out of the other.
How BAD the forgetting gets depends on how similar the tasks are. If A and B share a lot of structure -- say both are image classification on visually similar images -- the features transfer and forgetting is mild. If they are wildly different -- image classification then audio classification -- forgetting is brutal, because almost none of the learned structure is reusable. Nota bene: this is the same intuition that made transfer learning and fine-tuning (#69) work in the first place, just viewed from the dark side. Related tasks help each other; unrelated tasks trample each other.
So we have four broad families of defence. Constrain WHICH weights are allowed to move (EWC). Keep showing the model old data (replay). Refuse to touch old weights at all (progressive networks). Or preserve old BEHAVIOUR rather than old weights (LwF). Let's take them in turn.
The first real breakthrough, and still the cleanest idea in the bunch: figure out which weights MATTERED for the old task, and then protect those specifically while letting the rest roam free.
EWC (Kirkpatrick et al., 2017 -- a lovely paper with a title about "overcoming catastrophic forgetting") uses the Fisher information matrix to estimate how important each weight was for the previous task. The intuition is beautifully simple: a weight is important if wiggling it causes a big change in the loss. Weights with high Fisher information are the load-bearing walls -- touch them and the old task collapses. Weights with low Fisher information are decoration -- move them freely, the old task will not notice. So during training on the new task you add a penalty that pulls important weights back toward their old values, in proportion to how important they were.
import torch
import torch.nn as nn
from copy import deepcopy
class EWC:
"""Elastic Weight Consolidation: remember what mattered, protect it."""
def __init__(self, model, dataloader, device='cpu', lambda_ewc=1000.0):
self.model = model
self.lambda_ewc = lambda_ewc
self.device = device
# snapshot the weights AFTER training on the previous task
self.saved_params = {
n: p.clone().detach()
for n, p in model.named_parameters() if p.requires_grad
}
self.fisher = self._compute_fisher(dataloader)
def _compute_fisher(self, dataloader):
"""Diagonal Fisher information: E[grad^2] per parameter."""
fisher = {
n: torch.zeros_like(p)
for n, p in self.model.named_parameters() if p.requires_grad
}
self.model.eval()
for batch_x, batch_y in dataloader:
batch_x, batch_y = batch_x.to(self.device), batch_y.to(self.device)
self.model.zero_grad()
loss = nn.functional.cross_entropy(self.model(batch_x), batch_y)
loss.backward()
for n, p in self.model.named_parameters():
if p.requires_grad and p.grad is not None:
fisher[n] += p.grad.data.pow(2) # squared gradient = importance
for n in fisher:
fisher[n] /= len(dataloader)
return fisher
def penalty(self):
"""Fisher-weighted squared distance from the old weights."""
loss = 0.0
for n, p in self.model.named_parameters():
if p.requires_grad and n in self.fisher:
loss += (self.fisher[n] * (p - self.saved_params[n]).pow(2)).sum()
return self.lambda_ewc * loss
And you bolt that penalty straight onto the new task's loss in the training loop:
def train_with_ewc(model, new_task_loader, ewc, optimizer, epochs=10):
"""Learn the new task while EWC guards the old one."""
model.train()
for epoch in range(epochs):
running = 0.0
for batch_x, batch_y in new_task_loader:
task_loss = nn.functional.cross_entropy(model(batch_x), batch_y)
loss = task_loss + ewc.penalty() # <- the protection term
optimizer.zero_grad()
loss.backward()
optimizer.step()
running += loss.item()
if (epoch + 1) % 5 == 0:
print(f"epoch {epoch+1}: loss={running/len(new_task_loader):.4f}")
The lambda_ewc knob is the whole ballgame, and it is finicky. Crank it too high and the important weights freeze so hard the model cannot learn the new task at all. Set it too low and the protection is a whisper the gradients ignore, so the old task gets forgotten anyway. Finding the sweet spot is task-dependent and, honestly, a bit of a fiddle.
EWC has one more limitation worth naming, because it explains a lot of follow-up research. The diagonal Fisher approximation treats every weight as independent -- it ignores CORRELATIONS between parameters. The true Fisher matrix is parameters-by-parameters, which for a real network is astronomically large, so the diagonal is a pragmatic compromise. It works reasonably well, but it systematically underestimates how much parameters interact, and later methods (like the "online EWC" and "SI" variants) try to patch exactly that hole.
Completely different philosophy: stop fussing over which weights may move, and just keep SHOWING the model old data while it learns the new stuff. If the old examples are in the training mix, the gradients cannot forget what they never stopped seeing.
Experience replay keeps a buffer of old examples and stirs them into every new-task batch:
class ExperienceReplay:
"""Store a slice of each past task, replay it alongside new data."""
def __init__(self, buffer_size=1000):
self.buffer_size = buffer_size
self.buffer_x, self.buffer_y = [], []
def add_task_data(self, x, y, samples_per_task=None):
n = samples_per_task or self.buffer_size
idx = torch.randperm(len(x))[:n] # random slice (herding is smarter)
self.buffer_x.append(x[idx])
self.buffer_y.append(y[idx])
self._balance()
def _balance(self):
"""Keep the buffer bounded, split evenly across tasks."""
total = sum(len(bx) for bx in self.buffer_x)
if total <= self.buffer_size:
return
per_task = self.buffer_size // len(self.buffer_x)
self.buffer_x = [bx[:per_task] for bx in self.buffer_x]
self.buffer_y = [by[:per_task] for by in self.buffer_y]
def sample(self, batch_size):
all_x, all_y = torch.cat(self.buffer_x), torch.cat(self.buffer_y)
idx = torch.randperm(len(all_x))[:batch_size]
return all_x[idx], all_y[idx]
The obvious wart: you have to STORE data. In plenty of real settings that is a non-starter -- privacy rules forbid keeping user data around, or the original dataset is so vast that hoarding even a slice is painful. Generative replay dodges the whole problem with a trick I find genuinely elegant: train a generative model (a GAN or a VAE, straight from episode #55) on each task's data as you go, then, when you need "old" examples, SYNTHESIZE pseudo-data from the generator in stead of pulling real samples out of storage.
The beauty of generative replay is that your replay buffer is a MODEL, not a dataset. Its footprint is fixed no matter how many tasks stream past -- one generator, a few megabytes, versus an ever-growing warehouse of raw examples. The catch, of course, is that your protection is only as faithful as your generator: if the GAN produces blurry nonsense for task A, you are rehearsing blurry nonsense, and the old task drifts anyway. Garbage in, forgotten out.
Now a philosophy that swings the other way entirely: don't protect old weights, don't replay old data -- just NEVER touch the old weights, full stop. Progressive networks freeze the entire model from the previous task and bolt on a brand-new set of parameters (a fresh "column") for each new task. Lateral connections run from the old frozen columns into the new one, so the new task can still borrow features the old one learned, but it can never damage them.
import torch
import torch.nn as nn
class ProgressiveNet(nn.Module):
"""One frozen column per task, lateral connections for transfer."""
def __init__(self, input_dim, hidden_dim=128, n_classes=10):
super().__init__()
self.columns = nn.ModuleList()
self.laterals = nn.ModuleList()
self.input_dim, self.hidden_dim, self.n_classes = input_dim, hidden_dim, n_classes
self.add_column()
def add_column(self):
col_idx = len(self.columns)
self.columns.append(nn.ModuleDict({
'layer1': nn.Linear(self.input_dim, self.hidden_dim),
'layer2': nn.Linear(self.hidden_dim, self.hidden_dim),
'head': nn.Linear(self.hidden_dim, self.n_classes),
}))
if col_idx > 0:
self.laterals.append(nn.ModuleDict({
'lat1': nn.Linear(self.hidden_dim * col_idx, self.hidden_dim),
'lat2': nn.Linear(self.hidden_dim * col_idx, self.hidden_dim),
}))
for i in range(col_idx): # freeze every earlier column
for p in self.columns[i].parameters():
p.requires_grad = False
def forward(self, x, task_id):
col = self.columns[task_id]
if task_id == 0:
h1 = torch.relu(col['layer1'](x))
h2 = torch.relu(col['layer2'](h1))
else:
prev_h1, prev_h2 = [], []
for i in range(task_id):
p = self.columns[i]
a1 = torch.relu(p['layer1'](x))
prev_h1.append(a1)
prev_h2.append(torch.relu(p['layer2'](a1)))
lat = self.laterals[task_id - 1]
h1 = torch.relu(col['layer1'](x) + lat['lat1'](torch.cat(prev_h1, dim=-1)))
h2 = torch.relu(col['layer2'](h1) + lat['lat2'](torch.cat(prev_h2, dim=-1)))
return col['head'](h2)
The strength is absolute and it is lovely: zero forgetting, by construction. Old-task parameters literally cannot change, so there is nothing to forget -- no hyperparameter to tune, no penalty to balance, no buffer to store. The weakness is just as absolute: the model grows LINEARLY with the number of tasks. Ten tasks, ten columns. A hundred tasks and you are lugging around a hundred times the parameters plus a snarl of lateral connections. For a handful of well-defined tasks it is beautiful; as a lifelong-learning strategy it simply does not scale, and everyone knows it.
The last method is the cleverest, because it needs NEITHER old data NOR new parameters. Learning without Forgetting leans on knowledge distillation -- the teacher-student trick from the model-optimization episode (#120) -- to preserve old behaviour instead of old weights.
The move is subtle, so slow down for it. BEFORE you start training on the new task, run the new task's inputs through the current model and record what the OLD task heads output. Those recorded outputs become soft targets. Then, while you train on the new task, you add a distillation loss that says: "whatever else you do, keep producing the same old-head outputs you produced a moment ago." The model is, in effect, its own teacher for the old task.
import torch.nn as nn
import torch.nn.functional as F
def lwf_loss(model, new_data, new_labels, old_outputs, temperature=2.0, alpha=0.5):
"""Learn the new task while distilling the old model's own old-head behaviour."""
logits = model(new_data)
n_old = old_outputs.size(1)
old_logits, new_logits = logits[:, :n_old], logits[:, n_old:]
new_loss = F.cross_entropy(new_logits, new_labels) # the new task, hard labels
# old task: match the soft outputs the model gave BEFORE new-task training
old_soft = F.softmax(old_outputs / temperature, dim=1)
old_logsoft = F.log_softmax(old_logits / temperature, dim=1)
distill = F.kl_div(old_logsoft, old_soft, reduction='batchmean') * (temperature ** 2)
return alpha * distill + (1 - alpha) * new_loss
LwF is elegant precisely because it is so cheap -- no stored dataset, no growing architecture, just a recorded set of soft targets and a distillation term. But it comes with a real fine-print warning. It only works well when the old and new tasks are somewhat RELATED. The soft targets are the old heads' outputs on NEW data, and if the new data looks nothing like what the old task ever saw, those outputs are meaningless -- the distillation signal degenerates into noise, and you are asking the model to faithfully reproduce garbage. Distillation preserves behaviour only where the behaviour was defined in the first place.
Time for the honest part, because there is a wide gulf between the papers and the deployment, and I would rather you hear the boring truth from me than learn it the hard way at 2am.
Simple replay beats most of the clever stuff. This is the finding nobody wants on the poster. Storing even a tiny buffer -- 1 to 5% of the original data -- and mixing it into new training is startlingly effective and takes an afternoon to implement. A lot of elaborate methods struggle to clearly beat it once you tune both fairly.
Joint training on all data is still the ceiling. If you can afford to just retrain on everything, old plus new, do THAT. It is the upper bound every continual-learning method is quietly measured against. These techniques exist for the situations where you genuinely cannot retrain from scratch -- streaming data, privacy walls, compute budgets -- not as a default you reach for out of habit.
Task boundaries are a convenient lie. Most methods assume you know exactly when one task ends and the next begins. Reality rarely rings a bell. Distributions shift gradually and messily. The harder, more realistic setting is class-incremental learning (new classes show up over time and at test time you do NOT get told which task you're in), which is meaningfully tougher than the tidy task-incremental setting the benchmarks love.
Big pretrained models forget less -- and this is the practically useful one. Models pretrained on broad, diverse data (foundation models, #137) are naturally more resistant to forgetting. Fine-tune a well-pretrained model across a sequence of tasks and it forgets far, far less than a network trained from scratch, because its feature representations are already general enough to support many tasks at once. In 2026 the pragmatic continual-learning recipe for most teams is not an exotic algorithm at all -- it is "start from a strong foundation model, keep a small replay buffer, and fine-tune gently." Unsexy, and it wins.
Get your hands dirty before next time. Three tasks, climbing in difficulty:
Measure the forgetting, then soften it. Take the eight-line catastrophic-forgetting demo (task A = +1, task B = -1) and add a THIRD phase: after training B, mix in a small replay set of task-A examples (say 16 of them) and train a little more. Report the error on A before and after the replay phase, and in one sentence explain why even a handful of replayed examples pulls A's error back down.
Feel the lambda trade-off in EWC. Wire the EWC class into a two-task toy problem (two tiny classification tasks over the same input shape). Train task A, build the EWC object, then train task B with lambda_ewc set to 0, 100, and 100000 in turn. Report accuracy on BOTH tasks for each setting, and explain in one sentence what the two extremes buy you and cost you.
Count the cost of a progressive net. Instantiate ProgressiveNet, call add_column() five times to simulate five tasks, and write a small loop that sums p.numel() over all parameters after each added column. Plot or print the parameter count per task, then explain in one sentence why this growth curve is the method's fatal flaw for true lifelong learning.
We'll open next episode with full solutions, as always.
lambda;And here is the thread I'll leave dangling for next time. All of today assumed we have PLENTY of data for each new task -- enough to compute a Fisher matrix, fill a replay buffer, train a fresh column. But the most human kind of learning is the opposite: you see ONE example of a new thing, sometimes zero, and you generalise from it immediately. Show a child a single zebra and they'll spot the next one across a field. Our models, so far, have needed thousands of examples to learn anything at all. What would it take to learn from a handful -- or from nothing but a description? That is where we head next ;-)
@scipio