pip install torch) -- everything here runs on a CPU in a couple of seconds, no GPU needed;Learn AI Series):I closed #145 with a promise, so let me pay it off before anything else. We spent that whole episode on concept bottleneck models -- networks we DESIGNED to be transparent, with a human-readable waist bolted into the middle on purpose. And I ended by asking the obvious follow-up: what about the millions of models nobody built to be opened? The transformer you fine-tuned, the CNN you downloaded off the internet, the giant LLM sitting behind an API. How do you pry open a black box that was never meant to be pried open, and actually TRUST what you find inside? That question has an entire field behind it, and we are standing in it now. Welcome to explainability and interpretability ;-)
Let me start with the scene that makes this matter. Your model says this loan applicant should be rejected. The applicant asks: "Why?" Your model flags this X-ray as pneumonia. The radiologist asks: "What are you seeing?" Your model marks this transaction as fraud. The regulator asks: "On what basis?" In every one of those rooms, "the neural network said so" is not an answer. And in a growing pile of jurisdictions -- the EU AI Act, GDPR's so-called right to explanation -- it is not even legal. So this is not a nice-to-have. This is load-bearing.
House rules, same as always -- we settle last week's homework before we open anything new. Episode #145 was neuro-symbolic AI, and all three tasks were about feeling the mechanics in your own hands.
Exercise 1 -- Verify the TransE geometry by hand. Skip training entirely. Make a tiny TransE with 4 entities and 2 relations, manually SET the embeddings so that entity[PARIS] + relation[CAPITAL_OF] equals entity[FRANCE] exactly, then score a true triple and a false one and confirm the true one wins.
import torch
import torch.nn as nn
class TransE(nn.Module):
def __init__(self, n_entities, n_relations, embed_dim):
super().__init__()
self.entity_emb = nn.Embedding(n_entities, embed_dim)
self.relation_emb = nn.Embedding(n_relations, embed_dim)
def score(self, head, relation, tail):
h = self.entity_emb(head)
r = self.relation_emb(relation)
t = self.entity_emb(tail)
return -torch.norm(h + r - t, p=2, dim=-1) # higher = more plausible
PARIS, FRANCE, BERLIN, GERMANY = 0, 1, 2, 3
CAPITAL_OF, LOCATED_IN = 0, 1
model = TransE(4, 2, embed_dim=3)
with torch.no_grad():
model.entity_emb.weight[PARIS] = torch.tensor([1.0, 0.0, 0.0])
model.entity_emb.weight[FRANCE] = torch.tensor([1.0, 1.0, 0.0])
model.entity_emb.weight[BERLIN] = torch.tensor([0.0, 0.0, 1.0])
model.entity_emb.weight[GERMANY] = torch.tensor([5.0, 5.0, 5.0])
# PARIS + CAPITAL_OF must land exactly on FRANCE
model.relation_emb.weight[CAPITAL_OF] = torch.tensor([0.0, 1.0, 0.0])
model.relation_emb.weight[LOCATED_IN] = torch.tensor([0.0, 0.0, 0.0])
t = lambda i: torch.tensor([i])
true_score = model.score(t(PARIS), t(CAPITAL_OF), t(FRANCE)).item()
false_score = model.score(t(PARIS), t(CAPITAL_OF), t(BERLIN)).item()
print(f"true (Paris, capital_of, France): {true_score:.3f}")
print(f"false (Paris, capital_of, Berlin): {false_score:.3f}")
print("true scores higher:", true_score > false_score)
PARIS + CAPITAL_OF = [1,1,0], which is exactly FRANCE, so the distance is zero and the true triple scores 0.000. The false triple compares [1,1,0] against BERLIN = [0,0,1], a distance of about 1.732, so it scores -1.732. True wins, cleanly. The one sentence the exercise fished for: if the false triple had scored higher, the margin ranking loss relu(margin - pos + neg) would go positive, and its gradient would shove the embeddings around until the true triple outscores the false one by at least the margin -- that is the entire training signal, pushing true up and corrupted down.
Exercise 2 -- Break the fuzzy OR. Build the truth value for "penguin is a bird" from "has-feathers=0.99" and "can-fly=0.05", then show fuzzy logic is NOT boolean by evaluating fuzzy_and(x, fuzzy_not(x)) at x=0.5.
import torch
def fuzzy_and(a, b): return a * b
def fuzzy_or(a, b): return a + b - a * b
def fuzzy_not(a): return 1.0 - a
feathers = torch.tensor(0.99)
can_fly = torch.tensor(0.05)
is_bird = fuzzy_or(feathers, can_fly) # either is decent evidence of "bird"
print(f"penguin is a bird: {is_bird.item():.4f}")
x = torch.tensor(0.5)
contradiction = fuzzy_and(x, fuzzy_not(x)) # boolean logic: always 0
print(f"fuzzy_and(0.5, not 0.5) = {contradiction.item():.4f}")
The penguin lands at 0.9905 -- still very much a bird, because feathers alone almost settle it and the near-uselessness of "can fly" barely dents the OR. That is exactly the graceful behaviour you want. The second line is the sharp bit: in boolean logic x AND NOT x is a contradiction and equals zero, ALWAYS. Fuzzy logic hands you 0.25 instead. Why non-zero? Because a truth value of 0.5 half-satisfies both x and not x at once. The feature: contradictory graded evidence can coexist and gradients keep flowing. The bug: the law of non-contradiction quietly stops holding, so a fuzzy system can never fully rule anything out -- which is either liberating or terrifying depending on the day.
Exercise 3 -- Intervene on a concept bottleneck. Run a random input through the ConceptBottleneckModel, then override EACH concept to 1.0 one at a time and count how often the final class flips.
import torch
import torch.nn as nn
class ConceptBottleneckModel(nn.Module):
def __init__(self, input_dim=64, n_concepts=15, n_classes=200):
super().__init__()
self.concept_predictor = nn.Sequential(
nn.Linear(input_dim, 512), nn.ReLU(),
nn.Linear(512, n_concepts), nn.Sigmoid())
self.label_predictor = nn.Sequential(
nn.Linear(n_concepts, 128), nn.ReLU(),
nn.Linear(128, n_classes))
def forward(self, x, intervene=None):
concepts = self.concept_predictor(x)
if intervene is not None:
mask = (intervene >= 0) # -1 entries = leave alone
concepts = torch.where(mask, intervene, concepts)
return self.label_predictor(concepts), concepts
torch.manual_seed(0)
model = ConceptBottleneckModel()
x = torch.randn(1, 64)
base = model(x)[0].argmax(1).item()
flips = []
for c in range(15):
override = torch.full((1, 15), -1.0)
override[0, c] = 1.0
logits, _ = model(x, intervene=override)
if logits.argmax(1).item() != base:
flips.append(c)
print(f"base class: {base}")
print(f"{len(flips)}/15 concepts flip the prediction when forced on: {flips}")
Some concepts move the class, most do not. The concepts that flip it are the ones the final prediction is most SENSITIVE to -- pin them and you have found the load-bearing beams of this particular decision. And the two sentences the exercise wanted: this concept-level sensitivity is exactly the debugging a plain black-box classifier can never give you, because a black box only exposes raw input pixels as knobs, not named human concepts. When your CBM misfires, you can point at "it thought red-breast was true when it was not" -- when a black box misfires, you get a heatmap of pixels and a shrug. Right, homework settled. Now let us go crack open some boxes that were NOT built to open ;-)
Let me be precise about what "black box" even means, because people wave the phrase around loosely. A neural network is not black because we cannot see inside it -- we can print every single weight, every activation, every gradient. It is black because the computation is OPAQUE despite being fully visible. Millions of parameters, layers of nonlinear transformations, and even though every step is deterministic arithmetic -- matrix multiplies, ReLUs, softmaxes -- the emergent behaviour tells you nothing. You can stare at all 175 billion numbers and still have no earthly idea why it called this photo a cat and that photo a dog.
Having said that, why should you care? Three reasons, and none of them are academic.
Trust. People will not use a system they do not understand, and they are right not to. A radiologist is not going to sign off on an AI diagnosis that cannot point at what it found on the scan. Trust is not granted, it is earned, and "I am 99.4% confident, source: vibes" does not earn it.
Debugging. When a model fails, you need the failure MODE, not just the failure. Is it leaning on a spurious correlation -- the famous case of a pneumonia model that had really learned to detect which hospital's scanner took the image? Is it taking a shortcut? Did an adversarial input tip it over? You cannot fix what you cannot see, and accuracy on a test set hides all of this.
Compliance. This is the one that turned interpretability from a research hobby into a budget line. The EU AI Act, GDPR's right to explanation, financial regulations around automated credit decisions -- more and more, high-stakes AI has to be explainable BY LAW. Not because a professor thinks it is elegant, but because a regulator can fine you.
One quick distinction before we get to tools, because the two title words are not synonyms. Interpretability means the model is inherently understandable -- a linear regression (#10), a small decision tree (#17), you can read the logic straight off it. Explainability means the model is opaque and you generate a post-hoc STORY about what it did. Almost everything worth deploying today is opaque, so almost everything below is explainability. And a warning that will echo through this whole episode: a post-hoc story can be plausible, convincing, and WRONG. Keep that in your pocket.
If I had to bet on one attribution method surviving the decade, it is SHAP (SHapley Additive exPlanations). The reason is that it is the only one with an actual theorem behind it instead of a good intuition.
The idea comes from cooperative game theory. Imagine the features are players cooperating to produce a prediction, and you want to split the "payout" (how far this prediction sits from the average prediction) fairly among them. The Shapley value is the unique fair split, and it is the ONLY attribution that simultaneously satisfies three properties you actually want: local accuracy (the attributions sum exactly to the prediction), consistency (if a feature matters more in a new model, its attribution never goes down), and missingness (a feature the model ignores gets exactly zero credit). No other method ticks all three. That uniqueness is the whole selling point.
import torch
import torch.nn as nn
import itertools
import math
def compute_shapley_values(model, x, baseline, n_features):
"""Exact Shapley values -- exponential cost, for illustration only.
In production you use KernelSHAP / TreeSHAP / DeepSHAP instead."""
model.eval()
with torch.no_grad():
shapley = torch.zeros(n_features)
for i in range(n_features):
others = [j for j in range(n_features) if j != i]
total = 0.0
for size in range(len(others) + 1):
for subset in itertools.combinations(others, size):
subset = set(subset)
# prediction WITH feature i present
x_with = baseline.clone()
for j in subset | {i}:
x_with[j] = x[j]
val_with = model(x_with.unsqueeze(0)).item()
# prediction WITHOUT feature i
x_without = baseline.clone()
for j in subset:
x_without[j] = x[j]
val_without = model(x_without.unsqueeze(0)).item()
# weight = 1 / (n * C(n-1, |S|)) in Shapley form
s, n = len(subset), n_features
weight = (math.factorial(s) *
math.factorial(n - s - 1) /
math.factorial(n))
total += weight * (val_with - val_without)
shapley[i] = total
return shapley
Read what that loop is really doing: for each feature, it asks "how much does adding this feature change the prediction, averaged over every possible coalition of the other features?" That averaging-over-all-orderings is what makes it fair, and it is also what makes it expensive -- there are 2^n subsets for n features, so the exact version above dies past maybe fifteen or twenty features. Nobody runs the exact version. In practice you reach for approximations: KernelSHAP samples coalitions and fits a weighted linear regression, TreeSHAP exploits tree structure for exact values in polynomial time (perfect for the gradient boosting from #19), and DeepSHAP rides backprop to approximate values for neural nets cheaply. The payoff sentence you can say out loud in a compliance meeting, and which honours all three forementioned properties: "for THIS prediction, income contributed +0.31 and postcode contributed -0.12 relative to the average applicant." Concrete, additive, auditable. That is what a regulator wants to hear.
Before I get to the attention trap, let me hand you the tool I actually reach for on a neural net, because it is short and it is honest: integrated gradients (Sundararajan et al., 2017). The naive idea is "the gradient of the output with respect to each input tells you what matters." True-ish, but raw gradients are noisy and saturate -- a feature can be maxed out and locally flat, so its gradient reads zero even though it fully determined the answer. Integrated gradients fixes that by averaging the gradient along a straight path from a baseline (say, all-zeros) to the real input.
import torch
def integrated_gradients(model, x, baseline=None, steps=64, target=0):
"""Attribution = (x - baseline) * average gradient along the path."""
if baseline is None:
baseline = torch.zeros_like(x)
model.eval()
grads = torch.zeros_like(x)
for k in range(1, steps + 1):
alpha = k / steps
point = (baseline + alpha * (x - baseline)).clone().requires_grad_(True)
out = model(point.unsqueeze(0))[0, target]
out.backward()
grads += point.grad.detach()
avg_grad = grads / steps
return (x - baseline) * avg_grad # one attribution per input feature
That (x - baseline) * avg_grad gives you attributions that actually SUM to the difference in output between the baseline and the input (a completeness property, cousin to SHAP's local accuracy). Unlike a raw gradient, it does not get fooled by saturation, and unlike attention -- which we are about to trash -- it measures a real causal contribution to the output. This is my default first look at "which input tokens/pixels drove this."
LIME (Local Interpretable Model-agnostic Explanations) takes the lazy-but-clever route. Any model, no matter how gnarly, looks roughly LINEAR if you zoom in close enough on a single prediction. So: perturb the input a bunch of times around the point you care about, see how the black box responds, and fit a simple interpretable model (a weighted linear regression) to that little cloud of responses. The simple model is your explanation, valid locally.
import torch
class LIME:
"""Explain one prediction by fitting a local linear surrogate."""
def __init__(self, model, n_samples=1000):
self.model = model
self.n_samples = n_samples
def explain(self, x, n_show=5):
self.model.eval()
n_features = x.shape[0]
# binary masks: keep a feature (1) or replace it with baseline (0)
masks = torch.bernoulli(torch.full((self.n_samples, n_features), 0.5))
baseline = torch.zeros_like(x)
perturbed = baseline.unsqueeze(0).expand(self.n_samples, -1).clone()
for i in range(self.n_samples):
active = masks[i].bool()
perturbed[i, active] = x[active]
with torch.no_grad():
preds = self.model(perturbed).squeeze()
# samples closer to the original count more
distances = (masks - 1).pow(2).sum(dim=1).sqrt()
weights = torch.exp(-distances / (n_features * 0.25))
W = masks * weights.unsqueeze(1)
y = preds * weights
coefs = torch.linalg.lstsq(W, y).solution # the local linear model
top = coefs.abs().topk(n_show)
return {"features": top.indices.tolist(),
"weights": coefs[top.indices].tolist()}
LIME's charm is that it is genuinely model-agnostic -- it only ever needs input-in, prediction-out, so it works on a random forest, a transformer, a black-box API you have no source for, whatever. Its curse is INSTABILITY: change the random perturbations and you can get a noticeably different explanation for the exact same prediction. Run it twice, get two stories. SHAP does not have this problem because Shapley values are unique by construction. My honest take: LIME is a fine quick-and-dirty first look, but if the explanation is going into a report someone will be held accountable for, I want SHAP's determinism behind it. Do not build a compliance story on a method that changes its mind when you reseed the RNG.
Now the trap. If you have worked with transformers (#51-53) it is desperately tempting to grab the attention weights and call them an explanation. "Look, the model attended heavily to these three words, therefore these words caused the prediction." It looks like an explanation. It renders beautifully as a heatmap. And it is, at best, half-true.
def extract_attention_maps(model, input_ids, layer=-1):
"""Pull attention weights from one transformer layer."""
model.eval()
with torch.no_grad():
outputs = model(input_ids, output_attentions=True)
# attentions: tuple of (batch, n_heads, seq_len, seq_len)
attention = outputs.attentions[layer]
avg_attention = attention.mean(dim=1) # average across heads
cls_attention = avg_attention[0, 0, :] # [CLS] -> all tokens
return cls_attention
Here is why that heatmap lies to you. Attention is not attribution. It shows where the model LOOKS, not what CAUSES the output. A head can attend hard to a token and then a later layer can throw that information straight in the bin. Different heads do different jobs. Averaging across heads (like the code above does, for a pretty picture) smears together a head tracking syntax with a head tracking sentiment -- the average means nothing. And the killer, shown by Jain and Wallace in their bluntly-titled paper "Attention is not Explanation": you can often find COMPLETELY different attention weights that produce the exact same prediction. If two different "explanations" give the identical output, neither is faithfully explaining the decision. So what do you use instead? The integrated gradients from a few sections up, or plain gradient-based attribution -- both measure actual causal push on the output, not merely where the model turned its gaze. Attention maps are a lovely diagnostic for understanding the ARCHITECTURE. They are a bad witness in court.
SHAP and LIME answer "which INPUT FEATURES mattered?" -- which pixel, which token. But a stakeholder rarely thinks in pixels. They think in concepts. TCAV (Testing with Concept Activation Vectors, Kim et al., 2018) lets you ask the question at that higher level: does the human concept "stripes" actually influence the prediction "zebra"? Does "wrinkles" push the prediction toward "old"? That is an explanation a non-technical person can act on.
The trick is neat. Grab a bunch of examples that HAVE the concept (photos with stripes) and a bunch of random ones that do not. Look at the network's activations at some layer for both piles, and train a simple linear classifier to separate them. The direction perpendicular to that classifier's boundary -- the Concept Activation Vector -- IS the concept, expressed in the network's own internal language. Then you check how sensitive the class prediction is to nudging activations along that direction.
import torch
import torch.nn as nn
class TCAV:
"""Testing with Concept Activation Vectors."""
def __init__(self, model, layer_name):
self.model = model
self.layer_name = layer_name
self.activations = {}
for name, module in model.named_modules():
if name == layer_name:
module.register_forward_hook(self._hook)
def _hook(self, module, inp, out):
self.activations[self.layer_name] = out.detach()
def learn_concept_vector(self, concept_examples, random_examples):
"""Direction that separates 'has concept' from 'random' activations."""
self.model.eval()
with torch.no_grad():
self.model(concept_examples)
concept_acts = self.activations[self.layer_name].flatten(1)
self.model(random_examples)
random_acts = self.activations[self.layer_name].flatten(1)
X = torch.cat([concept_acts, random_acts])
y = torch.cat([torch.ones(len(concept_acts)),
torch.zeros(len(random_acts))])
cav = nn.Linear(X.shape[1], 1)
opt = torch.optim.Adam(cav.parameters(), lr=1e-3)
for _ in range(100):
loss = nn.functional.binary_cross_entropy_with_logits(
cav(X).squeeze(), y)
opt.zero_grad(); loss.backward(); opt.step()
return cav.weight.data.squeeze() # the CAV direction
TCAV boils all of that down to a single, gorgeously communicable number: what fraction of a class's predictions are pushed UP by the concept. "87% of this model's zebra predictions are positively influenced by the stripes concept." A product manager understands that sentence. A judge understands that sentence. And if you run it and discover that 60% of your "criminal risk" predictions are positively influenced by a concept that correlates with race, you have found something a pixel heatmap would have hidden from you forever. That is TCAV's real power -- it audits at the level humans actually reason and worry.
Everything so far generates explanations ABOUT a model from the outside. The most ambitious tribe in the field wants something harder: to reverse-engineer what the network actually COMPUTES, circuit by circuit, the way you would decompile a program. This is mechanistic interpretability, and the findings coming out of it are genuinely startling.
The simplest entry point is activation maximization -- instead of feeding data in and reading predictions out, you FREEZE the weights and optimise the INPUT to maximally excite one specific neuron. Whatever image or text emerges is that neuron's favourite thing, its concept.
import torch
import torch.nn as nn
def maximize_neuron(model, layer, neuron_idx, input_shape, steps=200, lr=0.1):
"""Find the input that most excites one neuron (freeze weights, train input)."""
captured = {}
handle = layer.register_forward_hook(
lambda m, i, o: captured.__setitem__("act", o))
x = torch.randn(input_shape, requires_grad=True)
opt = torch.optim.Adam([x], lr=lr)
for _ in range(steps):
model(x)
activation = captured["act"][0, neuron_idx] # the target neuron
loss = -activation # maximize -> minimize negative
opt.zero_grad(); loss.backward(); opt.step()
handle.remove()
return x.detach()
# toy demo: what does neuron 3 of a tiny net "want" to see?
net = nn.Sequential(nn.Linear(8, 16), nn.ReLU(), nn.Linear(16, 4))
dream = maximize_neuron(net, net[0], neuron_idx=3, input_shape=(1, 8))
print("input that maximally excites neuron 3:", dream.squeeze().round(decimals=2))
Run that on a real vision model and the "dreams" are how researchers found things like a neuron that fires for curves at a specific angle, or the now-famous multimodal neuron that responds to the Golden Gate Bridge whether you show it a photo, a drawing, or the literal words "Golden Gate". Beyond single neurons, people have found circuits -- little interpretable algorithms wired across layers, like the "induction heads" in transformers that learn to copy a pattern from earlier in the context (the mechanism underneath a lot of in-context learning from #144). And the strangest discovery, superposition: networks routinely represent MORE features than they have dimensions, by packing them into overlapping, nearly-orthogonal directions and relying on features rarely firing at once. Your intuition that "neuron = concept" is too clean -- reality is more like a compression scheme.
Be honest about the state of it: this is early, painstaking, mostly done on small or toy models, and scaling it to frontier LLMs is one of the hardest open problems in the whole safety agenda. But the glimpses are real. Neural networks are not inscrutable soup. There is structure in there -- interpretable, describable structure -- if you have the patience and the tools to look. And I find that quietly hopeful.
Get your hands dirty before the next one. Three tasks, climbing in difficulty:
Compare integrated gradients to a raw gradient. Build a tiny nn.Sequential regressor, pick one input, and compute both a plain single-point gradient AND the integrated-gradients attribution from a zero baseline. Print them side by side. In one sentence, explain a case where they would DISAGREE and why the integrated version is the one you would trust.
Feel LIME's instability. Take the LIME class, run explain on the SAME input three times WITHOUT fixing a seed, and print the top-3 features each time. Report whether the top feature stays put, then explain in two sentences why this instability is a genuine problem for a compliance report and how SHAP sidesteps it.
Fake a spurious correlation and catch it. Make a toy 2D classifier where the label secretly depends only on feature 0, but feature 1 is engineered to correlate with the label in your training sample. Train it, then use compute_shapley_values (or integrated gradients) on a test point where the correlation is BROKEN, and show the attribution reveals the model leaning on the wrong feature. Two sentences on why this is exactly the debugging power the black box denied you.
We open next episode with full solutions, as always.
And here is the thread I will leave hanging for next time. We have now spent an entire episode learning to see WHY a model decides what it decides. But seeing the reason is only step one. What happens when you look inside and you do not LIKE what you find -- when the model is optimising for something subtly, dangerously different from what you actually meant? When it learns to game your metric, or tells you what you want to hear instead of what is true? Understanding a system and CONTROLLING a system are very different problems, and the gap between them is where some of the most important work in the field lives right now. That is where we head next ;-)
@scipio