pip install torch scikit-learn xgboost) -- everything here runs comfortably on a laptop, no GPU required today;Learn AI Series):I ended episode #131 with a deliberately uncomfortable admission, and I promised we would come back to it. We spent that whole episode celebrating data that is proudly, irreducibly graph-shaped -- molecules, social networks, fraud rings -- and rightly so. But then I pointed out the twist that ought to nag at anybody who has built a hundred neural networks in a row: the overwhelming majority of the data actually sitting in the world's databases is not a molecule or a graph at all. It is boring, flat, tabular stuff. Rows and columns. The spreadsheets and SQL tables that quietly run every bank, shop and hospital on the planet. And on that kind of data -- the most common kind there is -- the fancy deep networks we have been lovingly hand-building very often lose to a humble gradient-boosted tree from episode #19.
So today we stare that embarrassment straight in the face. Why does deep learning struggle exactly where the data is most abundant? What have people tried to do about it? And -- the part you actually care about -- what should you reach for when the problem is a plain old table? Here we go.
As always, let's clear last time's graph homework first -- full code, no hand-waving.
Exercise 1 -- watch over-smoothing happen. The task was to rebuild the Cora classifier with 2, then 4, then 8 GCN layers, train each for 200 epochs, and then, for the 8-layer model, measure the average pairwise cosine similarity of the final node embeddings. The whole point is to see the accuracy collapse and connect it to the embeddings all drifting toward one vector.
import torch
import torch.nn as nn
import torch.nn.functional as F
from torch_geometric.nn import GCNConv
from torch_geometric.datasets import Planetoid
dataset = Planetoid(root='/tmp/Cora', name='Cora')
data = dataset[0]
class DeepGCN(nn.Module):
def __init__(self, in_dim, hidden, out_dim, n_layers):
super().__init__()
dims = [in_dim] + [hidden] * (n_layers - 1) + [out_dim]
self.convs = nn.ModuleList(
[GCNConv(dims[i], dims[i + 1]) for i in range(n_layers)]
)
def forward(self, data, return_embed=False):
x, edge_index = data.x, data.edge_index
for i, conv in enumerate(self.convs):
x = conv(x, edge_index)
if i < len(self.convs) - 1:
x = F.relu(x)
return x
def train_eval(n_layers):
torch.manual_seed(0)
model = DeepGCN(dataset.num_node_features, 64, dataset.num_classes, n_layers)
opt = torch.optim.Adam(model.parameters(), lr=0.01, weight_decay=5e-4)
for _ in range(200):
model.train(); opt.zero_grad()
out = model(data)
F.cross_entropy(out[data.train_mask], data.y[data.train_mask]).backward()
opt.step()
model.eval()
pred = model(data).argmax(dim=1)
acc = (pred[data.test_mask] == data.y[data.test_mask]).float().mean().item()
return acc
for n in (2, 4, 8):
print(f"{n} layers -> test acc {train_eval(n):.3f}")
Run it and you will watch the accuracy slide from a healthy ~0.81 at 2 layers down toward coin-flip territory by 8. Now the diagnosis -- the cosine similarity of the over-smoothed embeddings:
def mean_pairwise_cosine(embeds):
e = F.normalize(embeds, dim=1)
sim = e @ e.t() # all pairwise cosines
n = sim.shape[0]
off_diag = (sim.sum() - n) / (n * n - n) # exclude the diagonal (self-sim = 1)
return off_diag.item()
torch.manual_seed(0)
deep = DeepGCN(dataset.num_node_features, 64, dataset.num_classes, 8)
opt = torch.optim.Adam(deep.parameters(), lr=0.01, weight_decay=5e-4)
for _ in range(200):
deep.train(); opt.zero_grad()
out = deep(data)
F.cross_entropy(out[data.train_mask], data.y[data.train_mask]).backward()
opt.step()
deep.eval()
print(f"mean pairwise cosine (8 layers): {mean_pairwise_cosine(deep(data)):.3f}")
The two-sentence answer I was fishing for: the deeper the stack, the more rounds of neighbour-averaging you apply, and averaging is a smoothing operation -- so node representations converge toward a common vector and the mean pairwise cosine creeps up toward 1. Once every node looks like every other node, the classifier has nothing left to separate, and the accuracy falls off a cliff. Over-smoothing is not a training bug you can tune away; it is baked into what stacking message-passing layers does.
Exercise 2 -- GCN versus GAT, head to head. Train both variants on Cora five times each with different seeds, then report mean and standard deviation. Do the error bars overlap?
import numpy as np
from torch_geometric.nn import GATConv
class GNN(nn.Module):
def __init__(self, in_dim, hidden, out_dim, use_gat):
super().__init__()
if use_gat:
self.c1 = GATConv(in_dim, hidden, heads=4, concat=True)
self.c2 = GATConv(hidden * 4, out_dim, heads=1, concat=False)
else:
self.c1 = GCNConv(in_dim, hidden)
self.c2 = GCNConv(hidden, out_dim)
self.drop = nn.Dropout(0.5)
def forward(self, data):
x, ei = data.x, data.edge_index
x = self.drop(F.relu(self.c1(x, ei)))
return self.c2(x, ei)
def run(use_gat, seed):
torch.manual_seed(seed)
model = GNN(dataset.num_node_features, 64, dataset.num_classes, use_gat)
opt = torch.optim.Adam(model.parameters(), lr=0.01, weight_decay=5e-4)
for _ in range(200):
model.train(); opt.zero_grad()
out = model(data)
F.cross_entropy(out[data.train_mask], data.y[data.train_mask]).backward()
opt.step()
model.eval()
pred = model(data).argmax(dim=1)
return (pred[data.test_mask] == data.y[data.test_mask]).float().mean().item()
for name, flag in [("GCN", False), ("GAT", True)]:
accs = [run(flag, s) for s in range(5)]
print(f"{name}: {np.mean(accs):.3f} +/- {np.std(accs):.3f}")
On Cora you will typically see GAT land a point or two above GCN, but with standard deviations around 0.005-0.01 the error bars very often overlap. The honest one-liner: on this particular small, homophilous citation graph, attention's extra compute buys you a marginal, not-always-significant gain -- GAT earns its keep on messier, heterophilous graphs, not on the friendly benchmark. This is exactly the "measure, don't assume" discipline from episode #13, and it is the same lesson waiting for us today with tables.
Exercise 3 -- build a graph by hand and reason about hops. Construct a 6-node friend graph, run simple_message_pass twice, and confirm two-hop influence. Then: for a node whose farthest neighbour is 4 hops away, how many rounds until that signal can reach it?
def simple_message_pass(x, edge_index):
num_nodes = x.shape[0]
src, dst = edge_index
agg = torch.zeros_like(x)
agg.index_add_(0, dst, x[src])
deg = torch.zeros(num_nodes)
deg.index_add_(0, dst, torch.ones(src.shape[0]))
return agg / deg.clamp(min=1).unsqueeze(1)
# A line-ish graph: 0-1-2-3-4-5 (undirected), so node 0's farthest neighbour is 5.
edge_index = torch.tensor([
[0, 1, 1, 2, 2, 3, 3, 4, 4, 5],
[1, 0, 2, 1, 3, 2, 4, 3, 5, 4],
], dtype=torch.long)
x = torch.eye(6) # one-hot features so we can literally trace influence
h1 = simple_message_pass(x, edge_index)
h2 = simple_message_pass(h1, edge_index)
print("Node 0 after 2 rounds, has it 'seen' node 2?",
h2[0, 2].item() > 0) # True: node 2 is exactly two hops from node 0
The answer to the closing question: message passing moves information exactly one hop per round, so a signal four hops away needs a minimum of four rounds (four layers) before it can possibly reach the target node. That, incidentally, is the exact same reasoning that makes deep GNNs tempting -- and, as exercise 1 just showed, over-smoothing is the tax you pay for reaching too far. Right, graphs put to bed. On to the spreadsheets.
To understand the problem, think about why deep learning is so devastating on images and text in the first place. Images have spatial locality -- nearby pixels belong together, which is exactly the prior a convolution (episode #45) bakes in. Text has sequential structure -- word order carries meaning, which is what attention (episode #51) exploits. Both domains hand you a homogeneous input (all pixels, or all tokens) and, usually, oceans of data to pre-train on.
Tabular data has none of that. Not one of those properties. Let me be specific, because the specifics are the whole story:
Heterogeneous features. Column 1 is a person's age (continuous). Column 2 is their country (categorical, 195 possible values). Column 3 is their last purchase amount (continuous, heavy-tailed). Column 4 is a yes/no subscription flag (binary). Every column has a different scale, a different distribution, a different meaning. There is no sensible "convolution" you can slide across columns, because column 3 has nothing spatially in common with column 4.
No spatial or sequential structure. Shuffle the column order and you have lost exactly nothing -- the information content is identical. There is no "nearby" relationship between features for a convolutional prior to grab hold of.
Feature interactions are irregular. In an image, local pixel patterns matter everywhere in the frame. In a table, the interaction between "age" and "income" might be the crux of the whole problem, while "age" and "shoe size" is pure noise. Which interactions matter is dataset-specific, and the network has to discover that from scratch.
Smaller datasets. Most tabular problems have thousands to a few hundred thousand rows. Deep learning is happiest with millions. Trees, by contrast, are sample-efficient by design -- they greedily hunt for the single best split, and a good split needs surprisingly few examples to find.
Let's watch a plain MLP -- the naive deep-learning reflex -- have a go at a classic tabular dataset.
import torch
import torch.nn as nn
import numpy as np
from sklearn.datasets import fetch_california_housing
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import mean_squared_error
data = fetch_california_housing()
X_train, X_test, y_train, y_test = train_test_split(
data.data, data.target, test_size=0.2, random_state=42
)
scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)
X_test = scaler.transform(X_test)
class SimpleMLP(nn.Module):
def __init__(self, in_dim, hidden=128):
super().__init__()
self.net = nn.Sequential(
nn.Linear(in_dim, hidden), nn.ReLU(), nn.Dropout(0.1),
nn.Linear(hidden, hidden), nn.ReLU(), nn.Dropout(0.1),
nn.Linear(hidden, 1),
)
def forward(self, x):
return self.net(x).squeeze(-1)
model = SimpleMLP(X_train.shape[1])
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
X_tr = torch.tensor(X_train, dtype=torch.float32)
y_tr = torch.tensor(y_train, dtype=torch.float32)
for epoch in range(100):
model.train()
loss = nn.MSELoss()(model(X_tr), y_tr)
optimizer.zero_grad(); loss.backward(); optimizer.step()
model.eval()
with torch.no_grad():
pred = model(torch.tensor(X_test, dtype=torch.float32)).numpy()
print(f"MLP RMSE: {mean_squared_error(y_test, pred, squared=False):.4f}")
Note that even to get this far the MLP needed the StandardScaler -- feed it raw features and it barely trains at all. Hold that thought, because a tree would not have cared one bit about scaling. Now run the tree and compare.
Understanding why trees win teaches you more than any of the fancy alternatives. And this is not a fringe opinion, mind you -- multiple large-scale benchmarks have confirmed it. Grinsztajn and colleagues (2022) tested across 45 tabular datasets and found tree-based models beating deep learning on medium-sized data, with less tuning. The "deep learning conquers everything" narrative has a glaring exception, and it happens to be the most common data format in industry. Having said that, the reasons are wonderfully concrete:
Axis-aligned splits are natural for tables. A decision tree asks "is income > 50k?" -- a threshold on a single feature. A staggering number of real-world decision boundaries are approximately axis-aligned like that. A neural network has to approximate the same boundary with curved surfaces, which is overkill and wastes capacity.
Trees handle heterogeneous features natively. Categorical, continuous, missing -- a tree splits on any of them without a preprocessing pipeline. A neural network needs encoding, normalisation, and imputation before it can even start.
Trees are invariant to monotone transformations. Whether a feature ranges 0-1 or 0-1,000,000 does not change a single split. Neural nets are painfully sensitive to scale -- which is exactly why our MLP needed that scaler.
Boosting corrects its own mistakes. Each new tree in a gradient-boosting ensemble specifically targets the samples the previous trees got wrong (this is the residual-fitting idea from episode #19). It is a built-in form of hard-example mining that an MLP simply does not do on its own.
from sklearn.ensemble import GradientBoostingRegressor
gbm = GradientBoostingRegressor(n_estimators=200, max_depth=5, random_state=42)
gbm.fit(X_train, y_train) # note: NO scaling needed, the tree does not care
pred_gbm = gbm.predict(X_test)
print(f"GBM RMSE: {mean_squared_error(y_test, pred_gbm, squared=False):.4f}")
Run both and the tree almost certainly wins -- and it did so with default parameters and zero scaling. Not because the MLP is fundamentally incapable, but because it walked into the room with the wrong inductive biases for this kind of data. Keep that framing: it is not a story about intelligence, it is a story about priors.
So can we build a network with better priors? Google's TabNet (2019) was one of the first serious attempts to beat trees on their own turf. Its key insight is lovely: make feature selection step-wise, the way a decision tree does. At each decision step, an attention mechanism picks which features to look at, processes them, and hands off to the next step -- loosely echoing how a tree selects one feature to split on at each node.
Two components carry the whole design:
Sparse attention for feature selection. At each step a learnable mask selects a subset of features. Crucially the mask uses sparsemax, not softmax, so it produces genuine zeros -- features are truly ignored, not merely down-weighted. That is what makes TabNet interpretable: you can literally read off which features it used at each step.
Sequential multi-step processing. In stead of one forward pass, TabNet processes the data across several "steps", each seeing the original features through a different mask, and sums the outputs. Think of it as the network making a little sequence of decisions, each focused on a different corner of the feature space.
import torch.nn.functional as F
class TabNetStep(nn.Module):
"""A simplified single TabNet decision step."""
def __init__(self, in_dim, hidden_dim, out_dim):
super().__init__()
self.fc_attn = nn.Sequential(
nn.Linear(in_dim, in_dim), nn.BatchNorm1d(in_dim),
)
self.fc_process = nn.Sequential(
nn.Linear(in_dim, hidden_dim), nn.BatchNorm1d(hidden_dim), nn.ReLU(),
nn.Linear(hidden_dim, out_dim),
)
def forward(self, x, prior_scales):
# Which features matter for THIS step?
attn = torch.softmax(self.fc_attn(x) * prior_scales, dim=-1)
masked = attn * x
out = self.fc_process(masked)
# Features used a lot now get de-prioritised for later steps.
prior_scales = prior_scales * (1.0 - attn)
return out, attn, prior_scales
class MiniTabNet(nn.Module):
def __init__(self, in_dim, hidden=64, out_dim=1, n_steps=3):
super().__init__()
self.steps = nn.ModuleList(
[TabNetStep(in_dim, hidden, hidden) for _ in range(n_steps)]
)
self.head = nn.Linear(hidden, out_dim)
def forward(self, x):
prior = torch.ones_like(x) # all features start available
agg = torch.zeros(x.shape[0], 64, device=x.device)
for step in self.steps:
h, attn, prior = step(x, prior)
agg = agg + F.relu(h)
return self.head(agg).squeeze(-1)
TabNet's pitch is that it is interpretable and needs no feature engineering. In practice the results are, let me be honest, mixed. On some datasets it matches or beats XGBoost; on others it quietly underperforms, especially when you have not spent a fortune on hyperparameter tuning. It is a genuinely clever design that did not turn out to be a free lunch -- which is a recurring theme in this whole area.
SAINT (Self-Attention and Intersample Attention Transformer, 2021) is bolder. It applies transformer-style self-attention in two directions: across the features within a row (which columns relate to each other?), and across the rows within a batch (which other samples inform this one?).
The feature-wise half is the familiar Transformer of episode #52, just pointed sideways. Each feature becomes a "token": its value is embedded and added to a learned per-feature embedding, then multi-head self-attention lets the columns talk to each other -- "age" can attend to "income" to capture their joint effect without you ever hand-crafting an age_times_income column.
class FeatureSelfAttention(nn.Module):
"""Self-attention across features (columns) for each sample."""
def __init__(self, n_features, embed_dim, n_heads=4):
super().__init__()
self.feature_embeddings = nn.Embedding(n_features, embed_dim)
self.value_proj = nn.Linear(1, embed_dim)
self.attn = nn.MultiheadAttention(embed_dim, n_heads, batch_first=True)
self.norm = nn.LayerNorm(embed_dim)
def forward(self, x):
# x: (batch, n_features)
_, n_feat = x.shape
feat_ids = torch.arange(n_feat, device=x.device)
feat_emb = self.feature_embeddings(feat_ids) # (n_feat, embed_dim)
val_emb = self.value_proj(x.unsqueeze(-1)) # (batch, n_feat, embed_dim)
tokens = val_emb + feat_emb.unsqueeze(0) # value + column identity
attended, _ = self.attn(tokens, tokens, tokens)
return self.norm(tokens + attended) # residual, as always
The intersample half is the strange, interesting bit. For each feature, the model also attends across all the rows in the mini-batch, letting other training examples inform the current prediction -- a learned, soft cousin of k-nearest-neighbours baked right into the forward pass. The philosophical question SAINT poses is a good one: does the attention machinery that reshaped sequences and images actually transfer to tables? The answer, as far as the benchmarks go, is "partially". SAINT is competitive on many of them -- but it wants careful tuning and a large-enough dataset to justify all those parameters.
Let me be completely straight about what the evidence says, as of 2025-2026, because vendor blog posts will not:
Gradient boosting (XGBoost, LightGBM, CatBoost) wins on:
Deep learning approaches win on:
Here is the decision logic I actually carry around in my head, written as throwaway pseudo-code -- not a model, just a way of thinking:
# This is a decision aid, NOT something you fit. Read it, don't train it.
def choose_model(dataset_size, has_mixed_modalities, tuning_budget):
if has_mixed_modalities:
return "deep_learning" # you need one differentiable pipeline
if dataset_size < 10_000:
return "gradient_boosting" # trees own the small-data regime
if dataset_size < 50_000:
return "try_both" # genuinely competitive, so measure
if tuning_budget == "low":
return "gradient_boosting" # far less sensitive to hyperparameters
return "try_both" # at scale, deep learning starts to compete
Nota bene: notice how often the honest answer is simply "try both and measure". That is not a cop-out, it is the discipline -- the same one from episode #13, where we learned that a number on a held-out set beats an opinion every single time.
One of the loudest arguments for deep learning on tables is that it should abolish manual feature engineering. In stead of you hand-crafting age_times_income or log_purchase_amount, the network learns those transformations for you.
In theory, sure -- a wide-and-deep enough MLP can represent any feature interaction. In practice, hand-crafted features keep helping, even alongside deep models. Why? Because domain knowledge is free information that shrinks what the model has to discover from scratch. If you happen to know that "income relative to the regional median" matters more than raw income, encoding that directly spares the network from having to rediscover it from a limited pile of rows.
So the pragmatic move -- the one that quietly annoys purists on both sides -- is to do sensible feature engineering and use a model that can learn extra interactions on top. That is not cheating; it is efficiency. Even TabNet and SAINT are happier with a bit of thoughtful preprocessing under them.
The one place deep learning genuinely does dissolve feature engineering is when tabular data is fused with unstructured data -- images, text, audio. You cannot hand-engineer features out of raw pixels, but you can push them through a CNN (episode #45) or a Transformer and concatenate the result with your columns. When a prediction depends on both structured fields and a photo -- think property pricing from house features plus listing images -- an end-to-end neural network stops being a luxury and becomes the only sane architecture. That, right there, is the real frontier for deep learning on tables.
The tabular-deep-learning space is moving fast, so let me not pretend it is settled. FT-Transformer (2021) made a quietly important point: a straightforward Transformer over feature tokens -- without TabNet's step machinery or SAINT's intersample gymnastics -- is competitive with far simpler code.
# The spine of FT-Transformer: tokenise features, then a standard Transformer.
class FTTransformerSpine(nn.Module):
def __init__(self, n_features, embed_dim=32, depth=3, heads=8):
super().__init__()
self.tokenizer = FeatureSelfAttention(n_features, embed_dim, heads)
layer = nn.TransformerEncoderLayer(embed_dim, heads, batch_first=True)
self.encoder = nn.TransformerEncoder(layer, num_layers=depth)
self.cls = nn.Parameter(torch.randn(1, 1, embed_dim)) # a [CLS]-style token
self.head = nn.Linear(embed_dim, 1)
def forward(self, x):
tokens = self.tokenizer(x) # (batch, n_feat, embed)
cls = self.cls.expand(x.shape[0], -1, -1)
tokens = torch.cat([cls, tokens], dim=1) # prepend the CLS token
encoded = self.encoder(tokens)
return self.head(encoded[:, 0]).squeeze(-1) # read out from CLS
If that [CLS]-token readout looks familiar, it should -- it is exactly the trick BERT used back in episode #59, now pointed at columns in stead of words. More recent work goes further still: pre-training on large tabular corpora (the same recipe that made BERT and GPT what they are), and transfer learning across tabular datasets. It is early days, but the direction is unmistakable.
Still, the honest bottom line has not moved: if you have a bog-standard tabular problem, start with XGBoost or LightGBM. Train it in five minutes. Lock in your baseline. Then ask whether the marginal gain from a deep model justifies the extra complexity, training time, and tuning pain. For most tabular problems the answer is a cheerful "no". For the ones where it is "yes" -- multi-modal data, very large data, semi-supervised settings -- the architectures in this episode are your starting line.
Three to chew on before next time -- and, as always, I'll walk through full solutions at the top of the following episode.
Baseline the gap yourself. Take the California Housing setup from this episode and add an XGBoost regressor (xgboost.XGBRegressor, default params) alongside the MLP and the GradientBoostingRegressor. Report all three RMSEs. Then re-run the MLP without the StandardScaler and record what happens. Write two sentences: one on the tree-versus-MLP gap, one on why scaling mattered so much for only one of the three.
Make the MLP fight back. The plain MLP started with lousy priors -- so give it better ones. Add batch normalisation between layers, widen it, and train for 300 epochs with an LR scheduler. How close can you push the MLP's RMSE to the GBM's? Note the smallest change that helped the most -- that tells you which missing prior hurt worst.
Read TabNet's mind. Instantiate MiniTabNet on a small classification table of your choice (the sklearn breast-cancer dataset is fine), train it briefly, and for a handful of test rows print the attn mask from each of the three steps. Which features does it consistently select? Compare that shortlist against a GradientBoostingClassifier's feature_importances_ and write one sentence on whether the two methods agree about what matters.
We have spent this whole episode defending the honour of the humble tree, and rightly so. But notice the thread that kept resurfacing: deep learning's real advantage on tables shows up when data is plentiful -- and the cruel irony is that the tabular problems where we most want a good model are often the ones starved of rows. So here is the question that ought to itch: what if you could manufacture more rows? Not copy them, but generate believable, statistically faithful new examples to feed a data-hungry model -- and while you are at it, share a dataset without leaking a single real person's private record. That tension between "I need more data" and "I cannot expose the data I have" is exactly where we head next ;-)
@scipio