pip install torch scikit-learn); the tabular library sdv is optional and only needed if you want to skip writing CTGAN by hand;Learn AI Series):I closed episode #132 with a question I promised would itch, and I hope it did. We had just spent a whole episode defending the honour of the humble gradient-boosted tree on tabular data, and the sting in the tail was this: deep learning's real edge on tables only shows up when data is plentiful -- yet the tabular problems where we most desperately want a good model are exactly the ones starved of rows. So I asked: what if you could manufacture more rows? Not copy them -- generate believable, statistically faithful new examples. And while you are at it, share a dataset without leaking a single real person's private record.
That is today. Synthetic data -- data that no human ever produced, spun up by a model that learned what real data looks like. Here we go.
Let me anchor it with two stories, because the abstractions land better once you have a concrete itch. You are building a fraud-detection model. Your dataset has 10 million legitimate transactions and 847 fraudulent ones. You know the class-imbalance problem from episode #13, and you have already tried SMOTE, class weights, plain oversampling. But what if you could generate new fraudulent transactions that look realistic -- transactions preserving the statistical shape of real fraud without being carbon copies of the 847 cases you happen to have?
Now a different scene entirely. A hospital wants to share patient data with outside researchers, but privacy regulations flatly forbid it. Could you hand them a synthetic dataset that preserves the medical relationships -- age correlating with blood pressure, BMI predicting diabetes risk -- while containing precisely zero real patients? These are the two engines behind synthetic data: solving scarcity, and protecting privacy. A third motive -- augmentation for robustness -- rounds out the picture. All three are legitimate. All three have traps that will silently gut your model if you are not paying attention.
As always, homework first -- full code, no hand-waving.
Exercise 1 -- baseline the gap yourself. The task was to take the California Housing setup from #132, drop an XGBRegressor in alongside the MLP and the GradientBoostingRegressor, report all three RMSEs, and then re-run the MLP without the StandardScaler to watch what happens.
import numpy as np
import torch
import torch.nn as nn
from sklearn.datasets import fetch_california_housing
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import mean_squared_error
from sklearn.ensemble import GradientBoostingRegressor
from xgboost import XGBRegressor # pip install xgboost
data = fetch_california_housing()
X_tr_raw, X_te_raw, y_train, y_test = train_test_split(
data.data, data.target, test_size=0.2, random_state=42)
scaler = StandardScaler()
X_tr_scaled = scaler.fit_transform(X_tr_raw)
X_te_scaled = scaler.transform(X_te_raw)
def train_mlp(Xtr, Xte, epochs=100):
torch.manual_seed(0)
model = nn.Sequential(
nn.Linear(Xtr.shape[1], 128), nn.ReLU(), nn.Dropout(0.1),
nn.Linear(128, 128), nn.ReLU(), nn.Dropout(0.1),
nn.Linear(128, 1))
opt = torch.optim.Adam(model.parameters(), lr=1e-3)
xt = torch.tensor(Xtr, dtype=torch.float32)
yt = torch.tensor(y_train, dtype=torch.float32).unsqueeze(1)
for _ in range(epochs):
model.train()
loss = nn.MSELoss()(model(xt), yt)
opt.zero_grad(); loss.backward(); opt.step()
model.eval()
with torch.no_grad():
pred = model(torch.tensor(Xte, dtype=torch.float32)).squeeze(1).numpy()
return mean_squared_error(y_test, pred, squared=False)
gbm = GradientBoostingRegressor(n_estimators=200, max_depth=5, random_state=42)
gbm.fit(X_tr_raw, y_train)
rmse_gbm = mean_squared_error(y_test, gbm.predict(X_te_raw), squared=False)
xgb = XGBRegressor(n_estimators=300, max_depth=6, random_state=42)
xgb.fit(X_tr_raw, y_train)
rmse_xgb = mean_squared_error(y_test, xgb.predict(X_te_raw), squared=False)
print(f"MLP (scaled) RMSE: {train_mlp(X_tr_scaled, X_te_scaled):.4f}")
print(f"MLP (raw) RMSE: {train_mlp(X_tr_raw, X_te_raw):.4f}")
print(f"GBM RMSE: {rmse_gbm:.4f}")
print(f"XGBoost RMSE: {rmse_xgb:.4f}")
The two sentences I was fishing for: the trees (GBM and XGBoost) land a clearly lower RMSE than the scaled MLP, and XGBoost usually edges out plain GradientBoostingRegressor thanks to its regularisation -- the tree-versus-MLP gap is real and it is not close. And the MLP fed raw features barely trains at all (its RMSE balloons), because gradient descent chokes when one feature ranges 0-15 and another ranges 0-40,000; the trees, being invariant to monotone rescaling, did not even notice the difference. That asymmetry -- scaling is life-or-death for one model and a non-event for the other -- is the whole inductive-bias story from last episode, in one experiment.
Exercise 2 -- make the MLP fight back. Give the poor network better priors: batch-norm between layers, more width, 300 epochs, and a learning-rate scheduler. How close can you push it to the tree?
class BetterMLP(nn.Module):
def __init__(self, in_dim, hidden=256):
super().__init__()
self.net = nn.Sequential(
nn.Linear(in_dim, hidden), nn.BatchNorm1d(hidden), nn.ReLU(), nn.Dropout(0.1),
nn.Linear(hidden, hidden), nn.BatchNorm1d(hidden), nn.ReLU(), nn.Dropout(0.1),
nn.Linear(hidden, 1))
def forward(self, x):
return self.net(x).squeeze(-1)
torch.manual_seed(0)
model = BetterMLP(X_tr_scaled.shape[1])
opt = torch.optim.Adam(model.parameters(), lr=1e-3)
sched = torch.optim.lr_scheduler.CosineAnnealingLR(opt, T_max=300)
xt = torch.tensor(X_tr_scaled, dtype=torch.float32)
yt = torch.tensor(y_train, dtype=torch.float32)
for _ in range(300):
model.train()
loss = nn.MSELoss()(model(xt), yt)
opt.zero_grad(); loss.backward(); opt.step(); sched.step()
model.eval()
with torch.no_grad():
pred = model(torch.tensor(X_te_scaled, dtype=torch.float32)).numpy()
print(f"Better MLP RMSE: {mean_squared_error(y_test, pred, squared=False):.4f}")
You can shrink the gap substantially -- the better MLP gets meaningfully closer to the GBM, sometimes within spitting distance. The single change that helps most is almost always the batch normalisation: it keeps each layer's activations in a sane range through training, which is the very prior the raw features lacked. Width and the scheduler help at the margins, but if you ablate them one at a time you will find batch-norm is doing the heavy lifting. Even so -- notice how much effort that took to merely approach a tree you trained in five seconds with default parameters. That effort-to-payoff ratio is exactly why the practical advice stays "start with a tree".
Exercise 3 -- read TabNet's mind. Train MiniTabNet on a small classification table, print the attention mask from each of the three steps, and compare its shortlist against a GradientBoostingClassifier's feature_importances_.
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import GradientBoostingClassifier
# MiniTabNet / TabNetStep are the classes we defined in episode #132.
bc = load_breast_cancer()
Xtr, Xte, ytr, yte = train_test_split(bc.data, bc.target, test_size=0.2, random_state=0)
sc = StandardScaler()
Xtr = sc.fit_transform(Xtr); Xte = sc.transform(Xte)
torch.manual_seed(0)
net = MiniTabNet(in_dim=Xtr.shape[1], hidden=64, out_dim=1, n_steps=3)
opt = torch.optim.Adam(net.parameters(), lr=1e-3)
xt = torch.tensor(Xtr, dtype=torch.float32)
yt = torch.tensor(ytr, dtype=torch.float32)
for _ in range(200):
net.train()
loss = nn.BCEWithLogitsLoss()(net(xt), yt)
opt.zero_grad(); loss.backward(); opt.step()
# Re-run the steps by hand so we can capture each mask.
net.eval()
sample = torch.tensor(Xte[:16], dtype=torch.float32)
prior = torch.ones_like(sample)
with torch.no_grad():
for i, step in enumerate(net.steps):
_, attn, prior = step(sample, prior)
top = torch.topk(attn.mean(dim=0), 5).indices.tolist()
print(f"step {i+1} top features:", [bc.feature_names[j] for j in top])
gbc = GradientBoostingClassifier(random_state=0).fit(Xtr, ytr)
imp = np.argsort(gbc.feature_importances_)[::-1][:5]
print("GBM top features:", [bc.feature_names[j] for j in imp])
The one sentence: both methods keep gravitating to the same handful of "worst"-prefixed measurements (worst perimeter, worst area, worst concave points), so TabNet and the tree broadly agree about what matters -- which is reassuring, and also a quiet reminder that TabNet's headline selling point (interpretability) mostly re-derives what a tree's feature_importances_ hands you for free. Right -- exercises put to bed. On to making data out of thin air.
Before we reach for GANs and VAEs, let me be honest that some flavours of synthetic data are ancient, boring, and extremely well understood. You have already met most of them.
Random oversampling just duplicates minority-class examples. It generates nothing new -- it only reshapes the class distribution. Simple, occasionally effective, and it carries an obvious risk: the model happily memorises the copies.
SMOTE (Synthetic Minority Over-sampling Technique) interpolates between minority examples in feature space. Pick a sample, find its k nearest minority neighbours, and drop a fresh point somewhere on the line between them. That genuinely invents new data -- but only through linear interpolation, so it cannot capture any nonlinear manifold structure. If your minority class curls around in feature space, SMOTE will cheerfully place points in the empty region the curl encloses, inventing examples that could never occur.
Data augmentation for images -- random flips, rotations, colour jitter, cutout -- manufactures new training images from the ones you have. We leaned on this back in episode #43, and it remains one of the most reliably useful tricks in the whole field.
What makes the newer generative methods different -- and worth a whole episode -- is that they learn the distribution of the data and then sample from it. They do not interpolate between existing points; they model (approximately) the underlying data-generating process and conjure entirely new samples from it. That is a genuinely bigger hammer, and, as with all bigger hammers, it is easier to hit your own thumb.
CTGAN (Conditional Tabular GAN, 2019) adapts the GAN framework from episode #55 to the peculiar awfulness of tabular data. The core headache: tables mix column types. Some columns are continuous (normal, or heavy-tailed, or stubbornly multimodal), others are categorical (discrete, sometimes with hundreds of possible values). A vanilla image GAN, which assumes a nice homogeneous grid of pixels, simply cannot cope.
CTGAN answers with three ideas, and each one is worth understanding rather than memorising:
Mode-specific normalisation. For every continuous column, CTGAN fits a Gaussian Mixture Model to find its modes. In stead of squashing values into [0,1] and pretending the column is unimodal, it represents each value as a one-hot indicator of which Gaussian mode it belongs to, plus a normalised offset within that mode. That lets the generator reproduce genuinely multimodal columns (think "income", which clumps around several salary bands) without smearing them into one hump.
Conditional generation. During training the generator is conditioned on a randomly chosen column-and-category. This forces it to learn how to produce every category, including the rare ones -- which is precisely the thing you care about when your whole reason for being here is a starved minority class.
Training-by-sampling. To keep mode collapse at bay, CTGAN samples the training data so that rare categories still show up often enough to be learned, rather than being statistically drowned by the common ones.
import torch
import torch.nn as nn
class CTGANGenerator(nn.Module):
"""Simplified CTGAN generator structure."""
def __init__(self, noise_dim, cond_dim, output_dim, hidden=256):
super().__init__()
self.net = nn.Sequential(
nn.Linear(noise_dim + cond_dim, hidden), nn.BatchNorm1d(hidden), nn.ReLU(),
nn.Linear(hidden, hidden), nn.BatchNorm1d(hidden), nn.ReLU(),
nn.Linear(hidden, output_dim),
)
def forward(self, noise, cond):
return self.net(torch.cat([noise, cond], dim=1))
class CTGANDiscriminator(nn.Module):
"""Simplified CTGAN discriminator with PacGAN packing."""
def __init__(self, input_dim, cond_dim, hidden=256, pac=10):
super().__init__()
self.pac = pac
self.net = nn.Sequential(
nn.Linear((input_dim + cond_dim) * pac, hidden), nn.LeakyReLU(0.2), nn.Dropout(0.5),
nn.Linear(hidden, hidden), nn.LeakyReLU(0.2), nn.Dropout(0.5),
nn.Linear(hidden, 1),
)
def forward(self, data, cond):
x = torch.cat([data, cond], dim=1)
# PacGAN: pack several samples together to expose mode collapse.
x = x.reshape(-1, x.shape[1] * self.pac)
return self.net(x)
The PacGAN trick in the discriminator deserves a second look. In stead of judging one sample at a time, it evaluates packs of samples together. Why? Because a generator that has collapsed to a handful of modes will produce packs that look suspiciously self-similar, and a discriminator that sees the whole pack can smell that instantly. Judging samples individually hides the collapse; judging them in bunches reveals it. It is a lovely little piece of engineering.
In real life you would not hand-roll all this. You would reach for the sdv (Synthetic Data Vault) library, which wraps CTGAN behind a friendly interface:
# Using the SDV library (pip install sdv)
from sdv.single_table import CTGANSynthesizer
from sdv.metadata import SingleTableMetadata
# metadata = SingleTableMetadata()
# metadata.detect_from_dataframe(real_data)
# synthesizer = CTGANSynthesizer(metadata, epochs=300)
# synthesizer.fit(real_data)
# synthetic_data = synthesizer.sample(num_rows=10000)
TVAE takes the Variational Autoencoder road in stead of the adversarial one. Recall the autoencoder idea we brushed against around episode #26: encode data down into a compact latent space, then decode it back out. The "variational" part forces that latent space to be smooth and continuous, standing in for a plain Gaussian -- which means you can sample a random point from it and decode that point into a brand-new, plausible data row.
TVAE borrows CTGAN's mode-specific normalisation to handle the mixed column types, then trains with the classic VAE loss: a reconstruction term (how faithfully the decoder rebuilds the input) plus a KL-divergence term (how close the latent distribution stays to a standard Gaussian, so sampling behaves).
class TVAEEncoder(nn.Module):
def __init__(self, input_dim, latent_dim=128, hidden=128):
super().__init__()
self.shared = nn.Sequential(
nn.Linear(input_dim, hidden), nn.ReLU(),
nn.Linear(hidden, hidden), nn.ReLU(),
)
self.mu = nn.Linear(hidden, latent_dim)
self.logvar = nn.Linear(hidden, latent_dim)
def forward(self, x):
h = self.shared(x)
return self.mu(h), self.logvar(h)
class TVAEDecoder(nn.Module):
def __init__(self, latent_dim, output_dim, hidden=128):
super().__init__()
self.net = nn.Sequential(
nn.Linear(latent_dim, hidden), nn.ReLU(),
nn.Linear(hidden, hidden), nn.ReLU(),
nn.Linear(hidden, output_dim),
)
def forward(self, z):
return self.net(z)
def vae_loss(recon, original, mu, logvar):
recon_loss = nn.MSELoss()(recon, original)
kl_loss = -0.5 * torch.mean(1 + logvar - mu.pow(2) - logvar.exp())
return recon_loss + kl_loss
TVAE is usually faster to train than CTGAN -- there is no adversarial tug-of-war to babysit -- and considerably more stable. On many published benchmarks it matches CTGAN on synthetic-data quality, which surprises people who assume GANs always win. The trade-off is the familiar GAN-versus-VAE one: GANs tend to produce sharper, crisper samples, while VAEs lean toward slightly smoother, occasionally blurrier distributions. For tables that smoothness is often harmless, sometimes even a mild bonus, because it damps the overconfident spikes a GAN can hallucinate.
Swing over to vision and synthetic data gets genuinely spectacular. The core application is not art -- it is manufacturing labelled training images where real labelled data is scarce, expensive, or dangerous to collect.
Domain randomisation. Render 3D objects against wildly random backgrounds, lighting, textures, and camera angles. It looks like nonsense to a human. Yet models trained on this deliberately unrealistic soup often transfer beautifully to real photos, because the sheer diversity forces the network to learn the object rather than memorising a particular background. The randomness is the point.
Diffusion-based augmentation. Point a text-to-image model (episodes #84-#85) at your gaps. Want more images of "a stop sign half-buried in snow"? Generate a thousand of them. This is a godsend for rare conditions that are genuinely hard to photograph on demand -- you do not have to wait for the next blizzard.
Copy-paste augmentation. Segment objects out of some images and paste them onto different backgrounds. No generative model required, embarrassingly simple, and stubbornly effective for detection and segmentation tasks.
Here is the critical difference from the creative image generation everyone gushes about: for training data, diversity and label accuracy beat individual beauty every single time. A slightly weird-looking synthetic image carrying a correct bounding box is useful. A gorgeous, photorealistic synthetic image carrying a wrong label is worse than useless -- it actively teaches your model the wrong thing. Pretty is not the objective. Correct-and-varied is.
Right, here is where most people faceplant. They generate a pile of synthetic data, stir it into training, and simply hope. Proper evaluation is not optional, and it rests on three measurements that you must not skip.
Utility. Does the synthetic data actually help a downstream model? Train on real data alone, then on real-plus-synthetic, and compare on a held-out set of real data. If accuracy does not improve -- or worse, drops -- the synthetic data is not earning its place. This is the ultimate arbiter, and it is the same "measure, do not assume" discipline from episode #13 that keeps coming back to haunt us.
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import f1_score
import numpy as np
def evaluate_utility(real_train, real_test, synthetic, target_col):
"""Train on real vs. real+synthetic, evaluate on the real test set."""
X_real = real_train.drop(target_col, axis=1).values
y_real = real_train[target_col].values
X_test = real_test.drop(target_col, axis=1).values
y_test = real_test[target_col].values
X_synth = synthetic.drop(target_col, axis=1).values
y_synth = synthetic[target_col].values
# Real only
clf_real = RandomForestClassifier(n_estimators=100, random_state=42)
clf_real.fit(X_real, y_real)
score_real = f1_score(y_test, clf_real.predict(X_test), average='macro')
# Real + synthetic
X_combined = np.vstack([X_real, X_synth])
y_combined = np.concatenate([y_real, y_synth])
clf_combined = RandomForestClassifier(n_estimators=100, random_state=42)
clf_combined.fit(X_combined, y_combined)
score_combined = f1_score(y_test, clf_combined.predict(X_test), average='macro')
print(f"Real only F1: {score_real:.4f}")
print(f"Real+Synthetic F1: {score_combined:.4f}")
return score_real, score_combined
Fidelity. Do the statistics of the synthetic data match the real data? Check per-column distributions, the correlation matrix, and pairwise relationships. If age and income are plainly correlated in reality but arrive uncorrelated in your synthetic set, fidelity is low and any model trained on it will learn a fantasy. Fidelity is necessary but -- careful now -- not sufficient: a perfect copy of the real data has flawless fidelity and catastrophic privacy, which brings us to pillar three.
Privacy. Does the synthetic data quietly leak real individuals? The cheapest useful check is nearest-neighbour distance. For each synthetic record, find its closest real record. If synthetic points are hugging real points far tighter than real points hug each other, your generator is memorising training rows in stead of learning the distribution -- and that is a privacy breach wearing a synthetic-data costume.
from sklearn.neighbors import NearestNeighbors
def privacy_check(real_data, synthetic_data, threshold_percentile=5):
"""Flag synthetic samples that sit suspiciously close to real ones."""
nn = NearestNeighbors(n_neighbors=1)
nn.fit(real_data)
distances, _ = nn.kneighbors(synthetic_data)
# Baseline: how close are real points to each OTHER?
nn_real = NearestNeighbors(n_neighbors=2) # 2, because the nearest is itself
nn_real.fit(real_data)
real_distances, _ = nn_real.kneighbors(real_data)
real_distances = real_distances[:, 1] # skip the self-match
threshold = np.percentile(real_distances, threshold_percentile)
too_close = (distances.flatten() < threshold).mean()
print(f"Synthetic samples closer than {threshold_percentile}th pct: {too_close:.1%}")
return too_close
Nota bene: these three pillars pull against each other. Crank fidelity to the maximum and you drift toward memorising real records, wrecking privacy. Crank privacy to the maximum (add loads of noise) and you blur the very correlations that gave the data its utility. Good synthetic data lives in the uncomfortable middle, and the only way to know where you are standing is to measure all three -- never just the one that flatters you.
Class imbalance. Generating minority-class samples is the most common and often most effective use. But do not reflexively oversample to a tidy 50/50 -- sometimes 70/30 or 80/20 trains a better model, because you are not trying to fake a balanced world, you are trying to give the learner enough signal. Experiment, and let the held-out real set decide.
Privacy-preserving sharing. Hospitals, banks, and governments can hand out synthetic data for research when the real data legally cannot leave the building. Even imperfect synthetic data lets outsiders develop and debug methods, which is a genuine unlock -- the researchers finalise on the real data later, inside the walls.
Augmentation for robustness. Adding synthetic variation -- especially in computer vision -- pushes models to generalise rather than memorise. This one is thoroughly battle-tested and lives in production systems all over the place.
Cold start. Launching a recommender with zero interaction data? Synthetic usage patterns can bootstrap the system just enough to be useful while real behaviour accumulates and gradually takes over.
Rare events. If your dataset holds 50 fraud cases and you generate 5,000 synthetic ones, the synthetic fraud will faithfully echo the patterns in those 50 -- not the full, wild diversity of real-world fraud. You are amplifying a narrow keyhole view, not widening the window. More rows, same blind spots.
Causal relationships. Synthetic data preserves correlations, not causation -- and if you sat through episode #130 you know how big a landmine that is. A model trained on synthetic medical data can learn that drug X correlates with recovery while completely missing that a confounder was driving both. The generator copied the shadow, not the thing casting it.
Distributional shift. If the real world drifts over time (concept drift, episode #123), synthetic data minted from last year's distribution actively harms today's model. Generating more data from a stale distribution does not fix staleness -- it multiplies it.
False confidence. More data feels like better data. But synthetic data is a model of reality, not reality. Train on generated rows that obey your generator's assumptions and you will earn a warm, glowing confidence that is grounded in exactly nothing outside your own model's head.
The golden rule, and I will bold it because it is the single most important sentence in this episode: always evaluate synthetic data on a held-out set of real data. The real test set is sacred. Never let a synthetic row anywhere near it. It is the one honest voice in the room -- it tells you whether your synthetic-data strategy genuinely helped, or whether you have spent a week lovingly polishing a mirror.
Three to chew on before next time -- and, as always, I will walk through full solutions at the top of the following episode.
Watch SMOTE hallucinate. Build a tiny 2D minority class shaped like a crescent (a half-moon -- sklearn.datasets.make_moons will do nicely), then run SMOTE (imblearn.over_sampling.SMOTE) to oversample it. Scatter-plot the original points and the synthetic points in different colours. Write two sentences describing where SMOTE placed points that could never occur in the real crescent, and connect that back to the "linear interpolation cannot follow a curved manifold" claim from this episode.
Run the three pillars for real. Take any small tabular dataset with a target column (the sklearn breast-cancer set, framed as a DataFrame, is fine). Split it, fit a CTGANSynthesizer from sdv (or, if you would rather not install it, a quick TVAE-style autoencoder), and sample a synthetic set the same size as your training split. Now run all three checks from this episode -- evaluate_utility, a correlation-matrix comparison for fidelity, and privacy_check. Report one number per pillar and one sentence on which pillar looked weakest and why.
Break it on purpose. Construct a deliberately rare-event scenario: keep only ~30 examples of the positive class, generate 3,000 synthetic positives from just those 30, then train a classifier on real-plus-synthetic and evaluate on a held-out real test set that contains positives your 30 never represented (e.g. a different sub-cluster). Show that test performance on those unseen positives does not improve -- and ideally gets worse. One sentence on why "more synthetic rows" failed to buy you "more diversity".
We have spent this whole episode learning to manufacture data -- spinning up rows and images a model can feed on. But step back and notice the thing we have been quietly assuming the entire time: that we have the compute to train these generators, the storage to hold both the real and the synthetic sets, and the budget to run CTGAN for 300 epochs without a second thought. Every technique in this series so far has had a price tag we mostly ignored. Generating a million synthetic rows is cheap; serving a model to a million users is emphatically not. So the uncomfortable question I want you carrying into next time is the one nobody in the tutorials likes to ask out loud: what does all of this actually cost -- in GPUs, in dollars, in watts -- and how do you reason about that before the invoice arrives? That is where we head next ;-)
@scipio