Learn AI Series (#141) - Robotics and Embodied AI

scipio(71)

Published in

StemSocial

Words

4474

Reading

20 min

Listen

Play

Learn AI Series (#141) - Robotics and Embodied AI

What will I learn

The sim-to-real gap: why a policy that is flawless in simulation faceplants in reality, and how domain randomization quietly turns that into a problem we already know how to solve;
the three robot-learning paradigms -- imitation, reinforcement learning, and the hybrid that real systems actually ship -- and where each one breaks;
language-conditioned robotics: how a plain-English instruction becomes motor torque through a vision-language-action model, reusing the exact perception backbones from earlier in this series;
foundation models for robots, the brutal data bottleneck they run into, and the tricks (cross-embodiment, massive sim, teleop) people use to climb out of it;
the hardware reality -- sensors, actuators, on-board compute, safety -- that software people love to wave away and physics refuses to let them;
a small, runnable robot-learning pipeline in PyTorch that you can actually poke at.

Requirements

A working modern computer running macOS, Windows or Ubuntu;
Python 3.10+ with PyTorch installed (pip install torch) -- every snippet here is illustrative, nothing needs a GPU, a physics engine, or (thankfully) an actual robot to break;
You've been through the reinforcement learning arc (#102-116) and foundation models (#137). We lean on both, plus callbacks to attention (#51), CLIP-style multimodal perception (#75, #138), and edge AI (#122).

Difficulty

Beginner

Curriculum (of the `Learn AI Series`):

Learn AI Series (#141) - Robotics and Embodied AI

I ended last episode with a promise dressed up as a threat. We had spent #140 watching AI fold proteins, invent crystals and out-forecast supercomputers -- but always as a mind in a box, reading numbers and emitting numbers. And I asked what happens when you rip it OUT of the box and bolt it into something with motors, sensors, and a body that has to survive contact with the physical world.

That is today. Everything we've built across 140 episodes has lived in the digital world. Models process images, text and audio, but they never touch anything. They never bump a table, drop a glass, or lose their footing on a wet floor. Robotics is where AI meets physics, and physics -- unlike a validation set -- does not forgive sloppy abstractions. Let's dive right in ;-)

Solutions to episode #140's exercises

As always, we clear last time's homework before touching anything new. Episode #140 was Scientific AI, and the three tasks were about making our toy models a little more honest.

Exercise 1 -- Make the Evoformer admit how sure it is. The task: extend SimplifiedEvoformer so forward also returns a per-residue confidence vector, via a small linear head on the MSA output, averaged over the relatives and squashed with sigmoid. Here is the whole thing, confidence head and all.

import torch
import torch.nn as nn

class ConfidentEvoformer(nn.Module):
    """SimplifiedEvoformer (#140) + a per-residue confidence head."""
    def __init__(self, seq_dim=256, pair_dim=128, n_heads=8):
        super().__init__()
        self.row_attn = nn.MultiheadAttention(seq_dim, n_heads, batch_first=True)
        self.pair_proj = nn.Sequential(
            nn.Linear(pair_dim, pair_dim), nn.ReLU(),
            nn.Linear(pair_dim, pair_dim),
        )
        self.norm1 = nn.LayerNorm(seq_dim)
        self.norm2 = nn.LayerNorm(pair_dim)
        self.confidence = nn.Linear(seq_dim, 1)   # the new bit

    def forward(self, msa_repr, pair_repr):
        batch, n_seq, seq_len, dim = msa_repr.shape
        flat = msa_repr.reshape(batch * n_seq, seq_len, dim)
        attn_out, _ = self.row_attn(flat, flat, flat)
        msa_repr = self.norm1(msa_repr + attn_out.reshape(batch, n_seq, seq_len, dim))
        pair_repr = self.norm2(pair_repr + self.pair_proj(pair_repr))
        # average the confidence logits over the relatives, then squash to [0, 1]
        conf_logits = self.confidence(msa_repr).squeeze(-1)   # (batch, n_seq, seq_len)
        confidence = torch.sigmoid(conf_logits.mean(dim=1))   # (batch, seq_len)
        return msa_repr, pair_repr, confidence

model = ConfidentEvoformer()
msa = torch.randn(1, 4, 50, 256)
pair = torch.randn(1, 50, 50, 128)
_, _, conf = model(msa, pair)
print(f"confidence shape: {tuple(conf.shape)} (one number per residue)")
assert conf.shape == (1, 50)

The shape comes out (1, 50) -- exactly one confidence score per residue, which is the point. And WHY does a model that reports its own uncertainty beat one that only reports an answer? Because a scientist does not act on a prediction, they act on a prediction plus how much to trust it. AlphaFold's real confidence score (pLDDT) is not a nice-to-have -- it is the thing that tells a biologist which loops to believe and which to go verify in the lab. An answer with no error bar is a rumour.

Exercise 2 -- One GNN, two sciences. Feed MoleculeGNN a small-molecule graph and a crystal-cell graph of a DIFFERENT atom count, without touching the model, and confirm both spit out a single scalar. Then say what makes that possible.

class MoleculeGNN(nn.Module):
    """From #140: predict one bulk property from atoms + connectivity."""
    def __init__(self, atom_features=32, hidden=64, output=1):
        super().__init__()
        self.atom_embed = nn.Linear(atom_features, hidden)
        self.conv1 = nn.Linear(hidden, hidden)
        self.conv2 = nn.Linear(hidden, hidden)
        self.readout = nn.Linear(hidden, output)

    def forward(self, atom_feats, adj):
        h = torch.relu(self.atom_embed(atom_feats))
        h = torch.relu(self.conv1(adj @ h))
        h = torch.relu(self.conv2(adj @ h))
        graph_repr = h.mean(dim=0, keepdim=True)   # size-agnostic readout
        return self.readout(graph_repr)

gnn = MoleculeGNN(atom_features=32)

# (a) a 9-atom molecule
mol_feats = torch.randn(9, 32)
mol_adj = torch.eye(9)
# (b) a 12-atom crystal cell -- DIFFERENT node count, SAME model
crys_feats = torch.randn(12, 32)
crys_adj = torch.eye(12)

print("molecule ->", gnn(mol_feats, mol_adj).shape)   # (1, 1)
print("crystal  ->", gnn(crys_feats, crys_adj).shape) # (1, 1)

Both return shape (1, 1) -- a single scalar -- despite one graph having 9 nodes and the other 12. The trick lives entirely in h.mean(dim=0): the readout AGGREGATES over however many nodes there are, so a 9-row tensor and a 12-row tensor both collapse to one fixed-size vector before the final linear layer. That is precisely why "atoms as nodes" generalises across chemistry and materials -- the model never hard-codes how many atoms it will see, it just averages messages over whatever graph you hand it. Change the molecule, change the crystal, the architecture does not blink.

Exercise 3 -- Draw the tool-versus-scientist line yourself. I asked you to pick one system and argue exactly where it sits. My answer, for AlphaFold: it is a tool, and a magnificent one, but it is NOT a scientist. It depends on a human to choose which protein matters, to frame why the structure is worth predicting, and -- crucially -- to interpret what the fold implies for disease or drug design. AlphaFold predicts that a chain folds a certain way with startling accuracy, but it has handed us no new physics of why proteins fold as they do. For it to cross into autonomous discovery it would need to do more than predict -- it would need to form a hypothesis about folding mechanism, design an experiment to test it, and revise its own theory from the result. "It would need to understand" is not the argument; the argument is that understanding would let it generate the next question, and prediction alone never does. Right -- homework settled. Now we add gravity ;-)

Why robotics is hard in a way software simply isn't

When your chatbot hallucinates, a user gets a wrong sentence. When your robot hallucinates, it drives off a staircase. The physical world piles on constraints that pure software never has to face, and it is worth naming them precisely before we reach for solutions.

Continuous state and action spaces. A robot arm has 6 or 7 joints, each with a continuous angle, velocity and torque. The state space is not a grid you can enumerate -- it is a manifold. Unlike Atari (#107), where you pick one of a handful of buttons, here "the set of actions" is uncountable.

Real-time or bust. A walking robot needs control signals at 100 to 1000 Hz. Your policy can NOT take 200ms to think -- by then the robot has already met the floor. This alone rules out a lot of the heavy inference we've been casually assuming for 140 episodes.

Partial observability. Sensors are noisy, occluded and limited. A camera cannot see behind the robot. A force sensor gives you local pressure, not the full state of the object you are grasping. The policy is always reasoning from a keyhole view.

Safety is not a config flag. You cannot explore recklessly the way you would in simulation. An arm swinging at full torque in a factory can injure or kill. "Just let it try random things and learn" is a sentence that ends careers here.

Non-stationarity. The real world drifts. Lighting shifts through the day, surfaces change friction, objects deform, a gripper wears. The policy has to GENERALISE, not memorise a frozen world.

Having said that, none of this is a reason to give up on learning-based robotics -- it is a reason to be honest about what "learning" has to survive.

The sim-to-real gap

Training robots in the real world is slow, expensive and occasionally on fire. Training in simulation is fast and free. The catch, and it is the central catch of the whole field: policies trained in simulation tend to FAIL in reality. This is the sim-to-real gap.

A simulated robot enjoys perfect joint angles, frictionless-or-exactly-specified contacts, instant sensor readings and deterministic physics. A real robot has backlash in its gears, sticky joints, camera latency, and a gripper that bends a hair under load. The policy that scored perfectly in sim has never seen ANY of this, so the first time reality whispers something off-distribution, it face-plants.

Domain randomization is the beautifully brutish fix: instead of straining to make the simulation match reality exactly, you make it match everything. During training you randomize the simulation parameters -- friction, masses, lighting, camera pose, sensor noise, actuator delay -- across a wide distribution. The policy has to work across all of them at once, so it is forced to learn behaviours that are robust rather than behaviours that exploit one specific fake world.

import numpy as np

class DomainRandomizer:
    """Randomize simulation parameters for sim-to-real transfer."""
    def __init__(self):
        self.ranges = {
            'friction': (0.3, 1.5),        # coefficient of friction
            'mass_scale': (0.7, 1.3),      # object mass multiplier
            'actuator_noise': (0.0, 0.05), # noise added to actions
            'obs_noise': (0.0, 0.02),      # noise added to observations
            'latency_steps': (0, 3),       # sensor delay in timesteps
            'gravity_z': (-10.5, -9.0),    # gravity variation
        }

    def sample(self):
        """Sample one random environment configuration."""
        params = {}
        for key, (low, high) in self.ranges.items():
            if isinstance(low, int) and isinstance(high, int):
                params[key] = np.random.randint(low, high + 1)
            else:
                params[key] = np.random.uniform(low, high)
        return params

    def apply_obs_noise(self, obs, params):
        noise = torch.randn_like(obs) * params['obs_noise']
        return obs + noise

    def apply_action_noise(self, action, params):
        noise = torch.randn_like(action) * params['actuator_noise']
        return action + noise

randomizer = DomainRandomizer()
for _ in range(3):
    cfg = randomizer.sample()
    print(f"friction={cfg['friction']:.2f}  mass={cfg['mass_scale']:.2f}  "
          f"latency={cfg['latency_steps']} steps")

The insight worth tattooing on your arm: domain randomization turns sim-to-real into a GENERALIZATION problem, which is a problem we have spent this entire series learning to attack. Instead of asking "does this policy work in reality?" you ask "does this policy work in any environment drawn from a wide distribution?" -- and if the answer is yes, then reality is just one more environment it happens not to have seen. Nota bene: reality still has to fall INSIDE the randomization ranges. Randomize friction over too narrow a band and the real floor sits outside it, and you are right back where you started.

Robot learning paradigms

There are three main ways a robot learns to do a thing, and real systems mix them.

Imitation learning (learning from demonstrations). A human shows the robot what to do -- teleoperating an arm, say -- and the robot learns to replicate it. Treated as plain supervised learning (map observation to action), this is behavioral cloning, and it is as simple as it sounds.

class BehaviorCloningPolicy(nn.Module):
    """Learn to act by imitating expert demonstrations."""
    def __init__(self, obs_dim, action_dim, hidden=256):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(obs_dim, hidden), nn.ReLU(),
            nn.Linear(hidden, hidden), nn.ReLU(),
            nn.Linear(hidden, action_dim),
            nn.Tanh(),   # actions bounded to [-1, 1]
        )

    def forward(self, obs):
        return self.net(obs)

def train_behavior_cloning(policy, expert_obs, expert_actions, epochs=100):
    optimizer = torch.optim.Adam(policy.parameters(), lr=3e-4)
    dataset = torch.utils.data.TensorDataset(expert_obs, expert_actions)
    loader = torch.utils.data.DataLoader(dataset, batch_size=256, shuffle=True)
    for epoch in range(epochs):
        total = 0.0
        for obs_b, act_b in loader:
            loss = nn.functional.mse_loss(policy(obs_b), act_b)
            optimizer.zero_grad(); loss.backward(); optimizer.step()
            total += loss.item()
        if (epoch + 1) % 20 == 0:
            print(f"epoch {epoch+1}, loss {total/len(loader):.4f}")
    return policy

Behavioral cloning is simple but brittle, and the reason it breaks is worth understanding because it is subtle. It suffers from distribution shift -- sometimes called covariate shift or the compounding-error problem. The expert never made mistakes, so your training data contains ZERO examples of recovery. The moment the policy drifts a little off the expert's trajectory, it lands in a state it never trained on, makes a slightly worse prediction, drifts further, and the errors compound into a cascade. It is death by a thousand tiny deviations.

DAgger (Dataset Aggregation) patches exactly this: run the policy, collect the states it actually visits, ask the expert what they would have done in each of those states, add that to the dataset, retrain, repeat. You are teaching the model how to recover from its own drift. The price is an expert in the loop during training, which is not free.

Reinforcement learning (learning from rewards). The robot explores and learns from experience, the whole arc of episodes #102-116. For robotics, PPO (#109) and SAC (Soft Actor-Critic) dominate, because they cope well with continuous action spaces -- exactly the manifold-not-a-grid problem I flagged earlier.

Hybrid approaches are how serious robots actually get built: pretrain with imitation to get a sane starting policy (so RL is not exploring from pure noise), then fine-tune with RL to push past what the human demonstrator could do. Imitation bootstraps, RL refines. Here is the shape of that fine-tune, deliberately tiny so you can read it in one sitting:

def finetune_with_rl(policy, env_step, rollout_len=64, lr=1e-4):
    """Imitation bootstraps; RL refines. A minimal policy-gradient nudge
    on top of a behavior-cloned policy (see PPO/#109 for the real thing)."""
    optimizer = torch.optim.Adam(policy.parameters(), lr=lr)
    obs = env_step(None)             # reset -> initial observation
    log_probs, rewards = [], []
    for _ in range(rollout_len):
        mean = policy(obs)
        dist = torch.distributions.Normal(mean, 0.1)   # explore a little
        action = dist.sample()
        log_probs.append(dist.log_prob(action).sum())
        obs, reward = env_step(action.clamp(-1, 1))
        rewards.append(reward)
    returns = torch.tensor(rewards).flip(0).cumsum(0).flip(0)  # reward-to-go
    returns = (returns - returns.mean()) / (returns.std() + 1e-8)
    loss = -(torch.stack(log_probs) * returns).mean()  # REINFORCE objective
    optimizer.zero_grad(); loss.backward(); optimizer.step()
    return loss.item()

Notice the action.clamp(-1, 1) -- that is not cosmetic. In a body, an unclamped action is a joint asked to move past its physical stop. Even our toy loop respects the envelope, because the real one has to.

Dealing with the keyhole: observation history

Remember partial observability? A single camera frame does not tell you whether the ball is moving toward you or away. The standard trick is to feed the policy a short HISTORY of observations rather than a single snapshot, so velocity and intent become inferable. It is the same instinct behind sequence models (#48-49) -- give the network enough context to reconstruct what one frame hides.

class HistoryPolicy(nn.Module):
    """Stack the last k observations so the policy can infer motion."""
    def __init__(self, obs_dim, action_dim, k=4, hidden=256):
        super().__init__()
        self.k = k
        self.net = nn.Sequential(
            nn.Linear(obs_dim * k, hidden), nn.ReLU(),
            nn.Linear(hidden, action_dim), nn.Tanh(),
        )

    def forward(self, obs_window):
        # obs_window: (batch, k, obs_dim) -> flatten the time axis
        flat = obs_window.reshape(obs_window.shape[0], -1)
        return self.net(flat)

policy = HistoryPolicy(obs_dim=12, action_dim=7, k=4)
window = torch.randn(2, 4, 12)   # 2 robots, last 4 frames, 12-dim obs
print("actions ->", policy(window).shape)   # (2, 7)

Language-conditioned robotics

Now the part that genuinely excites me. The most striking recent development is robots that follow NATURAL LANGUAGE. Instead of programming a specific behaviour, you tell the robot what you want in plain English -- "pick up the red cup" -- and it figures out the motor commands.

The architecture is usually a vision-language-action (VLA) model: a vision encoder chews the camera input, a language encoder chews the instruction, and a policy head fuses the two and emits motor commands. The clever move is to reuse a pretrained vision-language model -- CLIP-style perception, the thing we met in #75 and #138 -- as the perception backbone, and train only the action head on the (scarce, precious) robot data.

class VisionLanguageActionPolicy(nn.Module):
    """Simplified VLA: vision + language -> robot actions."""
    def __init__(self, vision_dim=512, language_dim=512, action_dim=7):
        super().__init__()
        # In practice these would be PRETRAINED encoders (CLIP, etc.), frozen.
        self.vision_encoder = nn.Sequential(nn.Linear(vision_dim, 256), nn.ReLU())
        self.language_encoder = nn.Sequential(nn.Linear(language_dim, 256), nn.ReLU())
        self.policy_head = nn.Sequential(
            nn.Linear(512, 256), nn.ReLU(),
            nn.Linear(256, 128), nn.ReLU(),
            nn.Linear(128, action_dim), nn.Tanh(),
        )

    def forward(self, vision_features, language_features):
        v = self.vision_encoder(vision_features)
        l = self.language_encoder(language_features)
        fused = torch.cat([v, l], dim=-1)
        return self.policy_head(fused)

vla = VisionLanguageActionPolicy()
vision = torch.randn(1, 512)     # a frame, already encoded
language = torch.randn(1, 512)   # "pick up the red cup", already encoded
print("motor command ->", vla(vision, language).shape)   # (1, 7)

Google's RT-2 and its cousins showed that a large vision-language model can GROUND language in physical action -- the model does not merely know what a "red cup" looks like, it knows how to reach for one. And the scale of the pretrained backbone matters enormously: bigger perception models generalise to instructions the robot never saw in training, which is exactly the foundation-model promise (#137) crossing over into meat-space.

Foundation models for robots (and the data wall)

The foundation-model pattern is reaching robotics: train one big model on diverse robot data -- many robots, many tasks, many environments -- then fine-tune or prompt it per task. Lovely idea. It slams straight into a wall.

The wall is DATA. Internet-scale text and images are lying around by the exabyte. Internet-scale robot-interaction data is NOT -- you cannot crawl the web for a million examples of a robot folding laundry, because nobody uploaded them, because they do not exist. Current approaches attack the shortage from several angles at once: simulation at massive scale, teleoperation datasets recorded by humans puppeting real arms, cross-embodiment transfer (train on data pooled from different robot bodies), and even using video-prediction models trained on YouTube as implicit world models.

The Open X-Embodiment project is the poster child: it pools demonstrations from 22 different robot platforms, and the resulting policy generalises across robot morphologies it was not trained on. That is a genuinely hopeful signal -- it hints that a robot foundation model does not need one standardised body, it can learn from the whole zoo. Whether that scales the way language did is, honestly, still an open question in 2026.

The hardware reality software people wave away

Software folk chronically underestimate how hard hardware constrains robot AI. A few realities that the cloud let us forget:

Sensors define perception. RGB cameras, depth cameras (Intel RealSense, Azure Kinect), LiDAR, IMUs, force/torque sensors, joint encoders -- each hands you a different slice of the world. Your sensor suite is the hard ceiling on what any policy can possibly perceive. No amount of clever network fixes a sensor that cannot see the thing.

Actuators define action. Electric motors (fast, precise, force-limited), hydraulics (powerful, slow, messy), pneumatic muscles (compliant, a pain to control). The actuator type sets your control bandwidth and force envelope before a single line of policy code is written.

Compute is on-board. For a mobile robot you cannot ship inference to a cloud GPU -- the round-trip latency would tip it over. Jetson modules, neural accelerators, custom silicon. Edge AI (#122) is not an optimisation in robotics, it is the entry fee. Everything we said about quantization and pruning back then becomes load-bearing here.

Safety is mechanical, not a checkbox. Collaborative robots (cobots) need force-limiting control, hardware e-stops, and workspace monitoring -- sensors, control theory and mechanical design working together. You do not try/except your way out of a two-hundred-kilo arm moving where a human is standing. Even in software, though, a learned policy's raw output should never reach a motor unfiltered -- you clamp it to the joint limits AND cap how fast it is allowed to change, so a spike in the network output cannot become a violent lurch:

def safe_command(prev_action, raw_action, joint_limits, max_delta=0.1):
    """Filter a policy's raw output before it ever touches a motor.
    Clamp to physical limits, then limit the per-step CHANGE (rate limit)."""
    low, high = joint_limits            # tensors, one entry per joint
    clamped = torch.clamp(raw_action, low, high)          # respect the stops
    delta = torch.clamp(clamped - prev_action, -max_delta, max_delta)
    return prev_action + delta                            # bounded, smooth motion

prev = torch.zeros(7)
raw = torch.tensor([5.0, -9.0, 0.3, 0.0, 2.0, -0.4, 8.0])  # a wild network spike
limits = (torch.full((7,), -1.0), torch.full((7,), 1.0))
print("safe command ->", safe_command(prev, raw, limits).tolist())

This tiny guard is the software echo of the mechanical force-limiter: the network is allowed to be wrong, but it is NOT allowed to be wrong violently.

Exercises

Get your hands dirty before next time. Three tasks, climbing in difficulty:

Feel the compounding error. Take BehaviorCloningPolicy and a trivial 1D "track the target" environment you write yourself (state = position, expert action = move toward 0). Train on expert rollouts that always START near 0, then evaluate from a starting position FAR from 0. Watch it wander. Then write one sentence explaining why the failure is worse the further you start from the training distribution -- and name the fix from this episode.
Randomize until it transfers. Extend DomainRandomizer with a randomize_episode(env_params) method that returns a fresh config each episode, and run a loop of 1000 samples. Plot (or just bucket-count) the friction values to confirm they cover the full range. Then, in a comment, argue what happens to real-world transfer if your friction range is (0.9, 1.0) but the actual floor has friction 0.4.
Give the VLA a memory. Modify VisionLanguageActionPolicy to accept a short history of vision frames (like HistoryPolicy does) instead of a single frame, keeping the language input as one instruction. Confirm the output is still a single action vector. Then explain, in a sentence, one manipulation task that is IMPOSSIBLE from a single frame but solvable with a 4-frame window.

We'll open next episode with full solutions, as always.

Quick recap

Robotics is hard in ways software is not: continuous action manifolds, hard real-time control, partial observability, non-negotiable safety, and a world that keeps drifting under the policy's feet;
the sim-to-real gap is the central problem -- policies perfect in simulation fail in reality because of physics mismatch, sensor noise and actuator imperfection;
domain randomization bridges the gap by training across wide parameter distributions, converting sim-to-real into the generalization problem we already know how to fight (as long as reality falls inside your ranges);
three learning paradigms: behavioral cloning (simple, but distribution shift makes it brittle), RL (PPO/SAC for continuous control), and the hybrid -- imitation to bootstrap, RL to refine -- that real systems ship;
language-conditioned robotics uses vision-language-action models built on pretrained CLIP-style backbones (#75, #138) to turn plain-English instructions into motor commands;
foundation models for robots hit a hard data wall; cross-embodiment pooling (Open X-Embodiment) and massive simulation are the current ladders out;
hardware is not secondary -- sensors, actuators, on-board compute (edge AI, #122) and mechanical safety fundamentally bound what robot AI can do.

And here is the thread I want to leave dangling for next time. Everything today reacted to the world -- see a state, emit an action, see the next state, react again. Fast reflexes, no forethought. But the hardest problems, in a body or out of it, are not reflex problems -- they are the ones where the robot (or the agent) has to look several steps ahead, weigh options it has never tried, and commit to a PLAN before acting. What does it take to make a model that does not just react, but reasons about what to do next? That is where we head ;-)