# Learn AI Series (#81) - Pose Estimation and Tracking ![variant-a-12-green.png](https://images.hive.blog/DQmbC2LKb7BLTH4egg4if975nSnX46BrtSKKfa1JDatAaFq/variant-a-12-green.png) ### What will I learn - You will learn human pose estimation: detecting body keypoints like joints and limbs in images; - heatmap-based keypoint prediction: why models predict probability maps instead of raw coordinates; - top-down vs bottom-up approaches and when each one works best; - practical pose estimation with MediaPipe and MMPose/HRNet for real-time and research-grade accuracy; - object tracking across video frames: SORT, DeepSORT, and ByteTrack; - evaluation metrics: OKS for pose quality, MOTA and IDF1 for tracking quality; - how pose estimation and tracking combine for action recognition, sports analytics, and motion capture. ### Requirements - A working modern computer running macOS, Windows or Ubuntu; - An installed Python 3(.11+) distribution; - The ambition to learn AI and machine learning. ### Difficulty - Beginner ### Curriculum (of the `Learn AI Series`): - [Learn AI Series (#1) - What Machine Learning Actually Is](https://hive.blog/hive-196387/@scipio/learn-ai-series-1-what-machine-learning-actually-is) - [Learn AI Series (#2) - Setting Up Your AI Workbench - Python and NumPy](https://hive.blog/hive-196387/@scipio/learn-ai-series-2-setting-up-your-ai-workbench-python-and-numpy) - [Learn AI Series (#3) - Your Data Is Just Numbers - How Machines See the World](https://hive.blog/hive-196387/@scipio/learn-ai-series-3-your-data-is-just-numbers-how-machines-see-the-world) - [Learn AI Series (#4) - Your First Prediction - No Math, Just Intuition](https://hive.blog/hive-196387/@scipio/learn-ai-series-4-your-first-prediction-no-math-just-intuition) - [Learn AI Series (#5) - Patterns in Data - What "Learning" Actually Looks Like](https://hive.blog/hive-196387/@scipio/learn-ai-series-5-patterns-in-data-what-learning-actually-looks-like) - [Learn AI Series (#6) - From Intuition to Math - Why We Need Formulas](https://hive.blog/hive-196387/@scipio/learn-ai-series-6-from-intuition-to-math-why-we-need-formulas) - [Learn AI Series (#7) - The Training Loop - See It Work Step by Step](https://hive.blog/hive-196387/@scipio/learn-ai-series-7-the-training-loop-see-it-work-step-by-step) - [Learn AI Series (#8) - The Math You Actually Need (Part 1) - Linear Algebra](https://hive.blog/hive-196387/@scipio/learn-ai-series-8-the-math-you-actually-need-part-1-linear-algebra) - [Learn AI Series (#9) - The Math You Actually Need (Part 2) - Calculus and Probability](https://hive.blog/hive-196387/@scipio/learn-ai-series-9-the-math-you-actually-need-part-2-calculus-and-probability) - [Learn AI Series (#10) - Your First ML Model - Linear Regression From Scratch](https://hive.blog/hive-196387/@scipio/learn-ai-series-10-your-first-ml-model-linear-regression-from-scratch) - [Learn AI Series (#11) - Making Linear Regression Real](https://hive.blog/hive-196387/@scipio/learn-ai-series-11-making-linear-regression-real) - [Learn AI Series (#12) - Classification - Logistic Regression From Scratch](https://hive.blog/hive-196387/@scipio/learn-ai-series-12-classification-logistic-regression-from-scratch) - [Learn AI Series (#13) - Evaluation - How to Know If Your Model Actually Works](https://hive.blog/hive-196387/@scipio/learn-ai-series-13-evaluation-how-to-know-if-your-model-actually-works) - [Learn AI Series (#14) - Data Preparation - The 80% Nobody Talks About](https://hive.blog/hive-196387/@scipio/learn-ai-series-14-data-preparation-the-80-nobody-talks-about) - [Learn AI Series (#15) - Feature Engineering and Selection](https://hive.blog/hive-196387/@scipio/learn-ai-series-15-feature-engineering-and-selection) - [Learn AI Series (#16) - Scikit-Learn - The Standard Library of ML](https://hive.blog/hive-196387/@scipio/learn-ai-series-16-scikit-learn-the-standard-library-of-ml) - [Learn AI Series (#17) - Decision Trees - How Machines Make Decisions](https://hive.blog/hive-196387/@scipio/learn-ai-series-17-decision-trees-how-machines-make-decisions) - [Learn AI Series (#18) - Random Forests - Wisdom of Crowds](https://hive.blog/hive-196387/@scipio/learn-ai-series-18-random-forests-wisdom-of-crowds) - [Learn AI Series (#19) - Gradient Boosting - The Kaggle Champion](https://hive.blog/hive-196387/@scipio/learn-ai-series-19-gradient-boosting-the-kaggle-champion) - [Learn AI Series (#20) - Support Vector Machines - Drawing the Perfect Boundary](https://hive.blog/hive-196387/@scipio/learn-ai-series-20-support-vector-machines-drawing-the-perfect-boundary) - [Learn AI Series (#21) - Mini Project - Predicting Crypto Market Regimes](https://hive.blog/hive-196387/@scipio/learn-ai-series-21-mini-project-predicting-crypto-market-regimes) - [Learn AI Series (#22) - K-Means Clustering - Finding Groups](https://hive.blog/hive-196387/@scipio/learn-ai-series-22-k-means-clustering-finding-groups) - [Learn AI Series (#23) - Advanced Clustering - Beyond K-Means](https://hive.blog/hive-196387/@scipio/learn-ai-series-23-advanced-clustering-beyond-k-means) - [Learn AI Series (#24) - Dimensionality Reduction - PCA](https://hive.blog/hive-196387/@scipio/learn-ai-series-24-dimensionality-reduction-pca) - [Learn AI Series (#25) - Advanced Dimensionality Reduction - t-SNE and UMAP](https://hive.blog/hive-196387/@scipio/learn-ai-series-25-advanced-dimensionality-reduction-t-sne-and-umap) - [Learn AI Series (#26) - Anomaly Detection - Finding What Doesn't Belong](https://hive.blog/hive-196387/@scipio/learn-ai-series-26-anomaly-detection-finding-what-doesnt-belong) - [Learn AI Series (#27) - Recommendation Systems - "Users Like You Also Liked..."](https://hive.blog/hive-196387/@scipio/learn-ai-series-27-recommendation-systems-users-like-you-also-liked) - [Learn AI Series (#28) - Time Series Fundamentals - When Order Matters](https://hive.blog/hive-196387/@scipio/learn-ai-series-28-time-series-fundamentals-when-order-matters) - [Learn AI Series (#29) - Time Series Forecasting - Predicting What Comes Next](https://hive.blog/hive-196387/@scipio/learn-ai-series-29-time-series-forecasting-predicting-what-comes-next) - [Learn AI Series (#30) - Natural Language Processing - Text as Data](https://hive.blog/hive-196387/@scipio/learn-ai-series-30-natural-language-processing-text-as-data) - [Learn AI Series (#31) - Word Embeddings - Meaning in Numbers](https://hive.blog/hive-196387/@scipio/learn-ai-series-31-word-embeddings-meaning-in-numbers) - [Learn AI Series (#32) - Bayesian Methods - Thinking in Probabilities](https://hive.blog/hive-196387/@scipio/learn-ai-series-32-bayesian-methods-thinking-in-probabilities) - [Learn AI Series (#33) - Ensemble Methods Deep Dive - Stacking and Blending](https://hive.blog/hive-196387/@scipio/learn-ai-series-33-ensemble-methods-deep-dive-stacking-and-blending) - [Learn AI Series (#34) - ML Engineering - From Notebook to Production](https://hive.blog/hive-196387/@scipio/learn-ai-series-34-ml-engineering-from-notebook-to-production) - [Learn AI Series (#35) - Data Ethics and Bias in ML](https://hive.blog/hive-196387/@scipio/learn-ai-series-35-data-ethics-and-bias-in-ml) - [Learn AI Series (#36) - Mini Project - Complete ML Pipeline](https://hive.blog/hive-196387/@scipio/learn-ai-series-36-mini-project-complete-ml-pipeline) - [Learn AI Series (#37) - The Perceptron - Where It All Started](https://hive.blog/hive-196387/@scipio/learn-ai-series-37-the-perceptron-where-it-all-started) - [Learn AI Series (#38) - Neural Networks From Scratch - Forward Pass](https://hive.blog/hive-196387/@scipio/learn-ai-series-38-neural-networks-from-scratch-forward-pass) - [Learn AI Series (#39) - Neural Networks From Scratch - Backpropagation](https://hive.blog/hive-196387/@scipio/learn-ai-series-39-neural-networks-from-scratch-backpropagation) - [Learn AI Series (#40) - Training Neural Networks - Practical Challenges](https://hive.blog/hive-196387/@scipio/learn-ai-series-40-training-neural-networks-practical-challenges) - [Learn AI Series (#41) - Optimization Algorithms - SGD, Momentum, Adam](https://hive.blog/hive-196387/@scipio/learn-ai-series-41-optimization-algorithms-sgd-momentum-adam) - [Learn AI Series (#42) - PyTorch Fundamentals - Tensors and Autograd](https://hive.blog/hive-196387/@scipio/learn-ai-series-42-pytorch-fundamentals-tensors-and-autograd) - [Learn AI Series (#43) - PyTorch Data and Training](https://hive.blog/hive-196387/@scipio/learn-ai-series-43-pytorch-data-and-training) - [Learn AI Series (#44) - PyTorch nn.Module - Building Real Networks](https://hive.blog/hive-196387/@scipio/learn-ai-series-44-pytorch-nnmodule-building-real-networks) - [Learn AI Series (#45) - Convolutional Neural Networks - Theory](https://hive.blog/hive-196387/@scipio/learn-ai-series-45-convolutional-neural-networks-theory) - [Learn AI Series (#46) - CNNs in Practice - Classic to Modern Architectures](https://hive.blog/hive-196387/@scipio/learn-ai-series-46-cnns-in-practice-classic-to-modern-architectures) - [Learn AI Series (#47) - CNN Applications - Detection, Segmentation, Style Transfer](https://hive.blog/hive-196387/@scipio/learn-ai-series-47-cnn-applications-detection-segmentation-style-transfer) - [Learn AI Series (#48) - Recurrent Neural Networks - Sequences](https://hive.blog/hive-196387/@scipio/learn-ai-series-48-recurrent-neural-networks-sequences) - [Learn AI Series (#49) - LSTM and GRU - Solving the Memory Problem](https://hive.blog/hive-196387/@scipio/learn-ai-series-49-lstm-and-gru-solving-the-memory-problem) - [Learn AI Series (#50) - Sequence-to-Sequence Models](https://hive.blog/hive-196387/@scipio/learn-ai-series-50-sequence-to-sequence-models) - [Learn AI Series (#51) - Attention Mechanisms](https://hive.blog/hive-196387/@scipio/learn-ai-series-51-attention-mechanisms) - [Learn AI Series (#52) - The Transformer Architecture (Part 1)](https://hive.blog/hive-196387/@scipio/learn-ai-series-52-the-transformer-architecture-part-1) - [Learn AI Series (#53) - The Transformer Architecture (Part 2)](https://hive.blog/hive-196387/@scipio/learn-ai-series-53-the-transformer-architecture-part-2) - [Learn AI Series (#54) - Vision Transformers](https://hive.blog/hive-196387/@scipio/learn-ai-series-54-vision-transformers) - [Learn AI Series (#55) - Generative Adversarial Networks](https://hive.blog/hive-196387/@scipio/learn-ai-series-55-generative-adversarial-networks) - [Learn AI Series (#56) - Mini Project - Building a Transformer From Scratch](https://hive.blog/hive-196387/@scipio/learn-ai-series-56-mini-project-building-a-transformer-from-scratch) - [Learn AI Series (#57) - Language Modeling - Predicting the Next Word](https://hive.blog/hive-196387/@scipio/learn-ai-series-57-language-modeling-predicting-the-next-word) - [Learn AI Series (#58) - GPT Architecture - Decoder-Only Transformers](https://hive.blog/hive-196387/@scipio/learn-ai-series-58-gpt-architecture-decoder-only-transformers) - [Learn AI Series (#59) - BERT and Encoder Models](https://hive.blog/hive-196387/@scipio/learn-ai-series-59-bert-and-encoder-models) - [Learn AI Series (#60) - Training Large Language Models](https://hive.blog/hive-196387/@scipio/learn-ai-series-60-training-large-language-models) - [Learn AI Series (#61) - Instruction Tuning and Alignment](https://hive.blog/hive-196387/@scipio/learn-ai-series-61-instruction-tuning-and-alignment) - [Learn AI Series (#62) - Prompt Engineering - Getting the Most from LLMs](https://hive.blog/hive-196387/@scipio/learn-ai-series-62-prompt-engineering-getting-the-most-from-llms) - [Learn AI Series (#63) - Embeddings and Vector Search](https://hive.blog/hive-196387/@scipio/learn-ai-series-63-embeddings-and-vector-search) - [Learn AI Series (#64) - Retrieval-Augmented Generation (RAG) - Basics](https://hive.blog/hive-196387/@scipio/learn-ai-series-64-retrieval-augmented-generation-rag-basics) - [Learn AI Series (#65) - RAG - Advanced Techniques](https://hive.blog/hive-196387/@scipio/learn-ai-series-65-rag-advanced-techniques) - [Learn AI Series (#66) - Working with LLM APIs](https://hive.blog/hive-196387/@scipio/learn-ai-series-66-working-with-llm-apis) - [Learn AI Series (#67) - Building AI Agents (Part 1) - Foundations](https://hive.blog/hive-196387/@scipio/learn-ai-series-67-building-ai-agents-part-1-foundations) - [Learn AI Series (#68) - Building AI Agents (Part 2) - Advanced Patterns](https://hive.blog/hive-196387/@scipio/learn-ai-series-68-building-ai-agents-part-2-advanced-patterns) - [Learn AI Series (#69) - Fine-Tuning Language Models](https://hive.blog/hive-196387/@scipio/learn-ai-series-69-fine-tuning-language-models) - [Learn AI Series (#70) - Running Local Models](https://hive.blog/hive-196387/@scipio/learn-ai-series-70-running-local-models) - [Learn AI Series (#71) - Text Generation Techniques](https://hive.blog/hive-196387/@scipio/learn-ai-series-71-text-generation-techniques) - [Learn AI Series (#72) - Tokenization Deep Dive](https://hive.blog/hive-196387/@scipio/learn-ai-series-72-tokenization-deep-dive) - [Learn AI Series (#73) - LLM Evaluation](https://hive.blog/hive-196387/@scipio/learn-ai-series-73-llm-evaluation) - [Learn AI Series (#74) - The Hugging Face Ecosystem](https://hive.blog/hive-196387/@scipio/learn-ai-series-74-the-hugging-face-ecosystem) - [Learn AI Series (#75) - Multimodal Models - Text Meets Vision](https://hive.blog/hive-196387/@scipio/learn-ai-series-75-multimodal-models-text-meets-vision) - [Learn AI Series (#76) - Mini Project - Your Own AI Assistant](https://hive.blog/hive-196387/@scipio/learn-ai-series-76-mini-project-your-own-ai-assistant) - [Learn AI Series (#77) - Image Processing Fundamentals](https://hive.blog/hive-196387/@scipio/learn-ai-series-77-image-processing-fundamentals) - [Learn AI Series (#78) - Object Detection (Part 1) - Foundations](https://hive.blog/hive-196387/@scipio/learn-ai-series-78-object-detection-part-1-foundations) - [Learn AI Series (#79) - Object Detection (Part 2) - Modern Approaches](https://hive.blog/hive-196387/@scipio/learn-ai-series-79-object-detection-part-2-modern-approaches) - [Learn AI Series (#80) - Image Segmentation](https://hive.blog/hive-196387/@scipio/learn-ai-series-80-image-segmentation) - [Learn AI Series (#81) - Pose Estimation and Tracking](https://hive.blog/hive-196387/@scipio/learn-ai-series-81-pose-estimation-and-tracking) (this post) # Learn AI Series (#81) - Pose Estimation and Tracking ### Solutions to Episode #80 Exercises **Exercise 1:** Segmentation mask visualization toolkit. ```python import numpy as np class SegmentationVisualizer: """Visualize and evaluate segmentation masks.""" def __init__(self, class_colors): self.class_colors = class_colors self.num_classes = len(class_colors) def colorize(self, prediction): """Convert class label map to RGB image.""" h, w = prediction.shape rgb = np.zeros((h, w, 3), dtype=np.uint8) for cls_id, color in self.class_colors.items(): mask = prediction == cls_id rgb[mask] = color return rgb def overlay(self, image, prediction, alpha=0.5): """Blend colorized mask with original image.""" colored = self.colorize(prediction) if image.dtype != np.uint8: image = (image * 255).astype(np.uint8) blended = (alpha * colored.astype(np.float32) + (1 - alpha) * image.astype(np.float32)) return np.clip(blended, 0, 255).astype(np.uint8) def class_statistics(self, prediction): total = prediction.size print(f"Image: {prediction.shape[1]}x" f"{prediction.shape[0]} = {total} pixels") for cls_id, color in self.class_colors.items(): count = (prediction == cls_id).sum() pct = count / total * 100 present = "YES" if count > 0 else "no" print(f" Class {cls_id}: {count:>6d} px " f"({pct:>5.1f}%) [{present}]") def iou_per_class(self, prediction, ground_truth): ious = {} for cls_id in self.class_colors: pred_mask = prediction == cls_id gt_mask = ground_truth == cls_id inter = (pred_mask & gt_mask).sum() union = (pred_mask | gt_mask).sum() if union == 0: continue ious[cls_id] = inter / union miou = np.mean(list(ious.values())) if ious else 0 print(f"Per-class IoU:") for c, v in ious.items(): print(f" Class {c}: {v:.4f}") print(f" mIoU: {miou:.4f}") return ious, miou # Test: 4 classes, 200x200 colors = {0: (0, 0, 0), 1: (255, 0, 0), 2: (0, 255, 0), 3: (0, 0, 255)} vis = SegmentationVisualizer(colors) np.random.seed(42) gt = np.random.randint(0, 4, (200, 200)) vis.class_statistics(gt) ious, miou = vis.iou_per_class(gt, gt) # mIoU = 1.0 when prediction == ground truth ``` When prediction equals ground truth exactly, every class gets IoU = 1.0 and mIoU = 1.0. The moment you introduce even small errors, mIoU drops quickly -- a class that covers only 5% of the image can have its IoU destroyed by a handful of misclassified pixels. This is why mIoU is a stricter metric than pixel accuracy: accuracy lets you get away with ignoring small classes, mIoU does not. **Exercise 2:** Flexible U-Net with configurable depth. ```python import torch import torch.nn as nn class FlexibleUNet(nn.Module): """U-Net with configurable depth.""" def __init__(self, in_ch=3, num_classes=21, depth=4, base_channels=64): super().__init__() self.depth = depth self.first_forward = True # Build encoder self.encoders = nn.ModuleList() self.pools = nn.ModuleList() ch = in_ch for i in range(depth): out_ch = base_channels * (2 ** i) self.encoders.append(self._block(ch, out_ch)) self.pools.append(nn.MaxPool2d(2)) ch = out_ch # Bottleneck bot_ch = base_channels * (2 ** depth) self.bottleneck = self._block(ch, bot_ch) # Build decoder self.upconvs = nn.ModuleList() self.decoders = nn.ModuleList() ch = bot_ch for i in range(depth - 1, -1, -1): out_ch = base_channels * (2 ** i) self.upconvs.append( nn.ConvTranspose2d(ch, out_ch, 2, stride=2) ) self.decoders.append( self._block(out_ch * 2, out_ch) ) ch = out_ch self.out_conv = nn.Conv2d(ch, num_classes, 1) def _block(self, in_ch, out_ch): return nn.Sequential( nn.Conv2d(in_ch, out_ch, 3, padding=1), nn.BatchNorm2d(out_ch), nn.ReLU(True), nn.Conv2d(out_ch, out_ch, 3, padding=1), nn.BatchNorm2d(out_ch), nn.ReLU(True), ) def parameter_count(self): return sum(p.numel() for p in self.parameters() if p.requires_grad) def forward(self, x): skips = [] for enc, pool in zip(self.encoders, self.pools): x = enc(x) skips.append(x) if self.first_forward: print(f" Enc: {x.shape}") x = pool(x) x = self.bottleneck(x) if self.first_forward: print(f" Bot: {x.shape}") for up, dec, skip in zip( self.upconvs, self.decoders, reversed(skips) ): x = up(x) x = torch.cat([x, skip], dim=1) x = dec(x) if self.first_forward: print(f" Dec: {x.shape}") self.first_forward = False return self.out_conv(x) # Compare depths 2-5 print(f"{'Depth':<6} {'Params':>12} {'Output':>20}") print("-" * 40) for d in [2, 3, 4, 5]: model = FlexibleUNet(depth=d, base_channels=64) print(f"\n--- Depth {d} ---") out = model(torch.randn(1, 3, 256, 256)) params = model.parameter_count() print(f"{d:<6} {params:>12,} {str(out.shape):>20}") ``` Output spatial dimensions always match input regardless of depth -- the decoder upsamples back to exactly the original resolution. Deeper models have exponentially more parameters (depth 5 has roughly 16x more parameters than depth 2) because each level doubles the channel count. The practical sweet spot for most tasks is depth 3-4; depth 5 is overkill unless you have a very large dataset and significant GPU memory. **Exercise 3:** Dice coefficient monitor. ```python import torch import torch.nn.functional as F import numpy as np class DiceMonitor: """Track Dice scores across training batches.""" def __init__(self, num_classes): self.num_classes = num_classes self.reset() def reset(self): self.tp = np.zeros(self.num_classes) self.fp = np.zeros(self.num_classes) self.fn = np.zeros(self.num_classes) def update(self, predictions, targets): """predictions: (B,C,H,W), targets: (B,H,W).""" preds = predictions.argmax(dim=1) # (B,H,W) for c in range(self.num_classes): pred_c = (preds == c).float() targ_c = (targets == c).float() self.tp[c] += (pred_c * targ_c).sum().item() self.fp[c] += (pred_c * (1 - targ_c)).sum().item() self.fn[c] += ((1 - pred_c) * targ_c).sum().item() def compute(self): dice = {} for c in range(self.num_classes): denom = 2 * self.tp[c] + self.fp[c] + self.fn[c] if denom == 0: continue dice[c] = 2 * self.tp[c] / denom mean_dice = (np.mean(list(dice.values())) if dice else 0.0) return dice, mean_dice def report(self): dice, mean_d = self.compute() print(f"{'Class':>6} {'Dice':>8} {'Bar'}") print("-" * 40) for c, d in sorted(dice.items()): bar = "#" * int(d * 30) print(f"{c:>6} {d:>8.4f} {bar}") print(f"{'Mean':>6} {mean_d:>8.4f}") # Simulate 10 batches with improving predictions monitor = DiceMonitor(num_classes=4) np.random.seed(42) torch.manual_seed(42) gt = torch.randint(0, 4, (2, 32, 32)) for batch in range(10): # Gradually improve: start noisy, converge to GT noise_level = 1.0 - batch * 0.1 logits = F.one_hot(gt, 4).permute(0, 3, 1, 2).float() logits += torch.randn_like(logits) * noise_level monitor.update(logits, gt) monitor.report() # Verify perfect predictions -> Dice = 1.0 monitor.reset() perfect = F.one_hot(gt, 4).permute(0, 3, 1, 2).float() perfect *= 100 # high confidence monitor.update(perfect, gt) dice, mean_d = monitor.compute() print(f"\nPerfect predictions: mean Dice = {mean_d:.4f}") ``` The Dice score increases as the noise decreases, which is exactly what you want to see during training. With perfect predictions the Dice is 1.0 for every class. In real training, you'd call `reset()` at the start of each epoch, `update()` for every batch, and `report()` at the end -- giving you a live view of which classes the model is learning well and which ones are lagging behind. ### On to today's episode Here we go! The last four episodes built up a complete visual understanding stack: image processing fundamentals (#77), object detection foundations and modern approaches (#78-79), and pixel-level segmentation (#80). We can now tell _what_ is in an image, _where_ it is (bounding box), and _exactly which pixels_ belong to it (mask). That's incredibly powerful, but it still treats objects as blobs -- a person is just a labeled region of pixels. Humans aren't blobs. A person has structure -- arms, legs, a head, joints that bend in specific ways. Whether someone is standing, sitting, waving, or falling depends entirely on the spatial arrangement of their body parts. Segmentation tells you "these pixels are a person." Pose estimation tells you "this person's left elbow is at coordinates (342, 187) and their right knee is at (295, 410)." That structural information unlocks capabilities that detection and segmentation simply cannot provide ;-) ## What pose estimation actually outputs A pose estimator takes an image and returns a set of **keypoints** -- specific anatomical landmarks on the body. The COCO keypoint format defines 17 points: ``` 0: nose 1: left_eye 2: right_eye 3: left_ear 4: right_ear 5: left_shoulder 6: right_shoulder 7: left_elbow 8: right_elbow 9: left_wrist 10: right_wrist 11: left_hip 12: right_hip 13: left_knee 14: right_knee 15: left_ankle 16: right_ankle ``` Each keypoint is an `(x, y, confidence)` triplet. The model predicts not just where the joint is, but how sure it is about that prediction. A wrist hidden behind someone's back might get coordinates but a low confidence score -- the model is essentially saying "I think it's here, but I'm guessing." The keypoints are connected by a **skeleton** -- predefined edges between keypoints that represent bones and limbs. Shoulder-to-elbow, elbow-to-wrist, hip-to-knee, knee-to-ankle. The skeleton turns a bag of 17 scattered points into a recognizable human figure. ```python import numpy as np # COCO skeleton definition: pairs of keypoint indices COCO_SKELETON = [ (0, 1), (0, 2), (1, 3), (2, 4), # head (5, 6), # shoulders (5, 7), (7, 9), # left arm (6, 8), (8, 10), # right arm (5, 11), (6, 12), # torso (11, 12), # hips (11, 13), (13, 15), # left leg (12, 14), (14, 16), # right leg ] KEYPOINT_NAMES = [ "nose", "left_eye", "right_eye", "left_ear", "right_ear", "left_shoulder", "right_shoulder", "left_elbow", "right_elbow", "left_wrist", "right_wrist", "left_hip", "right_hip", "left_knee", "right_knee", "left_ankle", "right_ankle", ] def draw_skeleton(image, keypoints, threshold=0.3): """Draw keypoints and skeleton on image. keypoints: (17, 3) array of (x, y, confidence) """ import cv2 img = image.copy() h, w = img.shape[:2] # Draw keypoints as circles for i, (x, y, conf) in enumerate(keypoints): if conf < threshold: continue px, py = int(x), int(y) cv2.circle(img, (px, py), 4, (0, 255, 0), -1) # Draw skeleton lines for i, j in COCO_SKELETON: if (keypoints[i, 2] < threshold or keypoints[j, 2] < threshold): continue pt1 = (int(keypoints[i, 0]), int(keypoints[i, 1])) pt2 = (int(keypoints[j, 0]), int(keypoints[j, 1])) cv2.line(img, pt1, pt2, (255, 0, 0), 2) return img # Simulate a standing person keypoints = np.array([ [200, 50, 0.95], # nose [190, 45, 0.90], # left_eye [210, 45, 0.92], # right_eye [180, 55, 0.80], # left_ear [220, 55, 0.82], # right_ear [170, 120, 0.95], # left_shoulder [230, 120, 0.93], # right_shoulder [150, 200, 0.88], # left_elbow [250, 200, 0.87], # right_elbow [140, 280, 0.85], # left_wrist [260, 280, 0.83], # right_wrist [180, 280, 0.92], # left_hip [220, 280, 0.91], # right_hip [175, 380, 0.89], # left_knee [225, 380, 0.88], # right_knee [170, 470, 0.85], # left_ankle [230, 470, 0.84], # right_ankle ]) print(f"Keypoints shape: {keypoints.shape}") print(f"Mean confidence: {keypoints[:, 2].mean():.3f}") ``` ## Top-down vs bottom-up There are two fundamentally different strategies for multi-person pose estimation, and the choice between them has real practical consequences. **Top-down**: first detect all people in the image (using an object detector like YOLO or Faster R-CNN from episodes #78-79), then run a single-person pose estimator on each detected bounding box. The pose model never has to deal with multiple people -- it gets a cropped image containing exactly one person and predicts 17 keypoints. **Bottom-up**: first detect _all_ keypoints in the entire image at once (all the left elbows, all the right knees, everything), then figure out which keypoints belong to which person. The grouping step is the hard part -- if there are 5 people in the image, there are 5 left elbows, and you need to match each elbow to the correct person. ``` Top-down (two-stage): Image -> Detector -> [crop_1, crop_2, ...] -> Pose model -> keypoints per person Bottom-up (also two-stage, but different): Image -> Keypoint detector -> all keypoints -> Grouping -> keypoints per person ``` Top-down is generally more accurate because the pose model operates on clean, single-person crops. But it's slower when there are many people -- runtime scales linearly with the number of detections. If you have 30 people in the frame, you're running the pose model 30 times. Bottom-up is faster for crowded scenes because keypoint detection happens once regardless of how many people are present, but the grouping step can struggle when people overlap or stand very close together. For most practical applications (fitness apps, sports analytics, interactive installations), **top-down with a fast detector** is the right choice. You rarely have more than 10-15 people in the frame for these use cases, and accuracy matters more than throughput. For surveillance or crowd analysis where you might have 50+ people, bottom-up approches like OpenPose make more sense. ## Heatmap-based keypoint prediction Most modern pose estimators predict **heatmaps** rather than direct coordinate regression. For each keypoint, the model outputs a 2D probability map where the brightest spot indicates the predicted location. _Why heatmaps instead of just regressing (x, y) coordinates directly?_ Regressing exact coordinates is hard -- the loss landscape is steep and the model has to be precisely right. Heatmaps are a softer target: the ground truth is a 2D Gaussian centered on the true keypoint location, and the model learns to produce a similar blob. The predicted coordinate is just the argmax of the heatmap. Much easier to learn, and it naturally provides a confidence measure (how bright is the peak?). ```python import torch import torch.nn as nn import numpy as np def generate_heatmap(height, width, center_x, center_y, sigma=2.0): """Generate a Gaussian heatmap centered at (center_x, center_y).""" y_grid, x_grid = np.mgrid[0:height, 0:width] dist_sq = ((x_grid - center_x) ** 2 + (y_grid - center_y) ** 2) heatmap = np.exp(-dist_sq / (2 * sigma ** 2)) return heatmap # Ground truth: left shoulder at pixel (120, 85) gt_heatmap = generate_heatmap( 64, 64, center_x=30, center_y=21, sigma=2.0 ) print(f"Heatmap shape: {gt_heatmap.shape}") print(f"Peak value: {gt_heatmap.max():.3f}") print(f"Peak location: " f"{np.unravel_index(gt_heatmap.argmax(), gt_heatmap.shape)}") # The loss is MSE between predicted and GT heatmaps # For 17 keypoints: model outputs (B, 17, H, W) # Each channel is a heatmap for one keypoint gt_batch = torch.from_numpy( gt_heatmap ).unsqueeze(0).unsqueeze(0).float() pred = torch.randn(1, 1, 64, 64) loss_fn = nn.MSELoss() loss = loss_fn(pred, gt_batch) print(f"Heatmap MSE loss: {loss.item():.4f}") ``` The architecture follows a pattern we've seen many times now: encoder (backbone) compresses the image into features, then a decoder (deconvolution layers) expands back to spatial resolution. It's essentialy the U-Net thinking from episode #80 applied to keypoint prediction instead of pixel classification. ```python import torch import torch.nn as nn class SimplePoseNet(nn.Module): """Simplified pose estimation network. Backbone -> deconv upsampling -> heatmaps.""" def __init__(self, num_keypoints=17): super().__init__() self.backbone = nn.Sequential( nn.Conv2d(3, 64, 7, stride=2, padding=3), nn.BatchNorm2d(64), nn.ReLU(), nn.Conv2d(64, 128, 3, stride=2, padding=1), nn.BatchNorm2d(128), nn.ReLU(), nn.Conv2d(128, 256, 3, stride=2, padding=1), nn.BatchNorm2d(256), nn.ReLU(), ) self.deconv = nn.Sequential( nn.ConvTranspose2d(256, 128, 4, stride=2, padding=1), nn.BatchNorm2d(128), nn.ReLU(), nn.ConvTranspose2d(128, 64, 4, stride=2, padding=1), nn.BatchNorm2d(64), nn.ReLU(), nn.ConvTranspose2d(64, num_keypoints, 4, stride=2, padding=1), ) def forward(self, x): features = self.backbone(x) heatmaps = self.deconv(features) return heatmaps model = SimplePoseNet() img = torch.randn(1, 3, 256, 256) heatmaps = model(img) print(f"Input: {img.shape}") print(f"Output: {heatmaps.shape}") # (1, 17, 256, 256) -- one heatmap per keypoint def heatmaps_to_keypoints(heatmaps): """Extract (x, y, confidence) per keypoint.""" batch, num_kp, h, w = heatmaps.shape flat = heatmaps.view(batch, num_kp, -1) confidence, indices = flat.max(dim=-1) y = indices // w x = indices % w # Stack into (batch, num_keypoints, 3) return torch.stack([x, y, confidence], dim=-1) kps = heatmaps_to_keypoints(heatmaps) print(f"Keypoints: {kps.shape}") # (1, 17, 3) -- x, y, confidence per keypoint ``` The real production models (HRNet, ViTPose) use much deeper backbones, but the data flow is identical: image in, heatmaps out, argmax to get coordinates. The difference is in how the backbone maintains spatial information -- and that's where HRNet gets interesting. ## HRNet: maintaining resolution throughout **HRNet** (High-Resolution Network, Sun et al., 2019) takes a radically different approach from the standard "shrink then expand" pattern. Instead of progressively reducing resolution and then upsampling (like ResNet + deconv, or U-Net's encoder-decoder), HRNet **maintains high-resolution representations throughout the entire network**, running parallel branches at different resolutions and repeatedly exchanging information between them. ``` Stage 1: [High-res branch only] | Stage 2: [High-res] <-> [Medium-res] | | Stage 3: [High-res] <-> [Medium-res] <-> [Low-res] | | | Stage 4: [High-res] <-> [Medium-res] <-> [Low-res] <-> [Very-low-res] ``` At each stage, a new lower-resolution branch is added, and all branches exchange information through **multi-scale fusion**: each branch receives features from all other branches (upsampled or downsampled as needed). The high-res branch never loses spatial detail, and the low-res branches provide the semantic context. ```python import torch import torch.nn as nn import torch.nn.functional as F class MultiScaleFusion(nn.Module): """Fuse features across different resolutions.""" def __init__(self, channels_list): """channels_list: [high_ch, medium_ch, low_ch]""" super().__init__() self.num_branches = len(channels_list) self.fuse_layers = nn.ModuleList() for i in range(self.num_branches): branch_fuse = nn.ModuleList() for j in range(self.num_branches): if i == j: # Same resolution: identity branch_fuse.append(nn.Identity()) elif j > i: # Lower-res to higher-res: upsample branch_fuse.append(nn.Sequential( nn.Conv2d(channels_list[j], channels_list[i], 1), nn.BatchNorm2d(channels_list[i]), )) else: # Higher-res to lower-res: strided conv layers = [] for k in range(i - j): in_c = (channels_list[j] if k == 0 else channels_list[i]) layers.extend([ nn.Conv2d(in_c, channels_list[i], 3, stride=2, padding=1), nn.BatchNorm2d( channels_list[i]), ]) branch_fuse.append( nn.Sequential(*layers) ) self.fuse_layers.append(branch_fuse) def forward(self, inputs): """inputs: list of tensors at different resolutions.""" outputs = [] for i in range(self.num_branches): fused = None for j in range(self.num_branches): x = self.fuse_layers[i][j](inputs[j]) if j > i: # Upsample to match target size x = F.interpolate( x, size=inputs[i].shape[2:], mode="bilinear", align_corners=True ) if fused is None: fused = x else: fused = fused + x outputs.append(F.relu(fused)) return outputs # Test: 3 branches at different resolutions fusion = MultiScaleFusion([32, 64, 128]) inputs = [ torch.randn(1, 32, 64, 64), # high-res torch.randn(1, 64, 32, 32), # medium-res torch.randn(1, 128, 16, 16), # low-res ] outputs = fusion(inputs) for i, o in enumerate(outputs): print(f"Branch {i}: {inputs[i].shape} -> " f"{o.shape}") ``` _Why does this matter for pose estimation specifically?_ Because keypoint localization needs both local precision (exactly which pixel is the elbow) and global context (understanding that the blob of pixels near the torso is an arm, not a background object). The standard approach of downsampling then upsampling inevitably loses some spatial precision even with skip connections. HRNet avoids this entirely by never downsampling the high-resolution branch. The result: consistently more accurate keypoint predictions, especially for small or occluded joints. On the COCO keypoint benchmark, HRNet-W48 achieves around **75.1 AP** compared to SimpleBaseline (ResNet-152 + deconv) at **72.0 AP**. That 3-point improvement comes entirely from the architectural choice to maintain resolution. ## Practical pose estimation with MediaPipe For production use, you don't build from scratch. The ecosystem has mature models ready to deploy. **MediaPipe** (Google) is the go-to for real-time applications. It runs on CPU, on mobile devices, in the browser. It detects 33 body keypoints (more than COCO's 17), plus hands (21 keypoints each) and face (468 landmarks): ```python import mediapipe as mp import cv2 mp_pose = mp.solutions.pose mp_draw = mp.solutions.drawing_utils with mp_pose.Pose( static_image_mode=True, min_detection_confidence=0.5 ) as pose: image = cv2.imread("person.jpg") rgb = cv2.cvtColor(image, cv2.COLOR_BGR2RGB) results = pose.process(rgb) if results.pose_landmarks: h, w = image.shape[:2] for i, lm in enumerate( results.pose_landmarks.landmark ): px = int(lm.x * w) py = int(lm.y * h) print(f"Keypoint {i}: ({px}, {py}), " f"vis={lm.visibility:.2f}") # Draw skeleton on image mp_draw.draw_landmarks( image, results.pose_landmarks, mp_pose.POSE_CONNECTIONS ) cv2.imwrite("pose_result.jpg", image) ``` MediaPipe is remarkable because it runs at 30+ FPS _on a mobile phone_. The secret is a lightweight architecture (BlazePose) specifically designed for on-device inference. It uses a two-stage pipeline internally: a fast detector finds the person, then a regression network predicts keypoints from the crop. The regression network is tiny -- about 3.5 million parameters. **MMPose** (from the OpenMMLab ecosystem) offers research-grade accuracy with a zoo of pretrained models -- HRNet, ViTPose, RTMPose. It's heavier than MediaPipe but more acurate, and supports a wider range of pose types (body, hand, face, animal, whole-body): ```python # MMPose config-based inference from mmpose.apis import ( inference_topdown, init_model ) from mmpose.utils import register_all_modules register_all_modules() config = ( "configs/body_2d_keypoint/" "topdown_heatmap/coco/" "td-hm_hrnet-w48_8xb32-210e_coco-" "256x192.py" ) checkpoint = ( "https://download.openmmlab.com/mmpose/v1/" "body_2d_keypoint/topdown_heatmap/coco/" "td-hm_hrnet-w48_8xb32-210e_coco-" "256x192-0e67c616_20220913.pth" ) model = init_model(config, checkpoint, device="cpu") # Top-down: provide bounding boxes from detector bboxes = [{"bbox": [100, 50, 400, 500]}] results = inference_topdown( model, "person.jpg", bboxes ) keypoints = results[0].pred_instances.keypoints scores = results[0].pred_instances.keypoint_scores print(f"Keypoints: {keypoints.shape}") print(f"Scores: {scores.shape}") ``` The practical decision is straightforward: MediaPipe for real-time applications where speed matters (fitness apps, interactive installations, mobile deployment). HRNet/MMPose for offline processing where accuracy matters (sports analytics, research, medical assessment). Both produce the same output format -- keypoints with confidence scores -- so you can prototype with MediaPipe and swap in HRNet later if you need higher accuracy. ## Object tracking across frames Pose estimation tells you where people are in a single frame. **Tracking** tells you which person in frame _t_ is the same person in frame _t+1_. This sounds trivial -- _just match the closest detections, right?_ -- but it's genuinely hard. People look similar, they overlap, they walk behind pillars and reappear on the other side, they change direction abruptly. The classic approach is **SORT** (Simple Online and Realtime Tracking, Bewley et al., 2016), which uses two components: 1. **Kalman filter**: predict where each tracked object will be in the next frame based on its position and velocity 2. **Hungarian algorithm**: optimally match predictions to new detections using IoU as the cost metric ```python import numpy as np from scipy.optimize import linear_sum_assignment def compute_iou(box1, box2): x1 = max(box1[0], box2[0]) y1 = max(box1[1], box2[1]) x2 = min(box1[2], box2[2]) y2 = min(box1[3], box2[3]) inter = max(0, x2 - x1) * max(0, y2 - y1) a1 = (box1[2] - box1[0]) * (box1[3] - box1[1]) a2 = (box2[2] - box2[0]) * (box2[3] - box2[1]) union = a1 + a2 - inter return inter / max(union, 1e-6) class SimpleTracker: """Simplified SORT-style tracker.""" def __init__(self, max_age=5, iou_thresh=0.3): self.tracks = [] self.next_id = 0 self.max_age = max_age self.iou_thresh = iou_thresh def update(self, detections): """Match detections to existing tracks. detections: list of [x1, y1, x2, y2] Returns: list of (track_id, box) tuples """ if not self.tracks: # First frame: create tracks for all dets results = [] for det in detections: self.tracks.append({ "id": self.next_id, "box": det, "age": 0, }) results.append( (self.next_id, det) ) self.next_id += 1 return results # Build IoU cost matrix num_tracks = len(self.tracks) num_dets = len(detections) cost = np.zeros((num_tracks, num_dets)) for i, trk in enumerate(self.tracks): for j, det in enumerate(detections): cost[i, j] = 1.0 - compute_iou( trk["box"], det ) # Hungarian matching row_idx, col_idx = linear_sum_assignment(cost) matched_tracks = set() matched_dets = set() results = [] for r, c in zip(row_idx, col_idx): if cost[r, c] > (1 - self.iou_thresh): continue self.tracks[r]["box"] = detections[c] self.tracks[r]["age"] = 0 matched_tracks.add(r) matched_dets.add(c) results.append( (self.tracks[r]["id"], detections[c]) ) # Unmatched detections: new tracks for j in range(num_dets): if j not in matched_dets: self.tracks.append({ "id": self.next_id, "box": detections[j], "age": 0, }) results.append( (self.next_id, detections[j]) ) self.next_id += 1 # Age unmatched tracks, remove old ones for i in range(num_tracks): if i not in matched_tracks: self.tracks[i]["age"] += 1 self.tracks = [ t for t in self.tracks if t["age"] <= self.max_age ] return results # Simulate 3 frames of tracking tracker = SimpleTracker() frame1 = [[100, 100, 200, 300], [400, 100, 500, 300]] frame2 = [[110, 105, 210, 305], [410, 95, 510, 295]] frame3 = [[120, 110, 220, 310], [420, 90, 520, 290], [250, 200, 350, 400]] for i, dets in enumerate([frame1, frame2, frame3]): results = tracker.update(dets) print(f"Frame {i+1}:") for tid, box in results: print(f" Track {tid}: {box}") ``` SORT is fast but fragile. When a person is briefly occluded (walks behind a pillar, crouches behind a desk) and reappears, SORT often assigns them a **new ID** because the IoU between the predicted and actual positions is too low after the gap. This is called an **ID switch**, and it's the most common failure mode. **DeepSORT** (Wojke et al., 2017) fixes this by adding an **appearance model**. Each tracked object gets an appearance feature vector extracted by a small CNN trained on person re-identification data. When matching detections to tracks, DeepSORT uses both motion (Kalman prediction via IoU) and appearance (cosine similarity of feature vectors). This lets it re-identify a person after occlusion -- even if they've moved significantly, their appearance features still match ;-) **ByteTrack** (Zhang et al., 2022) takes yet another approach: instead of using appearance features, it makes better use of **low-confidence detections**. Most trackers throw away detections below a confidence threshold (say 0.5). ByteTrack keeps them and uses them in a second matching round. A briefly occluded person might produce a low-confidence detection that other trackers would discard but ByteTrack correctly associates with the existing track. Simple idea, big improvement -- ByteTrack achieved state-of-the-art results on the MOT17 benchmark without any appearance model at all. ## Where pose estimation meets tracking The most powerful applications combine both. Detect poses in each frame, track the skeletons across frames, and you get a continuous motion signal for each person: ``` Video frames -> Person detection -> Pose estimation -> Pose tracking -> Action analysis (YOLO, ep78-79) (HRNet/MediaPipe) (DeepSORT) (temporal model) ``` Each stage builds on what we've covered: detection (episodes #78-79), the CNN architectures that power pose estimation (#45-47), the U-Net-style encoder-decoder pattern (#80), and now tracking. The temporal model at the end might be an LSTM (episode #49), a transformer (episodes #52-53), or even a simple rule-based system ("if knee angle decreases below 90 degrees for more than 0.5 seconds -> classified as sitting down"). The applications are genuinely diverse: - **Sports analytics**: track player movements, analyze shooting form, measure stride length and cadence. The NBA uses optical tracking to generate the detailed player statistics you see in broadcasts. - **Physical therapy**: measure range of motion over time, detect compensation patterns (favoring one side), track rehabilitation progress objectively instead of relying on patient self-reporting. - **Sign language recognition**: track hand and body keypoints to decode signs. Each sign is a temporal sequence of poses, which maps naturally to sequence models from episodes #48-53. - **Action recognition**: classify the temporal sequence of poses -- walking, running, falling, fighting. Fall detection for elderly care is a major use case. - **Motion capture**: extract 3D body motion from 2D video. Much cheaper than marker-based mocap systems (which cost $50,000+), though less precise. ## Evaluation metrics for pose and tracking Pose estimation uses **OKS** (Object Keypoint Similarity) instead of IoU. OKS measures how close predicted keypoints are to ground truth, normalized by the object's scale and a per-keypoint constant that accounts for how precisely each keypoint can be annotated (the nose has a precise location; the hip center is ambiguous): ```python import numpy as np # COCO per-keypoint sigmas (annotation noise) COCO_SIGMAS = np.array([ 0.026, 0.025, 0.025, # nose, eyes 0.035, 0.035, # ears 0.079, 0.079, # shoulders 0.072, 0.072, # elbows 0.062, 0.062, # wrists 0.107, 0.107, # hips 0.087, 0.087, # knees 0.089, 0.089, # ankles ]) def compute_oks(pred_kps, gt_kps, gt_area): """Object Keypoint Similarity. pred_kps, gt_kps: (17, 3) -- x, y, visibility gt_area: bounding box area of the person """ visible = gt_kps[:, 2] > 0 if visible.sum() == 0: return 0.0 dx = pred_kps[:, 0] - gt_kps[:, 0] dy = pred_kps[:, 1] - gt_kps[:, 1] dist_sq = dx ** 2 + dy ** 2 scale = 2 * (COCO_SIGMAS ** 2) * gt_area oks_per_kp = np.exp(-dist_sq / (scale + 1e-6)) return float(oks_per_kp[visible].mean()) # Test: perfect prediction gt = np.random.rand(17, 3) gt[:, 2] = 1.0 # all visible perfect_oks = compute_oks(gt, gt, gt_area=10000) print(f"Perfect OKS: {perfect_oks:.4f}") # 1.0 # Test: slightly noisy prediction noisy = gt.copy() noisy[:, :2] += np.random.randn(17, 2) * 3 noisy_oks = compute_oks(noisy, gt, gt_area=10000) print(f"Noisy OKS: {noisy_oks:.4f}") ``` The sigmas encode domain knowledge: shoulders and hips have high sigma (0.079, 0.107) because annotators disagree more about their exact center -- the shoulder is a broad area, not a precise point. The nose has low sigma (0.026) because everyone agrees on where the nose is. This normalization means a 5-pixel error on the hip is penalized less than a 5-pixel error on the nose, which correctly reflects annotation difficulty. Tracking uses **MOTA** (Multiple Object Tracking Accuracy) and **IDF1**. MOTA penalizes missed detections, false positives, and ID switches. IDF1 measures how well the tracker maintains consistent identities -- it's the harmonic mean of identification precision and identification recall across the whole video. A tracker that detects everyone but keeps swapping IDs will have decent MOTA but poor IDF1. ## Samengevat - **Pose estimation** detects body keypoints (joints) in images, outputting `(x, y, confidence)` per keypoint plus a skeleton structure -- going beyond detection (where) and segmentation (which pixels) to understand body structure; - **heatmap-based prediction** is the standard approach: for each keypoint, the model outputs a 2D Gaussian probability map and the predicted coordinate is the argmax -- much easier to learn than direct coordinate regression; - **top-down** approaches detect people first then estimate pose per crop (more accurate, runtime scales with number of people); **bottom-up** detects all keypoints first then groups them (faster for crowds, harder grouping); - **HRNet** maintains high-resolution representations throughout the network with parallel multi-scale branches and repeated fusion -- avoiding the information loss from standard downsample-then-upsample architectures; - **MediaPipe** runs real-time on CPU and mobile (33 body + 21 hand + 468 face keypoints); **MMPose/HRNet** offers higher accuracy for offline processing; - **SORT** tracks objects using Kalman filters + Hungarian matching; **DeepSORT** adds appearance features for re-identification after occlusion; **ByteTrack** recovers low-confidence detections instead of discarding them; - **OKS** (Object Keypoint Similarity) evaluates pose accuracy with per-keypoint normalization based on annotation difficulty; **MOTA** and **IDF1** evaluate tracking quality; - combining pose estimation with tracking enables action recognition, sports analytics, motion capture, physical therapy, and sign language recognition -- any application that needs to understand human movement over time. With pose estimation and tracking, we've now covered the full spectrum of visual understanding: what's in the image (classification), where it is (detection), which pixels belong to it (segmentation), and how the body is structured and moving (pose + tracking). There's another dimension of visual perception we haven't explored yet though -- understanding text and characters within images. Machines reading signage, extracting text from documents, recognizing handwriting. That's a distinct set of techniques with its own architectures and challenges. ### Exercises **Exercise 1:** Build a **pose analysis toolkit**. Create a class `PoseAnalyzer` that: (a) takes a `(17, 3)` COCO-format keypoint array and computes joint angles using the arctangent of limb vectors -- specifically left elbow angle (shoulder-elbow-wrist), right elbow angle, left knee angle (hip-knee-ankle), and right knee angle, (b) implements a `classify_pose` method that uses simple angle thresholds to classify the pose as "standing" (both knee angles > 160), "sitting" (at least one knee angle < 120), "arms_raised" (both wrists above shoulders), or "unknown", (c) implements a `symmetry_score` method that compares left vs right side joint angles and returns a 0-1 score where 1.0 means perfectly symmetric, (d) test with three synthetic keypoint arrays representing standing, sitting, and arms-raised poses, verifying correct classification for each. **Exercise 2:** Implement a **multi-object tracker with ID management**. Create a class `MultiObjectTracker` that: (a) maintains a dictionary of active tracks, each with an ID, current bounding box, velocity estimate (simple: difference between last two positions), age (frames since creation), and hits (number of successful matches), (b) on each `update(detections)` call, predicts new positions using the velocity estimate, builds an IoU cost matrix between predictions and detections, solves the assignment with `scipy.optimize.linear_sum_assignment`, updates matched tracks, creates new tracks for unmatched detections, and increments age for unmatched tracks, (c) implements track lifecycle: a track is "tentative" until it accumulates 3 hits, then "confirmed"; a track is deleted after 5 frames without a match, (d) generate a synthetic 20-frame video sequence with 3 objects moving in straight lines (one disappears for 3 frames then reappears) and print per-frame track states. Verify that confirmed tracks maintain consistent IDs and that the reappearing object gets a new ID (since the old track was deleted). **Exercise 3:** Build an **OKS-based pose evaluation framework**. Create a class `PoseEvaluator` that: (a) implements `compute_oks(pred_kps, gt_kps, gt_area)` using the full COCO sigma values (provided: nose=0.026, eyes=0.025, ears=0.035, shoulders=0.079, elbows=0.072, wrists=0.062, hips=0.107, knees=0.087, ankles=0.089), (b) implements `compute_ap(predictions, ground_truths, oks_threshold)` that sorts predictions by score, matches to ground truths using OKS >= threshold, computes precision-recall, and returns AP (area under the PR curve), (c) generates a synthetic evaluation set: 10 "images" each with 1-3 ground truth poses, plus three detector profiles -- "perfect" (exact GT keypoints, score=1.0), "good" (GT + 3px Gaussian noise, random scores 0.7-0.99), and "poor" (GT + 15px noise, 40% of poses missed, random false positives), (d) prints a comparison table showing AP at OKS thresholds 0.50, 0.75, and 0.50:0.05:0.95 (COCO-style AP) for each detector. Verify that the perfect detector achieves AP=1.0 and that the gap between "good" and "poor" is larger at OKS=0.75 than at OKS=0.50. ### De groeten! @scipio

GYNOID MONUMENT D01