Learn AI Series):Exercise 1: Multi-band spectral subtractor -- generate 3s of clean tonal audio (3 harmonics at 200/400/800 Hz), add broadband noise at SNR=5dB. Estimate noise spectrum from first 0.5s (noise-only region), then subtract per-band with oversubtraction alpha=2.0 and spectral floor beta=0.01. Result: SNR improves from 5dB to roughly 12-15dB. The spectral floor prevents negative magnitudes (which cause "musical noise" artifacts). Per-band analysis shows the most improvement in bands where the signal-to-noise ratio was already moderate (200-800 Hz region), while very high frequency bands (above 2kHz) see less improvement because they were mostly noise to begin with.
Exercise 2: Adaptive echo canceller -- generate a 3s "far-end" signal (sum of 3 sinusoids), convolve it with a synthetic room impulse response (10-tap FIR with exponential decay), add local "near-end" speech (different frequency), and run NLMS adaptive filter (step size mu=0.5, filter length=15). Track convergence by measuring echo-return loss enhancement (ERLE) per frame. Result: the filter converges within 0.5-1.0s, ERLE rises from 0dB to 15-25dB, and the residual error drops below -20dB. The adapted filter coefficients approximate the true room impulse response -- plotting both side by side shows the match improving over time, with the largest coefficient errors in the tail taps (smallest true values).
Exercise 3: Waveform-domain denoiser with learned encoder-decoder -- generate 2s clean speech (5 harmonics with amplitude envelope), add Gaussian noise at SNR=0dB. Build a simple encoder (3 conv1d layers with stride-2 downsampling + ReLU) and symmetric decoder (3 transposed conv1d layers). Train for 200 epochs on 20 noisy/clean pairs using L1 loss. Result: SNR improves from 0dB to 5-8dB after training. The learned encoder filters show bandpass-like patterns -- lower filters capture low-frequency harmonics, higher filters capture the noise floor. Comparing waveform-domain vs spectrogram-domain (from exercise 1): waveform models avoid phase reconstruction artifacts but need more data to converge; spectrogram models converge faster but introduce musical noise at the phase boundaries.
Here we go! Episode one hundred. One hundred. When this series started with "what machine learning actually is" I genuinely did not expect we'd make it this far, but here we are and the topic fits perfectly for the milestone: models that combine two of the most fundamental human senses -- seeing and hearing ;-)
Humans are naturally multimodal creatures. You can understand someone in a noisy bar partly because you're reading their lips (even if you don't realize you're doing it). You turn toward a sudden sound because your eyes and ears triangulate together. A silent video of someone clapping is incomplete -- your brain fills in the expected sound whether you want it to or not. Audio-visual AI tries to capture exactly this kind of integration, and it unlocks capabilities that neither modality achieves on its own.
We've been building toward this for quite some time. Episode #75 covered multimodal text+vision. Episodes #92-99 covered the full audio domain -- from fundamentals through speech recognition, generation, classification, speaker identification, NLU, and enhancement. Today we bring audio and vision together into unified systems. This is where things get really interesting.
Consider speech recognition in a crowded room. The audio signal is a mess -- multiple people talking, background music, glasses clinking. An audio-only ASR system (the kind we built in episode #93) struggles badly in these conditions. But a camera pointed at the speaker's face provides a completly independent signal channel. Lip movements are NOT affected by acoustic noise. Combining both modalities makes the system robust in situations where either modality alone would fail.
This is not theoretical hand-waving. The McGurk effect demonstrates how tightly humans fuse audio and visual speech perception: when you hear "ba" but see lips forming "ga," your brain perceives "da" -- a sound that was neither heard nor seen. Your brain fuses the two modalities into a single percept automatically. Audio-visual models aim for the same kind of integration.
Let's start by building the fundamental building block: a system that extracts features from both modalities and learns to combine them.
Before we get into specific tasks, let's build the core fusion mechansim that every audio-visual system needs -- taking features from two different modalities and combining them into a unified representation:
import numpy as np
class AudioVisualFuser:
"""Combine audio spectral features with visual
spatial features using attention-weighted fusion."""
def __init__(self, audio_dim=40, visual_dim=32,
fused_dim=64):
self.rng = np.random.RandomState(42)
self.audio_dim = audio_dim
self.visual_dim = visual_dim
self.fused_dim = fused_dim
# Projection matrices
self.W_audio = self.rng.randn(
audio_dim, fused_dim) * 0.1
self.W_visual = self.rng.randn(
visual_dim, fused_dim) * 0.1
# Attention weights
self.W_attn = self.rng.randn(fused_dim, 1) * 0.1
def project(self, audio_feat, visual_feat):
"""Project both modalities into shared space."""
a_proj = np.tanh(audio_feat @ self.W_audio)
v_proj = np.tanh(visual_feat @ self.W_visual)
return a_proj, v_proj
def attention_fuse(self, a_proj, v_proj):
"""Attention-weighted fusion."""
# Stack modalities: (2, n_frames, fused_dim)
stacked = np.stack([a_proj, v_proj], axis=0)
# Compute attention scores
scores = np.tanh(stacked @ self.W_attn).squeeze(-1)
# Softmax over the 2 modalities
exp_s = np.exp(scores - scores.max(axis=0))
weights = exp_s / exp_s.sum(axis=0, keepdims=True)
# Weighted combination
fused = weights[0:1].T * a_proj + weights[1:2].T * v_proj
return fused, weights
def run_demo(self):
n_frames = 20
audio = self.rng.randn(n_frames, self.audio_dim)
visual = self.rng.randn(n_frames, self.visual_dim)
# Simulate noisy audio in frames 5-10
audio[5:10] += self.rng.randn(5, self.audio_dim) * 3.0
a_proj, v_proj = self.project(audio, visual)
fused, weights = self.attention_fuse(a_proj, v_proj)
print("=== Audio-Visual Fusion ===")
print(f"Audio features: {audio.shape}")
print(f"Visual features: {visual.shape}")
print(f"Fused features: {fused.shape}")
print(f"\nAttention weights (audio, visual):")
print(f" Clean frames (0-4): audio={weights[0,:5].mean():.3f}"
f" visual={weights[1,:5].mean():.3f}")
print(f" Noisy frames (5-9): audio={weights[0,5:10].mean():.3f}"
f" visual={weights[1,5:10].mean():.3f}")
print(f" Clean frames (10-19): audio={weights[0,10:].mean():.3f}"
f" visual={weights[1,10:].mean():.3f}")
fuser = AudioVisualFuser()
fuser.run_demo()
The attention mechanism is the key idea here. When the audio signal is clean, the model can rely heavily on audio features (which carry more information than lip movements for speech). When the audio is noisy, the attention shifts toward visual features. This adaptive weighting is exactly what makes audio-visual systems more robust than either modality alone -- the model learns to dynamically trust the cleaner signal.
Audio-visual ASR (AV-ASR) combines a visual encoder (processing lip/face video) with an audio encoder (processing spectrograms), fusing their representations before decoding text. Let's build a simplified version that demonstrates the key insight -- multimodal robustness under noise:
import numpy as np
class SimpleAVASR:
"""Audio-visual speech recognizer demonstrating
noise robustness through multimodal fusion."""
def __init__(self, sr=16000, n_mels=40, visual_dim=32):
self.sr = sr
self.n_mels = n_mels
self.visual_dim = visual_dim
self.rng = np.random.RandomState(42)
def synthesize_av_speech(self, word, duration=0.5):
"""Generate paired audio-visual data for a word."""
n = int(self.sr * duration)
t = np.arange(n) / self.sr
word_freqs = {
"hello": [200, 400, 800],
"world": [150, 350, 700],
"stop": [300, 500, 900],
"go": [180, 450, 750],
}
freqs = word_freqs.get(word, [200, 400, 800])
audio = sum(
(0.3 / (i + 1)) * np.sin(2 * np.pi * f * t)
for i, f in enumerate(freqs)
)
audio += self.rng.randn(n) * 0.02
# Visual: simulate lip aperture over time
n_frames = 10
lip_features = np.zeros((n_frames, self.visual_dim))
for i in range(n_frames):
phase = i / n_frames
aperture = 0.5 + 0.3 * np.sin(
2 * np.pi * freqs[0] / 200 * phase)
lip_features[i, :8] = aperture + self.rng.randn(8) * 0.1
lip_features[i, 8:16] = freqs[0] / 1000 + self.rng.randn(8) * 0.05
lip_features[i, 16:] = self.rng.randn(self.visual_dim - 16) * 0.1
return audio, lip_features
def mel_spectrogram(self, audio):
"""Simplified mel spectrogram extraction."""
n_fft, hop = 512, 160
n_frames = max(1, (len(audio) - n_fft) // hop)
spec = np.zeros((self.n_mels, n_frames))
window = np.hanning(n_fft)
for i in range(n_frames):
s = i * hop
frame = audio[s:s + n_fft]
if len(frame) < n_fft:
break
fft_mag = np.abs(np.fft.rfft(frame * window))
mel_bins = np.linspace(0, len(fft_mag),
self.n_mels + 1, dtype=int)
for m in range(self.n_mels):
lo, hi = mel_bins[m], mel_bins[m + 1]
if hi > lo:
spec[m, i] = np.mean(fft_mag[lo:hi])
return np.log(spec + 1e-9)
def extract_av_features(self, mel, visual):
"""Extract combined audio-visual feature vector."""
audio_feat = np.concatenate([
np.mean(mel, axis=1), np.std(mel, axis=1)])
visual_feat = np.concatenate([
np.mean(visual, axis=0), np.std(visual, axis=0)])
return np.concatenate([audio_feat, visual_feat])
def run(self):
words = ["hello", "world", "stop", "go"]
# Build training set
X_train, y_train = [], []
for wi, word in enumerate(words):
for _ in range(15):
audio, visual = self.synthesize_av_speech(word)
mel = self.mel_spectrogram(audio)
feat = self.extract_av_features(mel, visual)
X_train.append(feat)
y_train.append(wi)
X = np.array(X_train)
y = np.array(y_train)
mean, std = X.mean(axis=0), X.std(axis=0) + 1e-9
X_n = (X - mean) / std
# Train softmax classifier
W = self.rng.randn(X_n.shape[1], len(words)) * 0.01
b = np.zeros(len(words))
for epoch in range(300):
logits = X_n @ W + b
exp_l = np.exp(logits - logits.max(axis=1, keepdims=True))
probs = exp_l / exp_l.sum(axis=1, keepdims=True)
targets = np.zeros_like(probs)
targets[np.arange(len(y)), y] = 1.0
grad = probs - targets
W -= 0.1 * (X_n.T @ grad) / len(y)
b -= 0.1 * grad.mean(axis=0)
# Test across noise levels
print("=== AV-ASR Noise Robustness ===")
for noise in [0.0, 0.5, 2.0]:
av_ok = ao_ok = total = 0
for wi, word in enumerate(words):
for _ in range(5):
audio, visual = self.synthesize_av_speech(word)
audio += self.rng.randn(len(audio)) * noise
mel = self.mel_spectrogram(audio)
av_feat = self.extract_av_features(mel, visual)
av_n = (av_feat - mean) / std
if (av_n @ W + b).argmax() == wi:
av_ok += 1
# Audio-only: zero out visual features
a_only = np.zeros_like(av_feat)
a_only[:self.n_mels * 2] = av_feat[:self.n_mels * 2]
ao_n = (a_only - mean) / std
if (ao_n @ W + b).argmax() == wi:
ao_ok += 1
total += 1
print(f" Noise={noise:.1f}: "
f"AV={av_ok/total:.0%} Audio-only={ao_ok/total:.0%}")
asr = SimpleAVASR()
asr.run()
The comparison at different noise levels tells the whole story: when audio is clean, both approaches perform similarly. But as noise increases, the audio-visual system degrades much more gracefully because the visual channel remains unaffected by acoustic noise. This is exactly why production AV-ASR systems like AV-HuBERT (Meta) significantly outperform audio-only Whisper in noisy conditions.
Pure lip reading -- transcribing speech from video alone, with no audio at all -- is one of the hardest problems in multimodal AI. Humans are surprisingly bad at it (only about 10-30% of English phonemes are visually distinguishable -- "p", "b", and "m" all look identical on the lips). Yet modern models achieve remarkable accuracy on benchmark datasets.
The key insight is that even though individual phonemes are ambiguous, the temporal sequence of lip movements combined with language model priors can disambiguate quit a lot. A sequence of lip shapes that could be "bat", "pat", or "mat" gets resolved by context -- if the previous word was "rubber" then "mat" is far more likely ;-)
The LRS2 and LRS3 datasets (from BBC and TED talks) are the standard benchmarks. State-of-the-art lip reading achieves around 20-25% WER on LRS3 -- not as good as audio ASR, but impressive considering the inherent ambiguity of visual speech. The typical architecture uses a 3D CNN frontend (to capture short-term spatiotemporal motion) followed by a transformer temporal model and CTC loss for text prediction.
You're at a party. Multiple conversations happening simultaneously. Yet you can focus on one speaker and tune out the rest. How? Your brain uses both audio cues (pitch, timbre, spatial location) and visual cues (who's moving their lips, who's facing you) to separate the target speaker from the mixture.
Audio-visual source separation does the same thing. Given a video of multiple speakers and a mixed audio signal, separate each speaker's voice. Let's build a simplified version:
import numpy as np
class CocktailPartySolver:
"""Separate speakers using visual identity
as a guidance signal for spectral masking."""
def __init__(self, sr=16000):
self.sr = sr
self.rng = np.random.RandomState(42)
def synthesize_speaker(self, f0, formants, duration=3.0):
"""Generate speaker audio with distinct vocal identity."""
n = int(self.sr * duration)
t = np.arange(n) / self.sr
signal = sum(a * np.sin(2 * np.pi * f * t) for f, a in formants)
for h in range(2, 5):
signal += (0.2 / h) * np.sin(2 * np.pi * f0 * h * t)
signal *= 0.5 + 0.5 * np.sin(2 * np.pi * 3 * t)
return signal
def visual_identity(self, speaker_id):
"""Simulate face embedding for a speaker."""
self.rng.seed(speaker_id * 100)
emb = self.rng.randn(32)
return emb / np.linalg.norm(emb)
def spectrogram(self, audio):
"""Simple magnitude spectrogram."""
n_fft, hop = 512, 256
n_frames = (len(audio) - n_fft) // hop
spec = np.zeros((n_fft // 2 + 1, n_frames))
window = np.hanning(n_fft)
for i in range(n_frames):
s = i * hop
spec[:, i] = np.abs(np.fft.rfft(audio[s:s + n_fft] * window))
return spec
def spectral_mask(self, mixed_spec, target_spec):
"""Compute ideal ratio mask for target speaker."""
mask = np.zeros_like(mixed_spec)
mask[target_spec > mixed_spec * 0.3] = 1.0
kernel = np.ones(3) / 3
for i in range(mask.shape[0]):
mask[i] = np.convolve(mask[i], kernel, mode='same')
return np.clip(mask, 0, 1)
def run(self):
spk_a = self.synthesize_speaker(
f0=120, formants=[(500, 0.4), (1000, 0.3), (2200, 0.15)])
spk_b = self.synthesize_speaker(
f0=220, formants=[(800, 0.35), (1300, 0.25), (2700, 0.12)])
min_len = min(len(spk_a), len(spk_b))
mixed = spk_a[:min_len] + spk_b[:min_len]
spec_a = self.spectrogram(spk_a[:min_len])
spec_b = self.spectrogram(spk_b[:min_len])
spec_mix = self.spectrogram(mixed)
face_a = self.visual_identity(0)
face_b = self.visual_identity(1)
mask_a = self.spectral_mask(spec_mix, spec_a)
separated_a = spec_mix * mask_a
separated_b = spec_mix * (1 - mask_a)
def sdr_proxy(clean, estimate):
sig = np.sum(clean ** 2)
noise = np.sum((clean - estimate) ** 2)
return 10 * np.log10(sig / (noise + 1e-10))
print("=== Cocktail Party Separation ===")
print(f"Speaker A f0=120Hz, Speaker B f0=220Hz")
print(f"Mixed spectrogram: {spec_mix.shape}")
print(f"\nSeparation quality:")
print(f" Speaker A SDR: {sdr_proxy(spec_a, separated_a):.1f} dB")
print(f" Speaker B SDR: {sdr_proxy(spec_b, separated_b):.1f} dB")
print(f"\nVisual identity similarity:")
print(f" face_A . face_A: {np.dot(face_a, face_a):.3f}")
print(f" face_A . face_B: {np.dot(face_a, face_b):.3f}")
solver = CocktailPartySolver()
solver.run()
The Looking to Listen paper (Google, 2018) demonstrated this convincingly in a real system: show the model a video of two people talking simultaneously, and it separates each person's voice based on their face. The key insight is that lip movements are temporally correlated with the corresponding speaker's audio -- the model learns this correspondence during training. Having said that, the oracle mask we use above represents the ideal case; in practice the model has to predict the mask from the visual identity embedding, which is harder but still remarkably effective.
Given a video, where is the sound coming from? Is it the guitar on the left, the drums on the right, or the singer in the middle? Sound source localization identifies which visual regions correspond to the audio:
import numpy as np
class SoundLocalizer:
"""Locate sound sources in an image using
audio-visual feature correspondence."""
def __init__(self, grid_h=7, grid_w=7):
self.grid_h = grid_h
self.grid_w = grid_w
self.rng = np.random.RandomState(42)
self.W_audio = self.rng.randn(40, 32) * 0.1
def synthesize_scene(self):
"""Create a scene with 3 objects, one sounding."""
visual_map = self.rng.randn(self.grid_h, self.grid_w, 32) * 0.1
# Object 1: guitar (rows 2-4, cols 0-2)
guitar_feat = self.rng.randn(32) * 0.5
guitar_feat[0:8] = 1.0
for r in range(2, 5):
for c in range(0, 3):
visual_map[r, c] = guitar_feat + self.rng.randn(32) * 0.05
# Object 2: drums (rows 1-3, cols 4-6)
drum_feat = self.rng.randn(32) * 0.5
drum_feat[8:16] = 1.0
for r in range(1, 4):
for c in range(4, 7):
visual_map[r, c] = drum_feat + self.rng.randn(32) * 0.05
# Object 3: person (rows 4-6, cols 2-4)
person_feat = self.rng.randn(32) * 0.5
person_feat[16:24] = 1.0
for r in range(4, 7):
for c in range(2, 5):
visual_map[r, c] = person_feat + self.rng.randn(32) * 0.05
# Audio signal matches guitar
audio_feat = self.rng.randn(40) * 0.1
audio_feat[:8] = 1.0
return visual_map, audio_feat
def localize(self, visual_map, audio_feat):
"""Compute spatial attention heatmap."""
audio_proj = np.tanh(audio_feat @ self.W_audio)
audio_proj /= np.linalg.norm(audio_proj) + 1e-9
heatmap = np.zeros((self.grid_h, self.grid_w))
for r in range(self.grid_h):
for c in range(self.grid_w):
v = visual_map[r, c]
v_norm = v / (np.linalg.norm(v) + 1e-9)
heatmap[r, c] = np.dot(audio_proj, v_norm)
heatmap = (heatmap - heatmap.min()) / (heatmap.max() - heatmap.min() + 1e-9)
return heatmap
def run(self):
visual_map, audio_feat = self.synthesize_scene()
heatmap = self.localize(visual_map, audio_feat)
print("=== Sound Localization ===")
print("Heatmap (7x7 grid):")
print(" Guitar=rows 2-4 cols 0-2 Drums=rows 1-3 cols 4-6")
print(" Person=rows 4-6 cols 2-4 Audio=guitar sound\n")
for r in range(self.grid_h):
row_str = " "
for c in range(self.grid_w):
v = heatmap[r, c]
row_str += " ## " if v > 0.7 else (" ++ " if v > 0.4 else " .. ")
print(row_str)
print(f"\nGuitar region avg: {heatmap[2:5, 0:3].mean():.3f}")
print(f"Drums region avg: {heatmap[1:4, 4:7].mean():.3f}")
print(f"Person region avg: {heatmap[4:7, 2:5].mean():.3f}")
localizer = SoundLocalizer()
localizer.run()
The training trick in real systems: you don't need manual annotations of "the guitar is at pixel (234, 156)." Instead you use contrastive learning -- matching video frames with their corresponding audio (positive pairs) and mismatched audio (negative pairs). The model learns that the sounding object's visual features correlate with the audio features, and localization emerges as a natural byproduct. No expensive annotation required ;-)
The reverse problem: given a silent video, generate appropriate audio. A video of rain should produce rain sounds. A video of someone typing should produce keyboard clicks. A video of a dog barking should produce... well, a bark.
This combines video understanding (episode #83) with audio generation (episodes #94 and #96). The visual encoder extracts frame-level features, and an audio decoder generates a spectrogram conditioned on those features:
import numpy as np
class VideoToAudioGenerator:
"""Generate audio spectrograms from visual frame features."""
def __init__(self, n_mels=40, visual_dim=32, hidden_dim=64):
self.n_mels = n_mels
self.rng = np.random.RandomState(42)
self.W_enc = self.rng.randn(visual_dim, hidden_dim) * 0.1
self.b_enc = np.zeros(hidden_dim)
self.W_dec = self.rng.randn(hidden_dim, n_mels) * 0.1
self.b_dec = np.zeros(n_mels)
def simulate_video(self, event_type, n_frames=20):
"""Simulate visual features for sound-producing events."""
features = np.zeros((n_frames, 32))
if event_type == "clapping":
for i in range(n_frames):
phase = np.sin(2 * np.pi * 3 * i / n_frames)
features[i, :8] = phase * 0.8 + self.rng.randn(8) * 0.1
features[i, 8:16] = 0.3
elif event_type == "drumming":
for i in range(n_frames):
phase = np.sin(2 * np.pi * 5 * i / n_frames)
features[i, :8] = 0.2
features[i, 8:16] = abs(phase) * 0.9 + self.rng.randn(8) * 0.1
elif event_type == "waving":
for i in range(n_frames):
phase = np.sin(2 * np.pi * 2 * i / n_frames)
features[i, :8] = phase * 0.4
features[i, 16:24] = 0.5
return features
def generate_mel(self, visual_feats):
"""Predict mel spectrogram from visual features."""
hidden = np.tanh(visual_feats @ self.W_enc + self.b_enc)
return hidden @ self.W_dec + self.b_dec
def run(self):
events = ["clapping", "drumming", "waving"]
print("=== Video-to-Audio Generation ===")
mels = {}
for event in events:
visual = self.simulate_video(event)
mel = self.generate_mel(visual)
energy = np.mean(mel ** 2, axis=1)
mels[event] = mel
print(f"\n {event}: mel={mel.shape}"
f" mean_energy={np.mean(energy):.4f}"
f" peak_frame={energy.argmax()}")
print("\nCross-event mel similarity:")
for i, e1 in enumerate(events):
for e2 in events[i+1:]:
m1, m2 = mels[e1].flatten(), mels[e2].flatten()
cos = np.dot(m1, m2) / (np.linalg.norm(m1) * np.linalg.norm(m2) + 1e-9)
print(f" {e1} vs {e2}: {cos:.3f}")
gen = VideoToAudioGenerator()
gen.run()
Recent models like SpecVQGAN and Diff-Foley use codebook-based or diffusion-based generation rather than direct mel prediction, producing much higher quality output. The VGGSound dataset (300K+ video clips with audio labels) is the standard benchmark. None the less, the principle remains the same: extract temporal visual features, condition an audio generator on them, and produce audio that matches the visual content.
In episode #98 we recognized emotion from speech alone using prosodic features. Adding the visual channel -- facial expressions -- creates a much richer signal. The face shows micro-expressions that the voice might mask, and vocal tone reveals emotion that a poker face conceals:
import numpy as np
class AVEmotionSystem:
"""Multimodal emotion recognition
from face + audio features."""
def __init__(self):
self.emotions = ["neutral", "happy", "angry", "sad"]
self.rng = np.random.RandomState(42)
def synthesize_face_features(self, emotion, n_samples=15):
"""Simulate facial action unit features per emotion."""
profiles = {
"neutral": np.array([0.1, 0.1, 0.0, 0.0, 0.0, 0.0, 0.1, 0.0]),
"happy": np.array([0.0, 0.8, 0.0, 0.0, 0.7, 0.6, 0.2, 0.0]),
"angry": np.array([0.7, 0.0, 0.8, 0.6, 0.0, 0.0, 0.0, 0.5]),
"sad": np.array([0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.6, 0.7]),
}
base = profiles[emotion]
return np.array([
np.clip(base + self.rng.randn(8) * 0.15, 0, 1)
for _ in range(n_samples)])
def synthesize_audio_features(self, emotion, n_samples=15):
"""Simulate prosodic features per emotion."""
profiles = {
"neutral": np.array([150, 10, 0.3, 0.05, 1.0, 0.1]),
"happy": np.array([200, 30, 0.5, 0.1, 1.3, 0.3]),
"angry": np.array([170, 20, 0.6, 0.15, 1.1, 0.25]),
"sad": np.array([120, 8, 0.15, 0.03, 0.7, 0.05]),
}
base = profiles[emotion]
return np.array([
base + self.rng.randn(6) * base * 0.1
for _ in range(n_samples)])
def run(self):
X_face, X_audio, y = [], [], []
for ei, emo in enumerate(self.emotions):
faces = self.synthesize_face_features(emo)
audios = self.synthesize_audio_features(emo)
for i in range(len(faces)):
X_face.append(faces[i])
X_audio.append(audios[i])
y.append(ei)
X_face = np.array(X_face)
X_audio = np.array(X_audio)
y = np.array(y)
f_mean, f_std = X_face.mean(0), X_face.std(0) + 1e-9
a_mean, a_std = X_audio.mean(0), X_audio.std(0) + 1e-9
Xf_n = (X_face - f_mean) / f_std
Xa_n = (X_audio - a_mean) / a_std
X_combined = np.hstack([Xf_n, Xa_n])
# kNN evaluation (k=5, leave-one-out)
print("=== AV Emotion Recognition ===")
for name, X_data in [("Face only", Xf_n), ("Audio only", Xa_n),
("Audio+Visual", X_combined)]:
correct = 0
for i in range(len(X_data)):
dists = np.linalg.norm(
np.delete(X_data, i, axis=0) - X_data[i], axis=1)
y_rem = np.delete(y, i)
pred = int(np.bincount(y_rem[np.argsort(dists)[:5]]).argmax())
if pred == y[i]:
correct += 1
print(f" {name}: {correct / len(y):.1%}")
# Sarcasm: happy words, angry voice, neutral face
print("\n--- Sarcasm Scenario ---")
sarcasm_face = np.array([0.1, 0.2, 0.1, 0.1, 0.1, 0.1, 0.1, 0.0])
sarcasm_audio = np.array([170, 20, 0.6, 0.15, 1.1, 0.25])
sf_n = (sarcasm_face - f_mean) / f_std
sa_n = (sarcasm_audio - a_mean) / a_std
for name, feat, X_ref in [
("Face", sf_n, Xf_n), ("Audio", sa_n, Xa_n),
("Combined", np.concatenate([sf_n, sa_n]), X_combined),
]:
dists = np.linalg.norm(X_ref - feat, axis=1)
pred = int(np.bincount(y[np.argsort(dists)[:5]]).argmax())
print(f" {name} predicts: {self.emotions[pred]}")
system = AVEmotionSystem()
system.run()
On the IEMOCAP dataset, audio-visual models consistently outperform single-modality approaches by 5-10 percentage points. The gain is largest for emotions that are ambiguous in one modality but clear in the other -- sarcasm being the classic example (happy words, angry tone, flat face). A fused model catches what neither modality detects alone, which is exactly why the combined system scores higher in our demo too.
One of the most powerful ideas in audio-visual AI: use the natural correspondence between audio and video as free supervision. A video of a dog barking inherently contains aligned audio-visual information -- no human annotation needed.
Audio-Visual Correspondence (AVC) training is beautifully simple: given a video frame and an audio clip, predict whether they come from the same video or not. This binary classification task teaches the model about audio-visual semantics without any labels:
import numpy as np
class AVCPretrainer:
"""Self-supervised audio-visual correspondence:
does this audio match this video frame?"""
def __init__(self, embed_dim=32):
self.embed_dim = embed_dim
self.rng = np.random.RandomState(42)
self.W_visual = self.rng.randn(16, embed_dim) * 0.1
self.W_audio = self.rng.randn(16, embed_dim) * 0.1
def synthesize_pair(self, label, scene_type):
"""Generate matched (label=1) or mismatched (label=0)
audio-visual pair for a scene."""
scenes = {
"dog": {"visual": np.array([1,0,0,0,0.8,0.5,0,0,0.3,0,0,0,0,0,0,0]),
"audio": np.array([1,0,0,0,0.7,0.6,0,0,0.2,0,0,0,0,0,0,0])},
"guitar": {"visual": np.array([0,1,0,0,0,0,0.9,0.4,0,0,0.5,0,0,0,0,0]),
"audio": np.array([0,1,0,0,0,0,0.8,0.5,0,0,0.6,0,0,0,0,0])},
"rain": {"visual": np.array([0,0,1,0,0,0,0,0,0,0.7,0,0.8,0,0,0,0]),
"audio": np.array([0,0,1,0,0,0,0,0,0,0.6,0,0.7,0,0,0,0])},
}
scene = scenes[scene_type]
visual = scene["visual"] + self.rng.randn(16) * 0.1
if label == 1:
audio = scene["audio"] + self.rng.randn(16) * 0.1
else:
other = self.rng.choice([s for s in scenes if s != scene_type])
audio = scenes[other]["audio"] + self.rng.randn(16) * 0.1
return visual, audio
def train(self, n_epochs=200, lr=0.05):
scene_types = ["dog", "guitar", "rain"]
X_v, X_a, Y = [], [], []
for _ in range(30):
for scene in scene_types:
for label in [1, 0]:
v, a = self.synthesize_pair(label, scene)
X_v.append(v)
X_a.append(a)
Y.append(float(label))
X_v, X_a, Y = np.array(X_v), np.array(X_a), np.array(Y)
for epoch in range(n_epochs):
v_emb = np.tanh(X_v @ self.W_visual)
a_emb = np.tanh(X_a @ self.W_audio)
v_norm = v_emb / (np.linalg.norm(v_emb, axis=1, keepdims=True) + 1e-9)
a_norm = a_emb / (np.linalg.norm(a_emb, axis=1, keepdims=True) + 1e-9)
sim = np.sum(v_norm * a_norm, axis=1)
pred = 1 / (1 + np.exp(-sim * 5))
loss = -np.mean(Y * np.log(pred + 1e-9) + (1 - Y) * np.log(1 - pred + 1e-9))
if (epoch + 1) % 50 == 0:
acc = np.mean((pred > 0.5) == Y)
print(f"Epoch {epoch+1}: loss={loss:.3f} acc={acc:.1%}")
# Zero-shot retrieval demo
print("\n--- Zero-shot Audio-Visual Retrieval ---")
for scene in scene_types:
v, _ = self.synthesize_pair(1, scene)
v_e = np.tanh(v @ self.W_visual)
v_e /= np.linalg.norm(v_e) + 1e-9
print(f" Query: {scene} (visual)")
for s2 in scene_types:
_, a2 = self.synthesize_pair(1, s2)
a_e = np.tanh(a2 @ self.W_audio)
a_e /= np.linalg.norm(a_e) + 1e-9
cos = float(np.dot(v_e, a_e))
match = " <--" if s2 == scene else ""
print(f" vs {s2} audio: {cos:.3f}{match}")
trainer = AVCPretrainer()
trainer.train()
ImageBind (Meta, 2023) takes this concept to its logical extreme: it learns a joint embedding space across six modalities (images, text, audio, depth, thermal, IMU sensor data) using only image-text pairs as the anchor. Because images naturally co-occur with all other modalities, the image embedding becomes a bridge -- you can match audio to text even though the model was never explicitly trained on audio-text pairs. The audio of a barking dog maps to the same region as the image of a dog and the text "dog barking." That's a remarkably elegant solution to the multimodal alignment problem.
Let's build proper evaluation for multimodal systems. The core question is always: does combining modalities actually help compared to using each modality alone?
import numpy as np
class AVEvaluator:
"""Evaluate multimodal systems vs unimodal baselines."""
def __init__(self):
self.rng = np.random.RandomState(42)
def modality_gap_analysis(self):
"""Compare modalities across different conditions."""
conditions = {
"Clean": (0.92, 0.75, 0.95),
"Noisy audio": (0.60, 0.75, 0.82),
"Occluded face": (0.90, 0.45, 0.88),
"Both degraded": (0.55, 0.40, 0.68),
}
print("=== Modality Gap Analysis ===")
print(f"{'Condition':<18}{'Audio':>8}{'Visual':>8}{'AV':>8}{'Gain':>8}")
print("-" * 50)
for name, (a, v, av) in conditions.items():
gain = av - max(a, v)
print(f"{name:<18}{a:>7.0%}{v:>7.0%}{av:>7.0%}{gain:>+7.0%}")
def complementarity_score(self):
"""Measure how much modalities complement each other."""
n = 200
audio_ok = self.rng.random(n) > 0.15
visual_ok = self.rng.random(n) > 0.30
av_ok = self.rng.random(n) > 0.08
neither = ~audio_ok & ~visual_ok
rescued = av_ok & neither
print("\n=== Complementarity ===")
print(f"Audio correct: {audio_ok.mean():.1%}")
print(f"Visual correct: {visual_ok.mean():.1%}")
print(f"Both correct: {(audio_ok & visual_ok).mean():.1%}")
print(f"Audio-only right: {(audio_ok & ~visual_ok).mean():.1%}")
print(f"Visual-only right: {(~audio_ok & visual_ok).mean():.1%}")
print(f"AV rescued: {rescued.sum()}/{neither.sum()}")
def run(self):
self.modality_gap_analysis()
self.complementarity_score()
evaluator = AVEvaluator()
evaluator.run()
The modality gap analysis table is the most important table in any multimodal paper. It shows the accuracy of each modality alone versus the combined system across different conditions. The "Gain" column shows improvement over the best single modality. If the gain is zero or negative, your fusion mechanism is not working -- the combined system should always outperform the best individual modality, especially under degraded conditions.
We've covered a lot of ground in the audio domain over the last nine episodes -- from the raw physics of sound waves through speech recognition, synthesis, classification, speaker identification, voice NLU, enhancement, and now multimodal integration. The next milestone in this series is bringing all of that together into a practical project, and then we move into a completely different area of AI that involves learning through trial, error, and reward rather than labeled data. There's plenty more ground ahead.
Exercise 1: Build an audio-visual noise robustness analyzer. Create a class AVNoiseAnalyzer that generates 4 "words" (4 distinct frequency profiles, 0.5s each at sr=16000), with both audio (sinusoidal harmonics) and visual features (8-dim lip aperture patterns). Train a softmax classifier on combined AV features (20 samples per word). Then test at 5 noise levels (sigma = 0.0, 0.3, 0.6, 1.0, 2.0) applied ONLY to the audio channel. For each noise level, compute and print accuracy for: (a) audio-only, (b) visual-only, (c) audio+visual combined. Show the crossover point where audio-only drops below visual-only accuracy.
Exercise 2: Build an audio-visual synchrony detector. Create a class AVSyncDetector that generates 5s of paired audio-visual data (sr=16000, 25 visual frames/sec): audio is a series of 10 "impacts" (short bursts at random times), visual is a brightness spike at each impact time. Create both synchronized (audio and visual align) and desynchronized versions (visual shifted by 0.2s). Extract per-frame cross-correlation features between audio energy and visual brightness within a +/-0.3s window. Train a binary classifier (synchronized vs not). Print accuracy, and show the average cross-correlation peak position for sync vs desync pairs.
Exercise 3: Build an audio-visual retrieval system. Create a class AVRetriever with 5 "scene" categories (dog_barking, guitar_playing, rain, typing, clapping), each with distinct audio features (6-dim: spectral centroid, bandwidth, energy, zero-crossing rate, pitch, harmonicity) and visual features (6-dim: motion magnitude, color histogram peaks, edge density, texture roughness, brightness variance, spatial frequency). Generate 10 samples per scene. Project both modalities into a shared 8-dim embedding space using learned projection matrices (train with contrastive loss: matched pairs should have high cosine similarity, mismatched pairs low). Evaluate with recall@1: given an audio query, retrieve the nearest visual match (and vice versa). Print the 5x5 retrieval confusion matrix and overall recall@1 for both directions.
@scipio