I left you last time with a thought that's been buzzing round my head ever since: everything we've built so far, the visuals mostly follow the sound. Something goes loud, a shape gets big. The music kicks, the screen flashes. It works, but it's always one step behind, reacting after the fact, like a nod of the head to a beat. What I want today is something deeper - visuals that don't just follow the audio but seem to be the audio, made visible. Allez, this is the episode where we stop reacting to how loud the sound is and start listening to what it actually is :-).
Way back in episode 19 we did our first sound-reactive sketch, and it was honestly a bit of a party trick: grab the volume, scale a circle by it, done. Lovely for five minutes. But here's the thing that bugged me even then - a whole song, all its richness, its brightness, its melody, its rhythm, and we squashed it down to one number. One number! No wonder everything ended up pulsing the same dumb way. Today we go and get the other numbers. We're going to pull real musical features out of a sound - its brightness, the moments notes begin, the pulse of its beat, even which note is playing - and map each one to something different on screen. By the end you'll have a visualizer that feels like it genuinly understands the music, because in a small way it does.
Quick refresher on where episode 19 left us, because everything today grows out of it. The browser gives us an AnalyserNode that runs an FFT (a Fast Fourier Transform - the maths that splits a sound into its frequencies, which we met properly building synths back in episode 119). Out of it we can pull two things: the raw waveform, and the spectrum - how much energy sits at each frequency. That spectrum is the goldmine. Episode 19 barely scratched it.
// the setup from episode 19, our starting point.
const audioCtx = new AudioContext();
const analyser = audioCtx.createAnalyser();
analyser.fftSize = 2048; // -> 1024 frequency bins
const bins = analyser.frequencyBinCount; // = fftSize / 2 = 1024
// spectrum[i] = energy (0-255) at bin i's frequency. THIS is what we mine today.
const spectrum = new Uint8Array(bins);
function readSpectrum() {
analyser.getByteFrequencyData(spectrum); // fills the array each frame
}
Each slot in that spectrum array is a bin - a little frequency band - and the value (0 to 255) is how much energy is in it right now. Bin 0 is the lowest frequencies, the last bin the highest. To know what actual pitch a bin covers, you spread the sample rate across the bins:
// which real-world frequency (Hz) does bin i represent?
// the spectrum covers 0 Hz up to sampleRate/2 (the Nyquist limit).
function binToHz(i) {
return (i * audioCtx.sampleRate) / analyser.fftSize; // e.g. bin 10 ~ 215 Hz at 44.1kHz
}
Right, that's the raw material. Now instead of just summing it all into "loudness", let's ask the spectrum smarter questions. Each question is a feature, and each feature is going to drive a different part of the picture.
Even loudness we can do better than episode 19. There I just grabbed a peak value, which jitters like mad. What you actually want is RMS - root-mean-square - which is the energy of the whole frame, the thing your ear more or less hears as "how loud". Square every value, average them, take the square root. It's smoother and it means something.
// RMS = the "felt" loudness of the whole frame. smoother than peak.
function loudness(spectrum) {
let sum = 0;
for (let i = 0; i < spectrum.length; i++) {
const v = spectrum[i] / 255; // normalise 0..1
sum += v * v; // square (energy is amplitude squared)
}
return Math.sqrt(sum / spectrum.length); // mean, then root
}
One number still, but a good number. We'll use it for the overall energy of the scene - how much is happening, how bright and busy everything feels. But it's the floor, not the ceiling. Let's climb.
Here's my favourite, because it maps so beautifully to something visual. The spectral centroid is the "centre of mass" of the spectrum - the frequency around which the energy balances. If most of the energy is down low (a bass drum, a warm pad) the centroid is low, and we call that sound dark or warm. If there's lots of high-frequency energy (a cymbal, a hiss, a bright synth) the centroid is high, and we call it bright. Musicians literally use the word "bright", and now you can measure it.
// spectral centroid = energy-weighted average frequency = perceived "brightness".
// it's a weighted mean: each bin's frequency, weighted by how loud that bin is.
function centroid(spectrum) {
let weightedSum = 0, total = 0;
for (let i = 0; i < spectrum.length; i++) {
const energy = spectrum[i];
weightedSum += i * energy; // bin index weighted by its energy
total += energy;
}
if (total === 0) return 0;
const centroidBin = weightedSum / total; // the balance point, in bins
return binToHz(centroidBin); // convert to Hz for meaning
}
That formula is exactly the same "centre of mass" idea you'd use to balance weights on a ruler - sum of (position times weight) over sum of weights. It's the average bin, but weighted so loud bins pull the balance point toward them. And once you've got a brightness number, the mapping practically writes itself: bright sound, cool bright colour; dark sound, deep warm colour. Map the centroid straight onto hue.
// map brightness -> colour. dark sound = warm/red, bright sound = cool/blue.
// centroid roughly lives in 0..8000 Hz for music; clamp and remap to a hue.
function brightnessToHue(centroidHz) {
const b = Math.min(centroidHz / 6000, 1); // 0 (dark) .. 1 (very bright)
// 20 = warm red-orange, 220 = cool blue. lerp between them.
return 20 + b * 200; // remember lerp from episode 16!
}
Now the colour of your whole sketch shifts with the timbre of the music, not its volume. A muddy bass section glows deep amber, a shimmery high passage turns icy blue, and it happens all on its own. The first time I wired this up I just sat there feeding it different songs, grinning. It's such a direct little bridge between an ear-thing and an eye-thing. See where this is going? Each feature is a different sense of the sound, and we're painting each one separately.
Loudness and brightness are continuous - they drift up and down. But so much of music is events: the exact instant a drum hits, a note plucks, a word is sung. Catching those moments is called onset detection, and it lets us trigger discrete visual events - a flash, a particle burst, a shape spawning - precisely when something happens in the sound, instead of just smoothly following it.
The trick is spectral flux: measure how much the spectrum changed since the last frame. A steady held note barely changes frame to frame, so low flux. But the instant a new sound starts, a whole bunch of bins suddenly jump up together - a big spike in flux. We only count bins that got louder (that's the "half-wave rectified" bit), because a note starting is what we care about, not one fading.
// spectral flux = how much the spectrum GREW since last frame. spikes on note starts.
let prevSpectrum = new Uint8Array(analyser.frequencyBinCount);
function flux(spectrum) {
let sum = 0;
for (let i = 0; i < spectrum.length; i++) {
const diff = spectrum[i] - prevSpectrum[i]; // change in this bin
if (diff > 0) sum += diff; // only count INCREASES (onsets)
}
prevSpectrum.set(spectrum); // remember for next frame
return sum / spectrum.length; // average rise across bins
}
Now, a raw flux value on its own is noisy - it wiggles above zero constantly. To decide "yes, that's a real onset", you compare it to a running average of recent flux and only fire when it pokes well above the local norm. Adaptive thresholds like this are everywhere in signal work, because music gets louder and quieter and a fixed cutoff would never keep up.
// only call it an ONSET when flux jumps well above its own recent average.
let fluxHistory = [];
function isOnset(currentFlux) {
fluxHistory.push(currentFlux);
if (fluxHistory.length > 43) fluxHistory.shift(); // ~1 sec of history at 43fps
const avg = fluxHistory.reduce((a, b) => a + b, 0) / fluxHistory.length;
const threshold = avg * 1.5; // must beat 150% of recent avg
return currentFlux > threshold && currentFlux > 0.5; // and clear a noise floor
}
And now the payoff - when an onset fires, spawn something. This is where all our particle work (episodes 11 onward) comes flooding back. Every detected onset throws a burst of particles onto the canvas, coloured by the current brightness, so the timing comes from onsets and the colour comes from the centroid. Two features, two roles.
// on each detected onset, spawn a burst. timing from onsets, colour from brightness.
const particles = [];
function onOnset(hue) {
for (let n = 0; n < 30; n++) {
const angle = Math.random() * Math.PI * 2; // trig from episode 13!
const speed = 2 + Math.random() * 4;
particles.push({
x: width / 2, y: height / 2,
vx: Math.cos(angle) * speed,
vy: Math.sin(angle) * speed,
life: 1, hue,
});
}
}
Suddenly your sketch has rhythm. Things don't just breathe with the volume, they pop on the hits. That difference - continuous following versus discrete triggering - is honestly the single biggest step up from episode 19, and it's the thing that makes a visualizer feel alive rather than sleepy.
Onsets catch every little event, but often you want just the big regular pulse - the beat you'd tap your foot to. Full beat-tracking with tempo estimation gets properly hairy (there are whole PhD theses on it), but a cheap and cheerful version works great for visuals: watch the energy in the bass bins, and call it a beat whenever that energy spikes above its recent average. Kick drums and bass notes live down low, and they're usually what we feel as the beat.
// cheap beat detection: watch bass energy, fire when it spikes over recent average.
let energyHistory = [];
function detectBeat(spectrum) {
// sum only the low bins - the bass, where kicks live.
let bass = 0;
for (let i = 0; i < 16; i++) bass += spectrum[i]; // ~first 350 Hz
bass /= 16;
energyHistory.push(bass);
if (energyHistory.length > 43) energyHistory.shift(); // ~1 sec window
const avg = energyHistory.reduce((a, b) => a + b, 0) / energyHistory.length;
return bass > avg * 1.3 && bass > 100; // 30% above recent bass average = beat
}
The * 1.3 is a sensitivity dial you'll want to tune per track - some music is punchy and needs a higher bar, some is soft and needs a lower one. And to stop it firing ten times on one fat kick, add a tiny cooldown so it can't retrigger for, say, 200 milliseconds. That "trust a clock, add a refractory gap" idea is the same discipline we used scheduling rhythms back in episode 121.
// don't let one kick fire five beats. enforce a minimum gap between beats.
let lastBeat = 0;
function beatNow(spectrum, timeMs) {
if (timeMs - lastBeat < 200) return false; // refractory period: 200ms min gap
if (detectBeat(spectrum)) { lastBeat = timeMs; return true; }
return false;
}
Wire a beat to something big and structural - a full-screen pulse, a camera shake, a background colour swap - and reserve the finer onsets for the small sparkly detail. Layering a slow strong signal (beat) under a fast busy one (onsets) is what gives a visualizer a sense of hierarchy, the same way a song has both a backbone and its little flourishes.
Okay, this is the fancy one, and it still slightly delights me that it's possible in a browser. Everything so far has been about energy and timing. But music also has harmony - actual notes, C and E and G - and we can extract that too. The tool is the chromagram (or "chroma vector"), and the idea is gorgeous: fold the entire spectrum down into just twelve buckets, one per pitch class (C, C#, D... up to B), ignoring which octave. All the C's across the whole keyboard pile into the "C" bucket. What you get is a little 12-slot readout of which notes are sounding right now, regardless of octave.
// chromagram: fold ALL frequencies into 12 pitch classes (C, C#, D ... B).
// every octave of the same note lands in the same bucket. music's "colour" of harmony.
function chromagram(spectrum) {
const chroma = new Array(12).fill(0);
for (let i = 1; i < spectrum.length; i++) {
const hz = binToHz(i);
if (hz < 20) continue; // skip sub-audible rumble
// midi note number from frequency (69 = A4 = 440Hz), same formula as episode 126
const midi = 69 + 12 * Math.log2(hz / 440);
const pitchClass = ((Math.round(midi) % 12) + 12) % 12; // 0..11, wrapped
chroma[pitchClass] += spectrum[i]; // pile this bin's energy in
}
return chroma;
}
That 69 + 12 * log2(hz/440) is the inverse of the equal-temperament formula we used forwards last episode to turn MIDI notes into frequencies - here we run it backwards, from a frequency to a note number, then throw away the octave with % 12. Once you've got the twelve chroma values, the loveliest thing to do is find the dominant pitch class and map it to a hue - so the colour of your sketch tracks the actual harmony of the music, shifting as the chords change.
// map the DOMINANT pitch class to a hue -> colour follows the harmony.
// 12 notes spread evenly around the 360-degree colour wheel. very "episode 28".
function chromaToHue(chroma) {
let maxVal = 0, maxNote = 0;
for (let n = 0; n < 12; n++) {
if (chroma[n] > maxVal) { maxVal = chroma[n]; maxNote = n; }
}
return (maxNote / 12) * 360; // note 0 (C) -> 0deg, note 6 -> 180deg, etc.
}
Now here's a fun design decision: you've got two things that could drive hue - brightness (centroid) and harmony (chroma). Which wins? Neither, necessarily - you might let harmony pick the base hue and brightness pick the saturation or lightness. That's the real art of this whole episode, and it deserves its own little word.
You've now got five features - loudness, brightness, onsets, beats, and harmony - and the temptation is to slam all of them onto everything at once. Don't. A visualizer where every property jerks around to every feature just looks like noise, ironically. The music has structure; your mapping should have restraint. Pick a clear job for each feature.
// a MAPPING is a set of decisions: which feature drives which visual property.
// restraint beats chaos. give each feature ONE clear job.
const mapping = {
loudness: "overall scale + particle count", // energy -> how much is happening
brightness:"background lightness", // timbre -> light/dark of the scene
onsets: "spawn particle bursts", // events -> discrete pops
beats: "full-screen pulse + shake", // pulse -> big structural hits
harmony: "particle hue", // chords -> the colour palette
};
The reason this works is that it mirrors how we hear: we track the beat with our body, the melody with our attention, the brightness as a mood, all at once but on different channels. Give each channel its own visual lane and the picture stops fighting itself. When I get a mapping right, it stops feeling like "audio driving graphics" and starts feeling like one thing with two faces. Makes sense, right?
Let's braid the whole toolbox into one sketch. The brief: pull all five features every frame, then paint them on separate channels - background lightness from brightness, a big pulse on beats, particle bursts on onsets, particle colour from harmony, and the overall busyness from loudness. First the analysis pass, one tidy function that hands back everything:
// ANALYSIS: one pass per frame, returns every feature. clean separation.
function analyze(timeMs) {
analyser.getByteFrequencyData(spectrum);
const loud = loudness(spectrum);
const bright = centroid(spectrum);
const chroma = chromagram(spectrum);
const hue = chromaToHue(chroma); // harmony -> colour
const onset = isOnset(flux(spectrum)); // note starts
const beat = beatNow(spectrum, timeMs); // the pulse
return { loud, bright, hue, onset, beat };
}
Then the draw loop reads those features and gives each one its lane. Notice how little "audio" code there is here - by the time we're drawing, it's all just numbers with meaning, exactly as it should be.
// DRAW: each feature drives its own visual lane. no feature fights another.
let flash = 0; // beat flash amount, decays
function draw(p, timeMs) {
const f = analyze(timeMs);
// background lightness follows BRIGHTNESS (dark sound = dark scene)
const bg = 10 + Math.min(f.bright / 6000, 1) * 40;
p.background(bg);
// BEAT -> a full-screen flash we let decay away
if (f.beat) flash = 1;
flash *= 0.85; // ease back down (episode 16)
if (flash > 0.01) { p.fill(255, flash * 60); p.rect(0, 0, p.width, p.height); }
// ONSET -> spawn a burst, coloured by the current HARMONY
if (f.onset) onOnset(f.hue);
// draw + age every particle; overall size nudged by LOUDNESS
p.colorMode(p.HSB);
for (let i = particles.length - 1; i >= 0; i--) {
const pt = particles[i];
pt.x += pt.vx; pt.y += pt.vy; pt.life -= 0.02;
const size = (2 + f.loud * 20) * pt.life; // loud passages = bigger particles
p.noStroke();
p.fill(pt.hue, 80, 90, pt.life);
p.circle(pt.x, pt.y, size);
if (pt.life <= 0) particles.splice(i, 1);
}
}
Feed that a track with a clear beat and some harmonic movement - anything with a bassline and chords - and watch what happens. The room darkens and brightens with the timbre, the whole screen thumps on the kick, little coloured fireworks burst on every hi-hat and pluck, and the colour of those fireworks drifts as the chords change. That last part is the bit that gets people, because it's not obvious why the colours are shifting until you realise it's tracking the actual harmony. It genuinly feels like the sketch is listening, and in its small way it is.
When you want to push it further: smooth every feature with a little easing before you use it (raw features are jittery - a touch of lerp toward the new value each frame works wonders, straight out of episode 16), or occassionally let the beat trigger a slow key-change in a soft ambient bed underneath (episode 122), so the sound and picture feed each other. That loop - audio drives visuals drives audio - is a rabbit hole I'd happily lose you down :-).
AnalyserNode (getByteFrequencyData) holds far more, and today we mined it for five separate musical features, each driving a different part of the pictureSo that's the leap I promised at the end of last time. The picture doesn't trail the sound any more, one dumb step behind - it reads the sound the way we do, on separate channels, and paints each one where it belongs. Brightness became light, harmony became colour, the beat became a pulse, and the little events became little events on screen. It's still two systems, but for the first time they feel like one thing with two senses rather than a graphic politely nodding along to a speaker.
And that's got me itching, because we've spent this whole audio stretch building pieces - synths, sequencers, ambient beds, MIDI wiring, and now visuals that genuinly listen - all a bit separately, in their own little sketches. What would it take to fold all of it into one built thing? One system where the sound we generate and the picture that watches it are composed together, on purpose, as a single deliberate piece rather than a demo? That's the build I want to do with you next, and it's where this entire arc has quietly been heading. Get this visualizer reacting to your favourite track first - genuinly sit with it and tune the mappings until it feels right, that tuning is half the skill - and then come back, because next time we compose the whole thing as one :-).
Sallukes! Thanks for reading.
X
@femdev