I ended last time on a complaint, and it's been nagging at me ever since: everything we've built in this whole audio arc lives inside the browser tab, talking only to itself. Our code makes sound, our code makes pictures, they hold hands - but nothing outside can reach in and our sound can't reach out. I said I wanted a physical knob on a desk to drive that grain size, a real keyboard to trigger our clouds. Well, today we knock a hole in the wall. Today the browser tab stops being a sealed box and becomes one instrument in a much bigger room, wired up to hardware and to other programs. Allez, this is the episode where it all connects :-).
There are two languages we need for this, and they've been quietly running the entire world of electronic music and media art for decades. One is old, cranky, and everywhere - MIDI. The other is younger, flexible, and network-native - OSC. Learn to speak both and your creative code can talk to synths, drum machines, lighting rigs, VJ software, a phone on the same wifi, another program running on the same machine. It's a genuinly different feeling once your sketch is plugged in to the physical world instead of sealed behind glass. Let me show you the parts.
MIDI stands for Musical Instrument Digital Interface, and here's the wild bit - it was standardised in 1983 and has barely changed since. Every synth, every keyboard, every drum pad, every DAW controller you can buy speaks it. It is ancient by computing standards and it is completely unkillable, because it does one thing perfectly: it lets musical instruments send each other tiny messages like "play middle C, medium hard" or "this knob is now at 40%".
The key thing to understand before any code: MIDI does not send sound. It sends instructions. When you press a key on a MIDI keyboard, no audio travels down the cable - just a short message saying "note 60 pressed, velocity 100". Something on the other end (a synth, or our code) decides what that actually sounds like. That separation is the whole reason MIDI is so useful to us: the keyboard is just an input device throwing events, and we get to decide what those events mean.
// MIDI sends INSTRUCTIONS, never audio. a key-press is a tiny message:
// "note 60 (middle C) was pressed, velocity 100 (fairly hard)"
// WE decide what that turns into - a synth note, a visual, anything.
const exampleMessage = {
type: "noteon",
note: 60, // middle C (0-127, 60 = C4)
velocity: 100, // how hard (0-127); 0 means note-off
};
Modern browsers can speak MIDI directly through the Web MIDI API. No plugin, no library needed - just ask the browser for access and it hands you every MIDI device plugged into the machine.
You start by asking navigator.requestMIDIAccess(). It returns a promise (the browser may pop a permission prompt the first time), and what you get back is an object holding two Maps: inputs (devices sending to you, like keyboards) and outputs (devices you can send to, like a synth module).
// ask the browser for MIDI. returns a promise - user may see a permission prompt.
navigator.requestMIDIAccess().then((access) => {
console.log("MIDI ready!");
// inputs = things sending TO us (keyboards, pads, knobs)
for (const input of access.inputs.values()) {
console.log("found input:", input.name);
}
// outputs = things we can send TO (synth modules, drum machines)
for (const output of access.outputs.values()) {
console.log("found output:", output.name);
}
}).catch((err) => {
console.log("no MIDI access:", err); // some browsers/contexts refuse
});
One gotcha worth flagging early, because it cost me an afternoon once: Web MIDI needs a secure context. It works on https:// and on localhost, but not on a plain file:// page opened straight from disk. If requestMIDIAccess is undefined, that's usually why. Run a tiny local server (episode 80 covered spinning one up) and you're golden.
Once you've got an input, you attach an onmidimessage handler and every event that device sends comes flying in. Each message arrives as a little array of raw bytes in event.data, and that's where it gets interesting - we have to decode them ourselves.
// attach a listener to every input device - all their messages land here.
navigator.requestMIDIAccess().then((access) => {
for (const input of access.inputs.values()) {
input.onmidimessage = handleMIDI;
}
});
function handleMIDI(event) {
const [status, data1, data2] = event.data; // raw MIDI bytes
console.log("raw bytes:", status, data1, data2);
}
Three little numbers. To do anything useful we have to understand what they mean, and this is the one bit of MIDI that looks scary and genuinly isn't once someone draws it for you. So let me draw it for you.
A MIDI message is mostly three bytes. The first is the status byte and it's cleverly split down the middle. The top four bits (the high nibble) say what kind of message it is; the bottom four bits say which channel (1-16) it's on. The other two bytes are the data - for a note, that's which note and how hard.
// the STATUS byte packs two things into 8 bits:
// high nibble (top 4 bits) = message TYPE
// low nibble (bottom 4) = CHANNEL (0-15, shown to humans as 1-16)
function decodeStatus(status) {
const type = status & 0xf0; // mask off the channel -> just the type
const channel = status & 0x0f; // mask off the type -> just the channel
return { type, channel };
}
// the important type codes:
// 0x90 = note on 0x80 = note off 0xB0 = control change (knobs)
// 0xE0 = pitch bend 0xC0 = program change
That & 0xf0 / & 0x0f masking is the same bit-twiddling we met when we packed colours into pixels back in episode 10 - split one number into two by masking off the half you don't want. Once you see MIDI as "a type nibble and a channel nibble", the whole protocol stops being mysterious. Now let's turn those raw bytes into something we can actually work with.
// turn raw MIDI bytes into a friendly event object.
function parseMIDI(event) {
const [status, data1, data2] = event.data;
const type = status & 0xf0;
const channel = status & 0x0f;
// note-on with velocity 0 is secretly a note-OFF (a famous MIDI quirk!)
if (type === 0x90 && data2 > 0) {
return { kind: "noteon", note: data1, velocity: data2, channel };
}
if (type === 0x80 || (type === 0x90 && data2 === 0)) {
return { kind: "noteoff", note: data1, channel };
}
if (type === 0xb0) {
return { kind: "cc", controller: data1, value: data2, channel };
}
return { kind: "other", status, data1, data2 };
}
See that comment about velocity zero? That's one of MIDI's genuinly weird historical quirks - loads of keyboards send a "note on" with velocity 0 instead of a proper "note off", to save bandwidth back when every byte mattered. If you don't handle it, notes stick and drone forever. Ask me how I know :-). Always treat note-on-velocity-0 as a note-off and you'll save yourself that debugging session.
Right, the payoff. We've got a Tone.js synth from way back in episode 119. We've got parsed MIDI events. Let's wire the keyboard into the synth so pressing a physical key makes our code sing. The only missing piece is converting a MIDI note number (60) into something Tone.js understands (a note name like "C4", or a frequency in Hz).
// MIDI note number -> note name. the formula: 69 = A4 = 440Hz.
// Tone.js does the conversion for us with the "midi" unit.
function midiToNote(noteNumber) {
return Tone.Frequency(noteNumber, "midi").toNote(); // 60 -> "C4"
}
// or straight to frequency in Hz, if you want the raw number:
// freq = 440 * 2^((n - 69) / 12)
function midiToFreq(noteNumber) {
return 440 * Math.pow(2, (noteNumber - 69) / 12);
}
That 440 * 2^((n-69)/12) is worth pausing on - it's the equal temperament formula, the maths behind how a piano is tuned. Every semitone is the twelfth root of two apart, so twelve of them multiply up to exactly a doubling (an octave). We met this idea sideways when we built scales in episode 120; here it is as raw arithmetic. Anyway, now the wiring:
// a real MIDI keyboard now plays our software synth.
const synth = new Tone.PolySynth(Tone.Synth).toDestination();
function onKeyboard(event) {
const msg = parseMIDI(event);
if (msg.kind === "noteon") {
const note = Tone.Frequency(msg.note, "midi").toNote();
const vel = msg.velocity / 127; // MIDI 0-127 -> Tone 0-1
synth.triggerAttack(note, Tone.now(), vel); // press
} else if (msg.kind === "noteoff") {
const note = Tone.Frequency(msg.note, "midi").toNote();
synth.triggerRelease(note, Tone.now()); // release
}
}
Notice we use triggerAttack and triggerRelease seperately here, not the combined triggerAttackRelease we've leaned on all arc. That's because a real key press has an actual start and end - you hold it as long as you like - so we press on note-on and release on note-off, letting the player control the duration with their fingers. A PolySynth handles as many held notes at once as you throw at it, so chords just work. Plug in a keyboard, press a key, hear your own synth. That moment never gets old.
Notes are only half of MIDI's usefulness to a creative coder. The other half is Control Change messages - CC for short, status 0xB0. These are what every knob, fader, and slider on a MIDI controller sends. Each has a controller number (which knob) and a value from 0 to 127 (where it's turned to). This is exactly the "physical knob drives a parameter" thing I promised you last episode.
// CC messages = knobs and faders. controller number + value (0-127).
// map them onto anything: filter cutoff, grain size, a colour, whatever.
const params = { cutoff: 800, grainSize: 0.1, hue: 0 };
function onControl(event) {
const msg = parseMIDI(event);
if (msg.kind !== "cc") return;
const norm = msg.value / 127; // 0..1, easy to remap
if (msg.controller === 1) { // knob 1 -> filter cutoff
params.cutoff = 200 + norm * 4000; // 200Hz .. 4200Hz
filter.frequency.value = params.cutoff;
}
if (msg.controller === 2) { // knob 2 -> grain size (ep125!)
params.grainSize = 0.01 + norm * 0.3; // 10ms .. 310ms
grainPlayer.grainSize = params.grainSize;
}
if (msg.controller === 3) { // knob 3 -> a visual hue
params.hue = norm * 360;
}
}
There it is - remember the granular texture pad from last episode, where the mouse Y set the grain size? Swap the mouse for a real knob on your desk and it feels completely different to play. A knob has resistance, a detent, a physicality your hand understands that a mouse never gives you. This is the whole reason people spend money on hardware controllers for software instruments. And it's about six lines of code to hook one up. Makes sense, right?
We've been receiving, but MIDI goes both ways. If you've got an output device - a hardware synth, a drum machine, even a light controller that speaks MIDI - your code can drive it. You build the raw bytes yourself and call send. This turns your creative sketch into a sequencer that plays real hardware.
// send MIDI OUT: drive a hardware synth from our code.
// we build the raw status+data bytes by hand.
function sendNote(output, note, velocity, channel = 0) {
const noteOn = [0x90 | channel, note, velocity]; // OR the channel into the status
const noteOff = [0x80 | channel, note, 0];
output.send(noteOn); // play now
output.send(noteOff, performance.now() + 500); // schedule off in 500ms
}
// example: our generative melody (episode 120) driving a real synth.
navigator.requestMIDIAccess().then((access) => {
const out = [...access.outputs.values()][0]; // first output device
if (out) sendNote(out, 64, 100); // play E4 on the hardware
});
That 0x90 | channel is the mirror image of the decoding we did earlier - instead of masking the channel out we're OR-ing it in, packing type and channel back into one status byte. And notice output.send takes an optional timestamp as a second argument, so you can schedule events precisely in the future - the same "trust the clock, not setTimeout" lesson we learned building rhythms in episode 121. Now imagine our Euclidean drum patterns from that episode firing a real 808. That's the door MIDI-out opens.
MIDI is brilliant but it shows its age. Values are stuck at 0-127 (7 bits - coarse!), it's built around notes and channels, and it was designed for cables between boxes in one room. Enter OSC - Open Sound Control - designed in the late 90s specifically to fix MIDI's limits for the network era.
The difference that matters most: OSC messages are addressed, like little URLs, and carry any typed data you like. Instead of "CC number 2, value 64", an OSC message is "/synth/grainSize 0.087" - a readable address and a full-precision float. You invent the address space; there's no fixed vocabulary. That flexibility is why OSC is the glue between creative tools - TouchOSC on your phone, Max/MSP, Processing, VJ apps, lighting desks, they all speak it.
// an OSC message is an ADDRESS (like a URL path) plus typed arguments.
// you invent the addresses. full-precision numbers, strings, whatever.
const oscExamples = [
{ address: "/synth/freq", args: [440.0] }, // a float, not a 0-127 int
{ address: "/synth/grainSize", args: [0.087] }, // fine precision!
{ address: "/light/3/color", args: [255, 40, 90] }, // multiple args
{ address: "/vj/scene", args: ["forest"] }, // even strings
];
// compare to MIDI's rigid "channel + note/cc + 0-127". OSC is wide open.
Here's the catch, and it's an important one to understand or you'll be confused for an hour. OSC almost always travels over UDP on a network - and a browser cannot send raw UDP. Browsers only do HTTP and WebSockets. So the standard pattern is a tiny bridge: a little program (usually Node.js) that speaks UDP to the outside world on one side and WebSockets to your browser on the other. Your sketch talks OSC-over-WebSocket to the bridge; the bridge relays it as real UDP OSC to everything else.
// the osc.js library gives us OSC over WebSocket in the browser.
// it talks to a tiny Node bridge that relays to real UDP OSC on the network.
const oscPort = new osc.WebSocketPort({
url: "ws://localhost:8080", // our local bridge
metadata: true,
});
oscPort.on("ready", () => console.log("OSC bridge connected!"));
oscPort.open();
Don't let the bridge intimidate you - it's genuinly about fifteen lines of Node, and the osc.js project ships a ready-made one you can copy. The mental model is all that matters: browser <-> websocket <-> bridge <-> UDP <-> the world. Once that clicks, receiving messages is the easy part.
// receive OSC: match on the address, pull out the args. drive our params.
oscPort.on("message", (msg) => {
// msg.address is the path, msg.args is the array of typed values
if (msg.address === "/synth/grainSize") {
grainPlayer.grainSize = msg.args[0]; // a clean float, no 0-127 nonsense
}
if (msg.address === "/synth/freq") {
synth.frequency.value = msg.args[0];
}
if (msg.address === "/visual/color") {
const [r, g, b] = msg.args; // three args at once
bgColor = [r, g, b];
}
});
The lovely thing about address matching is how it scales. You can pattern-match with wildcards (/light/*/color to hit every light), namespace things sensibly, and add new controls without renumbering anything - none of MIDI's "wait, which CC number was the filter again?" pain. Your control surface becomes self-documenting.
Same as MIDI, OSC is bidirectional. Your sketch can send too - to control another program, push data to a phone, or drive a lighting rig from your visuals. You just build the message object and fire it down the same port.
// send OSC out - control other software or hardware from our sketch.
function sendOSC(address, ...args) {
oscPort.send({
address: address,
args: args.map((v) => ({ type: typeof v === "string" ? "s" : "f", value: v })),
});
}
// example: our particle count drives a light rig; our beat cues VJ software.
sendOSC("/light/brightness", 0.8);
sendOSC("/vj/trigger", "burst");
sendOSC("/synth/detune", -12.0);
Now picture the loop closing: a physical knob sends MIDI CC into your sketch, your sketch reacts visually and forwards an OSC message to a lighting desk that flashes the room in time. One turn of one knob, and code, sound, screen, and physical light all move together. That's not a browser tab any more - that's a little media installation, and you built the nervous system for it.
Let's tie the whole thing into one build you can actually perform with. The brief: a MIDI keyboard plays a Tone.js synth (episode 119), and every note fires a visual burst on a p5 canvas, and a MIDI knob (CC) reshapes the visuals live. If you don't have a MIDI controller handy, there are free on-screen ones and phone apps that send MIDI - or swap in the OSC path from your phone. The point is: an external device driving both sound and picture at once.
// HARDWARE-PLAYABLE INSTRUMENT: MIDI keys -> synth + visual bursts, knob -> shape.
const synth = new Tone.PolySynth(Tone.Synth).toDestination();
const bursts = []; // active visual events
let hueShift = 0; // driven live by a knob (CC)
navigator.requestMIDIAccess().then((access) => {
for (const input of access.inputs.values()) {
input.onmidimessage = route; // send every device's messages to one router
}
});
The router just parses each message and sends notes to the synth-plus-visual path and knobs to the parameter path - one clean switch on the message kind.
// route incoming MIDI: notes make sound + light, knobs reshape.
function route(event) {
const msg = parseMIDI(event);
if (msg.kind === "noteon") {
const note = Tone.Frequency(msg.note, "midi").toNote();
synth.triggerAttack(note, Tone.now(), msg.velocity / 127);
// spawn a visual burst whose size follows how hard the key was hit:
bursts.push({
x: Math.random() * 400, y: 200,
r: 0, max: 20 + msg.velocity, // harder hit = bigger burst
note: msg.note,
});
} else if (msg.kind === "noteoff") {
synth.triggerRelease(Tone.Frequency(msg.note, "midi").toNote(), Tone.now());
} else if (msg.kind === "cc" && msg.controller === 1) {
hueShift = (msg.value / 127) * 360; // knob 1 sweeps the colour of everything
}
}
And the p5 side draws and ages the bursts, colouring each by its note so the picture and the sound tell the same story - high notes one colour, low notes another, the knob sweeping the whole palette as you turn it.
// p5 draw: grow and fade each burst, colour it by pitch + the live knob.
function draw(p) {
p.background(15, 40); // slight trail (episode 16's alpha trick)
for (let i = bursts.length - 1; i >= 0; i--) {
const b = bursts[i];
b.r += (b.max - b.r) * 0.1; // ease outward toward its max size
const hue = (b.note * 3 + hueShift) % 360; // pitch + knob decide the colour
p.colorMode(p.HSB);
p.noFill();
p.stroke(hue, 80, 90, 1 - b.r / b.max); // fade as it grows
p.strokeWeight(2);
p.circle(b.x, b.y, b.r * 2);
if (b.r > b.max - 0.5) bursts.splice(i, 1); // retire finished bursts
}
}
Plug in a controller, hit a key, and a coloured ring blooms exactly as the note sounds - play a chord and three rings bloom together, turn the knob and the whole scene shifts hue under your hand. That is a physical instrument you built, sound and light welded to real keys and real knobs. Sit with it for a while, it's a genuinly magic feeling the first time your desk hardware lights up your own code.
When you want to push it: forward each note as an OSC message to a second sketch on another machine so two computers perform together, wire a CC knob to the granular grain size from last episode so you can play textures by hand, or send OSC from your phone's accelerometer so tilting the phone bends the sound. The wall's got a hole in it now - see how much you can pull through :-).
navigator.requestMIDIAccess()), no library needed. Just remember it wants a secure context - https:// or localhost, not file://0x90 note-on, 0x80 note-off, 0xB0 control-change) and a channel nibble - the same mask-and-split bit-twiddling we did on pixels in episode 10 (status & 0xf0, status & 0x0f)Tone.Frequency(n, "midi").toNote(), or the equal-temperament formula 440 * 2^((n-69)/12) - twelve semitones of the twelfth-root-of-two multiply up to an octave (episode 120)triggerAttack / triggerRelease seperately for real keys, so the player's finger controls the duration. A PolySynth gives you chords for free0xB0) are knobs and faders - a controller number plus a 0-127 value. Map them onto anything: filter cutoff, grain size (episode 125!), a colour. A real knob beats a mouse every time0x90 | channel) and output.send() to drive real hardware, with an optional timestamp for precise scheduling (episode 121's clock lesson)/synth/grainSize 0.087, full-precision, any types, an address space you invent. Browsers can't do raw UDP, so it runs over a tiny WebSocket bridge (browser <-> websocket <-> bridge <-> UDP <-> world). Match on the address, pull the args, doneSo that's the wall knocked through. Our sketches can now be played by real keyboards, twisted by real knobs, and can reach out to talk to phones, synths, lighting rigs, and other programs on the network. The browser tab was never really a box - it just needed a couple of protocols to become one node in a much bigger system. That's a genuinly big step: everything we make from here can be plugged in.
And that raises a thought I can't stop turning over. We've now got sound reacting to input, visuals reacting to sound, and hardware reacting to both - but so far our visuals have mostly followed the audio, one step behind, reacting after the fact. What if the two were bound so tightly, analysed so closely frame by frame, that the picture didn't just follow the sound but seemed to be the sound made visible? That takes everything we've built about listening to audio and pushes it much deeper, and it's exactly where we're headed next. Build the instrument first - genuinly plug something in and play it until it clicks - and then come back :-).
Sallukes! Thanks for reading.
X
@femdev