# Learn Creative Coding (#87) - Real-Time Data Streams ![cc-banner](https://images.hive.blog/DQmTUxQoFArrnLxQrqV2tPhMY6XhpoNn85wxFacvHV279kq/cc-banner-cyan.png) Last episode we turned text into visual material -- word frequency, sentence rhythm, sentiment analysis, character-to-color grids, Markov chain generation. Language became data, and data became art. Every technique from this arc applied: the `map()` function from episode 82 converted word counts to pixel sizes, log scaling from episode 83 handled Zipf's power law distributions, and the bar chart patterns from episode 84 showed sentiment arcs across stories. Text turned out to be one of the richest data sources we've worked with. But every dataset we've touched so far in this arc has been *static*. We loaded a file, parsed it, drew it. Done. The visualization was a snapshot -- a picture of data frozen in time. Real-world data doesn't sit still. Stock prices tick every second. Wikipedia gets edited in real time. Earthquakes happen. Tweets appear. Sensor readings stream in continuously. The data is *alive*, and the visualization should be alive too. This episode is about real-time data. We'll connect to live data sources using WebSockets and Server-Sent Events, build visualizations that scroll and update continuously, handle buffering and windowing (because you can't draw every data point forever), smooth noisy streams, detect significant events, and manage the practical problems of connections that drop and data that arrives faster than you can draw it. We used animation loops since episode 3 and real-time audio input in episode 19 -- now we're applying those same patterns to data streams. ## WebSockets: persistent two-way connections The normal way a browser talks to a server is HTTP: the browser asks, the server answers, connection closes. If you want new data, you ask again. That's polling -- and for real-time data, it's wasteful. You're constantly asking "anything new?" and the server keeps saying "no" until something actually happens. WebSockets fix this. A WebSocket is a persistent connection that stays open. Either side can send data at any time. The server can push new information the instant it arrives, without waiting for the browser to ask. ```javascript const ws = new WebSocket('wss://stream.example.com/prices'); ws.onopen = function() { console.log('connected'); }; ws.onmessage = function(event) { const data = JSON.parse(event.data); // data arrives whenever the server has something new processNewDataPoint(data); }; ws.onclose = function() { console.log('disconnected'); }; ws.onerror = function(err) { console.log('connection error'); }; ``` Four event handlers: `onopen` fires when the connection establishes, `onmessage` fires every time data arrives, `onclose` fires when the connection drops, and `onerror` handles problems. The `event.data` is usually a JSON string that you parse into whatever structure the server sends. The key mental shift: in a polling model, your code controls the timing. You decide when to fetch. With WebSockets, the *server* controls the timing. Data arrives whenever it arrives. Your visualization has to be ready to process it at any moment. That means your rendering loop and your data pipeline are decoupled -- data comes in asynchronously, and your `requestAnimationFrame` loop draws whatever's in the buffer at each frame. ## Server-Sent Events: one-way streams WebSockets are bidirectional -- both sides can send. But lots of real-time data is one-way: the server pushes, you receive. For those cases, Server-Sent Events (SSE) are simpler. The browser opens a standard HTTP connection that stays open, and the server sends events down it. No handshake upgrade, no special protocol, just a long-lived HTTP response. ```javascript const source = new EventSource('https://stream.example.com/updates'); source.onmessage = function(event) { const data = JSON.parse(event.data); processNewDataPoint(data); }; source.onerror = function() { // EventSource auto-reconnects by default console.log('connection lost, reconnecting...'); }; ``` `EventSource` has built-in reconnection. If the connection drops, the browser automatically tries to reconnect after a few seconds. You don't have to write retry logic. That's a big advantage over raw WebSockets, where you handle reconnection yourself. The trade-off: SSE only goes server-to-client. If you need to send messages back (like subscribing to specific channels), you can't do it through the SSE connection -- you'd make a separate HTTP request. For pure data consumption (price feeds, notification streams, activity logs), SSE is often the better choice. ## The data buffer: rolling window Here's the core problem with real-time visualization: data never stops. If you append every new data point to an array and draw all of them, your array grows forever and your frame rate drops to zero. You need a buffer with a fixed size -- a rolling window that keeps the most recent N data points and discards old ones. ```javascript const canvas = document.createElement('canvas'); canvas.width = 900; canvas.height = 400; document.body.appendChild(canvas); const ctx = canvas.getContext('2d'); const BUFFER_SIZE = 200; const buffer = []; function addDataPoint(value) { buffer.push({ value: value, time: Date.now() }); // keep only the last N points if (buffer.length > BUFFER_SIZE) { buffer.shift(); } } function draw() { ctx.fillStyle = '#0a0a1a'; ctx.fillRect(0, 0, 900, 400); if (buffer.length < 2) { requestAnimationFrame(draw); return; } ctx.beginPath(); for (let i = 0; i < buffer.length; i++) { const x = (i / (BUFFER_SIZE - 1)) * 860 + 20; const y = 380 - ((buffer[i].value + 5) / 50) * 360; if (i === 0) ctx.moveTo(x, y); else ctx.lineTo(x, y); } ctx.strokeStyle = 'rgba(120, 180, 255, 0.7)'; ctx.lineWidth = 1.5; ctx.stroke(); requestAnimationFrame(draw); } draw(); ``` The `buffer.shift()` drops the oldest point when the buffer is full. The drawing loop maps array index to x position, so the newest point is always on the right and old points scroll leftward as new ones arrive. It's a scrolling line chart -- the most fundamental real-time visualization. Stock tickers, heart monitors, network bandwidth gauges, temperature sensors -- they all use this pattern. The buffer size determines how much history you see. 200 points at 1 update per second = 3.3 minutes of history. At 10 updates per second = 20 seconds. Choose the buffer based on what temporal scale matters for your data. Too short and you miss trends. Too long and the line gets so compressed that individual variations disappear. ## Simulating real-time data for development You won't always have a live WebSocket to test against. Simulating a real-time stream lets you develop the visualization without needing a server. Use `setInterval` to emit fake data points at regular intervals: ```javascript function simulateStream(callback, intervalMs) { let value = 50; setInterval(function() { // random walk with drift value += (Math.random() - 0.48) * 3; value = Math.max(0, Math.min(100, value)); callback({ value: value, timestamp: Date.now() }); }, intervalMs); } // use it just like a real data source simulateStream(function(data) { addDataPoint(data.value); }, 100); // 10 updates per second ``` The random walk with slight downward drift (`0.48` instead of `0.5`) simulates a data stream that trends slightly down over time. Replace the random walk with whatever model matches your target data: sinusoidal for temperature, step functions for event counts, Brownian motion for prices. The key is that your visualization code doesn't care whether the data comes from a real server or from `setInterval` -- it just processes whatever arrives in the buffer. I use this pattern constantly when building data art. Get the visual right with fake data first, then swap in the real source later. Much faster than debugging visuals and network issues at the same time. ## Smoothing noisy data Raw real-time data is noisy. Sensor readings jitter. Network latency creates bursts and gaps. Price ticks bounce around the underlying trend. Drawing the raw values produces a jagged, jumpy line that obscures the pattern you care about. Smoothing reveals the signal underneath. The simplest smoother: exponential moving average (EMA). Each new smoothed value is a weighted blend of the previous smoothed value and the new raw value: ```javascript const canvas = document.createElement('canvas'); canvas.width = 900; canvas.height = 400; document.body.appendChild(canvas); const ctx = canvas.getContext('2d'); const BUFFER_SIZE = 200; const rawBuffer = []; const smoothBuffer = []; let smoothed = 50; const alpha = 0.15; // smoothing factor: 0 = no change, 1 = no smoothing function addDataPoint(value) { rawBuffer.push(value); if (rawBuffer.length > BUFFER_SIZE) rawBuffer.shift(); smoothed = smoothed * (1 - alpha) + value * alpha; smoothBuffer.push(smoothed); if (smoothBuffer.length > BUFFER_SIZE) smoothBuffer.shift(); } function drawLine(data, color) { if (data.length < 2) return; ctx.beginPath(); for (let i = 0; i < data.length; i++) { const x = (i / (BUFFER_SIZE - 1)) * 860 + 20; const y = 380 - ((data[i]) / 100) * 360; if (i === 0) ctx.moveTo(x, y); else ctx.lineTo(x, y); } ctx.strokeStyle = color; ctx.lineWidth = 1.5; ctx.stroke(); } function draw() { ctx.fillStyle = '#0a0a1a'; ctx.fillRect(0, 0, 900, 400); drawLine(rawBuffer, 'rgba(80, 100, 140, 0.3)'); // raw: faint drawLine(smoothBuffer, 'rgba(120, 180, 255, 0.8)'); // smooth: bright requestAnimationFrame(draw); } draw(); ``` The `alpha` parameter controls responsiveness. Low alpha (0.05) gives heavy smoothing -- the line barely reacts to individual data points, showing only broad trends. High alpha (0.5) gives light smoothing -- the line follows the raw data closely but still filters out frame-to-frame jitter. I usually start at 0.1-0.15 and adjust by eye. Drawing both raw and smoothed lines together is a nice visual effect in itself. The faint raw line jitters nervously while the bright smoothed line glides through it like a calm thread through chaos. It shows the relationship between signal and noise -- and for data art, that relationship *is* the artwork. ## Event detection: watching for spikes Sometimes you don't just want to show the stream -- you want to react to it. A sudden spike, a threshold crossing, a pattern that means something. Event detection turns passive visualization into active monitoring. ```javascript const canvas = document.createElement('canvas'); canvas.width = 900; canvas.height = 400; document.body.appendChild(canvas); const ctx = canvas.getContext('2d'); const BUFFER_SIZE = 200; const buffer = []; const events = []; let smoothed = 50; function addDataPoint(value) { const previous = smoothed; smoothed = smoothed * 0.85 + value * 0.15; buffer.push(value); if (buffer.length > BUFFER_SIZE) buffer.shift(); // detect spike: raw value deviates more than 15 from smoothed const deviation = Math.abs(value - smoothed); if (deviation > 15) { events.push({ index: buffer.length - 1, value: value, type: value > smoothed ? 'spike_up' : 'spike_down', age: 0 }); } // age out old events for (let i = events.length - 1; i >= 0; i--) { events[i].age++; events[i].index--; if (events[i].index < 0 || events[i].age > 120) { events.splice(i, 1); } } } function draw() { ctx.fillStyle = '#0a0a1a'; ctx.fillRect(0, 0, 900, 400); // draw the data line if (buffer.length > 1) { ctx.beginPath(); for (let i = 0; i < buffer.length; i++) { const x = (i / (BUFFER_SIZE - 1)) * 860 + 20; const y = 380 - (buffer[i] / 100) * 360; if (i === 0) ctx.moveTo(x, y); else ctx.lineTo(x, y); } ctx.strokeStyle = 'rgba(120, 180, 255, 0.6)'; ctx.lineWidth = 1.5; ctx.stroke(); } // draw event markers for (const evt of events) { const x = (evt.index / (BUFFER_SIZE - 1)) * 860 + 20; const y = 380 - (evt.value / 100) * 360; const fade = 1 - evt.age / 120; const color = evt.type === 'spike_up' ? `rgba(255, 120, 80, ${fade})` : `rgba(80, 200, 255, ${fade})`; ctx.beginPath(); ctx.arc(x, y, 6 + (1 - fade) * 10, 0, Math.PI * 2); ctx.strokeStyle = color; ctx.lineWidth = 2; ctx.stroke(); } requestAnimationFrame(draw); } draw(); ``` The event detection compares each raw value against the smoothed average. If the deviation exceeds a threshold, an event object is created with a type (spike up or down) and an age counter. Events fade out over time -- the expanding, fading ring effect gives a visual "ripple" that marks where something interesting happened. Old events scroll off the left edge as new data pushes in. This is the pattern behind every monitoring dashboard you've ever seen. Server load spike? Red dot. Traffic surge? Pulse. The threshold and the visual response are creative decisions. A lower threshold catches more events (noisier but more sensitive). A higher threshold catches only the dramatic ones. The visual response could be anything -- flashing colors, particle bursts, sound triggers. We built audio-reactive visuals in episode 19; the same approach works here, but triggered by data events instead of beat detection. ## Multiple streams: layered visualization Real systems don't produce one number. A weather station reports temperature, humidity, wind speed, and pressure simultaneously. A server reports CPU, memory, disk, and network. Visualizing multiple streams together reveals correlations that single-stream views hide. ```javascript const canvas = document.createElement('canvas'); canvas.width = 900; canvas.height = 500; document.body.appendChild(canvas); const ctx = canvas.getContext('2d'); const BUFFER_SIZE = 150; const streams = { temperature: { buffer: [], color: 'rgba(255, 130, 80, 0.7)', smoothed: 20 }, humidity: { buffer: [], color: 'rgba(80, 180, 255, 0.7)', smoothed: 60 }, windSpeed: { buffer: [], color: 'rgba(120, 220, 140, 0.7)', smoothed: 10 }, pressure: { buffer: [], color: 'rgba(200, 160, 255, 0.7)', smoothed: 1013 } }; const ranges = { temperature: { min: -10, max: 45 }, humidity: { min: 0, max: 100 }, windSpeed: { min: 0, max: 80 }, pressure: { min: 960, max: 1060 } }; function addStreamData(name, value) { const stream = streams[name]; stream.smoothed = stream.smoothed * 0.9 + value * 0.1; stream.buffer.push(stream.smoothed); if (stream.buffer.length > BUFFER_SIZE) stream.buffer.shift(); } function simulateWeather() { const t = Date.now() / 10000; addStreamData('temperature', 20 + Math.sin(t) * 8 + (Math.random() - 0.5) * 4); addStreamData('humidity', 60 + Math.cos(t * 0.7) * 20 + (Math.random() - 0.5) * 10); addStreamData('windSpeed', 15 + Math.sin(t * 1.3) * 12 + (Math.random() - 0.5) * 8); addStreamData('pressure', 1013 + Math.sin(t * 0.3) * 15 + (Math.random() - 0.5) * 3); } function draw() { ctx.fillStyle = '#0a0a1a'; ctx.fillRect(0, 0, 900, 500); const streamNames = Object.keys(streams); const laneHeight = 500 / streamNames.length; for (let s = 0; s < streamNames.length; s++) { const name = streamNames[s]; const stream = streams[name]; const range = ranges[name]; const laneTop = s * laneHeight; // lane separator if (s > 0) { ctx.beginPath(); ctx.moveTo(0, laneTop); ctx.lineTo(900, laneTop); ctx.strokeStyle = 'rgba(60, 70, 90, 0.3)'; ctx.lineWidth = 1; ctx.stroke(); } // label ctx.fillStyle = 'rgba(140, 150, 170, 0.4)'; ctx.font = '10px monospace'; ctx.textAlign = 'left'; ctx.fillText(name, 10, laneTop + 15); // draw stream line if (stream.buffer.length > 1) { ctx.beginPath(); for (let i = 0; i < stream.buffer.length; i++) { const x = (i / (BUFFER_SIZE - 1)) * 860 + 20; const normalized = (stream.buffer[i] - range.min) / (range.max - range.min); const y = laneTop + laneHeight - 10 - normalized * (laneHeight - 25); if (i === 0) ctx.moveTo(x, y); else ctx.lineTo(x, y); } ctx.strokeStyle = stream.color; ctx.lineWidth = 1.5; ctx.stroke(); // fill under the line const lastX = ((stream.buffer.length - 1) / (BUFFER_SIZE - 1)) * 860 + 20; ctx.lineTo(lastX, laneTop + laneHeight - 10); ctx.lineTo(20, laneTop + laneHeight - 10); ctx.closePath(); ctx.fillStyle = stream.color.replace('0.7', '0.08'); ctx.fill(); } } requestAnimationFrame(draw); } // simulate data arriving setInterval(simulateWeather, 200); draw(); ``` Four lanes, four streams, each normalized to its own range. Temperature in warm orange, humidity in cool blue, wind in green, pressure in purple. The filled area under each line gives visual weight -- you can see at a glance which streams are high and which are low. When temperature spikes and pressure drops simultaneously, the correlation is visible across lanes. The lane layout is one of many approaches. You could also overlay all streams on a single axis (with different y-scales -- gets confusing fast), use a stacked area chart (good for proportions), or give each stream its own visual treatment (temperature as a color gradient, wind as moving particles, humidity as opacity). The creative decision is which relationships you want to make visible. ## Connection health: showing the invisible Real-time connections fail. The server reboots. The network hiccups. The user's wifi drops for ten seconds. If your visualization just freezes and you don't tell the viewer what happened, they'll think it's broken. Showing connection status is a user experience requirement, but it's also a creative opportunity -- the gap in the data IS data. ```javascript const canvas = document.createElement('canvas'); canvas.width = 900; canvas.height = 300; document.body.appendChild(canvas); const ctx = canvas.getContext('2d'); let connected = true; let lastDataTime = Date.now(); const CONNECTION_TIMEOUT = 3000; // 3 seconds without data = "stale" const buffer = []; const BUFFER_SIZE = 200; function addDataPoint(value) { lastDataTime = Date.now(); connected = true; buffer.push({ value: value, connected: true }); if (buffer.length > BUFFER_SIZE) buffer.shift(); } function checkConnection() { const timeSinceData = Date.now() - lastDataTime; if (timeSinceData > CONNECTION_TIMEOUT) { connected = false; // insert a "gap" marker buffer.push({ value: null, connected: false }); if (buffer.length > BUFFER_SIZE) buffer.shift(); } } function draw() { ctx.fillStyle = connected ? '#0a0a1a' : '#0f0808'; ctx.fillRect(0, 0, 900, 300); // draw data line with gaps let inSegment = false; for (let i = 0; i < buffer.length; i++) { const point = buffer[i]; const x = (i / (BUFFER_SIZE - 1)) * 860 + 20; if (point.connected && point.value !== null) { const y = 280 - (point.value / 100) * 260; if (!inSegment) { ctx.beginPath(); ctx.moveTo(x, y); inSegment = true; } else { ctx.lineTo(x, y); } } else { if (inSegment) { ctx.strokeStyle = 'rgba(120, 180, 255, 0.7)'; ctx.lineWidth = 1.5; ctx.stroke(); inSegment = false; } // draw gap indicator ctx.fillStyle = 'rgba(255, 60, 60, 0.1)'; ctx.fillRect(x - 2, 0, 4, 300); } } if (inSegment) { ctx.strokeStyle = 'rgba(120, 180, 255, 0.7)'; ctx.lineWidth = 1.5; ctx.stroke(); } // connection status indicator ctx.beginPath(); ctx.arc(870, 20, 6, 0, Math.PI * 2); ctx.fillStyle = connected ? 'rgba(80, 200, 120, 0.8)' : 'rgba(255, 60, 60, 0.8)'; ctx.fill(); requestAnimationFrame(draw); } setInterval(checkConnection, 1000); draw(); ``` When data stops arriving, the background tints slightly red, the line breaks, and faint red columns mark the gap. A small status dot in the corner shows green (connected) or red (disconnected). The viewer knows instantly: "this isn't frozen, the data source went away." The gap handling is important for data integrity too. Without gap markers, the line would connect the last pre-gap point to the first post-gap point with a straight segment, implying a transition that never happened. The gap says "I don't know what happened here" -- which is honest. Showing absence of data is just as important as showing data. ## Batching fast data Some data sources are fast. Really fast. A cryptocurrency order book might update 50 times per second. A network packet capture might log thousands of events per second. Your screen refreshes at 60fps. If data arrives faster than your frame rate, you can't draw every individual point -- and you shouldn't try. The solution: batch processing. Collect all data points that arrive between frames, then process the batch at the start of each frame. ```javascript const canvas = document.createElement('canvas'); canvas.width = 900; canvas.height = 400; document.body.appendChild(canvas); const ctx = canvas.getContext('2d'); const pendingData = []; const processed = []; const MAX_POINTS = 300; function onDataArrived(value) { // data arrives at unpredictable rate -- just queue it pendingData.push(value); } function processBatch() { if (pendingData.length === 0) return; // option 1: use the average of the batch const avg = pendingData.reduce((s, v) => s + v, 0) / pendingData.length; // option 2: use the min and max (shows range) const min = Math.min(...pendingData); const max = Math.max(...pendingData); processed.push({ avg, min, max, count: pendingData.length }); if (processed.length > MAX_POINTS) processed.shift(); // clear the queue pendingData.length = 0; } function draw() { processBatch(); ctx.fillStyle = '#0a0a1a'; ctx.fillRect(0, 0, 900, 400); for (let i = 0; i < processed.length; i++) { const x = (i / (MAX_POINTS - 1)) * 860 + 20; const point = processed[i]; // draw min-max range as a vertical bar const yMin = 380 - (point.min / 100) * 360; const yMax = 380 - (point.max / 100) * 360; const yAvg = 380 - (point.avg / 100) * 360; ctx.fillStyle = 'rgba(120, 180, 255, 0.15)'; ctx.fillRect(x - 1, yMax, 2, yMin - yMax); // draw average as a dot ctx.beginPath(); ctx.arc(x, yAvg, 1.5, 0, Math.PI * 2); ctx.fillStyle = 'rgba(120, 180, 255, 0.7)'; ctx.fill(); } requestAnimationFrame(draw); } // simulate fast data: 30 updates per second setInterval(function() { onDataArrived(50 + (Math.random() - 0.5) * 40); }, 33); draw(); ``` Each frame processes all pending data points at once. Instead of drawing each individual value, it draws a summary: a faint vertical bar showing the min-max range, and a bright dot for the average. This is essentially what a candlestick chart does -- compressing variable-rate data into fixed-rate visual summaries. The bar thickness tells you how volatile the data was during that frame. Thin bars = stable. Thick bars = noisy. This pattern scales to any data rate. Ten updates per second? Each batch has one point, the bar and dot are the same. Ten thousand updates per second? Each batch aggregates ~166 points, and the min-max bar shows the actual spread. The visualization adapts automatically. ## Reconnection logic WebSocket connections die. The server restarts, the network blips, the client's laptop goes to sleep. A production real-time visualization needs reconnection logic -- automatic retry with backoff so you don't hammer a dead server. ```javascript function createReconnectingSocket(url, onMessage) { let ws = null; let retryDelay = 1000; const MAX_DELAY = 30000; let shouldReconnect = true; function connect() { ws = new WebSocket(url); ws.onopen = function() { console.log('connected'); retryDelay = 1000; // reset backoff on successful connection }; ws.onmessage = function(event) { onMessage(JSON.parse(event.data)); }; ws.onclose = function() { if (!shouldReconnect) return; console.log('disconnected, retrying in ' + retryDelay + 'ms'); setTimeout(connect, retryDelay); // exponential backoff: 1s, 2s, 4s, 8s, ... up to 30s retryDelay = Math.min(retryDelay * 2, MAX_DELAY); }; ws.onerror = function() { ws.close(); // triggers onclose -> reconnect }; } connect(); return { close: function() { shouldReconnect = false; if (ws) ws.close(); } }; } // usage const socket = createReconnectingSocket( 'wss://stream.example.com/data', function(data) { addDataPoint(data.value); } ); ``` The exponential backoff doubles the retry delay each time: 1 second, 2 seconds, 4, 8, 16, capped at 30. This prevents hammering a crashed server with thousands of reconnection atempts. When the connection succeeds, the delay resets to 1 second so the next failure recovers quickly. The `shouldReconnect` flag lets you cleanly close the connection without triggering the retry loop. Without it, calling `ws.close()` would fire `onclose` which would reconnect -- an infinite loop. ## Accumulation: building up over time Not all real-time data should scroll. Sometimes each new data point *adds* to the visualization permanently. A map of earthquakes filling up over hours. A network graph where each new message adds an edge. A timeline where events accumulate into density patterns. The visualization grows instead of scrolling. ```javascript const canvas = document.createElement('canvas'); canvas.width = 700; canvas.height = 700; document.body.appendChild(canvas); const ctx = canvas.getContext('2d'); // dark background drawn once ctx.fillStyle = '#0a0a1a'; ctx.fillRect(0, 0, 700, 700); function addEvent(x, y, magnitude) { // each event is a semi-transparent circle // over time they accumulate, building density const radius = magnitude * 8; const hue = 200 + magnitude * 15; const alpha = 0.03 + magnitude * 0.02; ctx.beginPath(); ctx.arc(x, y, radius, 0, Math.PI * 2); ctx.fillStyle = `hsla(${hue}, 50%, 50%, ${alpha})`; ctx.fill(); // bright core ctx.beginPath(); ctx.arc(x, y, 2, 0, Math.PI * 2); ctx.fillStyle = `hsla(${hue}, 60%, 70%, ${alpha * 3})`; ctx.fill(); } // simulate events arriving over time setInterval(function() { const x = Math.random() * 700; const y = Math.random() * 700; const mag = 0.5 + Math.random() * 4; addEvent(x, y, mag); }, 200); ``` No `clearRect` -- that's the trick. Each new event adds a semi-transparent circle on top of everything that came before. Areas with many events build up bright through the accumulated alpha. Areas with few events stay dark. After hundreds of events, a heat map emerges from the additive blending. The visualization is its own history. This is the technique behind one of my favourite data art patterns: the long-exposure data photograph. Just like a long-exposure camera shot accumulates light over time, the canvas accumulates data marks. The result shows not just where events happened but how *often* they happened at each location. Density becomes brightness. Frequency becomes opacity. Time collapses into a single image. ## Creative exercise: live data ticker Allez, time to bring it all together. We'll build a real-time ticker visualization that combines scrolling, smoothing, event detection, and connection status into a single cohesive piece. The data simulates a live stream (swap in a real WebSocket whenever you want -- the visualization code doesn't change). ```javascript const canvas = document.createElement('canvas'); canvas.width = 900; canvas.height = 500; document.body.appendChild(canvas); const ctx = canvas.getContext('2d'); const BUFFER_SIZE = 250; const rawBuffer = []; const smoothBuffer = []; const events = []; let smoothed = 50; let connected = true; let lastData = Date.now(); function addPoint(value) { lastData = Date.now(); connected = true; rawBuffer.push(value); if (rawBuffer.length > BUFFER_SIZE) rawBuffer.shift(); const prevSmooth = smoothed; smoothed = smoothed * 0.88 + value * 0.12; smoothBuffer.push(smoothed); if (smoothBuffer.length > BUFFER_SIZE) smoothBuffer.shift(); // event detection const dev = Math.abs(value - smoothed); if (dev > 12) { events.push({ idx: rawBuffer.length - 1, val: value, type: value > smoothed ? 'up' : 'down', age: 0 }); } for (let i = events.length - 1; i >= 0; i--) { events[i].age++; events[i].idx--; if (events[i].idx < 0 || events[i].age > 100) { events.splice(i, 1); } } } function toX(i) { return (i / (BUFFER_SIZE - 1)) * 840 + 30; } function toY(v) { return 460 - (v / 100) * 420; } function draw() { // connection check if (Date.now() - lastData > 2500) connected = false; ctx.fillStyle = connected ? '#0a0a1a' : '#100808'; ctx.fillRect(0, 0, 900, 500); // grid lines for (let v = 0; v <= 100; v += 25) { const y = toY(v); ctx.beginPath(); ctx.moveTo(30, y); ctx.lineTo(870, y); ctx.strokeStyle = 'rgba(40, 50, 65, 0.4)'; ctx.lineWidth = 1; ctx.stroke(); ctx.fillStyle = 'rgba(100, 110, 130, 0.3)'; ctx.font = '9px monospace'; ctx.textAlign = 'right'; ctx.fillText(v.toString(), 25, y + 3); } // raw line (faint) if (rawBuffer.length > 1) { ctx.beginPath(); for (let i = 0; i < rawBuffer.length; i++) { const x = toX(i); const y = toY(rawBuffer[i]); if (i === 0) ctx.moveTo(x, y); else ctx.lineTo(x, y); } ctx.strokeStyle = 'rgba(80, 110, 160, 0.2)'; ctx.lineWidth = 1; ctx.stroke(); } // smooth line (bright) if (smoothBuffer.length > 1) { ctx.beginPath(); for (let i = 0; i < smoothBuffer.length; i++) { const x = toX(i); const y = toY(smoothBuffer[i]); if (i === 0) ctx.moveTo(x, y); else ctx.lineTo(x, y); } ctx.strokeStyle = 'rgba(120, 180, 255, 0.8)'; ctx.lineWidth = 2; ctx.stroke(); } // events for (const evt of events) { if (evt.idx < 0 || evt.idx >= BUFFER_SIZE) continue; const x = toX(evt.idx); const y = toY(evt.val); const fade = 1 - evt.age / 100; // expanding ring ctx.beginPath(); ctx.arc(x, y, 5 + (1 - fade) * 15, 0, Math.PI * 2); ctx.strokeStyle = evt.type === 'up' ? `rgba(255, 120, 80, ${fade * 0.6})` : `rgba(80, 200, 255, ${fade * 0.6})`; ctx.lineWidth = 1.5; ctx.stroke(); // core dot ctx.beginPath(); ctx.arc(x, y, 3, 0, Math.PI * 2); ctx.fillStyle = evt.type === 'up' ? `rgba(255, 120, 80, ${fade})` : `rgba(80, 200, 255, ${fade})`; ctx.fill(); } // current value if (smoothBuffer.length > 0) { const current = smoothBuffer[smoothBuffer.length - 1]; ctx.fillStyle = 'rgba(200, 210, 230, 0.7)'; ctx.font = '16px monospace'; ctx.textAlign = 'right'; ctx.fillText(current.toFixed(1), 880, 25); } // connection dot ctx.beginPath(); ctx.arc(45, 20, 4, 0, Math.PI * 2); ctx.fillStyle = connected ? 'rgba(80, 200, 120, 0.7)' : 'rgba(255, 60, 60, 0.7)'; ctx.fill(); requestAnimationFrame(draw); } // simulate real-time data let simValue = 50; setInterval(function() { simValue += (Math.random() - 0.48) * 4; if (Math.random() < 0.02) simValue += (Math.random() - 0.5) * 30; // occasional spike simValue = Math.max(0, Math.min(100, simValue)); addPoint(simValue); }, 150); draw(); ``` Scrolling buffer. Faint raw line behind a bright smoothed line. Expanding ring markers on spikes. Grid lines with numeric labels. Current value readout. Connection status dot. The 2% random spike chance creates occasional dramatic moments -- the event markers light up, ripple outward, and fade. Between spikes, the smoothed line drifts peacefully. The whole thing feels alive in a way that static visualizations just can't. And here's the thing -- swap out the `setInterval` with a real WebSocket connection and the visualization works exactly the same way. That's the beauty of the buffer pattern: the visualization doesn't care where the data comes from. It's the same `addPoint()` function whether the data is simulated, loaded from a file, or streaming live from a server across the world. The data art arc has taken us from static file loading (episode 80-81) through visual mapping (82), geographic (83), temporal (84), network (85), text (86), and now real-time (this episode). Each one builds on the tools from the previous episodes. The `map()` function. Log scaling. Color palettes. The rolling buffer we built today is fundamentally the same temporal structure as the timeline from episode 84 -- it just happens to be updating in real time instead of drawn from a pre-loaded array. Same tools, different time scale. See where this is going? :-) Real-time data connects everything we've done in this arc. Geographic data arriving in real time becomes a live earthquake map. Text data streaming gives you real-time sentiment monitoring. Network data evolving over time shows you how a social graph grows. The real-time layer is what turns a data visualization into a data *experience* -- something that lives and breathes alongside the system it represents. ## 't Komt erop neer... - Real-time data arrives continuously from live sources -- sensor readings, price tickers, social feeds, server metrics. Static visualization shows a snapshot; real-time visualization shows a living stream. The data is alive, and the canvas should be too - WebSockets provide persistent bidirectional connections between browser and server. Data arrives via `onmessage` whenever the server pushes it -- no polling required. Four event handlers: `onopen`, `onmessage`, `onclose`, `onerror` - Server-Sent Events (SSE) are simpler for one-way streams. `EventSource` has built-in automatic reconnection. Good for price feeds, notification streams, activity logs -- anywhere the server pushes and you just consume - The rolling buffer is the core pattern: fixed-size array, `push()` new data, `shift()` old data. Buffer size determines how much history is visible. Drawing maps array index to x position so the newest point is always on the right and old data scrolls left - Simulating real-time data with `setInterval` lets you develop visualizations without a live server. Random walk, sinusoidal patterns, step functions -- match the simulation to your target data shape. The visualization code doesn't care whether data comes from a server or from `setInterval` - Exponential moving average (EMA) smooths noisy data: `smoothed = smoothed * (1 - alpha) + value * alpha`. Low alpha (0.05) = heavy smoothing, high alpha (0.5) = light smoothing. Drawing both raw and smoothed lines together shows the signal-noise relationship - Event detection compares raw values against the smoothed average. Deviations beyond a threshold trigger visual markers -- expanding rings, color flashes, particle bursts. The threshold controls sensitivity: lower catches more events, higher catches only dramatic ones - Multiple streams in a lane layout let you spot correlations between data dimensions. Each stream gets its own normalized y-range, its own color, and a subtle area fill. Temperature, humidity, wind, pressure -- when they move together, you see it across lanes - Connection health is visible data. When the stream goes silent, tint the background, break the line, mark the gap. A status dot (green/red) tells the viewer instantly whether data is flowing. Gaps in the data are just as informative as the data itself - Fast data needs batching: collect all points between frames, process the batch once per frame. Summarize with average + min-max range. The min-max bar shows volatility; the dot shows the center. This scales from 10 updates/sec to 10,000 without changing the rendering code - Exponential backoff for reconnection: retry after 1s, then 2s, 4s, 8s, up to a cap. Reset the delay on successful connection. Prevents hammering a crashed server while recovering quickly from brief network hiccups - Accumulation (no `clearRect`) turns each data point into a permanent semi-transparent mark. Over time, density builds through additive alpha. Areas with many events glow bright; sparse areas stay dark. Long-exposure data photography -- time collapses into a single image Sallukes! Thanks for reading. X @femdev

VR Data Visualization Learnings from the Place Viewer