pick call (episode 6 and 33 thinking, applied to traffic);Learn Zig Series):Last episode we built a reverse proxy that could route: given an incoming request, its longest-prefix matcher picked the one backend that a path was supposed to reach. That's the switchboard. Today we make the switchboard smart, because a route that points at exactly one machine is a single point of failure with extra steps. The reason reverse proxies conquered the web is not routing at all -- it's that one route can point at a whole pool of identical backends, and the proxy gets to choose which one handles each request. Spread the load, survive a dead box, drain a machine for maintenance without a single client noticing. Having said that, roll up your sleeves -- this is where the humble middleman earns its keep.
Think about what "load balancing" actually buys you, because the word gets thrown around like it's magic. You run three copies of your application server. Without a balancer, you'd have to tell clients about all three, and hope they spread themselves evenly (they never do). With a balancer out front, clients see one address, and the proxy quietly deals out requests across the three. If one dies, the proxy stops sending it traffic and nobody gets an error. If you need a fourth, you add it to the pool and it starts taking its share immediately. The client stays blissfully ignorant of the whole dance. That ignorance IS the product.
Remember last episode's Route: a prefix and a single upstream address. The entire change that unlocks load balancing is embarrassingly small in the type system -- we swap that one address for a slice of addresses, and add a little state to remember our place. Episode 6's lesson yet again: get the shape right and the logic follows.
const std = @import("std");
/// A pool of interchangeable backends behind one route. Instead of ep103's single
/// `upstream`, a Route now owns a slice of addresses (all serving the same content)
/// plus the tiny bit of state a balancer needs to pick between them. The addresses
/// are borrowed from config that outlives us -- same slice-as-a-view thinking as ep5.
const Backend = struct {
address: std.net.Address,
weight: u32 = 1, // relative share of traffic; 1 means "an ordinary box"
// In-flight request count, bumped when we hand this backend a request and
// dropped when the exchange finishes. Atomic because many threads touch it.
inflight: std.atomic.Value(u32) = std.atomic.Value(u32).init(0),
};
/// One routing rule, now pointing at a POOL. `prefix` still selects the rule by
/// longest-prefix match exactly as in ep103; `backends` is where the request can land.
const Route = struct {
prefix: []const u8,
backends: []Backend,
// Round-robin cursor for this route. Atomic so two threads incrementing it at
// once can't both read the same value and stampede onto the same backend (ep30).
cursor: std.atomic.Value(usize) = std.atomic.Value(usize).init(0),
};
Two small but important decisions are already baked in here. First, inflight and cursor are std.atomic.Value, not plain integers, because the whole point of a proxy is to handle many connections at once, and the moment two threads pick a backend simultaneously we're in the shared-mutable-state minefield we mapped back in episode 30. Second, Backend carries a weight -- most of the balancing strategies we're about to write reduce to "how do we honour that number", so we put it in the type from the start in stead of bolting it on later.
Round-robin is the "just take turns" strategy, and for a pool of roughly-equal backends handling roughly-equal requests, it's genuinely hard to beat. Request 1 goes to backend 0, request 2 to backend 1, request 3 to backend 2, request 4 back to backend 0, and around we go. The distribution is perfectly even by construction, there's no per-backend bookkeeping, and it's dirt cheap. The only thing that makes it non-trivial is that shared cursor.
/// Pick the next backend for `route` by round-robin. The single subtle line is the
/// atomic fetchAdd: it hands each caller a UNIQUE, monotonically increasing ticket,
/// even when a dozen threads call this at the same instant. We then map that ticket
/// onto a backend index with modulo. Without the atomic, two threads could read the
/// same cursor value, both pick backend 0, and the "balancing" would clump (ep30).
fn pickRoundRobin(route: *Route) usize {
// .monotonic ordering is all we need here: we don't care about ordering relative
// to OTHER memory, only that each ticket is distinct. That's the cheapest atomic.
const ticket = route.cursor.fetchAdd(1, .monotonic);
return ticket % route.backends.len;
}
Notice how the atomic does exactly the job a mutex would, but without the lock. fetchAdd is a single hardware instruction on any CPU you'll meet (lock xadd on x86, an ldadd-family instruction on ARM), so a thousand threads can be picking backends concurrently and none of them ever blocks. This is the payoff of episode 30's atomics chapter landing in a real scenario: we identified that the cursor is the only shared mutable state, we made just that one field atomic, and everything else stays lock-free and fast. Nota bene: the cursor keeps counting up forever and we lean on modulo to wrap it -- a usize overflow is 585 years away at a billion requests a second, so I'm comfortable not losing sleep over it ;-)
Round-robin is the default, not the law. Different services want different strategies, and hard-coding one into the proxy would be exactly the kind of rigidity we teach against. So we make the strategy a value -- a tagged union, the same tool we used to build a state machine in episode 33 -- and let each route carry the strategy it wants.
/// The balancing strategies our proxy understands. A tagged union (ep6, ep33) lets
/// us store the CHOICE as data and switch on it in one place, instead of scattering
/// `if (strategy == ...)` checks across the codebase. Adding a strategy later means
/// adding one variant and one switch arm -- the compiler then FORCES us to handle it
/// everywhere, which is exactly the exhaustiveness guarantee we want.
const Strategy = union(enum) {
round_robin,
random: u64, // carries a seed so the choice is reproducible in tests
least_connections,
weighted_round_robin,
};
/// The single entry point every request goes through. Given a route and a strategy,
/// return the INDEX of the backend to use. Pure-ish: it reads/updates the route's
/// atomic counters but performs no I/O, so we can test the distribution exhaustively.
fn pick(route: *Route, strategy: Strategy) usize {
return switch (strategy) {
.round_robin => pickRoundRobin(route),
.random => |seed| pickRandom(route, seed),
.least_connections => pickLeastConnections(route),
.weighted_round_robin => pickWeighted(route),
};
}
The beauty of switch over a tagged union in Zig is that it's exhaustive by default -- if I add a .ip_hash variant next month and forget to handle it here, the compiler stops the build cold and points at this exact function. That's not a nicety, it's a correctness guarantee: a load balancer that silently falls through to "do nothing" on an unhandled strategy is a 3am pager incident waiting to happen. The types make the omission impossible in stead of merely unlikely.
Here's the random strategy while we're at it -- sometimes you genuinely want a stateless pick (no shared cursor to contend on at all), and a decent PRNG gives you an even spread across enough requests:
/// Random selection. Stateless across calls -- no cursor contention whatsoever -- at
/// the cost of only being EVENTUALLY even, not perfectly even like round-robin. For
/// large request volumes the law of large numbers makes it indistinguishable in
/// practice, and it sidesteps the atomic entirely, which can matter under extreme load.
fn pickRandom(route: *Route, seed: u64) usize {
var prng = std.Random.DefaultPrng.init(seed);
return prng.random().uintLessThan(usize, route.backends.len);
}
Real pools are rarely uniform. You've got a shiny 32-core box next to a five-year-old server you haven't decommissioned yet, and sending them equal traffic is silly -- the old one will be gasping while the new one idles. Weighted round-robin fixes this by giving each backend a share proportional to its weight: a backend with weight 3 gets three requests for every one that a weight-1 backend gets.
The naive implementation (build an expanded list where each backend appears weight times, then round-robin over that) works but wastes memory and doesn't handle changing weights well. The classic smooth-weighted algorithm -- the one nginx actually ships -- keeps a running "current weight" per backend, and it's more elegant than it first looks.
/// Smooth weighted round-robin (the algorithm nginx uses). Each pick:
/// 1. add every backend's configured weight to its running current_weight,
/// 2. choose the backend with the highest current_weight,
/// 3. subtract the TOTAL weight from the winner's current_weight.
/// This spreads a weight-{5,1,1} pool as A B A C A A A (smooth), not A A A A A B C
/// (bursty), which keeps latency even instead of hammering the big box in a clump.
/// We store current_weight as a signed running total per backend in `cw`.
fn pickWeightedSmooth(backends: []const Backend, cw: []i64) usize {
var total: i64 = 0;
for (backends) |b| total += b.weight;
var best: usize = 0;
var best_cw: i64 = std.math.minInt(i64);
for (backends, 0..) |b, i| {
cw[i] += b.weight;
if (cw[i] > best_cw) {
best_cw = cw[i];
best = i;
}
}
cw[best] -= total;
return best;
}
Why bother with "smooth" instead of the expand-the-list version? Because burstiness is latency's enemy. If your weight-5 backend gets five requests back-to-back and then sits idle, its request queue spikes and drains, spikes and drains -- and a request unlucky enough to land at the top of a spike waits behind four others. The smooth version interleaves the heavy backend's turns evenly through the rotation, so every backend's queue depth stays as flat as its weight allows. Same average distribution, far better tail latency. This is the kind of second-order thing that separates a toy balancer from one you'd actually run, and it's why I reach for the nginx algorithm rather than the obvious one.
Round-robin and its weighted cousin are stateless about the backends -- they deal cards without looking at how fast each player is going through them. That's fine when every request costs about the same. But the moment request costs vary wildly -- one hits a cache and returns in a millisecond, the next runs a fat database report for ten seconds -- round-robin can pile three slow requests onto the same backend by sheer bad luck while its neighbours idle. Least-connections fixes this by looking at reality: send the next request to whichever backend currently has the fewest requests in flight.
/// Least-connections: choose the backend with the smallest live in-flight count.
/// This is the strategy that ADAPTS to real load -- a backend stuck on a slow request
/// naturally accumulates in-flight count and stops being chosen until it drains. We
/// read each backend's atomic `inflight` with .monotonic (a slightly stale read is
/// fine; we're picking a good backend, not proving a theorem). Ties break toward the
/// lowest index, which is arbitrary but deterministic -- and determinism is testable.
fn pickLeastConnections(route: *Route) usize {
var best: usize = 0;
var best_load: u32 = std.math.maxInt(u32);
for (route.backends, 0..) |*b, i| {
const load = b.inflight.load(.monotonic);
if (load < best_load) {
best_load = load;
best = i;
}
}
return best;
}
The critical detail that makes least-connections work is the bookkeeping around it: you have to bump inflight the instant you hand a backend a request, and drop it the instant the exchange finishes -- win, lose, or crash. Miss the decrement on an error path and that backend's count creeps upward forever, until the balancer decides it's permanently "busy" and stops sending it anything. This is a textbook place for Zig's defer to save your bacon, because defer runs on every exit from a scope, including the error returns you'd otherwise forget.
/// Increment on entry, guarantee the decrement on EVERY exit. `defer` is the whole
/// trick: it fires whether relay() returns normally, returns an error, or we bail on
/// a route-not-found -- so the in-flight count can never leak. Forgetting this is the
/// #1 way a hand-rolled least-connections balancer slowly poisons its own pool.
fn withInflightTracking(backend: *Backend, comptime relayFn: anytype, args: anytype) !void {
_ = backend.inflight.fetchAdd(1, .monotonic);
defer _ = backend.inflight.fetchSub(1, .monotonic);
try @call(.auto, relayFn, args);
}
Every strategy above is a pure function of its inputs -- indices and counters in, an index out, no I/O anywhere. That was a deliberate design choice, the exact same one that made episode 102's response framing bulletproof: isolate the bug-prone decision into something you can hammer with deterministic tests that run in microseconds and never touch the network. Let's prove round-robin is fair and weighted honours its weights.
test "round-robin distributes evenly across the pool" {
var backends = [_]Backend{
.{ .address = undefined },
.{ .address = undefined },
.{ .address = undefined },
};
var route = Route{ .prefix = "/", .backends = &backends };
// 3000 picks over 3 backends must land exactly 1000 each -- perfectly even.
var counts = [_]usize{ 0, 0, 0 };
var i: usize = 0;
while (i < 3000) : (i += 1) counts[pickRoundRobin(&route)] += 1;
try std.testing.expectEqual(@as(usize, 1000), counts[0]);
try std.testing.expectEqual(@as(usize, 1000), counts[1]);
try std.testing.expectEqual(@as(usize, 1000), counts[2]);
}
test "smooth weighted honours the weight ratio" {
const backends = [_]Backend{
.{ .address = undefined, .weight = 5 },
.{ .address = undefined, .weight = 1 },
.{ .address = undefined, .weight = 1 },
};
var cw = [_]i64{ 0, 0, 0 };
// Over 7 picks (total weight), backend 0 must win 5, the others 1 each.
var counts = [_]usize{ 0, 0, 0 };
var i: usize = 0;
while (i < 7) : (i += 1) counts[pickWeightedSmooth(&backends, &cw)] += 1;
try std.testing.expectEqual(@as(usize, 5), counts[0]);
try std.testing.expectEqual(@as(usize, 1), counts[1]);
try std.testing.expectEqual(@as(usize, 1), counts[2]);
// And after a full cycle the running weights return to zero -- proving the
// algorithm is stable and won't drift over millions of requests.
try std.testing.expectEqual(@as(i64, 0), cw[0] + cw[1] + cw[2]);
}
That second test is the one I'd defend hardest in code review, because it pins down the two properties that actually matter: the ratio is right (5:1:1), and the algorithm is stable (running weights sum back to zero after a full cycle, so it won't slowly drift after a billion requests). A distribution that's correct for the first hundred picks but skews after a million is a bug you'd never catch by eyeballing -- but a red test screams about it the moment a refactor breaks the invariant.
Here's the failure mode that load balancing exists to defeat: you pick a backend, dial it, and the connection is refused because that machine just fell over. A naive balancer forwards the client a 502 and calls it a day. A good balancer shrugs, picks a different backend, and the client never knows anything went wrong. That's failover, and it's a small loop around last episode's relay.
/// Try to serve `head` from `route`'s pool, failing over to the next backend when a
/// dial fails. We attempt up to backends.len distinct backends before giving up with
/// a 502. Each successful hand-off tracks in-flight count via defer so least-conn
/// stays honest. This is the difference between "one dead box = errors" and "one dead
/// box = a shrug" -- exactly the resilience the whole pattern is sold on.
fn serveWithFailover(route: *Route, strategy: Strategy, head: []const u8, client: std.net.Stream, scratch: []u8) void {
var attempts: usize = 0;
while (attempts < route.backends.len) : (attempts += 1) {
const idx = pick(route, strategy);
const backend = &route.backends[idx];
_ = backend.inflight.fetchAdd(1, .monotonic);
defer _ = backend.inflight.fetchSub(1, .monotonic);
// relay() is ep103's forwarder: dial the upstream, send the head, pump the
// reply back. If the DIAL fails we try another backend; if it fails MID-stream
// we can't retry (bytes already went to the client), so we surface that.
relay(backend.address, head, client, scratch) catch |err| {
std.log.warn("backend {d} failed ({s}); failing over", .{ idx, @errorName(err) });
continue; // next backend in the pool gets a shot
};
return; // success -- we're done
}
// Every backend in the pool refused us. NOW it's a genuine 502.
std.log.err("all {d} backends down for {s}", .{ route.backends.len, route.prefix });
sendError(client, 502, "Bad Gateway");
}
I want to be honest about the one thing this failover can't do, because glossing over it would be exactly the kind of hand-waving I promised you I wouldn't do. If a backend accepts the connection and dies halfway through streaming the response, we can't transparently retry -- we've already relayed the first chunk of a 200 OK to the client, and we can't un-send those bytes. Retrying would splice a second response onto a half-sent first one, which is worse than an honest error. So failover only covers connection-time failures (refused, timed out, no route to host), which in practice is the overwhelming majority of "backend is dead" situations. Full mid-stream resilience needs idempotency guarantees and response buffering that belong to a much heavier proxy than ours. Knowing precisely where your safety net ends is worth more than pretending it catches everything.
The integration point is last episode's handleClient. Almost nothing changes -- routing still runs first, then in stead of a single relay we call serveWithFailover, and we thread the strategy through. The elegance of having isolated the balancing behind one pick call is that the connection-handling code barely notices it exists.
/// ep103's handleClient, now load-balancing. Route selection is unchanged: longest-
/// prefix match picks the ROUTE. What's new is that the route owns a pool, and
/// serveWithFailover picks a BACKEND from it (per the route's strategy) with failover.
fn handleClient(router: *Router, strategy: Strategy, client: std.net.Stream, buf: []u8, scratch: []u8) void {
defer client.close();
const head_len = readHead(client, buf) catch return;
if (head_len == 0) return;
const head = buf[0..head_len];
const req = parseRequestLine(head) orelse {
sendError(client, 400, "Bad Request");
return;
};
const route = router.matchRoute(req.path) orelse {
sendError(client, 404, "Not Found");
return;
};
// The whole new behaviour lives behind this one call.
serveWithFailover(route, strategy, head, client, scratch);
}
And the main wiring, showing a pool with mixed weights so you can watch the distribution in the logs:
pub fn main() !void {
var gpa = std.heap.GeneralPurposeAllocator(.{}){};
defer _ = gpa.deinit();
const alloc = gpa.allocator();
// A pool of three backends behind /api, the first one weighted heavier because
// (pretend) it's the big box. In a real tool this whole table is ep20's JSON,
// parsed at startup -- hard-coded here so the balancing logic stands undressed.
var api_pool = [_]Backend{
.{ .address = try std.net.Address.parseIp("127.0.0.1", 9001), .weight = 3 },
.{ .address = try std.net.Address.parseIp("127.0.0.1", 9002), .weight = 1 },
.{ .address = try std.net.Address.parseIp("127.0.0.1", 9003), .weight = 1 },
};
var routes = [_]Route{
.{ .prefix = "/api", .backends = &api_pool },
};
var router = Router{ .routes = &routes };
const strategy: Strategy = .round_robin; // swap to .least_connections and re-run
const listen_addr = try std.net.Address.parseIp("127.0.0.1", 8080);
var server = try listen_addr.listen(.{ .reuse_address = true });
defer server.deinit();
std.log.info("balancing proxy listening on {any}", .{listen_addr});
const buf = try alloc.alloc(u8, 64 * 1024);
defer alloc.free(buf);
const scratch = try alloc.alloc(u8, 64 * 1024);
defer alloc.free(scratch);
while (true) {
const conn = server.accept() catch |err| {
std.log.warn("accept failed: {s}", .{@errorName(err)});
continue;
};
handleClient(&router, strategy, conn.stream, buf, scratch);
}
}
Fire up three episode 51-54 servers on 9001, 9002 and 9003, point a loop of requests at the proxy, and watch them spread:
$ for i in $(seq 1 9); do curl -s localhost:8080/api/ping; done
# round-robin: 9001 9002 9003 9001 9002 9003 9001 9002 9003
# kill the server on 9002, repeat:
$ for i in $(seq 1 6); do curl -s localhost:8080/api/ping; done
# every request still succeeds -- failover routes around the dead box, no 502s
The balancing decision itself is, once again, cheap enough to ignore -- round-robin is one atomic add, least-connections is a linear scan over a handful of atomic loads. For the tiny pools real proxies run (a route rarely fans out past a dozen backends), a linear scan wins, because the constant factors of anything cleverer don't pay off at that size and the branch predictor eats the small loop for breakfast. This is episode 34's lesson wearing a different hat: measure before you reach for a fancier data structure, and respect that a loop over a short list is frequently the fastest thing on the menu.
The real cost story, exactly as it was for routing, is the relaying and the concurrency -- how many exchanges you can pump at once. And here the atomics we chose start paying rent. Because the cursor and in-flight counters are lock-free, the balancing layer adds essentially zero contention as you scale from one worker to many; there's no mutex for threads to queue behind on every single request. When we do add real concurrency (a thread per connection from episode 64, or an event loop over poll), the balancer needs no changes at all -- that's the dividend of having made only the genuinely-shared state atomic and left everything else as plain, copyable values. You could see the shared state, so you could reason about it, so scaling it is mechanical rather than terrifying.
One design smell worth naming: notice we're picking a backend on every request with no memory of which client went where. That's correct for stateless backends, but a service that keeps session state in memory (rather than in a shared store) needs session affinity -- the same client landing on the same backend each time. That's a whole strategy of its own (usually hashing the client IP or a cookie), and deliberately out of scope today. And honestly, if you ever find yourself adding affinity to paper over backends that shouldn't be stateful, the better fix is almost always to make the backends stateless -- affinity is a workaround, not an architecture.
In C, this is the beating heart of nginx and HAProxy, and the smooth-weighted algorithm we wrote is literaly nginx's ngx_http_upstream_round_robin boiled down to its essence. What Zig bought us over the C originals is the atomics being first-class and typed -- std.atomic.Value(u32) carries its memory-ordering contract in the type, where the C equivalent is a naked _Atomic int and a lot of trusting the programmer to pass the right memory order to every atomic_fetch_add. Getting a memory order wrong in a C balancer is the kind of bug that only shows up under load on one customer's 64-core machine, once a fortnight. Zig doesn't make that bug impossible, but it makes it visible.
In Rust, you'd lean on tokio for the async runtime and very likely the tower ecosystem, where load balancing is a provided tower::balance layer with p2c (power-of-two-choices) ready to go. You'd get work-stealing async and a borrow-checked guarantee that your shared counters are accessed soundly (an AtomicUsize behind an Arc, the same atomic we used, just wrapped in Rust's ownership ceremony). The tradeoff is the familiar one: a deep, powerful stack you mostly trust versus our from-scratch version where you can point at the precise fetchAdd and know exactly which backend it's about to choose and why.
In Go, the standard library's httputil.ReverseProxy gives you the relay for free but -- interestingly -- not the load balancing; you supply the backend-selection in the Director/Rewrite hook yourself, which is basically our pick function wearing a Go coat. So Go hands you the plumbing and lets you keep the balancing brains, which is a pretty sane division of labour. A dozen lines of Go plus a round-robin counter and you've got what took us an episode -- and having built ours from the sockets and atomics up, you now know exactly what that Go counter needs to be (atomic!) and why a plain int++ in the Director is a data race that'll bite you the day traffic gets serious ;-)
Step back and look at how little genuinely-new machinery today required. The pool is just episode 5's slice. The lock-free cursor and in-flight counters are episode 30's atomics. The pluggable strategy is episode 6 and 33's tagged union. The pure-function-plus-tests rhythm is episode 102's, and the failover loop is a thin wrapper around last episode's relay. The one genuinely new idea -- and it's a portable one -- is that choosing among equivalent backends is a policy, cleanly separable from the mechanics of moving bytes, and that policy is small enough to hold in your head and test to exhaustion.
But our balancer still has a dangerous blind spot, and if you've been reading closely you already feel it. We only discover a backend is dead when we try it and the dial fails -- we're finding out about the fire by walking into the burning room. A real proxy knows a backend is sick before it sends a client there, by quietly checking each one on a schedule and pulling the unhealthy ones out of rotation until they recover. That's the piece that turns "fail over after an error" into "never route to a dead box in the first place", and it's where our little proxy grows up into something you'd actually trust in front of a service. That's next.
Bedankt en tot de volgende keer!
@scipio