rise/fall counters are the single most important knob for stopping a backend from flapping in and out of rotation on one unlucky timeout;pick from last episode so a drained pool returns an honest 503 in stead of knowingly dialing a corpse;Learn Zig Series):Last episode we taught the proxy to spread traffic across a pool and to fail over when a backend refused a connection. That failover is real resilience, and it's better than most hand-rolled proxies ever get -- but read it back and you'll notice something uncomfortable: we only find out a backend is dead by walking into the burning room. A client's request is the thing that discovers the fire. We dial the corpse, the dial fails, and only then do we shrug and try the next box. The client whose request happened to draw the dead backend eats the latency of a failed connect before we route around it. Do that at a thousand requests a second and "one dead box" becomes "a thousand clients each paying a connect-timeout tax until the pool sorts itself out". Having said that, this is the episode where the humble middleman finally grows up.
The fix has a name every ops person knows: health checking. Instead of learning a backend is sick from the requests that stumble onto it, the proxy checks each backend proactively, on its own schedule, and pulls the sick ones out of rotation before a single client is sent their way. When a backend recovers, the proxy notices that too and quietly slides it back into the pool. The client never sees any of it. That invisibility -- the same invisibility that made load balancing feel like magic last time -- is again the whole product. Let's build it, and let's build it honestly, because health checking is a swamp of subtle failure modes and I'd rather show you where the mud is than pretend the path is dry.
There are exactly two ways a proxy can learn a backend's health, and a grown-up proxy uses both. Active health checks are the proxy reaching out on a timer -- every couple of seconds it opens a probe connection (or sends a tiny GET /health) to each backend and notes whether it answered. Passive health checks are the proxy learning from the traffic it's already carrying -- when a real client request fails to connect to a backend, that's a data point too, and a fresher one than any scheduled probe.
Active checks catch a dead box proactively (you know within one probe interval, before any client is harmed) but they cost a trickle of extra connections and they only test what the probe tests. Passive checks cost nothing extra (you're already making the request) and they reflect real traffic, but they're reactive -- you only learn from a request that already suffered. Neither is enough alone. Active-only means a backend can die the instant after a probe and stay "healthy" for a full interval. Passive-only is exactly last episode's failover with a memory. Run both and the windows shrink until they nearly close: active checks bound the worst case, passive checks catch death the very next request. We'll wire up both.
Your first instinct is healthy: bool, and your first instinct is wrong -- or at least too coarse. A backend is not simply up or down; there's a genuinely-different third state, which is we have not checked it yet. At startup, before the first probe has run, a fresh backend deserves the benefit of the doubt: refusing traffic while we warm up would be a self-inflicted outage. So health is a small three-state enum, and thinking of it that way (episode 6's discipline again) prevents a whole category of "why is my brand-new pool serving 503s for the first two seconds" head-scratchers.
const std = @import("std");
/// Health status for a single backend. Three states, NOT a bool, because "we haven't
/// probed it yet" is genuinely distinct from "we probed it and it's down". A fresh
/// backend starts .unknown and is treated as ELIGIBLE until proven guilty -- we'd
/// rather send a request to an unprobed box than refuse traffic during warm-up.
const Health = enum { unknown, healthy, unhealthy };
Now we extend last episode's Backend to carry that health plus the little bit of bookkeeping the checker needs. Two of the new fields matter a lot, and one subtlety hides in which of them are atomic:
/// Last episode's Backend, now health-aware. `health` is read on the hot path by
/// pick() and written by the checker thread, so it MUST be atomic (ep30). The two
/// consecutive counters, by contrast, are touched ONLY by the checker thread -- a
/// single-writer discipline -- so they can stay plain integers. Arranging ownership
/// so that shared state is minimal beats slapping a mutex on everything.
const Backend = struct {
address: std.net.Address,
weight: u32 = 1,
inflight: std.atomic.Value(u32) = std.atomic.Value(u32).init(0),
// Shared across threads: the checker writes it, pick() reads it. Atomic.
health: std.atomic.Value(Health) = std.atomic.Value(Health).init(.unknown),
// Private to the checker thread. Single-writer, so plain ints are safe.
consecutive_ok: u32 = 0,
consecutive_fail: u32 = 0,
};
That split is the design in miniature. The decision (is this backend eligible?) is read constantly, from many threads, on the hottest path in the program -- so it's one atomic load. The reasoning behind the decision (how many probes in a row has it passed or failed?) is only ever computed in one place, by one thread, so it needs no synchronisation at all. Get the ownership right and the concurrency stops being scary.
Here's the single most important idea in this whole episode, and the one that separates a health check you'd trust from one that'll page you at 3am for no reason. Networks are noisy. A single probe can time out because a switch hiccuped, because the backend was briefly busy with a garbage-collection pause, because a cosmic ray looked at your packet funny. If one failed probe yanks a backend out of rotation, and one good probe slams it back in, your pool will flap -- backends oscillating in and out of service on random noise, which is worse than useless because it also churns connection state and confuses every dashboard you own.
The cure is hysteresis: require several consecutive results in the same direction before you believe them. A backend must fail fall probes in a row before we declare it unhealthy, and pass rise probes in a row before we declare it healthy again. Asymmetry is normal and deliberate -- you usually want to pull a sick box fast (low-ish fall) but re-admit a recovering one cautiously (higher rise), so a backend that's still thrashing doesn't get traffic the instant it manages one lucky response.
/// Health-check policy. `rise`/`fall` are the hysteresis knobs: a backend must pass
/// `rise` probes IN A ROW to be marked healthy, and fail `fall` IN A ROW to be marked
/// unhealthy. Requiring several consecutive results is what stops a backend flapping
/// in and out of rotation on one noisy timeout -- the most important tuning here.
const HealthPolicy = struct {
interval_ms: u64 = 2000, // how often to probe each backend
timeout_ms: u64 = 1000, // how long to wait for a single probe to answer
rise: u32 = 2, // consecutive successes needed to (re)admit a backend
fall: u32 = 3, // consecutive failures needed to evict a backend
};
The transition logic itself is a pure function of (current counters, this probe's outcome) -- no sockets, no clock, no I/O -- which, as last episode taught us with the balancing strategies, is exactly what makes it testable to exhaustion. Fold a probe result in, update the counters, and flip the atomic health only when a threshold is crossed:
/// Fold ONE probe result into a backend's health, applying hysteresis. Pure logic:
/// counters in, maybe a health transition out, zero I/O. The checker thread calls this
/// after every probe. Note the reset-on-flip: a success zeroes the failure streak and
/// vice-versa, so "consecutive" really means consecutive and a mixed run never trips.
fn applyProbe(b: *Backend, ok: bool, policy: HealthPolicy) void {
if (ok) {
b.consecutive_fail = 0;
b.consecutive_ok += 1;
if (b.consecutive_ok >= policy.rise) {
b.health.store(.healthy, .monotonic);
}
} else {
b.consecutive_ok = 0;
b.consecutive_fail += 1;
if (b.consecutive_fail >= policy.fall) {
b.health.store(.unhealthy, .monotonic);
}
}
}
Notice we reset the opposite counter on every result. That's what makes "consecutive" honest: a backend that goes fail, ok, fail, ok, fail never accumulates three fails in a row, so it never trips -- and a genuinely flapping backend like that arguably shouldn't be confidently marked either way. The counters encode "how sure are we, and in which direction", and the thresholds are where uncertainty becomes a decision.
What actually is a probe? At the simplest, layer 4: can we open a TCP connection to the backend at all? If the box is down, the kernel refuses the SYN or the connect times out. That's a cheap, meaningful liveness signal and it's where I'd start. At layer 7 you go further -- send GET /health HTTP/1.1 and insist on a 200 -- which catches the nastier failure where the process is accepting connections but is internally broken (database handle dead, disk full, deadlocked worker pool). The shape of the code is identical; the L7 version just writes a few bytes and reads the status line back. We'll do the L4 version and I'll be candid about the timeout.
/// One active probe: can we even open a TCP connection to this backend? A bare connect
/// is the cheapest liveness signal -- refused or timed-out both mean "down". For an L7
/// check you'd send `GET /health` and require a 200; same structure, a few more bytes.
/// We treat ANY error as a failed probe, because from the balancer's chair a refused
/// connect, a timeout, and no-route-to-host all mean the identical thing: don't send
/// users here. Nota bene: a production probe needs a real connect timeout (non-blocking
/// connect + poll), otherwise a black-holed backend hangs the checker for the OS default.
fn probeTcp(address: std.net.Address, timeout_ms: u64) bool {
_ = timeout_ms; // see the note above -- wire SO_SNDTIMEO / non-blocking connect here
const stream = std.net.tcpConnectToAddress(address) catch return false;
stream.close();
return true;
}
I'm flagging that discarded timeout_ms on purpose, because it's the exact spot a naive health checker quietly betrays you. If a backend doesn't refuse the connection but just swallows it -- a black hole, a machine whose network stack is wedged, a firewall dropping packets silently -- then a blocking connect waits for the operating system's default timeout, which can be over a minute. One black-holed backend can stall the entire single-threaded checker, so every other backend's probe is delayed too, and your fast health checker becomes a slow one for the whole pool. The fix is a non-blocking connect followed by a poll with your own deadline, which I'm leaving as the obvious hardening step rather than drowning this listing in socket-option ceremony. Knowing the trap is there is half the battle.
The probes run on their own thread so the accept loop never blocks waiting for one (episode 64 gave us std.Thread.spawn, episode 70 the timing intuition). The loop is deliberately dull: wake, probe every backend of every route, fold each result in, sleep, repeat -- until someone sets the stop flag. And it is the only writer of health state and the only toucher of the consecutive counters, which is precisely the single-writer discipline that lets the hot path read health locklessly and never see a torn value.
/// The active health-check loop. Runs on its own thread (ep64) and is the SOLE writer
/// of health state + the sole toucher of the consecutive counters -- that single-writer
/// discipline is what lets pick() read `health` locklessly. `stop` is checked with
/// .acquire so we see the main thread's .release store promptly on shutdown (ep30).
fn healthLoop(routes: []Route, policy: HealthPolicy, stop: *std.atomic.Value(bool)) void {
while (!stop.load(.acquire)) {
for (routes) |*route| {
for (route.backends) |*b| {
const ok = probeTcp(b.address, policy.timeout_ms);
applyProbe(b, ok, policy);
}
}
std.Thread.sleep(policy.interval_ms * std.time.ns_per_ms);
}
}
A real deployment would probe backends concurrently (one slow probe shouldn't delay the rest, per the black-hole warning above) and would jitter the interval a touch so a hundred proxies don't all hammer the same backend on the same tick. But the skeleton is exactly this: a background heartbeat that keeps everyone's health field honest.
pick to skip the sickAll that probing is worthless unless the balancer acts on it. Last episode's pick chose among all backends blindly; now it must only ever hand out an eligible one. Eligibility is a one-liner with a deliberate subtlety baked in:
/// A backend is eligible for traffic unless we've AFFIRMATIVELY marked it unhealthy.
/// The subtlety: .unknown counts as eligible. A backend we haven't probed yet gets the
/// benefit of the doubt -- pulling traffic during warm-up would be a self-own. Only a
/// backend we positively decided is down gets excluded. A single relaxed atomic load.
fn isEligible(b: *const Backend) bool {
return b.health.load(.monotonic) != .unhealthy;
}
Now round-robin becomes health-aware round-robin: advance the shared cursor exactly as before (still one atomic fetchAdd, still lock-free, straight out of last episode), but skip any backend that isn't eligible, trying at most backends.len steps. If we walk the whole pool and everyone is down, we return null -- and returning null is the honest thing, because knowingly dialing a backend we just decided is dead would be lying to ourselves.
/// Round-robin that respects health. Same atomic cursor as ep104, but we SKIP backends
/// that aren't eligible, bounded to backends.len attempts so a fully-down pool can't
/// loop forever. Returning null means "everyone is down" -- the caller then serves an
/// honest 503 instead of pretending it can help. THIS is where all the probing pays off.
fn pickHealthyRoundRobin(route: *Route) ?usize {
var tries: usize = 0;
while (tries < route.backends.len) : (tries += 1) {
const idx = route.cursor.fetchAdd(1, .monotonic) % route.backends.len;
if (isEligible(&route.backends[idx])) return idx;
}
return null; // pool fully drained -- nobody healthy to serve
}
The other strategies from last episode (least-connections, weighted) get the identical treatment -- filter to eligible backends first, then apply the policy among the survivors. I'll spare you the near-duplicate listings; the pattern is what matters, and the pattern is "health is a gate that sits in front of whatever balancing policy you chose".
Active checks bound our worst case to one probe interval. Passive checks slam that window shut. When a real request fails to even connect to a backend, we shouldn't wait up to two seconds for the next scheduled probe to notice the obvious -- we should react on the spot. This is the circuit breaker pattern, and there's a neat way to fit it in without breaking our single-writer discipline.
The trick is asymmetry: the request path is allowed to trip a backend down but never to bring it up. Tripping down is a single atomic store(.unhealthy), which is safe from any thread. Bringing a backend back up stays the exclusive job of the active checker's applyProbe, so the consecutive counters remain single-writer and untouched by request threads. Any thread can yank a dead box out instantly; only the deliberate, hysteresis-guarded checker can slide one back in.
/// Passive circuit-breaker trip. A live request just failed to connect to this backend,
/// so we short-circuit it to .unhealthy RIGHT NOW rather than waiting an interval for
/// the active checker. Crucially, the request path may only ever trip DOWN -- recovery
/// stays the checker's exclusive job via applyProbe. That asymmetry keeps the counters
/// single-writer while still letting ANY thread evict a corpse the instant it's found.
fn tripUnhealthy(b: *Backend) void {
b.health.store(.unhealthy, .monotonic);
}
I like this design because the concurrency argument is airtight and you can see it at a glance: writes to health come from two places (checker and request threads) but they're all plain atomic stores, so they can't tear; writes to the counters come from exactly one place (the checker), so they need no atomics at all. No mutex, no lock, no lost update, and every line of the reasoning fits in your head. That's the payoff of thinking about who-owns-what before reaching for a lock.
Now last episode's serveWithFailover gets its health upgrade. It asks the pool only for eligible backends via pickHealthyRoundRobin; a connect failure both fails over to the next backend and trips the loser's breaker so the very next request skips it entirely. And when the pool is genuinely empty of healthy backends, we return 503 Service Unavailable -- which is the semantically correct code for "the service is temporarily unable to help", distinct from the 502 Bad Gateway you'd send for a backend that answered with garbage.
/// Failover, now health-aware. We only ever pick ELIGIBLE backends; a connect failure
/// both fails over AND trips that backend's breaker (passive check) so subsequent
/// requests skip it without paying the failed-connect tax. An empty healthy set is an
/// honest 503 -- "no healthy backend right now" -- not last episode's 502.
fn serveWithFailover(route: *Route, head: []const u8, client: std.net.Stream, scratch: []u8) void {
var attempts: usize = 0;
while (attempts < route.backends.len) : (attempts += 1) {
const idx = pickHealthyRoundRobin(route) orelse break; // nobody healthy left
const backend = &route.backends[idx];
_ = backend.inflight.fetchAdd(1, .monotonic);
defer _ = backend.inflight.fetchSub(1, .monotonic);
relay(backend.address, head, client, scratch) catch |err| {
std.log.warn("backend {d} failed ({s}); tripping + failing over", .{ idx, @errorName(err) });
tripUnhealthy(backend); // passive: yank it now, don't wait for the prober
continue;
};
return; // served cleanly
}
std.log.err("no healthy backend for {s}", .{route.prefix});
sendError(client, 503, "Service Unavailable");
}
Read the two halves together and you see the belt-and-suspenders in action. The active checker keeps health roughly current in the background; the request path both consumes that health (via pickHealthyRoundRobin) and feeds it (via tripUnhealthy). A backend that dies is out of rotation within one probe interval at worst, and within one request at best.
Because applyProbe and pickHealthyRoundRobin are pure logic over in-memory state, we can nail the tricky behaviours -- hysteresis and pool-drain -- in tests that run in microseconds and never open a socket. The hysteresis test is the one I'd defend hardest in review, because it pins down the exact anti-flap guarantee the whole episode rests on: one bad probe must not unseat a healthy backend.
test "hysteresis: a single failure does not unseat a healthy backend" {
const policy = HealthPolicy{ .rise = 2, .fall = 3 };
var b = Backend{ .address = undefined };
// Two good probes in a row bring an unknown backend up (rise = 2).
applyProbe(&b, true, policy);
applyProbe(&b, true, policy);
try std.testing.expectEqual(Health.healthy, b.health.load(.monotonic));
// ONE failure must NOT evict it -- this is the anti-flap promise, in one assertion.
applyProbe(&b, false, policy);
try std.testing.expectEqual(Health.healthy, b.health.load(.monotonic));
// But three failures in a row (fall = 3) do evict it.
applyProbe(&b, false, policy);
applyProbe(&b, false, policy);
try std.testing.expectEqual(Health.unhealthy, b.health.load(.monotonic));
}
test "a fully drained pool yields no backend" {
var backends = [_]Backend{
.{ .address = undefined },
.{ .address = undefined },
};
for (&backends) |*b| b.health.store(.unhealthy, .monotonic);
var route = Route{ .prefix = "/", .backends = &backends };
// Everyone down -> pick returns null -> caller serves 503, never dials a corpse.
try std.testing.expectEqual(@as(?usize, null), pickHealthyRoundRobin(&route));
}
Two tests, and between them they cover the two ways this feature can betray you: flapping (backend evicted on noise) and false confidence (dialing a backend we know is dead). A red bar on either would scream the moment a refactor broke the invariant -- which is exactly the safety net you want around code that decides whether real users get served.
The main from last episode barely changes. We build the same pool, then spawn the checker thread before the accept loop and join it on the way out via defer. The accept loop itself is unchanged in spirit -- route, then serveWithFailover -- it just quietly benefits from a health field that's being kept honest behind its back.
pub fn main() !void {
var gpa = std.heap.GeneralPurposeAllocator(.{}){};
defer _ = gpa.deinit();
const alloc = gpa.allocator();
var api_pool = [_]Backend{
.{ .address = try std.net.Address.parseIp("127.0.0.1", 9001), .weight = 3 },
.{ .address = try std.net.Address.parseIp("127.0.0.1", 9002), .weight = 1 },
.{ .address = try std.net.Address.parseIp("127.0.0.1", 9003), .weight = 1 },
};
var routes = [_]Route{
.{ .prefix = "/api", .backends = &api_pool },
};
var router = Router{ .routes = &routes };
const policy = HealthPolicy{};
var stop = std.atomic.Value(bool).init(false);
// The checker runs in the background so the accept loop never blocks on a probe.
const checker = try std.Thread.spawn(.{}, healthLoop, .{ &routes, policy, &stop });
defer {
stop.store(true, .release); // ask the loop to finish...
checker.join(); // ...and wait for it, so no thread outlives main.
}
const listen_addr = try std.net.Address.parseIp("127.0.0.1", 8080);
var server = try listen_addr.listen(.{ .reuse_address = true });
defer server.deinit();
std.log.info("health-checked proxy on {any}", .{listen_addr});
const buf = try alloc.alloc(u8, 64 * 1024);
defer alloc.free(buf);
const scratch = try alloc.alloc(u8, 64 * 1024);
defer alloc.free(scratch);
while (true) {
const conn = server.accept() catch continue;
// routing is unchanged from ep103; serveWithFailover now respects health.
const route = router.matchRoute("/api") orelse { conn.stream.close(); continue; };
serveWithFailover(route, "", conn.stream, buf);
conn.stream.close();
_ = scratch;
}
}
Fire up three backends on 9001-9003, point a loop of requests at the proxy, then kill one backend mid-loop and watch what's different from last episode:
$ for i in $(seq 1 9); do curl -s localhost:8080/api/ping; done
# all three answer in turn: 9001 9002 9003 9001 9002 9003 ...
# now kill 9002 and keep the requests coming:
$ kill %2
# ep104: every request that DREW 9002 first paid a failed-connect tax, then failed over.
# ep105: within one probe interval 9002 is marked unhealthy and SKIPPED entirely --
# no client ever draws it again until it comes back and passes `rise` probes.
That's the whole upgrade in one observable behaviour: the dead box stops costing anybody anything, almost immediately, and re-admits itself cautiously when it recovers. No 502s, no connect-timeout tax, no flapping.
In C, this is bread-and-butter nginx (health_check in the plus build, plus the passive max_fails/fail_timeout in open-source nginx) and HAProxy, whose option httpchk and rise/fall/inter directives are literaly the knobs we just built -- I did not pick those field names by accident, they're HAProxy's own vocabulary. What Zig bought us over the C originals is, once again, the atomics carrying their memory-ordering contract in the type: std.atomic.Value(Health) versus a naked _Atomic int and a lot of trusting the programmer to pass the right order to every access. The single-writer discipline that keeps our counters lock-free is achievable in C too, but nothing in C reminds you which fields are shared -- here the std.atomic.Value wrapper is a standing sign that says "many threads, tread carefully", and the plain u32 next to it says "one thread, relax".
In Rust, you'd reach for tokio and probably the tower ecosystem again, where health and load-shedding live as composable Service layers, and the borrow checker would force the shared/private split we arrived at by discipline -- a health behind an AtomicU8 in an Arc, the private counters owned by the checker task and simply not shared, so a stray access from a request task wouldn't compile. That's the recurring trade: Rust makes the mistake impossible at the cost of ceremony; our Zig version leaves the door unlocked but makes the correct arrangement small enough to hold in your head and verify by eye.
In Go, you'd write almost exactly this by hand (the standard library gives you httputil.ReverseProxy for the relay but no health checking at all), a background goroutine ranging over backends on a time.Ticker, flipping an atomic.Bool or a mutex-guarded map. It'd be a dozen tidy lines -- and having built ours from the probe and the atomic up, you now know precisely why that goroutine needs the atomic, why one bad tick shouldn't flip the flag, and why the recovery path deserves more consecutive successes than the eviction path deserves failures ;-)
Step back and look at what the reverse proxy became over three episodes: a routing switchboard (ep103), a load balancer across a pool (ep104), and now a self-healing front door that quietly evicts and re-admits backends without a client ever noticing. The genuinely new idea today was small and portable -- health is a gate that sits in front of your balancing policy, kept honest by a background heartbeat and a circuit breaker, tuned by a couple of hysteresis counters that stop the whole thing from flapping on noise. That exact pattern is a Kubernetes readiness probe, an AWS target-group health check, an Envoy outlier detector. Build it once from the sockets up and you'll recognise it in every piece of infrastructure you ever touch.
And with that, our little proxy is genuinely done -- not toy-done, but "I understand every atomic in it" done, which is a far better place to stand than "I trust the framework". So the next stretch of the series turns a corner. All through this networking arc we've leaned on slices, hash maps, and pools as if they were furniture -- always just there, provided by the standard library, used without a second thought. But a systems programmer who can only use the container the stdlib happened to ship is at its mercy the day its trade-offs don't fit. Pointers (episode 8) and allocators (episode 26) were the warm-up; from here we go back to first principles and build the fundamental data structures ourselves, from the simplest node-and-pointer arrangement upward, and learn exactly why each one makes the machines we've been writing fast. That's next.
Thanks for reading -- catch you in the next episode!
@scipio