Content-Length and chunked bodies) instead of leaning on the EOF crutch from Part 1;Learn Zig Series):Last time we built a load tester that worked -- and I told you, right at the end, that it also lied in three specific ways. It opened a fresh TCP connection for every single request, so a fat slice of every "latency" we measured was really the kernel doing a three-way handshake. It reported nothing until the run was over, so a five-minute test kept you in the dark for five minutes. And it had exactly one speed: flat-out. Today we fix all three, and in the process we bump into the single most misunderstood idea in the whole discipline of load testing -- coordinated omission -- which is a fancy name for the way naive testers accidentally hide the exact latency spikes you ran the test to find. Having said that, roll up your sleeves, because Part 2 is where this tool grows teeth.
Same rules of engagement as last time: point this ONLY at a server you own or a throwaway local one. Everything below I ran against the little HTTP server we built in episodes 51 through 54, on 127.0.0.1:8080. We're extending the exact code from episode 101 -- the Sample struct, the oneRequest function, the worker pool, the percentile function -- so if that's hazy, keep Part 1 open in a tab.
The Part 1 oneRequest did a full tcpConnectToAddress on every call, sent Connection: close, and let the server hang up. Clean, correct, and slow -- against localhost the handshake is cheap, but against anything across a real network the TCP setup can dwarf the actual request, and you end up benchmarking round-trip-time, not the server. Real HTTP clients hold a connection open and pump many requests down it (that's what Connection: keep-alive and HTTP/1.1's default persistence buy you). So the first job is to let each worker connect once and then loop, reusing the same stream.
/// A single persistent connection a worker owns for its whole run. Connect once
/// (ep 21), then fire many requests down the same socket. `alive` lets us drop
/// back to a fresh connect if the server ever closes on us mid-run.
const Conn = struct {
stream: std.net.Stream,
alive: bool,
fn open(addr: std.net.Address) !Conn {
return .{ .stream = try std.net.tcpConnectToAddress(addr), .alive = true };
}
fn close(self: *Conn) void {
if (self.alive) self.stream.close();
self.alive = false;
}
};
The request bytes change in one small-but-crucial way: we swap Connection: close for Connection: keep-alive, which is a promise to the server that we intend to send more requests, so please don't hang up. That single header flip is the entire protocol-level difference -- everything else that follows is us dealing with the consequences of the server keeping the socket open.
/// Same as Part 1's builder, but we now ask the server to KEEP the connection.
/// The cost of that convenience is that we can no longer use "socket closed" as
/// our end-of-response signal -- the whole next section exists to pay that bill.
fn buildRequest(alloc: std.mem.Allocator, host: []const u8, path: []const u8) ![]u8 {
return std.fmt.allocPrint(alloc,
"GET {s} HTTP/1.1\r\n" ++
"Host: {s}\r\n" ++
"User-Agent: zig-loadtest/0.2\r\n" ++
"Connection: keep-alive\r\n\r\n",
.{ path, host },
);
}
Here's the part everyone underestimates. In Part 1 we read until EOF -- stream.read returned 0, the server had closed, done. That was only legal because we'd said Connection: close. The moment we keep the socket open, EOF never comes between requests, and a reader that waits for it will hang forever after the first response. We now have to know exactly where one response ends so the next one can begin, and HTTP/1.1 gives us precisely two ways to know: a Content-Length header telling us the body's byte count, or Transfer-Encoding: chunked where the body arrives in length-prefixed pieces (we dissected both back in episode 84). An honest load tester has to handle both, because you don't get to pick which one the server under test uses.
Let me name the shape of a parsed response first, so every function downstream is honest about what it produces (episode 6 discipline, same as the Sample struct):
/// The result of framing ONE response off a shared, streaming buffer. `consumed`
/// is the total bytes this response occupied (headers + body) so the caller can
/// slide the buffer forward and start reading the NEXT response where this ended.
const Response = struct {
status: u16,
consumed: usize,
keep_alive: bool, // did the server agree to keep the socket open?
};
Now the framing itself. We find the end of the headers (\r\n\r\n), pull the status off the first line exactly like Part 1, then decide the body length from the headers. This is the fiddly, bug-prone code -- which, per the lesson we keep hammering, means it's exactly what we isolate as a pure function and unit-test without a socket in sight.
/// Parse ONE HTTP/1.1 response out of `buf`. Returns null if we don't yet have
/// a complete response (caller should read more bytes and retry). This is a pure
/// function of a byte slice -- no I/O -- so it's the piece we hammer with tests.
fn frameResponse(buf: []const u8) ?Response {
// 1. Headers end at the first blank line. No blank line yet -> need more bytes.
const head_end = std.mem.indexOf(u8, buf, "\r\n\r\n") orelse return null;
const headers = buf[0..head_end];
const body_start = head_end + 4;
// 2. Status code: the three digits after "HTTP/1.1 " (Part 1's parseStatus).
const status = parseStatus(headers);
// 3. Persistence: HTTP/1.1 keeps the connection alive UNLESS told otherwise.
const keep = !containsHeaderValue(headers, "connection", "close");
// 4. Body length: Content-Length is the common, easy case.
if (headerValue(headers, "content-length")) |cl| {
const len = std.fmt.parseInt(usize, std.mem.trim(u8, cl, " "), 10) catch return null;
if (buf.len < body_start + len) return null; // body not fully arrived yet
return .{ .status = status, .consumed = body_start + len, .keep_alive = keep };
}
// 5. Otherwise it may be chunked -- hand off to the chunk walker.
if (containsHeaderValue(headers, "transfer-encoding", "chunked")) {
const body_len = chunkedBodyLen(buf[body_start..]) orelse return null;
return .{ .status = status, .consumed = body_start + body_len, .keep_alive = keep };
}
// 6. No length and not chunked: server will close to signal end (like Part 1).
return .{ .status = status, .consumed = buf.len, .keep_alive = false };
}
The chunked walker is the trickiest morsel. A chunked body is a run of <hex-length>\r\n<that many bytes>\r\n pieces, terminated by a zero-length chunk 0\r\n\r\n. We walk it piece by piece and return the total byte length only once we've seen the terminating zero chunk -- returning null the instant we run out of buffer, which is the reader's cue to fetch more and try again:
/// Walk a chunked body and return its TOTAL length in bytes (including the size
/// lines and the terminating 0-chunk), or null if the body isn't complete yet.
/// Each chunk: "\r\n\r\n", ended by a "0\r\n\r\n".
fn chunkedBodyLen(body: []const u8) ?usize {
var pos: usize = 0;
while (true) {
const nl = std.mem.indexOfPos(u8, body, pos, "\r\n") orelse return null;
const size = std.fmt.parseInt(usize, body[pos..nl], 16) catch return null;
const data_start = nl + 2;
if (size == 0) return data_start + 2; // 0-chunk: +2 for the final CRLF
const next = data_start + size + 2; // skip data AND its trailing CRLF
if (next > body.len) return null; // chunk data not all here yet
pos = next;
}
}
Two small header helpers keep frameResponse readable -- a case-insensitive value lookup and a "does this header contain this token" check. Headers are case-insensitive by spec, and forgetting that is a bug that only shows up against one particular server, which is the worst kind:
/// Case-insensitive lookup of a header's value, e.g. headerValue(h, "content-length").
/// Returns the slice after the colon, still needing a trim. Null if absent.
fn headerValue(headers: []const u8, name: []const u8) ?[]const u8 {
var it = std.mem.splitSequence(u8, headers, "\r\n");
while (it.next()) |line| {
const colon = std.mem.indexOfScalar(u8, line, ':') orelse continue;
if (std.ascii.eqlIgnoreCase(line[0..colon], name)) return line[colon + 1 ..];
}
return null;
}
/// True if header `name` exists AND its value contains `token` (case-insensitive).
/// Used for "Connection: close" and "Transfer-Encoding: chunked" detection.
fn containsHeaderValue(headers: []const u8, name: []const u8, token: []const u8) bool {
const val = headerValue(headers, name) orelse return false;
var buf: [64]u8 = undefined;
if (val.len >= buf.len) return false;
const lower = std.ascii.lowerString(buf[0..val.len], val);
return std.mem.indexOf(u8, lower, token) != null;
}
And now the tests -- pure input, pure output, deterministic, no network. This is where framing bugs go to die:
test "frameResponse handles Content-Length" {
const r = frameResponse("HTTP/1.1 200 OK\r\nContent-Length: 5\r\n\r\nhello").?;
try std.testing.expectEqual(@as(u16, 200), r.status);
try std.testing.expectEqual(@as(usize, 44), r.consumed);
try std.testing.expect(r.keep_alive);
}
test "frameResponse returns null on a partial body" {
// Content-Length says 5 but only 2 bytes arrived -> not complete yet.
try std.testing.expect(frameResponse("HTTP/1.1 200 OK\r\nContent-Length: 5\r\n\r\nhi") == null);
}
test "chunkedBodyLen sums chunks and the terminator" {
// "5\r\nhello\r\n0\r\n\r\n" -> 5 data + framing + 0-chunk = 14 bytes total.
try std.testing.expectEqual(@as(usize, 14), chunkedBodyLen("5\r\nhello\r\n0\r\n\r\n").?);
// A missing terminator means "not done yet".
try std.testing.expect(chunkedBodyLen("5\r\nhello\r\n") == null);
}
That partial-body test is the one that saves your afternoon. Under real load a read() returns whatever bytes happened to arrive, which is almost never a whole response on a boundary. A framer that assumes each read gives it a complete response works perfectly on localhost and then falls apart the instant there's a network in between -- the classic "works on my machine" load-testing bug.
Part 1 stayed silent until the join. That's fine for a five-second test and maddening for a five-minute one. What we want is a line printed every second -- requests so far, current throughput, a rough p99 -- so you watch the server buckle in real time in stead of performing an autopsy afterward. The trick is doing this without a lock on the hot path, because a mutex every request would be us benchmarking our own contention (episode 30's whole point).
The move: each worker keeps a single atomic counter of completed requests -- a u64 it bumps with a relaxed fetchAdd, which is essentially free. A separate reporter thread wakes once a second, reads all the counters, and prints a delta. We deliberately do NOT try to compute exact percentiles live (that'd need the sorted latency pile); the live line is a cheap pulse, and the precise stats still come at the end where sorting is safe and single-threaded.
/// Per-worker live counters. Only `done` is touched on the hot path, with the
/// cheapest possible ordering -- we don't publish other memory through it, we
/// just want a monotonically-rising number the reporter can sample (ep 30).
const Counters = struct {
done: std.atomic.Value(u64) = .{ .raw = 0 },
failed: std.atomic.Value(u64) = .{ .raw = 0 },
};
/// The reporter thread: every second, sum all workers' counters and print the
/// delta since last tick. It reads only atomics -- never touches a worker's
/// Sample list -- so it can't race the workers writing their own private piles.
fn reporter(all: []Counters, running: *std.atomic.Value(bool)) void {
const stderr = std.io.getStdErr().writer();
var last: u64 = 0;
while (running.load(.monotonic)) {
std.time.sleep(std.time.ns_per_s);
var total: u64 = 0;
var failed: u64 = 0;
for (all) |*c| {
total += c.done.load(.monotonic);
failed += c.failed.load(.monotonic);
}
stderr.print("[live] {d} reqs, {d}/s, {d} failed\n",
.{ total, total - last, failed }) catch {};
last = total;
}
}
Notice the live line goes to stderr, not stdout. The final machine-readable summary belongs on stdout; the human-facing progress pulse belongs on stderr, so you can pipe the real result somewhere and still watch the show in your terminal. That's a small Unix-hygiene detail (episode 24), but it's the difference between a tool that composes and one that dumps everything into one stream and makes you grep.
Now the idea that separates a real load tester from a toy, and that our Part 1 tool got wrong in the most seductive way. In Part 1, each worker did: fire a request, wait for the response, fire the next. That's a closed-loop model -- the load is coupled to the server's own speed. Sounds fine, right? It is a catastrophe for latency measurement, and here's why. Suppose the server freezes for one full second. In a closed loop, your workers are all blocked on that frozen request -- they simply don't send the requests they would have sent during that second. So the freeze produces exactly ONE slow sample per worker, when in reality thousands of requests should have been affected. Your p99 comes back looking healthy because the tester politely stopped generating load precisely when the server was in trouble. This is coordinated omission: the load generator "coordinates" with the server's stalls and omits the very measurements that matter. Gil Tene named it, and once you see it you can't unsee it.
The fix is to schedule requests against a wall clock, not against the previous response. If you want 5,000 req/s, then request number n is supposed to be sent at time start + n/5000 seconds, full stop -- whether or not the previous one has come back. When the server stalls, the requests pile up behind their scheduled send-times, and each one's latency is measured from when it should have gone out, not from when we finally got around to it. That's an open-loop model, and it's the only way the p99 tells the truth. Let me show the honest latency calculation, because it's a two-line change that fixes the entire lie:
/// Open-model request timing. `scheduled_ns` is when this request was SUPPOSED
/// to be sent (start + n * interval). We measure latency from THAT instant, not
/// from when we actually sent it -- so a backed-up queue shows up as high
/// latency on every delayed request, which is the whole point (anti-CO).
fn timedOpen(conn: *Conn, req: []const u8, scratch: []u8, scheduled_ns: u64, clock: *std.time.Timer) Sample {
const now = clock.read();
// If we're already past the scheduled time, that lateness is REAL latency
// the user experienced -- bake it in instead of pretending it didn't happen.
const backlog = if (now > scheduled_ns) now - scheduled_ns else 0;
const service = oneRequestOn(conn, req, scratch); // the actual send+recv time
return .{
.latency_ns = service.latency_ns + backlog,
.status = service.status,
.ok = service.ok,
};
}
That backlog term is the entire cure. In the closed model it's structurally always zero -- you never send late because you never send until you're ready. In the open model, when the server stalls for a second, every request that was scheduled during that second carries a growing backlog, and your latency histogram fills with the truth in stead of a comfortable fiction. The oneRequestOn helper is just Part 1's oneRequest adapted to fire down an existing Conn and frame the response with our new frameResponse in stead of reading to EOF:
/// Part 1's oneRequest, upgraded: reuse an open connection and frame the reply
/// properly. On any I/O error we mark the conn dead so the worker reconnects,
/// and STILL return a Sample -- a failed request under load is data, not noise.
fn oneRequestOn(conn: *Conn, req: []const u8, scratch: []u8) Sample {
var timer = std.time.Timer.start() catch unreachable;
conn.stream.writeAll(req) catch {
conn.alive = false;
return .{ .latency_ns = timer.read(), .status = 0, .ok = false };
};
var total: usize = 0;
while (total < scratch.len) {
const n = conn.stream.read(scratch[total..]) catch { conn.alive = false; break; };
if (n == 0) { conn.alive = false; break; } // server hung up
total += n;
// Stop as soon as we have ONE complete response -- don't wait for EOF.
if (frameResponse(scratch[0..total])) |resp| {
conn.alive = conn.alive and resp.keep_alive;
return .{ .latency_ns = timer.read(), .status = resp.status, .ok = resp.status >= 200 and resp.status < 400 };
}
}
return .{ .latency_ns = timer.read(), .status = 0, .ok = false };
}
The rate-limited worker ties it together: it computes each request's scheduled send-time from a fixed interval, sleeps until that instant (if it's ahead of schedule), reuses its connection, and reconnects on the fly if the server ever drops it. The per-worker rate is the global target divided across the pool:
/// One open-model worker. It targets `interval_ns` between its OWN requests,
/// measures latency against the schedule (anti-coordinated-omission), reuses one
/// connection, and reconnects transparently if the server closes on it.
fn rateWorker(
addr: std.net.Address,
req: []const u8,
interval_ns: u64,
out: *std.ArrayList(Sample),
counters: *Counters,
running: *std.atomic.Value(bool),
scratch: []u8,
) void {
var clock = std.time.Timer.start() catch unreachable;
var conn = Conn.open(addr) catch return;
defer conn.close();
var n: u64 = 0;
while (running.load(.monotonic)) {
const scheduled = n * interval_ns; // when request n SHOULD go out
const now = clock.read();
if (now < scheduled) std.time.sleep(scheduled - now); // ahead? wait our turn
if (!conn.alive) conn = Conn.open(addr) catch { std.time.sleep(interval_ns); continue; };
const sample = timedOpen(&conn, req, scratch, scheduled, &clock);
out.append(sample) catch {};
_ = counters.done.fetchAdd(1, .monotonic);
if (!sample.ok) _ = counters.failed.fetchAdd(1, .monotonic);
n += 1;
}
}
The orchestration is Part 1's spawn, sleep, signal, join, merge with two additions: we hand each worker its slice of the target rate, and we spin up the reporter thread alongside the workers. I'll show just the changed heart of run, since the allocation-of-per-worker-lists scaffolding is verbatim from episode 101:
// Split the target rate across workers. 5000 req/s over 50 workers = 100/s
// each = a 10ms interval per worker. Integer-safe: guard the zero-rate case.
const per_worker_hz = if (target_rps > worker_count) target_rps / worker_count else 1;
const interval_ns = std.time.ns_per_s / per_worker_hz;
const counters = try alloc.alloc(Counters, worker_count);
defer alloc.free(counters);
for (counters) |*c| c.* = .{};
var running = std.atomic.Value(bool).init(true);
const rep = try std.Thread.spawn(.{}, reporter, .{ counters, &running });
for (lists, bufs, counters, 0..) |*l, *b, *c, i| {
l.* = std.ArrayList(Sample).init(alloc);
b.* = try alloc.alloc(u8, 64 * 1024);
threads[i] = try std.Thread.spawn(.{}, rateWorker,
.{ list_addr, request, interval_ns, l, c, &running, b.* });
}
std.time.sleep(duration_ms * std.time.ns_per_ms);
running.store(false, .monotonic);
for (threads) |t| t.join();
rep.join(); // stop the live pulse before we print the final verdict
Run it against a healthy local server and the live pulse ticks by, then the final verdict lands on stdout:
$ zig build run -- --rate 5000 --workers 50 --duration 10
[live] 4998 reqs, 4998/s, 0 failed
[live] 9997 reqs, 4999/s, 0 failed
[live] 14996 reqs, 4999/s, 0 failed
...
requests : 49981 (49981 ok, 0 failed)
achieved : 4998 req/s (target 5000)
p50 : 0.71 ms
p95 : 1.83 ms
p99 : 3.90 ms
Now the payoff. Point the SAME tool, open-model, at a server you deliberately hobble (add a sleep(500ms) to one route), and compare it to what the closed-loop Part 1 tool reported against the identical stall:
closed-loop (Part 1) : p99 512 ms <- one slow sample per worker, rest "fine"
open-model (Part 2) : p99 1730 ms <- every request queued behind the stall
The Part 1 number is a lie of omission. During that half-second freeze, a 5,000 req/s target means ~2,500 requests should have been sent and made to wait -- the open model counts all their backlog, the closed model simply never sent them and reported a rosy p99. If you've ever had a tool tell you "p99 is 50ms" and then watched real users rage about multi-second hangs, coordinated omission was very likely the culprit.
Keep-alive is the big win here, and it's worth being precise about why. In Part 1, every request paid for a connect() -- a full round trip before a single byte of the actual request went out, plus a fresh entry churning through the kernel's socket table and leaving TIME_WAIT litter behind. Reusing the connection means we pay that once per worker and then amortise it across thousands of requests. Against localhost the effect is modest (the loopback handshake is nearly free); across a real network it's the difference between measuring the server and measuring the speed of light down a fiber. And there's a subtle honesty bonus: with one connection per worker, the descriptor count is bounded by the worker count, so we sidestep the ProcessFdQuotaExceeded wall (episode 71) that the connection-per-request model slams into at high concurrency.
The open-model scheduling costs almost nothing on the hot path -- a subtraction and a conditional sleep -- but it changes what the tool is. It's now a machine that offers a fixed load and reports what the server does under it, in stead of a machine that goes as fast as the server lets it and calls the result "performance". Those are genuinely different experiments, and the open one is the one that predicts production, where your users arrive on their schedule, not yours.
One honest limitation I'll flag rather than hide: our sleep-based pacing is only as precise as the OS scheduler, which on a stock kernel means millisecond-ish granularity. At very high target rates (hundreds of thousands per second) the per-request sleep stops being accurate and you'd batch requests per tick or move to a busy-wait -- a real refinement, but out of scope for a tool that's already grown enough teeth for one day. Nota bene: knowing where your tool stops being accurate is itself a feature; a benchmark that can't tell you its own error bars is one you shouldn't trust.
In C, keep-alive and framing are exactly the same bytes -- you'd recv into a growing buffer and scan for \r\n\r\n and Content-Length by hand, precisely as we did. What Zig gave us for free is the ?Response optional forcing us to handle the "response not complete yet" case at the call site; in C that partial-read state is an easy if to forget, and forgetting it is the segfault-under-load bug that never reproduces in the debugger. The atomics for the live counters map straight onto C11 _Atomic -- same model, same relaxed ordering -- so that part ports almost mechanically.
In Rust, you'd likely lean on tokio and represent each connection as a task holding its TcpStream, using tokio::time::interval for the open-model pacing (which is genuinely nicer than hand-rolled sleep math). Crates like hyper would hand you HTTP/1.1 framing for free, chunked decoding included -- a real productivity win, at the cost of a dependency tree between your timer and the socket. The borrow checker guarantees your per-task Sample vectors don't race, which is the same property we got from giving each worker its own list, just enforced by the compiler in stead of by discipline. For a tool whose entire job is to measure the wire, some days I still want to see the exact read the clock wraps -- which is the whole argument for the from-scratch version.
In Go, this is nearly a freebie: net/http's Transport does keep-alive and connection pooling automatically, http.Client frames the response for you, and a time.Ticker drives the open-model schedule in a few lines. It's no accident that vegeta -- probably the best-known coordinated-omission-aware load tester -- is written in Go and pins its whole design on exactly the open-model insight we built by hand today. What our version buys is the same thing it always buys in this series: you can point at the precise byte where the clock starts, the precise place the backlog is added, and know with certainty why your p99 says what it says. For learning, seeing the machinery is the value; for shipping load tests tomorrow, honestly, reach for vegeta and now you'll understand what it's doing under the hood ;-)
Step back and count how much of today was, again, assembled from parts we'd already sharpened. The persistent connection is episode 21's socket with its lifetime stretched; the response framing is episode 84's HTTP/1.1 rules turned into code; the live counters are episode 30's atomics doing their lightest possible job; the stderr-vs-stdout split is episode 24's output hygiene. The genuinely new idea -- coordinated omission -- isn't Zig at all, it's a way of thinking about measurement that will change how you read every latency number for the rest of your career. That's the best kind of mini project: you finish with a tool AND a permanently upgraded pair of eyes.
We've now spent a good long run building programs that DO things -- servers, proxies, scanners, testers -- and lived off Zig's standard-library containers (ArrayList, HashMap) as if they were free. They aren't free; somebody wrote them, and the choices inside them are the difference between a tool that flies and one that crawls. So the next stretch turns the lens inward, onto the data structures themselves -- how they're actually built, byte by byte, and why the shape of your data quietly decides the speed of everything on top of it. The load tester runs and tells the truth; next we go one layer deeper and build the machinery it stood on.
Bedankt en tot de volgende keer!
@scipio