Taming the Unary Code: Writing a Custom Stream Codec in C++20 from Scratch

Hi Hive! Today I want to share the results of a purely academic and sports-oriented engineering experiment. I have written cast — a custom lossless streaming data codec implemented in pure C++20.

Essentially, this is a "hand-built bicycle," and my main goal was to answer a simple question: is it possible to use unary coding in real-time streaming data without triggering the classic "unary explosion" of file size, and turn it into a rock-solid transport for low-end hardware?

You can check out the source code directly on GitHub: github.com/seminthai/cast

The Idea and the "Unary Explosion" Problem

The initial concept was very straightforward: operate at the level of a single nibble (4 bits). This approach guarantees ultra-low latency — the pipeline consumes data chunk by chunk on the fly, without needing to know or wait for what comes next in the stream. The stream is encoded in pairs of numbers from the 0–3 range, and the maximum length of the unary code per pair is strictly limited to 6 bits.

However, unary coding has a massive flaw — it spawns ones with exponential speed as soon as the numbers get even slightly larger than zero. If you try to encode a raw stream "face-on," the output file will instantly blow up.

Salvation via Compile-Time Preprocessing

To tame the unary beast, I implemented an aggressive data preprocessing pipeline before the encoding stage. Its sole purpose is to multiply zeros in the stream by any means necessary.

The pipeline works as follows:

1. Delta Encoding: flattens smooth data changes into near-zero values.

2. Gray Code (1 to 3 passes): smooths out bit-flips and jumps when crossing the boundaries of powers of two.

3. Inversion: if the stream still contains more ones than zeros, we simply flip all the bits.

To avoid any runtime performance penalties, I offloaded the brute-force evaluation of the optimal compression strategy entirely to the compiler using C++20 templates (template<bool Delta, uint8_t Gray, bool Invert>). As a result, the selected strategy encodes the data via fast Look-Up Tables (LUTs) that completely fit into the CPU's L1 cache.

The "Cheat": Matrix Unary Hash and LUTs

For maximum throughput, I ditched bit-by-bit write loops in favor of precomputed LUTs. The table was generated using a "vertical" matrix unary coding scheme.

Let's take a pair of numbers [1,3] as an example:

• First Layer (numbers > 0): both match → [1, 1]

• Second Layer (numbers > 1): the first one drops out (we write a closing 0 flag), the second keeps growing →[0, 1]

• Third Layer (numbers > 2): the first is long gone, the second keeps growing → [1] (for the maximum value of 3, a closing zero is omitted since a value of 4 is impossible).

Now, remove the layer boundaries and flatten the bits into a single string. We get a clean hash ready for the table: 11011 (5 bits total).

Here is how it looks in the source code:

alignas(64) constexpr CodeInfo LUT_ENCODE = {
    { {0b00, 2},     {0b010, 3},    {0b0110, 4},   {0b0111, 4} },
    { {0b100, 3},    {0b1100, 4},   {0b11010, 5},  {0b11011, 5} }, // Here is our pair -> 0b11011
    { {0b1010, 4},   {0b11100, 5},  {0b111100, 6}, {0b111101, 6} },
    { {0b1011, 4},   {0b11101, 5},  {0b111110, 6}, {0b111111, 6} }
};

The alignas(64) specifier aligns the tables perfectly with the CPU cache line boundaries. The decoder does not run any loops and never breaks the CPU branch predictor (a completely branchless approach) — it simply pulls values instantly from the L1 cache via REVERSE_LUT.

Frame Architecture: The Network Transport

All processing occurs in independent blocks (frames) of a fixed size (512 bytes by default, adjustable via CLI).

The structure of an encoded frame:
[Technical Header: 1 byte] + [Payload Bitstream] + [Zero-aligned tail of the last byte]

The header overhead is negligible (~0.2%). But the killer feature here is frame independence. If you pipe this stream over the network (via UDP, for instance) and a packet gets lost, the bit synchronization of the remaining stream does NOT break. The very next frame will be decoded perfectly.

Stress Test Results (From Logs to Massive RAW Video)

I threw several heavy real-world data types at the codec. The critical condition: data restoration must be 100% lossless, verified bit-by-bit via fc /b or diff.

All tests were executed on modest laptop hardware: Core i5 vPro with only 4 GB of RAM.

1. Windows Binary System Logs (System.evtx, ~21 MB):

• Original size: 20,975,616 bytes

• Compressed size: 15,730,560 bytes (Compression ratio: ~25%)

• Compression speed: 34.1 MB/sec | Decompression speed: 43.0 MB/sec

2. Large IoT Telemetry Text Dataset (iot_telemetry_data.csv, ~62 MB):

• Original size: 61,926,558 bytes

• Compressed size: 54,824,254 bytes (Compression ratio: ~11.5%)

• Compression speed: 37.2 MB/sec | Decompression speed: 39.8 MB/sec

3. The Ultimate Crash Test: Massive Raw Video Stream (Night city drive, rain, windshield droplets, glare, 720p, no audio, ~7.5 GB):
Streaming directly through the Windows console pipe conveyor:
type raw_720p_video.avi | cast.exe -c - - | cast.exe -d - restored_pipe.avi

• Original size: 7,502,661,288 bytes

• Compressed size: 6,877,351,176 bytes (Compression ratio: ~8.3%, saving ~624 MB!)

• Execution time under load: ~10 minutes (617,456 ms)

• Pipeline throughput speed: stable 11.58 MB/sec

More than 14.6 million independent frames flew through the codec in a continuous stream. Thanks to the fixed block size, RAM consumption never exceeded a few kilobytes. The algorithm operated strictly "in line," demonstrating a textbook linear complexity O(N) with zero memory leaks on highly constrained hardware.

Where does the algorithm fail? As expected, on audio streams (16-bit PCM). Audio waves contain too much chaotic noise in their least significant bits. Delta encoding fails to find patterns, zeros are non-existent, and the unary code instantly triggers an "explosion," expanding audio files by 10–15%.

Conclusion

Obviously, this is not Huffman coding, nor is it a competitor to giants like zstd. However, as an experimental network transport for "flat" data, binary logs, or heavy raw streams running under tight memory constraints — the concept has fully proven its viability. Unary code can be tamed if you handle the preprocessing right.

You can explore the project source code, investigate BitStream mechanics, and check the LUT configurations directly in the GitHub repository:
👉 github.com/seminthai/cast

Looking forward to your constructive thoughts in the comments!

#programming #development #technology #cpp #opensource