Most of us treat compressors as sealed boxes: bytes go in, smaller bytes come out, gratitude is expressed. I wanted to open the box, so I built Brick — a file compressor implementing the Re-Pair algorithm in Rust, with parallel processing, optional AES-256-GCM encryption, and CRC32 integrity checks.
Re-Pair is beautifully simple to state. Scan the input and count every pair of adjacent symbols. Take the most frequent pair, invent a new symbol for it, and replace every occurrence. Record the rule — 'symbol 256 means byte A followed by byte B' — and repeat, now counting pairs that may include your invented symbols. Stop when no pair occurs often enough to pay for its rule. What remains is a short sequence plus a grammar, and the grammar rebuilds the original exactly. Compression falls out of the fact that repetition is structure, and structure can be named.
The engineering is where it gets interesting. Pair counting and replacement are embarrassingly parallel, so Brick uses Rayon to spread both across every core; on large files this is the difference between seconds and minutes. The archive format stores rules, sequence and the original filename behind magic bytes and a CRC32. Encryption, when enabled, seals the payload with AES-256-GCM — authenticated encryption, so tampering is detected rather than silently decrypted into garbage.
My favourite feature is pure paranoia: after compressing, Brick decompresses its own output in memory and verifies it byte-for-byte against the input before declaring success. A compressor that cannot round-trip its own archive is a data-destruction tool with good marketing, and the check has caught real bugs during development.
Honest limitations, because every project has them: key derivation is a single SHA-256 of the passphrase — a real KDF like Argon2 would resist brute force far better; already-compressed input (JPEG, ZIP) does not shrink further, as with every compressor; and the whole file is processed in memory. None of these are hidden in the repo, and reciting them is half the fun of explaining the project.
If you want one exercise that teaches data structures, ownership, parallelism and file formats simultaneously, write a compressor. You feel every allocation. The code and demo GIFs are on GitHub.
