Aqueous Archive Format
I have been working on the Aqueous Archive Format for many years, and the recent advancement of AI significantly accelerated the process. I believe this is the best way to use AI: human sets the goals and ideas, machine experiments and verifies them. (Anyone with a hands-on experience with compression algorithm knows those jobs are very annoying to do.)
It is an openly specified and implemented format designed around the concept of lossless recompression; a file in an Aqueous archive is described by a formula, a small stack-machine program over stored blocks that applies Wasm modules carried in the archive itself, so new transforms need no update to the extractor. Since the Wasm module itself is untrusted, a decompressor only runs known modules natively; everything else goes into a sandbox. If you are into the compression, you'll recognize that's ZPAQ and yes, it's the same concept done with modern tools much more carefully.
Aqueous also supports preset dictionaries, random access to individual files, appending, encryption and rich metadata, and every aspect of the format is specified (still a draft, so it may change, but not drastically), with room for recovery records and overlay archives that the code doesn't implement yet. (Well, I had to start with the specification because the implementation was costly back then. Now it is paying back by rigorously guiding coding agents.)
Okay, let's talk about actual numbers. Aqueous as of 2026-11-01 compresses Matt Mahoney's 10GB benchmark into less than 2.5 GB (Apple M4, 10 cores):
method size time
------------------------------ ----------- ----------------------------------------------
tar | zstd -3 -T10 4.68 GB compression: 33 s
tar | zstd -3 -T10 --long=31 4.17 GB compression: 36 s
zpaq 7.15 3.70 GB unknown
aqr (level mid, aka -5) 2.498 GB compression: 164 s (61 MB/s, 3.1 GB peak)
descompression: 38.8 s (258 MB/s, 2.6 GB peak)
To be clear, I think this is still slow; my original goal for the level mid was 100 MB/s compression and yet some formats are just difficult to optimize, hence some discrepancies. Compression also includes the round-trip check of every file so that that affects the timing. Still it makes Aqueous a good candidate for the Pareto frontier. (Not exhaustively checked though!)
There are many more things to talk about (memory optimization, streaming, deduplication, similarity-based optimizations etc.) but hopefully this introduction is interesting enough to you. The official Windows GUI and multi-platform CLI are available, so try it and please tell me your comments and opinions!
2 points · 1 comment
Mentions
Aqueous Archive Format I have been working on the Aqueous Archive Format for many years, and the recent advancement of AI significantly accelerated the process. I believe this is the best way to use AI: human sets the goals and ideas, machine experiments and verifies them. (Anyone with a hands-on experience with compression algorithm knows those jobs are very annoying to do.) It is an openly specified and implemented format designed around the concept of lossless recompression; a file in an Aqueous archive is described by a formula, a small stack-machine program over stored blocks that applies Wasm modules carried in the archive itself, so new transforms need no update to the extractor. Since the Wasm module itself is untrusted, a decompressor only runs known modules natively; everything else goes into a sandbox. If you are into the compression, you'll recognize that's ZPAQ and yes, it's the same concept done with modern tools much more carefully. Aqueous also supports preset dictionaries, random access to individual files, appending, encryption and rich metadata, and every aspect of the format is specified (still a draft, so it may change, but not drastically), with room for recovery records and overlay archives that the code doesn't implement yet. (Well, I had to start with the specification because the implementation was costly back then. Now it is paying back by rigorously guiding coding agents.) Okay, let's talk about actual numbers. Aqueous as of 2026-11-01 compresses Matt Mahoney's 10GB benchmark into less than 2.5 GB (Apple M4, 10 cores): method size time ------------------------------ ----------- ---------------------------------------------- tar | zstd -3 -T10 4.68 GB compression: 33 s tar | zstd -3 -T10 --long=31 4.17 GB compression: 36 s zpaq 7.15 3.70 GB unknown aqr (level mid, aka -5) 2.498 GB compression: 164 s (61 MB/s, 3.1 GB peak) descompression: 38.8 s (258 MB/s, 2.6 GB peak) To be clear, I think this is still slow; my original goal for the level mid was 100 MB/s compression and yet some formats are just difficult to optimize, hence some discrepancies. Compression also includes the round-trip check of every file so that that affects the timing. Still it makes Aqueous a good candidate for the Pareto frontier. (Not exhaustively checked though!) There are many more things to talk about (memory optimization, streaming, deduplication, similarity-based optimizations etc.) but hopefully this introduction is interesting enough to you. The official Windows GUI and multi-platform CLI are available, so try it and please tell me your comments and opinions!
Aqueous 2026-10-31 스냅샷을 만들었습니다. 지나치게 많은 변경점이 있었는데 그 모든 목록에 대해서는 github.com/lifthrasiir/aqueous/releases/tag/v0.0.… 를 참고하시고... 가장 중요한 것으로는 윈도 GUI와 인스톨러가 생겼다는 것! 왠지 몰라도 Aqueous에만 있는 3단 진행 표시줄을 구경할 수 있습니다. (각각 찔러보기probe - 분석하기analysis - 압축하기compression에 대응) 요즘은 LLM으로 딸깍 해 주면 GUI와 인스톨러가 생겨나는 시대로군요...
Show HN: Aqueous Archive Format
The current implementation is written in Rust but an independent reference implementation is planned using WAH (lifthrasiir/wah), my C single-header Wasm implementation. Take a look at it and tell me what you think about it---thanks!
I have worked on Aqueous Archive Format lifthrasiir/aqueous) for many years, and especially very hard for recent months. It is a new-generation archive format that intelligently detects recompressible portions of compressed files while being fast enough for casual uses.
몇 년간 계획해 왔고 지난 몇 달동안 꾸준히 작업해 온 Aqueous 압축 포맷을 처음으로 공개했다. lifthrasiir/aqueous 압축되는 파일을 분석해서 재압축 가능한 부분을 지능적으로 감지하는 차세대 압축 포맷을 컨셉으로 만들었는데, 더럽게 어려웠다. 명세랍시고 쓰는 것도 힘들었고, 클로드 아니었으면 여기까지 오지도 못했을 것이다. 현재 구현은 Rust로 되어 있는데 압축 해제 부분은 참조 구현을 C로도 작성할 예정이다(그렇다, WAH는 이 원대한 프로젝트의 일부였던 것이다). 많관부.