kosh

Findings

Every number here was produced by a command in this repository, on the machine that ran it. If a claim in the README is not in this file, it is not a claim.

Reproduce everything with:

kosh bench --markdown docs/bench-table.md    # codec comparison
pytest -q                                    # 43 tests, including the savings assertions

1. Codec comparison

Fixed corpus, seed 20261111, generated by kosh/core/bench.py: prose, CSV records, log lines and one megabyte of random bytes. Same bytes to every codec.

file	raw	gzip -9	xz -9	zip -9	kosh zstd 19	kosh brotli 11	kosh lzma 9	best
noise.bin	1.00 MB	1,000,328	1,000,108	1,000,426	1,000,471	1,000,445	1,000,545	xz -9
prose.txt	453.68 KB	53,462	46,204	53,560	46,829	48,591	46,637	xz -9
records.csv	634.52 KB	167,778	77,252	167,880	100,429	63,624	77,687	kosh brotli 11
service.log	2.62 MB	267,951	115,948	268,053	136,998	109,521	116,386	kosh brotli 11

Speeds, from the same run:

codec	prose	records	log	noise
gzip -9	4.9 MB/s	5.6 MB/s	11.6 MB/s	17.4 MB/s
xz -9	1.1 MB/s	1.2 MB/s	0.8 MB/s	1.6 MB/s
zip -9	4.6 MB/s	4.5 MB/s	9.6 MB/s	12.2 MB/s
kosh zstd 19	1.0 MB/s	1.3 MB/s	0.5 MB/s	2.9 MB/s
kosh brotli 11	0.35 MB/s	0.27 MB/s	0.27 MB/s	0.8 MB/s
kosh lzma 9	1.2 MB/s	0.9 MB/s	0.9 MB/s	1.7 MB/s

What this says

• Brotli 11 is the right default for records and logs — 18% smaller than xz -9 on the CSV, 5.5% smaller on the log.
• xz wins on prose by 1.4% and on incompressible data by a rounding error. Kosh’s LZMA path is the same LZMA2; the difference is container overhead and block framing, which is the honest cost of a seekable container.
• Zstandard is a speed choice, not a size choice. At level 19 it is 34% larger than Brotli 11 on the log while being only 1.8× faster. If size is the goal, do not pick zstd 19.

2. Lossless re-compression

kosh optimize, measured on the files described. Every one round-trips byte-exactly (pixels for PNG, entries for ZIP, payload for GZIP).

case	before	after	cut	engine
PNG saved at compress_level=1 (900×600 RGB)	113,356	31,170	72.5%	zopfli
Screenshot PNG, already level 6 (1024×768)	4,344	3,727	14.2%	zopfli
notes.txt.gz, level 1	1,790	1,080	39.6%	zopfli
report.txt.gz, level 6	287	274	4.5%	zopfli
ZIP of text, deflate level 1	5,782	4,998	13.6%	zopfli
JPEG with ~4 KB EXIF	44,514	43,526	2.2%	strip

What this says

• The win comes from what the file was saved with, not from Kosh being cleverer than the original encoder. A file saved at maximum effort gains nothing, and Kosh reports “no gain” rather than rewriting it.
• A second pass gains nothing: after the first run the PNG is already at the limit of this technique, and the tool says so.

3. The Zopfli budget

Zopfli’s cost per byte is not linear in any useful way. Measured on a 900×600 RGB PNG (1.62 MB of filtered image data) on this machine:

pass	time	output
zlib -9	0.5 s	41,260 B
zopfli, 1 iteration	74 s	32,879 B

Zopfli buys 20% over zlib for 148× the time. That is a fine trade for a build step and a terrible one for an interactive tool, so the encoder has a budget:

• ≤ 256 KB of payload: full iterations (5 / 15 / 40 by level).
• ≤ 1 MB: at most 3 iterations.

• 1 MB: zlib -9, and the report says · zlib-9.

• Level 3 (> 1 MB): Zopfli with one iteration up to 4 MB, because the user explicitly asked to wait.

4. Verification cost

kosh verify on a 5 MB, 8-entry container: decompresses every block and recomputes every entry hash — the same work as an extraction without the writes. --quick checks the footer, the manifest digest and every block’s bounds without decompressing; on the same file that is dominated by reading the footer and the table of contents.

5. Rejected ideas

• Re-implementing preflate inside Kosh. ziptastic already carries it as src/precompressors/preflate_wrap.cpp. Duplicating it would break the family’s single-purpose rule and produce a second, divergent implementation.
• Writing a CHCE magic for the container. A decoder that sees CHCE would try to read a .kosh as a bitstream. Distinct magic, explicit detection, deferral to the owner of the format.
• GPU paths. WABBIE used CUDA for its BSC stage. The family rule is no GPU paths, and blocks that compress independently already parallelise on cores.
• A fifth codec. Adding Zstandard’s dictionary training or a custom LZ would produce numbers that look like progress and are not. The benchmark table above is the argument: the wins are in format transforms, not entropy coding.
• Making optimize default to level 3. It would look better in a table and make the tool unusable on a folder of photos.

6. Open questions

• Windows behaviour of atomic_write + os.replace on a file that is open in another program is untested. It should fail loudly; it has not been proven.
• tar.zst extraction uses a streaming reader, so --only selection still walks the archive. A random-access index would need a different container.
• The web UI has no authentication because it binds to loopback. Binding it to 0.0.0.0 is possible and would be unsafe; the flag exists, the warning does not.

7. The gain estimator (EN-3)

kosh scan answers “would the engine help?” before the time is spent. It takes a 1 MiB slice from the middle of the file — a tar’s first megabyte is headers, a log’s first megabyte is the least representative part of it — compresses that slice with the engine itself, and extrapolates by size. An estimate from a different codec would be a guess wearing the engine’s name.

Measured against ziptastic 1.4.2 at its default level, full file versus estimate:

file	raw	predicted	actual	error
prose.txt	911,564	81,473	81,473	+0.0%
records.csv	1,184,607	229,858	231,440	−0.7%
service.log	3,557,560	155,540	152,259	+2.2%
noise.bin	400,000	400,108	400,108	+0.0%

Worst case 2.2%, inside the ±10% the roadmap asked for. The estimate costs one subprocess on a megabyte: 1.5 s for the prose file against 8 s for the full run, and the gap widens with file size, which is the point.

A first attempt used lzma at preset 6 on a front-of-file sample as a cheap proxy and was wrong by +35.5% on prose and −19.7% on the log. Recorded because the plausible method was the inaccurate one.

8. Corruption fuzzing (CT-4)

verify must fail closed on any mutation: refuse it, or accept it only when the container still parses to exactly the same structure. Anything else is a false “intact”, and that is the one failure a verifier cannot have.

Three real defects came out of the first run, none of which a hand-written test had found:

Defect	Consequence	Fix
Renaming the version key in the table of contents ("version" → "wersion")	read_toc defaulted to version 1, and the manifest-digest check only runs for version 2 and above — one byte switched the integrity check off	version is required and must be a positive integer; a version-2 container without a digest is refused
Damaging the START_MAGIC	the file fell through to the generic branch and reported “intact (hash only)”	a footer that says KOSH with a header that does not is damage; so is a .kosh that does not parse
A flipped bit in the table-of-contents length field	the reader sought before the start of the file and raised OSError — a crash, not a refusal	the length is bounded by the file minus its own header and footer

Exhaustive single-bit sweep on a 461-byte container — 3 688 mutations, every bit of every byte — now refuses or accepts-with-identical-structure in all of them. The random fuzzer runs 400 mutations on every push and 100 000 nightly.

This site is open source. Improve this page.