LEXORA
Layered EXact-text Repository & Adaptive compression architecture. A text-cleaning and exact-compression system for document collections: canonicalize conservatively, then compress with guaranteed byte-exact recovery — decode(encode(C)) == C for every input.
-informational){kind=icon}
round-trip byte-exact
container .lxz v0
RAM budget ~4 GiB
No releases yet · no stable API · install from a source checkout · pytest -q is the local gate
Clean conservatively. Compress exactly.
LEXORA ingests text, canonicalizes it into deterministic UTF-8 bytes, and then compresses those bytes with a guarantee that holds for every input: the decompressed stream is byte-identical to the canonical stream. The core invariant is deliberate — canonicalization may lose information relative to the original source; compression after canonicalization must never lose a single byte.
Byte-exact losslessness
The round-trip is verified before anything is written: decode(encode(C)) == C. A failed comparison is a blocker, not a warning.
Conservative canonicalization
Versioned, deterministic rule set (lexcan-0.1: CAN-R01/R02/R05) with golden test vectors and a whitelist contract — non-matching bytes pass through unchanged.
Cost-driven selection
Candidates compete against pinned baselines under a total-serialized-byte cost model — every byte of header, index and payload counts, not just the compressed stream.
measured, not assumed.lxz v0 container
Frozen v0 layout per ADR-0007: 128-byte header, one complete zstd frame per block, per-block SHA-256, 40-byte trailer. Every block is decode-verified and the write is atomic.
experimental · major 0A real CLI
Five tested subcommands — canon, zround, pipe, pack, unpack — with a stable exit-code contract (0, 2–7) exercised by the test suite.
Built for low resources
Designed to run within a ~4 GiB RAM budget using bounded, streaming/block-based processing. Multilingual coverage (English, Hindi, Gujarati) is a design target.
bounded blocksWhat the tool looks like
The experimental PySide6 GUI (milestones M1–M3, not released) shows a document collection, the cost-selected codec result and the round-trip verdict in one window. The figures below are the only published numbers — the Tier-1 smoke snapshot, baseline-vs-baseline.
Interface illustration — LEXORA is a Phase 1 prototype with no releases and no stable API; the figures shown are the project's published T1-gutenberg-p0 smoke snapshot (baseline-vs-baseline).
A pipeline with one hard boundary
Extraction and canonicalization are allowed to be lossy relative to the source. Everything after the canonical bytes exist must round-trip exactly. That boundary is where the guarantee lives.
Ingest
TXT/PDF files; OCR optional for scanned pages. (file ingestion planned)
Canonicalize
Versioned, deterministic UTF-8 bytes. may be lossy vs source
Compress
Cost-selected codec; candidate vs pinned baselines.
Verify
decode(encode(C)) == C enforced before writing.
Archive
.lxz v0 container now; .lxdb planned.
Quickstart
Install from a source checkout — no published package exists yet.
# editable install (Qt-free core) pip install -e . # + test suite, or + experimental PySide6 GUI pip install -e ".[dev]" pip install -e ".[gui]" python3 -m pytest -q # 192 passed locally lexora --help # lexora 0.0.1.dev1
Run the pipeline
canonicalize → compress → verify, then pack into a container.
# raw → canonical → zstd frame → verify lexora pipe --in raw.txt --out out.zst # or step by step lexora canon --in raw.txt --out canon.txt lexora zround --in canon.txt --out frame.zst # wrap canonical bytes into .lxz v0 lexora pack --in canon.txt --out archive.lxz lexora unpack --in archive.lxz --out back.txt
6) is a blocker severity.Measured results — reported honestly
There is no validated LEXORA-vs-baseline result yet. The figures below are the only published numbers: a Tier-1 smoke snapshot comparing frozen generic baselines against each other, not a gate verdict. Every number carries its mandatory tier label.
Corpus tier: T1-gutenberg-p0
| File | Split | Input B | Zstd-19 B (ratio) | XZ-9 B (ratio) |
|---|---|---|---|---|
| gutenberg-1184.txt | heldout | 2,787,124 | 782,892 (0.2809) | 776,472 (0.2786) |
| gutenberg-1342.txt | train | 772,386 | 226,028 (0.2926) | 223,512 (0.2894) |
| gutenberg-1400.txt | train | 1,058,204 | 324,526 (0.3067) | 321,664 (0.3040) |
| gutenberg-1661.txt | train | 607,606 | 191,034 (0.3144) | 188,860 (0.3108) |
| gutenberg-2701.txt | train | 1,276,315 | 422,220 (0.3308) | 418,184 (0.3276) |
| gutenberg-76.txt | train | 622,460 | 193,262 (0.3105) | 191,064 (0.3069) |
Acceptance gates (defined, not yet met)
A candidate codec does not exist yet, so no Pass/Partial/Fail row can be filled.
Peak RSS (measured once, R3 artifact): zstd-19 ≈ 85,956 KiB · xz-9 ≈ 35,348 KiB. Budget target ≈ 4 GiB (REF-4G).
Known caveats, recorded honestly: 2 of 8 requested texts were discarded by the fetch size guard; held-out is a single lineage, so statistical strength is nil; the corpus (~7.1 MB) is far below the ≥1 GiB protocol target.
Clone it, install it, run the gate
LEXORA is an early Phase 1 prototype. Nothing here is a shipped or released feature — install from source and treat the test suite as the contract.
$ git clone
git clone https://github.com/drsamonline/lexora.git
cd lexora
pip install -e ".[dev]"
python3 -m pytest -q
Python ≥3.12,<4.0 · dependency zstandard pinned to a tested range · GUI is an opt-in [gui] extra.
What you get
- canonicalize.py — conservative v0 rule set with golden vectors
- zstd_core.py — frame codec with round-trip verification
- container.py —
.lxzv0 pack/unpack (ADR-0007) - cli.py — the five-subcommand CLI with stable exit codes
- benchmarks/ — harness + frozen corpus manifest
No releases, no download artifacts, and no coverage/release badges exist — those claims are not yet true.
Straight answers
Q01Is LEXORA released or production-ready?
.lxz v0 container, a benchmark harness, a CLI, and experimental PySide6 GUI milestones).Q02Does LEXORA beat Zstd, Brotli or XZ?
Q03What exactly does "byte-exact" mean here?
C, the guarantee is decode(encode(C)) == C for every input. It is enforced at build time — a mismatch is treated as a blocker (exit code 6), not a warning.Q04Can canonicalization lose data?
Q05Which languages and inputs are supported?
.txt and .pdf (text layer first, OCR optional for scans) across Latin, Devanagari and Gujarati scripts. Current reality: byte-level canonicalization of raw text streams is implemented; TXT/PDF file ingestion is not implemented yet, and the build scope is English-only, with Hindi/Gujarati deferred (not blocking).Q06What is the licence, and who maintains it?
pytest -q plus a docs link check before every commit.