feat: Add optional lz4 compression support for arrays passed via base64 or binref encoding

[angela-ko]

Relevant issue or PR

n/a

Description of changes

Adds optional lz4 compression for arrays serialized via json+base64 and json+binref encodings, opt-in through a new TESSERACT_COMPRESSION config var (currently lz4 or unset). lz4 was chosen as the first option because it's a small, single dependency with fast round-trips. Further codecs can be added later without touching the encode/decode call sites.

TESSERACT_COMPRESSION only controls how a Tesseract encodes its output. It has no effect on inputs: each array carries its own "compression" field, so a Tesseract decodes any valid input — compressed or not — regardless of config. This means you can pipe a compressed output straight into another Tesseract without configuring the receiver.

Key details:

• lz4 is a new runtime dependency for all Tesseract builds.

• Arrays are compressed individually rather than compressing the whole buffer, which preserves offset-based random access into binref files. Because compressed length isn't derivable from shape/dtype, the binref buffer spec is extended from <path>:<offset> to <path>:<offset>:<compressed_size>, and a "compression" field is added to the array data dict. The buffer regex and validation pattern are updated accordingly; readers require compressed_size when compression is set.

• TESSERACT_COMPRESSION is a runtime config var, so through tesseract run it must be forwarded into the container with -e (a shell-level prefix stays on the host):

  # binref
  $ tesseract run -e TESSERACT_COMPRESSION=lz4 vectoradd apply -f json+binref -o /tmp/out @inputs.json
  {"result": {..., "data": {"buffer": "arr.bin:0:35", "encoding": "binref", "compression": "lz4"}}}
  
  # base64
  $ tesseract run -e TESSERACT_COMPRESSION=lz4 vectoradd apply -f json+base64 @inputs.json
  {"result": {..., "data": {"buffer": "<base64>", "encoding": "base64", "compression": "lz4"}}}

Testing done

CI. New unit tests cover base64 and binref compress/decompress round-trips, the extended buffer-spec parsing, and the "missing compressed_size" error path; end-to-end tesseract run tests exercise TESSERACT_COMPRESSION=lz4 for both encodings.

New benchmark, too.

[angela-ko]

@dionhaefner @nmheim Let me know if this is what you meant by testing compression in tesseract?

[dionhaefner]

That's a good start, thanks @angela-ko ! As next step, please add minimal, meaningful end-to-end tests that cover this functionality - which I expect are going to fail because I do see some issues with how the new lz4 dependency is added :)

Once everything is passing end-to-end I'll have a closer look at the design choices here.

[dionhaefner]

And please outline your rationale for choosing lz4 specifically as part of the PR body.

[codecov]

Codecov Report

❌ Patch coverage is 75.00000% with 16 lines in your changes missing coverage. Please review.
✅ Project coverage is 78.03%. Comparing base (dd88dfd) to head (7eb2075).

Files with missing lines	Patch %	Lines
tesseract_core/runtime/array_encoding.py	75.00%	7 Missing and 2 partials ⚠️
tesseract_core/sdk/tesseract.py	68.42%	3 Missing and 3 partials ⚠️
tesseract_core/runtime/cli.py	83.33%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #579      +/-   ##
==========================================
- Coverage   78.15%   78.03%   -0.12%     
==========================================
  Files          39       39              
  Lines        4632     4685      +53     
  Branches      754      770      +16     
==========================================
+ Hits         3620     3656      +36     
- Misses        709      720      +11     
- Partials      303      309       +6     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[PasteurBot]

Benchmark Results

ℹ️ No baseline found — all benchmarks marked as new.

Benchmarks use a no-op Tesseract to measure pure framework overhead.

Benchmark	Baseline	Current	Change	Status
api/apply_1,000	-	0.604ms	new	🆕
api/apply_100,000	-	0.596ms	new	🆕
api/apply_10,000,000	-	0.593ms	new	🆕
cli/apply_1,000	-	1746.945ms	new	🆕
cli/apply_100,000	-	1717.080ms	new	🆕
cli/apply_10,000,000	-	1764.094ms	new	🆕
decoding/base64_1,000	-	0.037ms	new	🆕
decoding/base64_100,000	-	0.654ms	new	🆕
decoding/base64_10,000,000	-	67.462ms	new	🆕
decoding/base64+lz4_1,000	-	0.041ms	new	🆕
decoding/base64+lz4_100,000	-	0.692ms	new	🆕
decoding/base64+lz4_10,000,000	-	115.795ms	new	🆕
decoding/binref_1,000	-	0.204ms	new	🆕
decoding/binref_100,000	-	0.243ms	new	🆕
decoding/binref_10,000,000	-	11.696ms	new	🆕
decoding/binref+lz4_1,000	-	0.210ms	new	🆕
decoding/binref+lz4_100,000	-	0.299ms	new	🆕
decoding/binref+lz4_10,000,000	-	41.466ms	new	🆕
decoding/json_1,000	-	0.109ms	new	🆕
decoding/json_100,000	-	9.114ms	new	🆕
decoding/json_10,000,000	-	1080.938ms	new	🆕
encoding/base64_1,000	-	0.041ms	new	🆕
encoding/base64_100,000	-	0.149ms	new	🆕
encoding/base64_10,000,000	-	27.703ms	new	🆕
encoding/base64+lz4_1,000	-	0.047ms	new	🆕
encoding/base64+lz4_100,000	-	0.351ms	new	🆕
encoding/base64+lz4_10,000,000	-	91.256ms	new	🆕
encoding/binref_1,000	-	0.322ms	new	🆕
encoding/binref_100,000	-	0.498ms	new	🆕
encoding/binref_10,000,000	-	19.290ms	new	🆕
encoding/binref+lz4_1,000	-	0.332ms	new	🆕
encoding/binref+lz4_100,000	-	0.715ms	new	🆕
encoding/binref+lz4_10,000,000	-	83.501ms	new	🆕
encoding/json_1,000	-	0.154ms	new	🆕
encoding/json_100,000	-	13.362ms	new	🆕
encoding/json_10,000,000	-	1432.884ms	new	🆕
http/apply_1,000	-	3.242ms	new	🆕
http/apply_100,000	-	9.538ms	new	🆕
http/apply_10,000,000	-	782.331ms	new	🆕
roundtrip/base64_1,000	-	0.093ms	new	🆕
roundtrip/base64_100,000	-	0.718ms	new	🆕
roundtrip/base64_10,000,000	-	92.851ms	new	🆕
roundtrip/base64+lz4_1,000	-	0.103ms	new	🆕
roundtrip/base64+lz4_100,000	-	0.958ms	new	🆕
roundtrip/base64+lz4_10,000,000	-	200.016ms	new	🆕
roundtrip/binref_1,000	-	0.549ms	new	🆕
roundtrip/binref_100,000	-	0.756ms	new	🆕
roundtrip/binref_10,000,000	-	30.219ms	new	🆕
roundtrip/binref+lz4_1,000	-	0.563ms	new	🆕
roundtrip/binref+lz4_100,000	-	1.035ms	new	🆕
roundtrip/binref+lz4_10,000,000	-	123.009ms	new	🆕
roundtrip/json_1,000	-	0.276ms	new	🆕
roundtrip/json_100,000	-	20.543ms	new	🆕
roundtrip/json_10,000,000	-	2492.259ms	new	🆕

Benchmark details

• Runner: Linux 6.17.0-1018-azure x86_64

[dionhaefner]

Taking shape – let's get some clarity on high-level design decisions before diving into details.

[dionhaefner]

Thanks @angela-ko. Looking real good now, just a last few comments.

[dionhaefner]

@angela-ko Thanks for the feature! Please have a look at the updated PR body - remember to only link to public resources next time.