Finetuned Needle checkpoint converts to a CQ4 bundle that emits token garbage (stock converts fine)

[noelsaw1]

Converting a finetuned cactus-compute/needle checkpoint with cactus convert --bits 4 produces a
bundle that loads and serves but emits repeating token garbage for every prompt. The stock Needle
checkpoint converts and serves coherently through the exact same path. I've isolated the fault to the CQ
quantization core (not rounding tolerance, not the remap, not the toolchain, not the fusion passes).

The weights are healthy: they survive naive per-group int4 round-trip at ~0.90 task accuracy and run at
0.938 accuracy at full precision. Only Cactus's CQ4 conversion breaks them.

Environment

• cactus 2.0.1 — the current latest (Homebrew cactus-compute/cactus stable 2.0.1; GitHub latest
  release v2.0.1, 2026-07-09; PyPI cactus-compute 2.0.1). macOS (Apple Silicon), --backend cpu.
• Base model: cactus-compute/needle (26M encoder-decoder tool-caller), finetuned locally via
  needle finetune (JAX/Flax)
• Convert env: pip install cactus-compute (torch 2.13, transformers 5.5)

Note: pip install cactus grabs an unrelated static-site-generator package (currently 3.3.x) — the
engine is cactus-compute on PyPI / the Homebrew cactus-compute/cactus tap. Verified this repro is on
the newest engine release (2.0.1).

Reproduction

1. Finetune the stock Needle on any small tool-calling set (mine: a 165-example single-tool routing set,
   20 epochs, lr=3e-5 — the default finetune_local LR).
2. Export the finetuned JAX .pkl → HF NeedleForCausalLM safetensors (my remap validates 228/228
   tensors against the stock HF reference, transpose orientation unambiguous).
3. cactus convert <finetuned-hf> <out> --bits 4 → clean bundle (114 converted / 114 fallback).
4. cactus serve <out> --port 8085 --backend cpu

Then:

# plain prompt
curl -s localhost:8085/v1/chat/completions -d '{"model":"<bundle>","messages":[{"role":"user","content":"What is the capital of France? Answer in one word."}],"max_tokens":24}'
# -> content: "virtualvirtualvirtualvirtual..."   (finetuned)
# -> content: ""  (stock needle-cq4, coherent null — Needle is a tool-caller)

# routing/tool prompt
# finetuned -> "extendedextendeddishdishstrenuous..."
# stock     -> well-formed tool call

What I ruled out (isolation)

Hypothesis	Test	Result
4-bit rounding is too lossy for these weights	naive per-group int4 (group=64) round-trip, re-eval	survives — 0.896 route acc
The finetuned weights are broken	full-precision (JAX) eval	fine — 0.938 route acc
Remap / HF export is wrong	validate every tensor vs stock HF ref	228/228 within tol
Toolchain / engine build	convert stock through same venv+engine	coherent output
Fusion passes	reconvert with --no-fuse-rms-norm --no-fuse-rope --no-fuse-attention --no-fuse-attention-block --no-fuse-add-clipped	same garbage

The decisive signal: survives naive int4 but garbles under CQ4 ⟹ the fault is CQ-specific
(the Hessian/rotation-codebook calibration — each bundle carries hessian_metadata.json), and it appears
to only be robust for the stock weight distribution. I also swept earlier finetune epochs (less drift from
stock); the least-drifted checkpoint that still learned the task survives naive int4 better than the
fully-trained one, yet still garbles under CQ4 — so it's not simply "how far the finetune drifted."

Questions for maintainers

1. Is converting a finetuned Needle to native CQ4 supported today, or is CQ calibration effectively
   pinned to the released stock checkpoint?
2. Can convert accept calibration data (or a calibration-free / higher-bit path) so finetuned
   encoder-decoder models can be quantized robustly?
3. Is there a recommended path to serve a finetuned Needle natively (vs. running the JAX checkpoint
   full-precision behind a shim)?

Happy to share the exact repro artifacts (finetuned HF checkpoint, the int4-proxy screen, before/after
serve transcripts).

[ayaangazali]

Some triage findings from reading the conversion path that may explain the stock-vs-finetuned difference:

1. Calibration is silently skipped in the cactus convert path. _convert_from_source (python/cactus/cli/model.py) invokes the CQ converter with only --model/--out/--bits — --calibration-manifest is never passed. With an empty manifest, collect_manifest_hessians produces zero samples for every GPTQ-eligible tensor, so in non-strict mode each one hits "expected GPTQ target had zero samples" (python/cactus/convert/cli.py ~L521) and quantization falls back to data-free RTN with input_scale=None. A stock checkpoint matches the distribution the codebooks/rotations were tuned around; a finetuned checkpoint with shifted activation statistics gets no per-checkpoint correction at all, which is consistent with CQ4 degrading to garbage only for the finetune.

2. Embeddings/lm_head are pinned to CQ4-orthogonal with GPTQ off regardless of --bits (python/cactus/convert/model_adapters/policy.py ~L63-71), so those tensors also never receive calibration even when a manifest is supplied.

Two things that would help confirm: does the garbage persist if you run the converter module directly with a --calibration-manifest built from a few hundred tokens of your finetune's domain data? And does an FP16 (unquantized) bundle of the same finetuned checkpoint behave correctly, isolating this to the CQ step?

If the maintainers agree calibration exposure is the right direction, I'd be happy to contribute the scoped slice: plumb --calibration-manifest through cactus convert and emit a loud warning when GPTQ targets end up with zero samples instead of silently falling back.