mistral3: diagnosis of long-context degradation on Mistral-Medium-3.5-128B

[danielhanchen]

Summary

Adds docs/diagnosis/ with a structured writeup of why Mistral-Medium-3.5-128B
degrades on long generations under llama.cpp / llama-server, and what the
experiments rule out.

This is documentation only — no code changes. It is intended as a starting
point for a real fix; the experimental harness for each phase is reproducible
from the artefacts described in the docs.

The TL;DR matches what unsloth has already published on
unsloth/Mistral-Medium-3.5-128B-GGUF:
the model's GGUF support is WIP, and the long-context degeneration occurs
regardless of who converted the GGUF.

What the experiments ruled out

• Tokenization (vLLM render, llama-server tokenize, mistral-common, and HF
  AutoTokenizer all produce byte-identical 434-token streams for the same
  fixture).
• Chat template (4 templates render identical token streams; the only diff
  vs upstream is a {%- if false %} no-op assertion).
• GGUF metadata (all numerical config matches HF text_config.rope_parameters,
  including yarn_beta_fast=4, beta_slow=1, factor=64, freq_base=1e6,
  original_context_length=4096, yarn_log_mul=1).
• KV cache dtype (F16, BF16, F32 all degrade the same way).
• Flash attention on/off.
• cuBLAS build flag + GGML_CUDA_FORCE_CUBLAS_COMPUTE_32F=1.
• Sampler — matched temp=0.1, top_p=1, top_k=64, min_p=0.05, seed=42, and
  sweeps over repetition_penalty, frequency_penalty, dry_multiplier.
• Architecture code in src/models/mistral3.cpp vs HF
  transformers/models/ministral3/modeling_ministral3.py — equivalent residual /
  RMSNorm / SwiGLU / RoPE flow; YARN attn_factor resolves to 1.0 on both
  sides for mscale=mscale_all_dim=1.0.

What is interesting

• HF transformers BF16 inference of the FP8 safetensors also degenerates
  on the same input, ruling out Q8_0 quantization or llama.cpp-specific kernel
  bugs as the unique cause.
• vLLM FP8 with --attention-backend FLASH_ATTN does not degenerate. It
  is the only setup in our matrix that stays coherent past ~1000 generated
  tokens.

Suggested next steps

1. Per-layer activation dump comparing vLLM and HF/llama.cpp on the same input
   — find the first layer where they diverge meaningfully.
2. If divergence localises to RMSNorm or attention softmax, force FP32
   accumulators in those ops in ggml-cuda for LLM_ARCH_MISTRAL3.
3. Sweep quants (Q8_0 / Q6_K / Q4_K_M / F16-from-FP8) for perplexity vs base.
4. Optional: a PyTorch reference forward pass driven by the GGUF weights via
   gguf-py and the HF arch, token-by-token.

Test plan

• CI: docs-only change; no behaviour to validate.
• Reviewer reads docs/diagnosis/mistral-medium-3.5-long-context.md and
  confirms the conclusions are reproducible from the listed artefacts.

[danielhanchen]

Update — first divergence pinned, see commit e617ca3

The two servers (vLLM FP8 and llama-server Q8_0) produce a byte-identical 13-token prefix then diverge at token 14. The top-2 candidates are the same on both, but the relative ranking is flipped:

rank	vLLM	logprob	llama-server	logprob
1	a	−0.314	Fl	−0.289
2	Fl	−1.314	a	−1.430

Both trajectories are individually clean for ~600–1000 generated tokens; only llama-server's degenerates after that into broken syntax (pygame.display.set mode((800, 600) with the underscore in set_mode lost).

When vLLM's full output is fed back as a fixed prefix to llama-server at checkpoints 50/200/500/1000/1400, llama-server's top-1 next-token agrees with vLLM at every checkpoint. So this is not a per-step disagreement — it's the cumulative effect of one early ~1-nat ranking flip pushing the path into a degenerate attractor.

Q4_K_M (74 GiB GGUF) makes the same wrong choice at token 14 and degenerates more downstream — uniform across quants.

GGML_CUDA_FORCE_CUBLAS_COMPUTE_32F=1 only helps the FP16 cuBLAS path; Q8_0 / Q4_K_M dequant + matmul takes a different code path. The targeted experiment that we have not been able to run is forcing FP32 accumulators in ggml-cuda/mmq*.cu (or wherever the dequant matmul lives) for LLM_ARCH_MISTRAL3 only.

Reproducer scripts are in docs/diagnosis/repro_*.py. Both run against the OpenAI Chat Completions API on a vLLM server (port 8765) and a llama-server (port 8766).