v0.7.4: malformed-args → tool-error channel + 32GB eval tier (#102)

* Args-validation: route malformed args through tool-error channel

Models occasionally emit a structurally valid tool call with malformed
args content (e.g. arguments="" instead of arguments="{}"). Pydantic
rejected at ToolCall construction, crashing the stage with
ValidationError. Observed at 86% of error rows on Qwen3-Next prompt
mode (77/89), same family on Qwen3.6 (rig-02).

This is conceptually "tool called with bad args" — the call exists,
the inputs are wrong — same as FileNotFoundError at runtime. Should
ride the tool-error channel with max_tool_errors=2 budget, not crash.

- ToolCall / TextResponse: BaseModel → @DataClass. args is no longer
  validated at construction; ResponseValidator enforces dict-shape.
  Audit: no .model_* API on ToolCall anywhere in forge.
- ResponseValidator: new args-shape branch after unknown-tool check.
  Unknown-tool runs first (cheap; no point validating args on a
  hallucinated tool name).
- nudges.tool_arg_validation_nudge: schema-derived message naming the
  tool, the received args type, and the required JSON-object shape.
- inference: parse-error nudges drain max_tool_errors (record_result)
  not max_retries (record_retry). Message prefix
  [ToolArgValidationError] vs [UnknownTool].
- Exhaustion message simplified: includes which budget and nudge kind.

Smoke (Ministral-3-14B-Reasoning, 26 scenarios × 25 runs prompt-mode):
score 78.77% (vs v0.7.0 baseline 79.5% at n=50). Delta within
±1.6% noise band — no regression. Patch never tripped on this model;
this is a regression check, full bake on a model that hits the path
to follow.

884 tests pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* batch_eval/sampling: 32GB-eval lineup — Qwen3.6-27B, Qwen3.6-35B-A3B-UD, Nemotron-3-Nano

GGUF + server-flag + sampling-default entries for the three rig-02
32GB-tier models added for the v0.7.1 run. Launch config of record
for eval_results_rig-02_v0.7.1.jsonl (31,200 rows).

* clients: route malformed tool-call args through the tool-error channel

Unify all OpenAI-shape clients on one decode_tool_args helper
(clients/base.py): JSON-string args are parsed; malformed or non-dict
payloads ride through on the ToolCall as raw (non-dict) args instead of
collapsing to a TextResponse (openai_compat, vllm, llamafile) or raising
(anthropic streaming). ResponseValidator's args-shape check then routes
them to the tool-error channel + max_tool_errors budget — the same lane
as a runtime tool error — rather than a retry nudge.

Completes the client normalization #86 started (one decoder, all
clients) and keeps fail-loud (never coerced to {}). Also closes an
unguarded json.loads crash in the anthropic streaming finalize.

Behavior change: structural malformed-args now drains max_tool_errors
(2), not max_retries (3), in proxy mode. Wire-invisible; no public
signature changes.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* review fixes: honest ToolCall.args type + de-stale comments + llamafile malformed test

From the helper-vs-inline structural review (verdict: keep the shared
decode_tool_args helper). Two correctness/honesty caveats actioned:

- ToolCall.args annotated dict[str, Any] while the runtime contract now
  intentionally allows non-dicts (the docstring already says so). Widen
  to Any so the type stops lying.
- Stale comments in openai_compat/vllm streaming finalize still claimed
  malformed args yield a retry-driving TextResponse; corrected to the
  raw-args → tool-error-channel routing (they invited the exact drift
  the helper prevents).
- Add an explicit llamafile malformed-args test (non-stream native path).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* guardrails: make tool-call faults consistent across all three modes + expose proxy max_tool_errors

From the H1/H2 design review. Two consistency gaps closed:

1. The Guardrails middleware facade recorded EVERY validator failure as a
   retry (max_retries) and returned action='retry' — diverging from
   run_inference/proxy, where malformed args drain the tool-error budget
   and tool-call faults ride the tool channel. The facade now:
     - routes malformed args (tool_arg_validation) to max_tool_errors,
     - returns a new action='tool_error' for tool-call faults (unknown
       tool name OR malformed args), with nudge.role='tool' so callers
       emit the correction on the tool-result channel.
   Channel vs budget are now two explicit kind-sets in nudge.py
   (TOOL_CHANNEL_KINDS ⊃ TOOL_ERROR_KINDS); unknown-tool rides the tool
   channel but still drains the retry budget, matching run_inference.
   _TOOL_ERROR_KINDS moved from inference.py to the shared nudge module.

2. Proxy exposed max_retries but not max_tool_errors, hiding the budget
   exactly where malformed-arg recovery now matters. Added --max-tool-errors
   (default 2) threaded ProxyServer → HTTPServer → handler → ErrorTracker.

Nobody depends on the middleware facade yet, so the CheckResult.action
addition is free. Channel parity in run_inference unchanged (it emits
role=tool for list-branch corrections regardless of nudge.role).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* steps: guard arg-match prereqs against non-dict args + ADR-016

Closes the last crash vector from the design review (hole 4):
StepTracker.check_prerequisites did args.get(match_arg), which raises on
a non-dict args. ResponseValidator fences this before dispatch in the
runner/proxy, but a granular caller that bypasses check() could reach it
directly. Treat non-dict args as unsatisfied (block, don't crash).

ADR-016 records the malformed-args → tool-error-channel decision in its
honest framing: a native-mode conditioning bet, not an ontology claim;
prompt mode degrades to the prior retry shape; the tool-error budget
coupling is deliberate but revisitable. CHANGELOG held for release time.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* eval data: tag generations + add 32GB tier (v0.7.4)

Inject a per-row `gen` field so one dashboard can fold eval waves run
against different code states. gen is a comparability epoch, not a
release version: v0.6.0 -> gen 1 (carries the Anthropic ablation +
Retired-tier models, neither re-run since), and v0.7.0 plus the new
32GB tier -> gen 2.

Rename the 32GB wave to eval_results_v0.7.4.jsonl (its landing release)
and keep it as a separate file beside v0.7.0 — same gen, distinct wave,
so each wave keeps its own landing commit for reproducibility.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* eval dashboard: eval-generation badges + retired toggle

report.py now accepts multiple result files and keeps the newest gen per
config (dedup_latest_gen), so the board folds all generations into one
view. Lagging rows (gen < newest) get a superscript badge backed by a
commit/date legend; Retired-tier models are carried forward but hidden
by default (--include-retired, or a sidebar checkbox in the HTML). Adds
MODEL_FAMILIES entries for the 6 32GB models so they render clean family
names and cross-backend keys instead of raw GGUF stems.

React dashboard: Show-retired checkbox (dimmed rows + a "retired" pill),
superscript gen badges with provenance tooltips from the data blob.

Regenerated docs/results/ from the three gen-tagged datasets.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* release: 0.7.4 — version bump, CHANGELOG, registry 32GB tier

Bump 0.7.3 -> 0.7.4 and add the 0.7.4 CHANGELOG entry (malformed args ->
tool-error channel; 32GB eval tier + dashboard eval-generations). Move
the 6 32GB models (Mistral-Small-3.2, Qwen3.5/3.6 27-35B, Nemotron-3
Nano) from Unpublished to Current now that they're in the published eval,
and reword the tier definitions for the dashboard's eval generations.

Also scrub a stale bring-up note from the Qwen3.5-122B footnote (it leaked
operator smoke-probe process into a public doc) and exclude the built
dashboard dist/ from the hatchling sdist sweep.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* changelog: de-escalate dataclass entry from BREAKING to Changed

The ToolCall/TextResponse pydantic->dataclass move only breaks callers
who serialized these via the pydantic .model_* API or relied on
construction-time validation; attribute reads and keyword construction
are unchanged. Reserving BREAKING for forced-migration changes (cf.
0.7.3 --mode rename) keeps the badge meaningful.

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Author: antoinezambelli

CHANGELOG.md

@@ -2,6 +2,23 @@
 
 All notable changes to forge are documented here.
 
+## [0.7.4] — 2026-06-03
+
+Malformed tool-call arguments now self-correct on the tool-error channel, and the eval suite gains its first model-size upgrade — a 32GB tier (Qwen3.5 / 3.6 27–35B, Nemotron-3 Nano, Mistral-Small-3.2) surfaced in the dashboard alongside the existing 8–14B lineup.
+
+### Added
+- **Proxy `--max-tool-errors`** (default 2) — bounds consecutive tool-argument errors per request, mirroring the `WorkflowRunner` budget. Threaded through `ProxyServer` and the HTTP handler.
+- **32GB model tier** in the published eval and dashboard: Mistral-Small-3.2 24B, Qwen3.5 27B / 35B-A3B, Qwen3.6 27B / 35B-A3B, Nemotron-3 Nano 30B-A3B (moved Unpublished → Current in the [Model Registry](docs/MODEL_REGISTRY.md)).
+- **Eval-generation tracking in the dashboard.** Results gathered against different code states fold into a single view, deduped to the newest generation per config. Runs not yet re-swept (e.g. the Anthropic ablation) are carried forward and superscript-badged with a commit/date legend; Retired-tier models are carried forward but hidden behind a `Show retired` toggle.
+
+### Changed
+- **Malformed tool-call arguments ride the tool-error channel.** A model that emits a structurally valid call whose `arguments` are unparseable or not an object is now corrected via a tool-error result (`role="tool"`, anchored to its `tool_call_id`) draining `max_tool_errors`, uniformly across all OpenAI-shape clients and all three integration modes (`WorkflowRunner`, proxy, `Guardrails` facade). This supersedes 0.7.3's "malformed args drive a retry nudge" behavior. The change is a native-mode conditioning bet — a small model plausibly self-corrects better on the channel it was pretrained on than via a trailing user nudge; in prompt mode the tool role is downgraded to a user message, so behavior there is unchanged. See [ADR-016](docs/decisions/016-malformed-args-tool-error-channel.md).
+- **`Guardrails.check()` gains `action="tool_error"`** for tool-call faults (unknown tool, malformed args) so middleware loops account for them on the tool channel. No consumers depended on the prior action vocabulary.
+- **`ToolCall` / `TextResponse` are now plain dataclasses** (`args: Any`); arg-shape validation moved to `ResponseValidator`. Attribute access and keyword construction are unchanged — but the pydantic `.model_*` API on these two exported types is gone, and construction no longer raises on a non-dict `args`. Only affects callers that serialized these objects via pydantic or relied on construction-time validation.
+
+### Fixed
+- **Non-object tool args no longer crash the parser.** Previously `arguments` decoding to a list / scalar / `null` raised at `ToolCall` construction; it is now caught at validation and routed to the tool-error channel. `StepTracker.check_prerequisites` additionally guards against a non-dict `args` reaching a direct dispatch.
+
 ## [0.7.3] — 2026-06-01
 
 Native-first proxy. With native function calling now well-supported across modern local models, the proxy defaults to — and is optimized for — native tool calling, forwarding the client's OpenAI `tools` / `messages` to the backend verbatim. Prompt-injection remains available as an explicit opt-in for llama.cpp / llamafile backends that lack a function-calling template, but it is no longer the default path. This release also folds in the OpenAI-compatible client and several proxy / eval fixes that landed on `main` since 0.7.2.

docs/MODEL_REGISTRY.md

@@ -4,9 +4,9 @@ Every model forge knows about, classified by eval-suite status.
 
 ## Status meanings
 
-- **Current** — in the v0.7.0 eval suite. Numbers in [`docs/results/`](results/) and the [dashboard](results/dashboard.html).
+- **Current** — in the published eval. The dashboard folds multiple eval *generations* into one view (the v0.7.0 8–14B lineup, plus the v0.7.4 32GB tier); runs not yet re-swept against the latest code — e.g. the Anthropic ablation — are carried forward and superscript-tagged. Numbers in [`docs/results/`](results/) and the [dashboard](results/dashboard.html).
 - **Retired** — appeared in a prior eval suite, cut from the current one. Either too weak (bare scores below the threshold for informative comparison) or superseded by a newer family member. Sampling defaults retained for backward compatibility.
-- **Unpublished** — sampling defaults are present (either dogfooded by forge consumers, or staged for a future eval), but no eval numbers have been published. Forge will work with them; performance is undocumented.
+- **Unpublished** — sampling defaults are present, but no eval numbers have been published. Forge will work with these models; performance is undocumented.
 
 Sampling values are sourced from the model's HuggingFace card unless noted. Values are verified one model at a time — see [`src/forge/clients/sampling_defaults.py`](../src/forge/clients/sampling_defaults.py) for the authoritative map.
 
@@ -25,6 +25,12 @@ Sampling values are sourced from the model's HuggingFace card unless noted. Valu
 | Granite 4.1 8B | Q4_K_M, Q8_0 | 0.0³ | 1.0 | 0 | — | — | — | (IBM convention, unconfirmed) |
 | Gemma-4 E4B-it | Q4_K_M, Q8_0 | 1.0 | 0.95 | 64 | — | — | — | [HF](https://huggingface.co/google/gemma-4-e4b-it) |
 | Phi-4 | Q4_K_M | — | — | — | — | — | — | (no formal recommendation⁴) |
+| Mistral Small 3.2 24B Instruct 2506 | Q4_K_M | 0.15 | — | — | — | — | — | [HF](https://huggingface.co/mistralai/Mistral-Small-3.2-24B-Instruct-2506) |
+| Qwen3.5 27B | Q4_K_M | 1.0 | 0.95 | 20 | 0.0 | — | 1.5 | [HF](https://huggingface.co/Qwen/Qwen3.5-27B) |
+| Qwen3.5 35B-A3B | Q4_K_M | 1.0 | 0.95 | 20 | 0.0 | — | 1.5 | [HF](https://huggingface.co/Qwen/Qwen3.5-35B-A3B) |
+| Qwen3.6 27B | Q4_K_M | 1.0 | 0.95 | 20 | 0.0 | — | 0.0⁶ | [HF](https://huggingface.co/Qwen/Qwen3.6-27B) |
+| Qwen3.6 35B-A3B UD | Q4_K_M | 1.0 | 0.95 | 20 | 0.0 | — | 1.5 | [HF](https://huggingface.co/Qwen/Qwen3.6-35B-A3B) |
+| Nemotron-3 Nano 30B-A3B | Q4_K_M | 0.6 | 0.95 | — | — | — | —⁷ | [HF](https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16) |
 | Claude Haiku 4.5⁵ | — | — | — | — | — | — | — | (SDK-managed) |
 | Claude Sonnet 4.6⁵ | — | — | — | — | — | — | — | (SDK-managed) |
 | Claude Opus 4.6⁵ | — | — | — | — | — | — | — | (SDK-managed) |
@@ -33,7 +39,9 @@ Sampling values are sourced from the model's HuggingFace card unless noted. Valu
 ² Ministral-3 Reasoning cards show `top_p=0.95` in code examples but do NOT include it in the formal "Recommended Settings" section. Add explicitly if you want to follow the examples.
 ³ Granite 4.1 sampling mirrors the Granite 4.0 IBM convention (greedy decoding); marked unconfirmed pending IBM publication for the 4.1 family specifically.
 ⁴ Phi-4: no formal sampling recommendation from any official source (Microsoft HF card, model docs). Falls through to backend defaults.
-⁵ **Claude numbers in v0.7.0 docs are from the v0.6.0 dataset.** The Anthropic ablation was not re-run in v0.7.0 due to cost (~$272 for the full 11,700-row matrix). Backend support is unchanged; numbers are stable to within tool-error-channel sensitivity (small).
+⁵ **Claude numbers are carried forward from the v0.6.0 dataset** — gen 1 on the dashboard, superscript-tagged. The Anthropic ablation has not been re-run since, owing to cost (~$272 for the full 11,700-row matrix). Backend support is unchanged; numbers are stable to within tool-error-channel sensitivity (small).
+⁶ Qwen3.6 27B (dense) deliberately diverges from its A3B siblings: its card drops the `presence_penalty=1.5` the MoE variants recommend, so forge sends `0.0` (no penalty).
+⁷ Nemotron-3 Nano: the card splits sampling into a Reasoning preset (T=1.0, top_p=1.0) and a Tool-calling preset (T=0.6, top_p=0.95); the tool-calling preset is used here, with thinking enabled via `chat_template_kwargs`.
 
 ---
 
@@ -59,27 +67,21 @@ Sampling params staged, no eval data published. Forge supports these — perform
 |---|---|---|---|---|---|---|---|
 | Qwen3 4B Instruct 2507 | Q4_K_M | 0.7 | 0.8 | 20 | 0.0 | — | [HF](https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507) |
 | Qwen3 4B Thinking 2507 | Q4_K_M | 0.6 | 0.95 | 20 | 0.0 | — | [HF](https://huggingface.co/Qwen/Qwen3-4B-Thinking-2507) |
-| Qwen3.5 27B | Q4_K_M | 1.0 | 0.95 | 20 | 0.0 | presence_penalty=1.5 | [HF](https://huggingface.co/Qwen/Qwen3.5-27B) |
-| Qwen3.5 35B-A3B | Q4_K_M | 1.0 | 0.95 | 20 | 0.0 | presence_penalty=1.5 | [HF](https://huggingface.co/Qwen/Qwen3.5-35B-A3B) |
-| Qwen3.6 35B-A3B UD | Q4_K_M | 1.0 | 0.95 | 20 | 0.0 | presence_penalty=1.5 | [HF](https://huggingface.co/Qwen/Qwen3.6-35B-A3B) |
-| Qwen3.5 122B-A10B | Q4_K_M | 0.7 | 0.8 | 20 | — | balanced profile; thinking mode separate run⁶ | [HF](https://huggingface.co/Qwen/Qwen3.5-122B-A10B) |
+| Qwen3.5 122B-A10B | Q4_K_M | 0.7 | 0.8 | 20 | — | balanced (instruct) preset; thinking mode is separate⁸ | [HF](https://huggingface.co/Qwen/Qwen3.5-122B-A10B) |
 | Qwen3-Coder 30B-A3B Instruct | Q4_K_M | 0.7 | 0.8 | 20 | — | repeat_penalty=1.05 | [HF](https://huggingface.co/Qwen/Qwen3-Coder-30B-A3B-Instruct) |
 | Qwen3-Coder-Next 80B-A3B | Q4_K_M | 1.0 | 0.95 | 40 | — | coder fine-tune over Qwen3-Next | [HF](https://huggingface.co/Qwen/Qwen3-Coder-Next) |
 | Qwen3-Next 80B-A3B Instruct | Q4_K_M | 0.7 | 0.8 | 20 | 0.0 | hybrid-attention MoE; thinking-mode profile | [HF](https://huggingface.co/Qwen/Qwen3-Next-80B-A3B-Instruct) |
-| Mistral Small 3.2 24B Instruct 2506 | Q4_K_M, Q8_0 | 0.15 | — | — | — | — | [HF](https://huggingface.co/mistralai/Mistral-Small-3.2-24B-Instruct-2506) |
 | Devstral Small 2 24B Instruct 2512 | Q4_K_M, Q8_0 | 0.15 | — | — | — | — | [HF](https://huggingface.co/mistralai/Devstral-Small-2-24B-Instruct-2512) |
-| Mistral Small 4 119B 2603 UD | Q4_K_M | 0.7 | — | — | — | `chat_template_kwargs.reasoning_effort="high"`⁷ | [HF](https://huggingface.co/mistralai/Mistral-Small-4-119B-2603) |
+| Mistral Small 4 119B 2603 UD | Q4_K_M | 0.7 | — | — | — | `chat_template_kwargs.reasoning_effort="high"`⁹ | [HF](https://huggingface.co/mistralai/Mistral-Small-4-119B-2603) |
 | Gemma-4 26B-A4B-it | Q4_K_M (UD), Q8_0 | 1.0 | 0.95 | 64 | — | — | [HF](https://huggingface.co/google/gemma-4-26b-a4b-it) |
 | Gemma-4 31B-it | Q4_K_M | 1.0 | 0.95 | 64 | — | — | [HF](https://huggingface.co/google/gemma-4-31b-it) |
-| NVIDIA Nemotron-3 Super 120B-A12B UD | Q4_K_M | 1.0 | 0.95 | — | — | thinking on; low_effort, force_nonempty_content via `chat_template_kwargs`⁸ | [HF](https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16) |
-| Nemotron-3 Nano 30B-A3B | Q4_K_M | 0.6 | 0.95 | — | — | tool-calling preset (deterministic); thinking on via `chat_template_kwargs`⁹ | [HF](https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16) |
-| gpt-oss 120b | Q4_K_M | 1.0 | 1.0 | 0 | 0.0 | `chat_template_kwargs.reasoning_effort="medium"`; **do not** set repeat/presence penalties¹⁰ | [HF](https://huggingface.co/openai/gpt-oss-120b) |
-
-⁶ Qwen3.5-122B-A10B: smoke probe starts with `--reasoning-budget 0` (instruct mode). Bumping to thinking mode is a separate run if smoke clean. Card lists a "balanced" preset of T=0.7 / top_p=0.8 / top_k=20.
-⁷ Mistral-Small-4 card gives T=0.7 for `reasoning_effort="high"` and "between 0.0 and 0.7" for `reasoning_effort="none"` (task-dependent). High-effort profile picked as the safer default; top_p/top_k not specified on the card.
-⁸ Nemotron-3 Super: card recommends T=1.0 / top_p=0.95 across all tasks. `low_effort` reins in over-thinking; `force_nonempty_content` so the model emits something substantive instead of empty `<think>` blocks.
-⁹ Nemotron-3 Nano: card splits sampling into "Reasoning" (T=1.0, top_p=1.0) and "Tool-calling" (T=0.6, top_p=0.95) presets. Currently using the tool-calling preset to test whether F3 deception observed at higher T is temperature-tunable.
-¹⁰ gpt-oss-120b: per the llama.cpp maintainer guide, **explicitly do not set `repeat_penalty` or `presence_penalty`** — they degrade output. Registry omission == None == field omitted from request body. `reasoning_effort` adjustable via `chat_template_kwargs`; "medium" is the current default, bring down if overthinking observed.
+| NVIDIA Nemotron-3 Super 120B-A12B UD | Q4_K_M | 1.0 | 0.95 | — | — | thinking on; low_effort, force_nonempty_content via `chat_template_kwargs`¹⁰ | [HF](https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16) |
+| gpt-oss 120b | Q4_K_M | 1.0 | 1.0 | 0 | 0.0 | `chat_template_kwargs.reasoning_effort="medium"`; **do not** set repeat/presence penalties¹¹ | [HF](https://huggingface.co/openai/gpt-oss-120b) |
+
+⁸ Qwen3.5-122B-A10B: values are the card's "balanced" instruct preset (T=0.7 / top_p=0.8 / top_k=20). Thinking mode uses different sampling and is tracked as a separate configuration.
+⁹ Mistral-Small-4 card gives T=0.7 for `reasoning_effort="high"` and "between 0.0 and 0.7" for `reasoning_effort="none"` (task-dependent). High-effort profile picked as the safer default; top_p/top_k not specified on the card.
+¹⁰ Nemotron-3 Super: card recommends T=1.0 / top_p=0.95 across all tasks. `low_effort` reins in over-thinking; `force_nonempty_content` so the model emits something substantive instead of empty `<think>` blocks.
+¹¹ gpt-oss-120b: per the llama.cpp maintainer guide, **explicitly do not set `repeat_penalty` or `presence_penalty`** — they degrade output. Registry omission == None == field omitted from request body. `reasoning_effort` adjustable via `chat_template_kwargs`; "medium" is the current default, bring down if overthinking observed.
 
 ---