Skip to content

Commit d784364

Browse files
kevinkevin
authored andcommitted
docs: cite harness sensitivity paper
1 parent b42b9ed commit d784364

3 files changed

Lines changed: 20 additions & 4 deletions

File tree

README.ko.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -316,7 +316,7 @@ Helm은 dedicated workspace에서 동작하며, 기존 시스템을 read-only co
316316
317317
## 연구 배경
318318
319-
Helm의 설계 방향은 [Harness Design Determines Operational Stability in Small Language Models](https://arxiv.org/abs/2605.12129) 의 결과와 일치합니다. 이 논문은 planning, verification, recovery harness가 소형 언어 모델의 운영 안정성에 어떻게 영향을 주는지 실험적으로 연구합니다.
319+
Helm의 설계 방향은 [Harness Design Determines Operational Stability in Small Language Models](https://arxiv.org/abs/2605.12129) 의 결과와 일치합니다. 이 논문은 planning, verification, recovery harness가 소형 언어 모델의 운영 안정성에 어떻게 영향을 주는지 실험적으로 연구합니다. Helm의 adaptive harness 방향은 [It's Not the Capability: Harness Sensitivity Is Non-Monotone Across LLM Agent Tiers](https://arxiv.org/abs/2605.26731) 의 결과와도 연결됩니다. 이 후속 논문은 harness strictness를 모든 모델에 일괄 적용하기보다 모델 타입과 실패 패턴에 맞춰 선택해야 함을 보여줍니다.
320320
321321
Helm 인용:
322322

README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -316,7 +316,7 @@ See deeper comparisons in [`docs/comparisons/`](docs/comparisons/).
316316
317317
## Research background
318318
319-
Helm's design follows the findings in [Harness Design Determines Operational Stability in Small Language Models](https://arxiv.org/abs/2605.12129), which experimentally studies how planning, verification, and recovery harnesses affect operational stability.
319+
Helm's design follows the findings in [Harness Design Determines Operational Stability in Small Language Models](https://arxiv.org/abs/2605.12129), which experimentally studies how planning, verification, and recovery harnesses affect operational stability. Its adaptive-harness direction is also informed by [It's Not the Capability: Harness Sensitivity Is Non-Monotone Across LLM Agent Tiers](https://arxiv.org/abs/2605.26731), which shows that harness strictness should be selected by model type and failure mode rather than applied uniformly.
320320
321321
Cite Helm:
322322

docs/research-background.md

Lines changed: 18 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,18 +2,23 @@
22

33
Helm is a practical operations layer for long-lived agent workspaces. Its design is grounded in a simple observation: repeated agent work does not fail only because a model is too weak. It often fails because the surrounding harness does not make planning, verification, recovery, and auditability explicit.
44

5-
That framing is aligned with the arXiv paper:
5+
That framing is aligned with two arXiv papers:
66

77
> Yong Eun Cho, ["Harness Design Determines Operational Stability in Small Language Models"](https://arxiv.org/abs/2605.12129), arXiv:2605.12129, 2026.
88
9-
The paper studies how different harness conditions affect the operational performance of small language models. It compares raw model-only prompting, minimal wrapper tags, and a structured `plan -> execute -> verify -> recover` pipeline across multiple 2-3B parameter models and task types.
9+
> Yong Eun Cho, ["It's Not the Capability: Harness Sensitivity Is Non-Monotone Across LLM Agent Tiers"](https://arxiv.org/abs/2605.26731), arXiv:2605.26731, 2026.
10+
11+
The first paper studies how different harness conditions affect the operational performance of small language models. It compares raw model-only prompting, minimal wrapper tags, and a structured `plan -> execute -> verify -> recover` pipeline across multiple 2-3B parameter models and task types.
12+
13+
The follow-up paper widens that question across model tiers and shows that harness sensitivity is non-monotone: stricter harnesses can hurt a frontier chat model, help a frontier reasoning model, and interact differently with constrained models depending on instruction-following quality and failure mode.
1014

1115
Key takeaways for Helm:
1216

1317
- **Harness design is operational infrastructure.** Agent performance depends not only on the model call, but also on the surrounding workflow that shapes planning, execution, validation, and recovery.
1418
- **Minimal scaffolding is not automatically safer.** A thin wrapper can fail to improve stability, and in some cases can perform worse than model-only prompting.
1519
- **Verification and recovery need first-class representation.** Reliable agent work needs explicit checks, catch points, and recovery paths rather than best-effort prompt instructions.
1620
- **Small models benefit from structured operating loops.** Smaller models can become more usable when the runtime supplies a disciplined task loop and state boundary.
21+
- **Harness policy should be adaptive, not one-size-fits-all.** The right amount of structure depends on the model's failure pattern: format violations, wrong-file edits, missing changes, and unresolved verifier failures call for different runtime responses.
1722

1823
Helm applies those lessons at the workspace level. Instead of trying to be another agent runtime, it provides the operational substrate around existing runtimes:
1924

@@ -40,4 +45,15 @@ The paper should be read as research background, not as a claim that Helm is the
4045
doi = {10.48550/arXiv.2605.12129},
4146
url = {https://arxiv.org/abs/2605.12129}
4247
}
48+
49+
@misc{cho2026harnesssensitivity,
50+
title = {It's Not the Capability: Harness Sensitivity Is Non-Monotone Across LLM Agent Tiers},
51+
author = {Yong Eun Cho},
52+
year = {2026},
53+
eprint = {2605.26731},
54+
archivePrefix= {arXiv},
55+
primaryClass = {cs.AI},
56+
doi = {10.48550/arXiv.2605.26731},
57+
url = {https://arxiv.org/abs/2605.26731}
58+
}
4359
```

0 commit comments

Comments
 (0)