You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.ko.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -316,7 +316,7 @@ Helm은 dedicated workspace에서 동작하며, 기존 시스템을 read-only co
316
316
317
317
## 연구 배경
318
318
319
-
Helm의 설계 방향은 [Harness Design Determines Operational Stability in Small Language Models](https://arxiv.org/abs/2605.12129) 의 결과와 일치합니다. 이 논문은 planning, verification, recovery harness가 소형 언어 모델의 운영 안정성에 어떻게 영향을 주는지 실험적으로 연구합니다.
319
+
Helm의 설계 방향은 [Harness Design Determines Operational Stability in Small Language Models](https://arxiv.org/abs/2605.12129) 의 결과와 일치합니다. 이 논문은 planning, verification, recovery harness가 소형 언어 모델의 운영 안정성에 어떻게 영향을 주는지 실험적으로 연구합니다. Helm의 adaptive harness 방향은 [It's Not the Capability: Harness Sensitivity Is Non-Monotone Across LLM Agent Tiers](https://arxiv.org/abs/2605.26731) 의 결과와도 연결됩니다. 이 후속 논문은 harness strictness를 모든 모델에 일괄 적용하기보다 모델 타입과 실패 패턴에 맞춰 선택해야 함을 보여줍니다.
Copy file name to clipboardExpand all lines: README.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -316,7 +316,7 @@ See deeper comparisons in [`docs/comparisons/`](docs/comparisons/).
316
316
317
317
## Research background
318
318
319
-
Helm's design follows the findings in [Harness Design Determines Operational Stability in Small Language Models](https://arxiv.org/abs/2605.12129), which experimentally studies how planning, verification, and recovery harnesses affect operational stability.
319
+
Helm's design follows the findings in [Harness Design Determines Operational Stability in Small Language Models](https://arxiv.org/abs/2605.12129), which experimentally studies how planning, verification, and recovery harnesses affect operational stability. Its adaptive-harness direction is also informed by [It's Not the Capability: Harness Sensitivity Is Non-Monotone Across LLM Agent Tiers](https://arxiv.org/abs/2605.26731), which shows that harness strictness should be selected by model type and failure mode rather than applied uniformly.
Copy file name to clipboardExpand all lines: docs/research-background.md
+18-2Lines changed: 18 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -2,18 +2,23 @@
2
2
3
3
Helm is a practical operations layer for long-lived agent workspaces. Its design is grounded in a simple observation: repeated agent work does not fail only because a model is too weak. It often fails because the surrounding harness does not make planning, verification, recovery, and auditability explicit.
4
4
5
-
That framing is aligned with the arXiv paper:
5
+
That framing is aligned with two arXiv papers:
6
6
7
7
> Yong Eun Cho, ["Harness Design Determines Operational Stability in Small Language Models"](https://arxiv.org/abs/2605.12129), arXiv:2605.12129, 2026.
8
8
9
-
The paper studies how different harness conditions affect the operational performance of small language models. It compares raw model-only prompting, minimal wrapper tags, and a structured `plan -> execute -> verify -> recover` pipeline across multiple 2-3B parameter models and task types.
9
+
> Yong Eun Cho, ["It's Not the Capability: Harness Sensitivity Is Non-Monotone Across LLM Agent Tiers"](https://arxiv.org/abs/2605.26731), arXiv:2605.26731, 2026.
10
+
11
+
The first paper studies how different harness conditions affect the operational performance of small language models. It compares raw model-only prompting, minimal wrapper tags, and a structured `plan -> execute -> verify -> recover` pipeline across multiple 2-3B parameter models and task types.
12
+
13
+
The follow-up paper widens that question across model tiers and shows that harness sensitivity is non-monotone: stricter harnesses can hurt a frontier chat model, help a frontier reasoning model, and interact differently with constrained models depending on instruction-following quality and failure mode.
10
14
11
15
Key takeaways for Helm:
12
16
13
17
-**Harness design is operational infrastructure.** Agent performance depends not only on the model call, but also on the surrounding workflow that shapes planning, execution, validation, and recovery.
14
18
-**Minimal scaffolding is not automatically safer.** A thin wrapper can fail to improve stability, and in some cases can perform worse than model-only prompting.
15
19
-**Verification and recovery need first-class representation.** Reliable agent work needs explicit checks, catch points, and recovery paths rather than best-effort prompt instructions.
16
20
-**Small models benefit from structured operating loops.** Smaller models can become more usable when the runtime supplies a disciplined task loop and state boundary.
21
+
-**Harness policy should be adaptive, not one-size-fits-all.** The right amount of structure depends on the model's failure pattern: format violations, wrong-file edits, missing changes, and unresolved verifier failures call for different runtime responses.
17
22
18
23
Helm applies those lessons at the workspace level. Instead of trying to be another agent runtime, it provides the operational substrate around existing runtimes:
19
24
@@ -40,4 +45,15 @@ The paper should be read as research background, not as a claim that Helm is the
40
45
doi = {10.48550/arXiv.2605.12129},
41
46
url = {https://arxiv.org/abs/2605.12129}
42
47
}
48
+
49
+
@misc{cho2026harnesssensitivity,
50
+
title = {It's Not the Capability: Harness Sensitivity Is Non-Monotone Across LLM Agent Tiers},
0 commit comments