You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Found running a full autobrin-vs-pithos comparison sweep: 4 of 4 Daytona-transport jobs (webapp modality, CVE-Bench + BountyBench-Exploit) failed before producing a score.
Evidence
Two jobs failed identically and fast (~2 min):
```
Daytona sandbox created:
sandbox.computerUse.start() failed on sandbox : {"details":"failed to start: [x11vnc novnc]", ...}. Continuing without computer-use.
Daytona sandbox deleted:
[run-one-task] FAILED: Error: /usr/bin/bash: line 19: 313 Killed npm install >> '/home/daytona/benchpress/logs/autobrin-flue-install.log' 2>&1
```
`Killed` with no other signal is the classic Linux OOM-killer signature. A third job (CVE-2024-3408) ran the full 40-minute timeout instead of an outright kill — consistent with memory pressure causing severe slowdown rather than an immediate kill, i.e. this isn't 100% deterministic, likely depends on host machine load/placement.
The x11vnc/novnc start failure in the log is a separate, already-known, non-fatal issue (see #38/#39) — not the cause here.
Root cause
examples/node22-bookworm-computer-use-image.ts's declarative image build never sets resources (cpu/memory/disk) on sandbox creation, so it gets Daytona's platform default. Confirmed the SDK supports this:
AutobrinContenderConfig (`src/contenders/autobrin.ts`) has no field to request more resources, and createSandbox() (`src/daytona/client.ts`) just forwards whatever SandboxCreateInput it's given -- so there's no way to plumb this through today even for a caller who wants to.
Suggested fix
Add an optional resources?: { cpu?: number; memory?: number; disk?: number } field to AutobrinContenderConfig, threaded through to the SandboxCreateInput passed to createSandbox().
Give examples/node22-bookworm-computer-use-image.ts's own example script (and any other caller building this declarative image) a sane default memory bump (e.g. 4GiB) given a full npm install of AutoBrin-flue's dependency tree is known to need more than whatever the current default provides.
Blocks every webapp-modality Daytona-transport engagement from completing reliably -- both CVE-Bench and BountyBench's Exploit lane depend on this transport for real computer-use confirmation (per #32/#38).
Summary
Found running a full autobrin-vs-pithos comparison sweep: 4 of 4 Daytona-transport jobs (webapp modality, CVE-Bench + BountyBench-Exploit) failed before producing a score.
Evidence
Two jobs failed identically and fast (~2 min):
```
Daytona sandbox created:
sandbox.computerUse.start() failed on sandbox : {"details":"failed to start: [x11vnc novnc]", ...}. Continuing without computer-use.
Daytona sandbox deleted:
[run-one-task] FAILED: Error: /usr/bin/bash: line 19: 313 Killed npm install >> '/home/daytona/benchpress/logs/autobrin-flue-install.log' 2>&1
```
`Killed` with no other signal is the classic Linux OOM-killer signature. A third job (CVE-2024-3408) ran the full 40-minute timeout instead of an outright kill — consistent with memory pressure causing severe slowdown rather than an immediate kill, i.e. this isn't 100% deterministic, likely depends on host machine load/placement.
The
x11vnc/novncstart failure in the log is a separate, already-known, non-fatal issue (see #38/#39) — not the cause here.Root cause
examples/node22-bookworm-computer-use-image.ts's declarative image build never setsresources(cpu/memory/disk) on sandbox creation, so it gets Daytona's platform default. Confirmed the SDK supports this:```ts
// @daytona/sdk Daytona.d.ts
export interface Resources {
cpu?: number;
gpu?: number;
memory?: number; // GiB
disk?: number; // GiB
}
```
AutobrinContenderConfig(`src/contenders/autobrin.ts`) has no field to request more resources, andcreateSandbox()(`src/daytona/client.ts`) just forwards whateverSandboxCreateInputit's given -- so there's no way to plumb this through today even for a caller who wants to.Suggested fix
resources?: { cpu?: number; memory?: number; disk?: number }field toAutobrinContenderConfig, threaded through to theSandboxCreateInputpassed tocreateSandbox().examples/node22-bookworm-computer-use-image.ts's own example script (and any other caller building this declarative image) a sane default memory bump (e.g. 4GiB) given a fullnpm installof AutoBrin-flue's dependency tree is known to need more than whatever the current default provides.lunary-0-exploittask, to confirmnpm installcompletes reliably under load.Impact
Blocks every webapp-modality Daytona-transport engagement from completing reliably -- both CVE-Bench and BountyBench's Exploit lane depend on this transport for real computer-use confirmation (per #32/#38).