Skip to content

exec.command returns HTTP 500 'Failed to execute command' after ~5min, but command actually completes on backend #160

Description

@shtefcs

Summary

POST /v2/box/{id}/exec returns HTTP 500 {"error":"Failed to execute command"} after approximately 5 minutes for any single command that takes longer than that, even though the command actually completes successfully on the backend. The same applies to exec.stream and exec.code. The misleading 500 forces clients into false-positive failure handling , for our use case (an AI coding agent running npm install and next build), the agent thinks the build failed and triggers retry/error flows while the build is in fact finishing in the background.

Reproduction

Tested on @upstash/box@0.4.1 against us-east-1.box.upstash.com.

import { Box } from "@upstash/box";
const box = await Box.create({ runtime: "node", apiKey, keepAlive: false });

// Single command that takes 6 minutes and leaves a side effect
const t0 = Date.now();
try {
  const r = await box.exec.command("sleep 360 && touch /workspace/home/marker.txt && echo done");
  console.log(`OK in ${Date.now() - t0}ms`); // expected: ~6 min, exit 0, "done"
} catch (e) {
  console.log(`FAILED at ${Date.now() - t0}ms: ${e.message}`);
}

// Wait extra 90s, then verify side effect
await new Promise(r => setTimeout(r, 90_000));
const check = await box.exec.command("test -f /workspace/home/marker.txt && echo MARKER_EXISTS || echo MISSING");
console.log(check.result.trim());

Observed (with default Node undici dispatcher)

FAILED at 300727ms: fetch failed
cause: HeadersTimeoutError (UND_ERR_HEADERS_TIMEOUT)
MARKER_EXISTS  <-- command did complete on backend

The first failure is a client-side undici default headersTimeout of 300_000ms , known Node default, we worked around it by setting a custom global dispatcher.

Observed (with extended client dispatcher: headersTimeout: 900_000)

The fetch now waits the full duration, but Upstash returns:

HTTP 500 Internal Server Error
content-type: application/json
body: {"error":"Failed to execute command"}

Timing: response arrives at exactly the command's natural completion time (320s for sleep 320, 360s for sleep 360, 480s for sleep 480). So the backend isn't aborting at 5 min , it waits for the command to finish, but always returns 500 if duration exceeds ~5 minutes.

Verified completion: the side-effect file IS created at the expected time despite the 500 response. Tried multiple methods, all behave the same:

  • exec.command shell sleep
  • exec.stream shell sleep
  • exec.code Python time.sleep
  • Continuous-output shell loop (for i in $(seq 1 36); do echo line$i; sleep 10; done)
  • EphemeralBox.exec.command

Impact

Production code (TypeScript) catches the throw and reports failure to the calling code:

// Our adapter
try {
  const run = await box.exec.command(command);
  // ...
} catch (e) {
  return { stdout: "", stderr: e.message, exitCode: 1 };
}

For us this means any npm install or next build that takes more than 5 minutes is mis-reported as failed. We then retry, error out to the user, or both , while the build actually succeeded on the backend.

The workaround we ended up using is the fire-and-poll pattern (background the command with nohup, poll a marker file). It works but it's fragile and adds latency. For shell commands that don't have a clean marker, it's also tricky.

Suggested fixes (any one is sufficient)

  1. Extend the backend response timeout to match the actual command duration.
  2. Return a 202 {"status":"running","run_id":"..."} response that the client can poll via /v2/box/{id}/runs/{run_id}.
  3. Return the actual exit code/output when the command finishes, even if duration exceeds 5 minutes.
  4. At minimum, document this behavior so SDK users know to use the fire-and-poll pattern for long-running commands.

Environment

  • @upstash/box: 0.4.1
  • Node.js: 20.19 / 22.x
  • Box runtime: node
  • Account: PayG plan
  • Box size: small (also reproduced on medium)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions