fix: print "job timed out"/"job cancelled" instead of "context cancelled"

[mastermanas805]

Description

Bare context canceled was leaking to user-facing job logs from the Kubernetes runner (kubernetes/runner.go) and the executor's command phase. Adds internal/job.FormatJobError: context.DeadlineExceeded → "job timed out", context.Canceled → "job cancelled". Routed through the three sites that surface this error.

Fixes #3384

Server-side, a UI cancel and job-level timeout both arrive as context.Canceled (the agent can't distinguish locally), so Canceled → "cancelled" is honest; only DeadlineExceeded (set by the agent itself) → "timed out".

[moskyb]

@mastermanas805 can you go into a bit more detail about how/where you're seeing context cancelled/deadline exceeded errors in the logs? when i cancel jobs i get the standard "terminated" message:

i agree that better error messages are a very good thing, so if you can provide some steps to reproduce the errors you're seeing, i'd really appreciate it.

[mastermanas805]

You're seeing "terminated" because the non-k8s path works fine — the leak only shows up under the kubernetes runner, which is why this has been hard to pin down historically.

Added kubernetes/repro_3384_test.go (552bd2a) that spins up a real kubernetes.Runner, cancels its context, and prints the resulting job log line.

On main:

user-visible job log line on main: "Error running job: context canceled"

On this branch:

POST-FIX job log line: "Error running job: job cancelled"

Only the agent/run_job.go call site differs.

The two paths diverge because of how process.Run is implemented. *process.Process.Run (regular agent) blocks on command.Wait(), which returns *exec.ExitError with signal info — so on cancel you land in isExitError && isExitSignaled at executor.go:1135 and get the "interrupted by a signal: terminated" line from your screenshot. *kubernetes.Runner.Run is a different beast — it's a socket server, not a wrapped subprocess, so when its context cancels it just returns ctx.Err() raw (kubernetes/runner.go:122-123). That used to flow straight through agent/run_job.go:333 into the log. The existing test at kubernetes_test.go:292 even documents this with an errors.Is(err, context.Canceled) tolerance.

To reproduce live: run anything via agent-stack-k8s (or --kubernetes-exec) and cancel from the UI. Job-level timeouts come through the same way server-side.

The two executor.go changes are defensive — the executor (bootstrap) runs inside the user's container in k8s and as a child process otherwise, so the same wrapping covers any ctx error that slips into the default branch of the command-phase switch, e.g. an early setup failure while the context is already cancelled. I don't have a clean repro for those, so happy to drop them and keep the fix to just the k8s call site if you'd rather.

[mastermanas805]

@moskyb

[moskyb]

heya, the test above was performed on a live instance of the agent-stack-k8s. if you could produce evidence of these errors making it to logs in live agent instances — not in tests where you're manually cancelling the context — i would really appreciate it. context cancelled/deadline timeout errors will only make it into job logs if the agent's context gets cancelled, which is a code path that i don't think actually gets exercised in production workloads.