Design: container exit and output observability (#909)

[BatmanByte]

Container Exit and Output Observability

Status and scope

This proposal replaces the pre-#988 design in Discussion #967.

It addresses #909: init stdout and stderr are not durable, and an unread pipe can block the workload. It also completes the resource-lifecycle work exposed by always-on output draining.

#988 already made the box exit code durable. This document treats that behavior as a prerequisite; it does not redesign terminal box state.

The design is based on origin/main, including #1022. Constants such as write-batch size, queue capacity, and shutdown timeouts are intentionally not chosen yet: they require a guest-side virtiofs benchmark.

Goals

1. Drain init output continuously, whether or not a client attaches.
2. Persist init output so boxlite logs works after the box has stopped.
3. Never let a slow filesystem or log reader block the container workload.
4. Report loss and incomplete capture honestly rather than presenting it as a clean log.
5. Bound memory, file descriptors, tasks, and on-disk usage.
6. Keep the first implementation local: guest capture and local reads. Cloud propagation is a later, separately shippable layer.

Non-goals

• Keeping a VM alive after init exits for interactive debugging.
• Persisting output from arbitrary exec sessions. Only init is captured to disk.
• Re-attach semantics beyond the existing delayed-attach window.
• Long-term log archival or integration with external log systems.
• Treating Stopped as evidence that capture completed successfully.

Current model and constraints

#1022 changes execution output from attach-driven reads to one long-lived pump per stream. The pump appends to a bounded ring used by Attach. Every Stdout and Stderr data event carries that stream's absolute byte offset; an empty event with offset == total_bytes marks that stream's end. It is the required base for this proposal.

flowchart LR
    P["init / exec process"] --> K["kernel pipe"]
    K --> O["OutputManager pump"]
    O --> R["1 MiB attach ring"]
    O --> S["CaptureSink (init only)"]
    S --> L["output.log on virtiofs"]

Loading

The important constraints are:

• A pipe has one reader. A second drain task would either race the attach path or require another bounded relay.
• The only permitted backpressure is the kernel pipe's native backpressure to the process. Neither the ring nor disk capture may await a downstream consumer.
• shutdown_all() sends SIGKILL and returns before the process is necessarily dead.
• Init exiting does not imply that every inherited stdout/stderr writer has exited. sh -c 'child & exit' is the minimal counterexample.
• The registry still retains completed SDK and init sessions indefinitely. feat(exec): drain guest output before attach #1022 releases explicitly ephemeral SSH sessions, but holding an unreleased ExecHandle still retains stdin and PTY file descriptors.

Attach loss contract

Offsets are per-stream source-byte positions: an offset is the number of bytes emitted on that file descriptor before the event's payload. It starts at zero when the process stream is created and does not reset for Attach. A new Attach starts with an expected offset of zero for each stream. If a received offset is ahead of that expected value, the reader reports the difference as loss, flushes only that stream's UTF-8 decoder, and then decodes the payload.

The terminal empty event makes the same contract complete when no data event for a stream survived the shared ring: total_bytes > expected reports the missing tail or the entire stream. These positions are not output.log file offsets; CRI headers and stdout/stderr interleaving make the two domains different. The optional fields preserve compatibility with older guests; a reader that receives no offset treats that event as contiguous and cannot infer a gap.

Design summary

Decision	Choice
Output source	One OutputManager pump per session; capture is a consumer of that pump.
Capture scope	Init only.
Log writer	The guest writes a dedicated virtiofs share; the host reads it after the box stops.
Backpressure	A bounded channel with try_send; full means dropping the newest capture chunk.
Capture format	CRI-compatible records plus a private boxlite stream for metadata.
Completeness	A matching begin and end, plus zero end.dropped, are required for lossless completion.
Startup guarantee	begin is synchronously written and fsynced before Container.Start.
Shutdown	Kill the complete container cgroup, join pumps, close the one sender, then await the writer.
Execution cleanup	Convert finished sessions to bounded tombstones and release handles, rings, and reaper slots.
Deletion	capture_logs and remove-on-stop are rejected as incompatible options.

Data flow and backpressure

The capture sink is pushed from OutputManager::push(). It does not maintain a cursor into the attach ring. A pull model would silently fall behind ring eviction, or would force the ring lifetime to follow the slowest consumer.

There are three buffers:

Buffer	Full behavior	Affected consumer
Kernel pipe	Blocks the process writer	The workload itself
Attach ring	Drops oldest chunks; offset gaps expose the exact missing source bytes	Attach clients
Capture channel	Drops newest bytes	Durable log

Only the pipe may block. Awaiting channel capacity, filesystem writes, or an attach client would propagate pressure back through the pump and recreate #909.

CaptureSink is one bounded channel and one writer task. The sender is stored only in OutputState.sink; pumps borrow it through push() and never clone it. This gives shutdown a single owner:

Running(sender, pump join handles)
  -> all pumps reached EOF and joined
  -> take and drop the sole sender
  -> writer receives None, finishes the stream, writes end, flushes, exits

If the bounded writer wait times out, the writer task is aborted and the run is incomplete. It must not continue writing after the shutdown RPC has returned.

Drop accounting

The sink keeps two per-stream counters:

• pending_dropped: bytes not yet described by a drop metadata record.
• total_dropped: bytes lost during the entire run; monotonic and never cleared.

On TrySendError::Full, record_drop(stream, len) increments both. Before a later data chunk is sent, the producer tries to enqueue Chunk::Drop(pending_dropped). Only a successful enqueue clears pending_dropped; it never clears total_dropped.

The following edge case is intentional and needs a unit test: with a capacity-one channel, a successful Drop fills the slot again, so the immediately following Data send can still be full. That data must be counted again.

The writer reads total_dropped from shared Arc<AtomicU64> values when it writes end. It must not receive the final total through the bounded channel: that message could be dropped precisely when the total is most important. Pumps have joined before the read, so no concurrent increment remains.

Log format and integrity model

When capture is enabled, the host creates a dedicated logs share at:

host:  ~/.boxlite/boxes/{box}/logs/{container}/output.log
guest: /run/boxlite/logs/output.log

The guest is the only writer. The path is not supplied over gRPC: the guest derives it from the fixed capture-share mount and opens the final file with O_NOFOLLOW.

The file is a CRI-compatible extension. A regular record is:

<RFC3339Nano> <stream> <F|P> <content>

stdout and stderr contain application output. The private boxlite stream contains structured metadata that an application cannot forge by printing matching text:

2026-07-27T03:14:15.926535897Z boxlite F {"run":"b3f1c0a4-7d2e-4a91-8c55-0e6f2ab41d90","event":"begin"}
2026-07-27T03:14:16.031204118Z stdout F Starting build...
2026-07-27T03:14:18.902133871Z boxlite F {"run":"b3f1c0a4-7d2e-4a91-8c55-0e6f2ab41d90","event":"drop","stdout":8192,"stderr":0}
2026-07-27T03:14:19.402771005Z boxlite F {"run":"b3f1c0a4-7d2e-4a91-8c55-0e6f2ab41d90","event":"end","dropped":{"stdout":8192,"stderr":0}}

This is not standard CRI because boxlite is not a standard CRI stream. Generic CRI readers that only filter stdout and stderr still see application payload; boxlite logs and the future remote endpoint are the only readers required to interpret metadata.

The writer never converts payload with from_utf8_lossy. Content is framed at 16 KiB, with P fragments followed by F for a newline-terminated line. A final line without a newline remains a trailing P; readers do not add a newline for it. Therefore binary data and split UTF-8 sequences are preserved without inventing replacement characters.

Run records and reader behavior

Every Container.Init attempt receives a host-generated UUID in LogCapture.run_id. A guest-memory counter is insufficient because the same log file survives VM restarts.

Readers group metadata by UUID and classify each run independently:

Metadata for one run	Meaning
begin + end, end.dropped is zero	Lossless completion
begin + end, end.dropped is non-zero	Writer closed normally, but content was lost
begin without end	Abnormal interruption; the tail may be missing
end without begin	The retained window lost the beginning through rotation
No begin in the retained file	Capture state is unknown; never report completion

The four normal states apply only to structurally valid metadata. Duplicate begin, duplicate end, end before begin, or invalid metadata JSON is reported as corrupted/unknown, never as successful capture. boxlite logs renders the newest run's status and warns if any retained earlier run is incomplete.

The file rotates in the guest at 10 MiB × 5 using a temporary file and renames. Follow mode must retry an ENOENT between rename steps. Rotation can remove the first record of an older run, which is why an isolated end is a retention condition rather than a crash diagnosis.

Startup and shutdown

Startup barrier

begin is a startup barrier, not an ordinary queued item:

1. Open the fixed log path in append mode.
2. Write the begin record and fsync it once.
3. Construct the writer task using the same append target.
4. Only then call Container.Start.

Failure at any of these steps fails Container.Init. A caller who explicitly requested capture should not discover after execution that no capture was ever active. The one startup fsync is deliberately outside the hot write path; whether it is acceptable is part of the required benchmark.

Shared drain sequence

Natural init exit and host-driven Guest.Shutdown must invoke the same bounded sequence:

0. Wait for the init ExitSlot.
1. Kill remaining processes in the complete container cgroup and wait for reaping.
2. Wait for every init pump to observe EOF and join.
3. Take and drop the sole CaptureSink sender.
4. Await writer completion: flush partial framing, write end(total_dropped), flush.
5. Continue shutdown even if any bounded wait expires.

Step 1 is required even after init's exit slot becomes ready. A background child can retain a pipe write end; killing only registered exec PIDs leaves the pump waiting for EOF indefinitely. The existing reboot also kills such children, but too late for capture: the behavioral change is the ordering.

Both paths persist the exit record before draining. Capture drain precedes sync(). The existing 500 ms network-output grace window remains separate from capture drain; it still protects in-flight attach responses, not file correctness.

An end record proves that the writer completed its own close path. It does not prove losslessness; only end.dropped == 0 proves that no capture chunks were dropped. A timeout, writer failure, or hard VM kill naturally leaves a begin without end; no unreliable crash-time status write is required.

Session and storage lifecycle

After every stream reaches EOF, a session gets a delayed attach grace period. If no attach is active at expiry, release the attach ring. An attach that ends after expiry restarts the timer.

At the same point, replace a finished session with a tombstone:

Release	Retain
ExecHandle, including stdin and PTY fds	Exit code
Attach ring	Per-stream total-byte summary
Reaper slot and watch resources	Terminal session identity

Tombstones are bounded by TTL and LRU size. An init tombstone is exempt because there is only one per box and its exit status is the box exit status. Before tombstone eviction, Wait returns the retained status, and late Attach returns an empty terminal event for each enabled stream with its retained offset == total_bytes, then EOF. Because the reader starts each stream at expected offset zero, those events report the exact per-stream loss without a control event. SendInput returns FailedPrecondition. After eviction, these requests return NotFound.

Capture consumes at most 50 MiB per box (10 MiB × 5). Capture and remove-on-stop are incompatible because removal deletes the box directory containing the log. sanitize() rejects that option combination instead of silently changing box lifecycle semantics.

Protocol and user-facing behavior

ContainerInitRequest gains:

message LogCapture {
  string run_id = 1;
}

optional LogCapture log_capture = 7;

The presence of log_capture enables capture; an absent message disables it. The host produces a UUID, while the guest parses the UUID at its gRPC boundary into an internal CaptureConfig. A malformed or empty UUID fails Init. No guest path is accepted from the host.

Other public behavior:

• capture_logs is create-time only because the share is mounted only at create time.
• boxlite logs reads container logs when available, follows rotation, and keeps console output behind an explicit flag.
• box.wait() and boxlite wait are independent of capture and return the persisted terminal status and exit code.
• Attach is still single-consumer and has no transport-resume contract. A future reconnect design can reuse these offsets only if it retains the per-stream expected offsets; the protocol does not itself permit a second Attach.

Delivery plan

First, run a disposable guest benchmark against /run/boxlite/logs/ to measure 4 KiB write latency and throughput, rotation cost, and the one startup fsync. It supplies the batch, queue, and timeout constants; it is not a merged feature.

flowchart TB
    P1["PR1: #1022 drain and ring"] --> P2["PR2: terminal session cleanup"]
    P1 --> P3["PR3: minimal durable capture"]
    P3 --> P4["PR4: positioned drop records"]
    P3 --> P5["PR5: boxlite logs"]
    P3 --> P6["PR6: SDK propagation"]
    P3 --> P7["PR7: REST/cloud propagation"]
    P7 --> P8["PR8: remote log endpoint"]
    P9["PR9: wait"]

Loading

PR	Scope	Dependency
PR1	#1022: always-on pump, 1 MiB ring, per-stream offset and terminal total_bytes, and AsyncFd reads	—
PR2	Grace-period reclamation, tombstones, and terminal Wait/Attach/SendInput semantics	PR1
PR3	Dedicated share, LogCapture, startup barrier, writer, rotation, total drop accounting, shared cgroup teardown, and option validation	PR1
PR4	Best-effort positioned drop metadata using source-byte offsets; PR3 already reports the total in end	PR3
PR5	boxlite logs, completeness rendering, rotation follow, and console flag	PR3
PR6	Node, Python, Go, and C SDK propagation	PR3
PR7	REST, OpenAPI, cloud API, and runner propagation	PR3
PR8	Remote log reading endpoint	PR7
PR9	box.wait, boxlite wait, and SDK exposure	—

PR3 is a minimum usable vertical slice: capture cannot ship without the startup barrier, bounded drain, rotation, and truthful end.dropped. PR4 may follow because it only improves where loss is displayed. PR7 must not expose a cloud capture option until PR8 makes captured logs readable remotely.

Validation matrix

Scenario	Expected result
Init writes more than a pipe buffer with no Attach, then exits 23	No deadlock; box stops with exit code 23
Attach after the ring wraps	A per-stream offset jump reports loss and resets only that stream's decoder
One stream is fully evicted before Attach	Its terminal total_bytes reports the exact loss even without a surviving data event
2,000 completed execs	File descriptor count does not grow with exec count
Tombstone before and after eviction	Retained terminal semantics, then NotFound; init is retained
Short-lived init with capture and no Attach	begin, output, and zero-loss end are present
Capture channel overflow	end.dropped equals full-run loss, not only the final pending delta
Capacity-one queue after a successful drop event	A second full Data send is counted
Begin fsynced, VM killed before Start	begin exists without end; report interruption, not success
Init runs sh -c 'child & exit'	Cgroup cleanup closes inherited writers; capture reaches end
Writer failure or VM SIGKILL during capture	No end; reader reports incomplete capture
Invalid run UUID, symlinked log path, or read-only share	Init fails before Start and does not write an unsafe target
Multiple runs and rotation removes an old begin	Runs remain distinct; reader reports retention loss rather than a false crash
capture_logs with remove-on-stop	Validation rejects the conflicting options

Rejected alternatives

• A shim monitor that streams output and writes container state: fix(run): run COMMAND as the container init (docker semantics) #988 already establishes guest-written exit state plus host shim watching, and detached capture must not depend on a long-lived shim monitor.
• A second output pump plus relay pipe: it creates another bounded buffer whose overflow either blocks the workload or silently loses unaccounted bytes.
• Metadata injected into stdout/stderr: application output can forge matching text and the log no longer preserves payload as output.
• A sidecar status file with pessimistic initialization: it adds another format and still cannot reliably write during the failures it is meant to describe.
• A host-supplied guest log path: a high-privilege guest must not trust a variable path that it can derive safely from a fixed share.

[DorianZheng]

do we think we can reuse attach API as the CaptureSink so we can simplify the protocol