Skip to content

feat(agent): surface runtime resolution diagnostics#2842

Open
RitwijParmar wants to merge 1 commit into
TracecatHQ:mainfrom
RitwijParmar:codex/tracecat-agent-runtime-diagnostics
Open

feat(agent): surface runtime resolution diagnostics#2842
RitwijParmar wants to merge 1 commit into
TracecatHQ:mainfrom
RitwijParmar:codex/tracecat-agent-runtime-diagnostics

Conversation

@RitwijParmar

@RitwijParmar RitwijParmar commented Jun 10, 2026

Copy link
Copy Markdown

References #2656.

This adds a metadata-only runtime diagnostics path for agent runs, so an operator can tell what Tracecat actually resolved before or after a model call without exposing prompts or secrets.

What changed:

  • Adds RuntimeResolution, a structured diagnostics model for the selected runtime, model route, prompt/config shape, tool/MCP/subagent/skill counts, approval settings, and resume/fork flags.
  • Threads that metadata through the sandbox result protocol, socket writer, loopback result, executor result, and durable agent output.
  • Emits a runtime_resolution stream event in the Claude sandbox path before the model call starts, which should help debug failures that happen before a final result exists.
  • Adds equivalent result metadata for the legacy pydantic-ai runtime path.
  • Documents what is included and, more importantly, what is intentionally excluded. No prompt bodies, tool names, MCP headers, OAuth tokens, secret values, or resolved variables are serialized.
  • Adds focused unit coverage for metadata-only serialization, result-envelope round-trip behavior, stream-event shape, and resolved Claude runtime counts.

Validation:

  • uv run ruff check tracecat/agent/common/types.py tracecat/agent/schemas.py tracecat/agent/common/protocol.py tracecat/agent/common/socket_io.py tracecat/agent/executor/loopback.py tracecat/agent/executor/activity.py tracecat/agent/common/stream_types.py tracecat/agent/runtime/claude_code/runtime.py tracecat/agent/runtime/pydantic_ai/runtime.py packages/tracecat-ee/tracecat_ee/agent/workflows/durable.py tests/unit/test_agent_runtime_resolution.py
  • uv run basedpyright tracecat/agent/common/types.py tracecat/agent/schemas.py tracecat/agent/common/protocol.py tracecat/agent/common/socket_io.py tracecat/agent/executor/loopback.py tracecat/agent/executor/activity.py tracecat/agent/common/stream_types.py tracecat/agent/runtime/claude_code/runtime.py tracecat/agent/runtime/pydantic_ai/runtime.py packages/tracecat-ee/tracecat_ee/agent/workflows/durable.py tests/unit/test_agent_runtime_resolution.py
  • uv run python -m py_compile ...
  • git diff --check

I also tried uv run pytest tests/unit/test_agent_runtime_resolution.py -q, but this local checkout cannot start Tracecat's global Postgres-backed test fixture because no Postgres server is listening on 127.0.0.1:5432. The tests were collected, then blocked during fixture setup before the test bodies ran.


Summary by cubic

Adds metadata-only runtime diagnostics for agent runs so operators can see which runtime, model route, and config were resolved without exposing prompts or secrets. Surfaces as a pre-call stream event and in final results to help debug early failures. Supports #2656.

  • New Features
    • Added RuntimeResolution diagnostics model (runtime, model provider/name/route, prompt lengths, output type, tool/MCP/subagent/skill counts, approvals, resume/fork flags).
    • Emits a runtime_resolution stream event before the model call in the Claude runtime; includes the same metadata in result envelopes and durable workflow output.
    • Added equivalent diagnostics for the legacy pydantic-ai runtime path.
    • Threaded diagnostics through the sandbox protocol, socket writer, loopback, executor, and output schemas.
    • Explicitly excludes prompts, tool names, MCP headers, tokens, secrets, and resolved variables.
    • Added docs page on runtime diagnostics and focused unit tests for serialization, stream event shape, and count logic.

Written for commit 5aa29ad. Summary will update on new commits.

Review in cubic

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 13 files

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.

Re-trigger cubic

@zeropath-ai

zeropath-ai Bot commented Jun 10, 2026

Copy link
Copy Markdown

No security or compliance issues detected. Reviewed everything up to 5aa29ad.

Security Overview
Detected Code Changes

| Change Type | Relevant files

... (code changes summary truncated to fit VCS comment limits.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant