feat(agent): surface runtime resolution diagnostics#2842
Open
RitwijParmar wants to merge 1 commit into
Open
Conversation
|
✅ No security or compliance issues detected. Reviewed everything up to 5aa29ad. Security Overview
Detected Code Changes| Change Type | Relevant files ... (code changes summary truncated to fit VCS comment limits.) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
References #2656.
This adds a metadata-only runtime diagnostics path for agent runs, so an operator can tell what Tracecat actually resolved before or after a model call without exposing prompts or secrets.
What changed:
RuntimeResolution, a structured diagnostics model for the selected runtime, model route, prompt/config shape, tool/MCP/subagent/skill counts, approval settings, and resume/fork flags.runtime_resolutionstream event in the Claude sandbox path before the model call starts, which should help debug failures that happen before a final result exists.Validation:
uv run ruff check tracecat/agent/common/types.py tracecat/agent/schemas.py tracecat/agent/common/protocol.py tracecat/agent/common/socket_io.py tracecat/agent/executor/loopback.py tracecat/agent/executor/activity.py tracecat/agent/common/stream_types.py tracecat/agent/runtime/claude_code/runtime.py tracecat/agent/runtime/pydantic_ai/runtime.py packages/tracecat-ee/tracecat_ee/agent/workflows/durable.py tests/unit/test_agent_runtime_resolution.pyuv run basedpyright tracecat/agent/common/types.py tracecat/agent/schemas.py tracecat/agent/common/protocol.py tracecat/agent/common/socket_io.py tracecat/agent/executor/loopback.py tracecat/agent/executor/activity.py tracecat/agent/common/stream_types.py tracecat/agent/runtime/claude_code/runtime.py tracecat/agent/runtime/pydantic_ai/runtime.py packages/tracecat-ee/tracecat_ee/agent/workflows/durable.py tests/unit/test_agent_runtime_resolution.pyuv run python -m py_compile ...git diff --checkI also tried
uv run pytest tests/unit/test_agent_runtime_resolution.py -q, but this local checkout cannot start Tracecat's global Postgres-backed test fixture because no Postgres server is listening on127.0.0.1:5432. The tests were collected, then blocked during fixture setup before the test bodies ran.Summary by cubic
Adds metadata-only runtime diagnostics for agent runs so operators can see which runtime, model route, and config were resolved without exposing prompts or secrets. Surfaces as a pre-call stream event and in final results to help debug early failures. Supports #2656.
RuntimeResolutiondiagnostics model (runtime, model provider/name/route, prompt lengths, output type, tool/MCP/subagent/skill counts, approvals, resume/fork flags).runtime_resolutionstream event before the model call in the Claude runtime; includes the same metadata in result envelopes and durable workflow output.Written for commit 5aa29ad. Summary will update on new commits.