diff --git a/content/docs/observability/features/meta.json b/content/docs/observability/features/meta.json index d1837b110b..5b9706b15b 100644 --- a/content/docs/observability/features/meta.json +++ b/content/docs/observability/features/meta.json @@ -14,6 +14,7 @@ "user-feedback", "log-levels", "agent-graphs", + "tool-calls", "masking", "mcp-tracing", "multi-modality", diff --git a/content/docs/observability/features/tool-calls.mdx b/content/docs/observability/features/tool-calls.mdx new file mode 100644 index 0000000000..98fb639ea9 --- /dev/null +++ b/content/docs/observability/features/tool-calls.mdx @@ -0,0 +1,61 @@ +--- +description: Langfuse automatically renders tool definitions and tool calls from your agent traces, making it easy to debug function calling. +sidebarTitle: Tool Calls +--- + +import { GhDiscussionsPreview } from "@/components/gh-discussions/GhDiscussionsPreview"; + +# Tool Call Visualization + +Tool calls are the heartbeat of agents. Langfuse automatically renders all tools available to an LLM at the top of each generation, allowing you to instantly see if the LLM selected the right one. + + + ![Tool call visualization](/images/changelog/2025-11-05-langfuse-for-agents/lw4-d3-tools.png) + + +## What's Displayed + +**Tool Definitions**: All tools available to the LLM during that interaction: + +- Tool name and call status +- Call count badge ("called", "called 2x", "not called") +- Expandable description and parameter schema +- Toggle between formatted and JSON views + +**Tool Calls in Chat**: Called tools appear with their arguments and call IDs. Numbering matches the available tools list for easy validation. + +**Automatic Sorting**: Called tools appear first, followed by unused tools. + +## Supported Frameworks + +Tool visualization works automatically with: + +- OpenAI (Chat Completions & Responses API) +- Google Gemini / Vertex AI +- Vercel AI SDK +- LangGraph / LangChain +- Pydantic AI +- Microsoft Agent Framework + +No changes to your instrumentation code required. If you're using tools with a supported framework, visualization appears automatically. + +## How Tools Are Detected + +Langfuse detects **available tools** from supported framework and OpenTelemetry metadata, normalizes them into the generation input, and renders them at the top of each generation. Langfuse detects **called tools** from the model output and renders them inline in the chat view with their arguments and call IDs. + +For the exact OpenTelemetry and framework metadata shapes that are remapped into `input.tools`, `tool_definitions`, `tool_calls`, and `tool_call_names`, see [Tool definitions and tool calls](/integrations/native/opentelemetry#tool-definitions-and-tool-calls) in the OpenTelemetry mapping docs. + +## Why This Matters + +1. **Faster debugging**: Instantly see if the right tools were available and called +2. **Better prompt engineering**: Understand which tool descriptions lead to actual usage +3. **Cost optimization**: Identify unused tools to reduce token usage + +## Related + +- [Agent Graphs](/docs/observability/features/agent-graphs) +- [Observation Types](/docs/observability/features/observation-types) + +## GitHub Discussions + + diff --git a/content/integrations/native/opentelemetry.mdx b/content/integrations/native/opentelemetry.mdx index 46b219c6ad..05d591e52d 100644 --- a/content/integrations/native/opentelemetry.mdx +++ b/content/integrations/native/opentelemetry.mdx @@ -442,6 +442,21 @@ These attributes are applied to individual observations (spans) within a trace ( | `version` | The [version](/docs/observability/features/releases-and-versioning) of the observation. | • `langfuse.version`: `string` | | `environment` | The deployment [environment](/docs/observability/features/environments) where the observation was generated. | • `langfuse.environment`
• `deployment.environment`
• `deployment.environment.name` | +### Tool definitions and tool calls [#tool-definitions-and-tool-calls] + +Langfuse normalizes available tool definitions and called tool invocations into dedicated observation fields. Available tools are attached to the generation input as `input.tools`; named tools are also stored in `tool_definitions` for filtering, exports, and dashboards. Called tools are parsed from the generation output and stored in `tool_calls` and `tool_call_names`. + +| Langfuse concept | Source attribute or shape | Mapping behavior | +| ------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Available tools | `gen_ai.tool.definitions` | Moves valid named tool definitions to `input.tools` and removes the duplicate metadata attribute. Each item must have a tool name, matching the OpenTelemetry GenAI tool definition schema. | +| Available tools | `ai.prompt.tools` | Moves Vercel AI SDK tool definitions to `input.tools` and removes the duplicate metadata attribute. Named tools are extracted into `tool_definitions`; provider tools without names, such as `{ type: "web_search_preview" }`, remain in `input.tools` for display. | +| Available tools | `model_request_parameters.function_tools` | Moves pydantic-ai function tools to `input.tools` and removes the nested `function_tools` key when valid. | +| Available tools | `llm.tools..tool.json_schema` | Moves valid indexed OpenInference-style tool schema attributes to `input.tools` in index order and removes the moved attributes. | +| Available tools fallback | Observation `metadata.tools` | Compatibility fallback for already-shaped tool metadata. Only arrays where every item is a named tool definition are moved; generic user metadata such as `{ id: "hammer" }` or `{ type: "manual" }` is preserved. | +| Called tools | `output.tool_calls`, `output.toolCalls`, `choices[].message.tool_calls`, message `tool_calls`, AI SDK `tool-call` parts, Anthropic `tool_use` parts | Extracts tool invocation payloads into `tool_calls` and tool names into `tool_call_names`. | + +If a tool definition source is moved into `input.tools`, Langfuse removes the duplicate source metadata to keep observations clean. If the value does not look like a supported tool definition, Langfuse leaves it untouched in metadata. + **Filtering by metadata key in Langfuse** diff --git a/pages/docs/observability/features/tool-calls.mdx b/pages/docs/observability/features/tool-calls.mdx new file mode 100644 index 0000000000..2b4a41a81a --- /dev/null +++ b/pages/docs/observability/features/tool-calls.mdx @@ -0,0 +1,61 @@ +--- +description: Langfuse automatically renders tool definitions and tool calls from your agent traces, making it easy to debug function calling. +sidebarTitle: Tool Calls +--- + +# Tool Call Visualization + +Tool calls are the heartbeat of agents. Langfuse automatically renders all tools available to an LLM at the top of each generation, allowing you to instantly see if the LLM selected the right one. + + + ![Tool call visualization](/images/changelog/2025-11-05-langfuse-for-agents/lw4-d3-tools.png) + + +## What's Displayed + +**Tool Definitions**: All tools available to the LLM during that interaction: + +- Tool name and call status +- Call count badge ("called", "called 2x", "not called") +- Expandable description and parameter schema +- Toggle between formatted and JSON views + +**Tool Calls in Chat**: Called tools appear with their arguments and call IDs. Numbering matches the available tools list for easy validation. + +**Automatic Sorting**: Called tools appear first, followed by unused tools. + +## Supported Frameworks + +Tool visualization works automatically with: + +- OpenAI (Chat Completions & Responses API) +- Google Gemini / Vertex AI +- Vercel AI SDK +- LangGraph / LangChain +- Pydantic AI +- Microsoft Agent Framework + +No changes to your instrumentation code required—if you're using tools with a supported framework, visualization appears automatically. + +## How Tools Are Detected + +Langfuse detects **available tools** from supported framework and OpenTelemetry metadata, normalizes them into the generation input, and renders them at the top of each generation. Langfuse detects **called tools** from the model output and renders them inline in the chat view with their arguments and call IDs. + +For the exact OpenTelemetry and framework metadata shapes that are remapped into `input.tools`, `tool_definitions`, `tool_calls`, and `tool_call_names`, see [Tool definitions and tool calls](/integrations/native/opentelemetry#tool-definitions-and-tool-calls) in the OpenTelemetry mapping docs. + +## Why This Matters + +1. **Faster debugging**: Instantly see if the right tools were available and called +2. **Better prompt engineering**: Understand which tool descriptions lead to actual usage +3. **Cost optimization**: Identify unused tools to reduce token usage + +## Related + +- [Agent Graphs](/docs/observability/features/agent-graphs) +- [Observation Types](/docs/observability/features/observation-types) + +## GitHub Discussions + +import { GhDiscussionsPreview } from "@/components/gh-discussions/GhDiscussionsPreview"; + +