LLMAgent per-turn request configuration: ToolChoice, ResponseFormat, and metadata

[weeco]

Problem Statement

LLMAgent.executeSingleTurn() builds the llm.Request with only Messages and Tools populated. The underlying llm.Request type supports several other important fields that are inaccessible from the agent level:

• ToolChoice: Controls whether tools are used (auto, none, required, specific). Useful for forcing tool use on the first turn or disabling it on the final turn.
• ResponseFormat: Controls output structure (text, json_object, json_schema). Essential for agents that must produce structured output.
• Options: Provider-specific parameters (temperature, top_p, max_tokens). Currently only configurable at the model level, not per-request.
• Metadata: Request-level metadata for tracing and observability.

The only workaround is implementing a ModelInterceptor that mutates info.Req before it reaches the model — significant boilerplate for what should be a configuration option.

This makes LLMAgent too rigid for many real agent patterns:

• "Force the agent to use a tool on the first turn, then let it decide" — not possible
• "Always respond in JSON matching this schema" — requires a custom interceptor
• "Use temperature 0 for tool calls, 0.7 for final responses" — requires a custom interceptor
• "Attach trace IDs to every request" — requires a custom interceptor

Proposed Solution

Agent-level request configuration

Add options to LLMAgent for fields that should apply to every request:

agent, _ := llmagent.New("data-extractor", prompt, model,
    llmagent.WithResponseFormat(&llm.ResponseFormat{
        Type: "json_schema",
        JSONSchema: &llm.JSONSchema{
            Name:   "extraction_result",
            Schema: extractionSchema,
        },
    }),
    llmagent.WithToolChoice(&llm.ToolChoice{Type: "auto"}),
    llmagent.WithRequestMetadata(map[string]string{"service": "extraction"}),
)

Per-turn request hooks

For dynamic per-turn configuration, add a request builder hook:

// RequestHook modifies the LLM request before each model call.
// Receives the request and current invocation metadata.
type RequestHook func(ctx context.Context, req *llm.Request, inv *InvocationMetadata)

agent, _ := llmagent.New("assistant", prompt, model,
    llmagent.WithRequestHook(func(ctx context.Context, req *llm.Request, inv *agent.InvocationMetadata) {
        if inv.Turn() == 0 {
            // Force tool use on first turn
            req.ToolChoice = &llm.ToolChoice{Type: "required"}
        }
        // Attach trace context
        req.Metadata["trace_id"] = getTraceID(ctx)
    }),
)

Implementation

In executeSingleTurn(), after building the base request, apply agent-level configuration:

req := &llm.Request{
    Messages: reqMessages,
}
if a.config.tools != nil {
    req.Tools = a.config.tools.List()
}
// Apply agent-level config
if a.config.responseFormat != nil {
    req.ResponseFormat = a.config.responseFormat
}
if a.config.toolChoice != nil {
    req.ToolChoice = a.config.toolChoice
}
if a.config.requestMetadata != nil {
    req.Metadata = a.config.requestMetadata
}
// Apply per-turn hook
if a.config.requestHook != nil {
    a.config.requestHook(ctx, req, inv)
}

Use Case Example

Structured data extraction agent:

agent, _ := llmagent.New("extractor",
    "Extract structured data from the user's input.",
    model,
    llmagent.WithResponseFormat(&llm.ResponseFormat{
        Type: "json_schema",
        JSONSchema: &llm.JSONSchema{
            Name:   "contact_info",
            Schema: json.RawMessage(`{"type":"object","properties":{"name":{"type":"string"},"email":{"type":"string"}}}`),
        },
    }),
    llmagent.WithToolChoice(&llm.ToolChoice{Type: "none"}), // No tools needed
)

Dynamic per-turn configuration:

agent, _ := llmagent.New("research-agent", prompt, model,
    llmagent.WithTools(toolRegistry),
    llmagent.WithRequestHook(func(ctx context.Context, req *llm.Request, inv *agent.InvocationMetadata) {
        if inv.Turn() == 0 {
            req.ToolChoice = &llm.ToolChoice{Type: "required"} // Must use tools first
        } else if inv.Turn() >= 3 {
            req.ToolChoice = &llm.ToolChoice{Type: "none"} // Synthesize, stop calling tools
        }
    }),
)

Why This Matters

• Structured output is essential: Many agents exist specifically to extract or generate structured data. Without ResponseFormat at the agent level, every structured-output agent needs a custom interceptor.
• Tool control enables patterns: "Always use a tool first" (research agents), "Never use tools" (pure generation), "Use this specific tool" (guided workflows) are all common patterns that require ToolChoice.
• Reduces interceptor boilerplate: The request hook is a much lighter-weight alternative to ModelInterceptor for the common case of "I just need to set a field on the request."
• Per-turn flexibility: Agent behavior often needs to change across turns (explore → synthesize, broad → specific). Per-turn hooks enable this without complex state management in interceptors.