Bug: _extract_anthropic_tokens raises AttributeError on MessageDeltaUsage, causing incorrect output token counts in streaming #754

Summary

When using Anthropic's streaming API (messages.create(stream=True)), the instrumentation silently fails to record the final output token count from message_delta chunks. This means observed JUDGMENT_USAGE_OUTPUT_TOKENS values for streaming calls are wrong — they reflect only the early estimate from message_start, not the authoritative count from the final message_delta.

Root Cause

_extract_anthropic_tokens in src/judgeval/instrumentation/llm/llm_anthropic/messages.py uses direct attribute access:

input_tokens = usage.input_tokens if usage.input_tokens is not None else 0
cache_read = usage.cache_read_input_tokens if ... else 0
cache_creation = usage.cache_creation_input_tokens if ... else 0

However, the Anthropic SDK's MessageDeltaUsage (returned by message_delta stream events) only defines output_tokens. Accessing .input_tokens, .cache_read_input_tokens, or .cache_creation_input_tokens on it raises AttributeError.

This error is caught by the dont_throw decorator wrapping yield_hook in immutable_wrap_sync_iterator, which logs "[Caught] An exception was raised in yield_hook" and returns None — silently skipping the entire message_delta token update.

Impact

• All streaming Anthropic calls (stream=True) record incorrect output token counts in spans.
• The JUDGMENT_USAGE_OUTPUT_TOKENS span attribute is set from message_start (early estimate) and never updated from message_delta (the authoritative final count).
• The bug is invisible to end users since dont_throw suppresses the exception — they must enable debug logging to see it.

Reproduction

from anthropic.types import MessageDeltaUsage
from judgeval.instrumentation.llm.llm_anthropic.messages import _extract_anthropic_tokens

delta_usage = MessageDeltaUsage(output_tokens=42)
_extract_anthropic_tokens(delta_usage)  # AttributeError: 'MessageDeltaUsage' object has no attribute 'input_tokens'

Fix

Replace direct attribute access with getattr(..., None) or 0:

def _extract_anthropic_tokens(usage: Usage | MessageDeltaUsage) -> Tuple[int, int, int, int]:
    input_tokens = getattr(usage, "input_tokens", None) or 0
    output_tokens = getattr(usage, "output_tokens", None) or 0
    cache_read = getattr(usage, "cache_read_input_tokens", None) or 0
    cache_creation = getattr(usage, "cache_creation_input_tokens", None) or 0
    return (input_tokens, output_tokens, cache_read, cache_creation)

This handles both Usage (all fields present) and MessageDeltaUsage (only output_tokens) without raising.

Related

• Added in PR Add llm streaming (JUD-1720) #525 (Add llm streaming, Aug 2025)
• MessageDeltaUsage is documented at https://docs.anthropic.com/en/api/messages-streaming (message_delta event)