Skip to content

[Bugfix] Sanitize malformed tool call recipients in Harmony parser #31677

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
eous wants to merge 1 commit into
base: main
Choose a base branch
from
Open

[Bugfix] Sanitize malformed tool call recipients in Harmony parser #31677

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 23 additions & 0 deletions tests/entrypoints/openai/parser/test_harmony_utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -888,6 +888,29 @@ def test_commentary_with_function_recipient_creates_function_call(self):
assert output_items[0].call_id.startswith("call_")
assert output_items[0].id.startswith("fc_")

def test_malformed_recipient_with_channel_token_is_sanitized(self):
"""Test that malformed recipients containing <|channel|> are sanitized.

The model sometimes outputs malformed sequences like
'functions.bash<|channel|>commentary' instead of 'functions.bash'.
This test verifies the sanitization handles this case.
"""
message = Message.from_role_and_content(
Role.ASSISTANT, '{"command": "date"}'
)
message = message.with_channel("commentary")
# Simulate malformed recipient from model output
message = message.with_recipient("functions.bash<|channel|>commentary")

output_items = parse_output_message(message)

assert len(output_items) == 1
assert isinstance(output_items[0], ResponseFunctionToolCall)
assert output_items[0].type == "function_call"
# The function name should be sanitized to just "bash"
assert output_items[0].name == "bash"
assert output_items[0].arguments == '{"command": "date"}'

def test_commentary_with_python_recipient_creates_reasoning(self):
"""Test that commentary with recipient='python' creates reasoning items."""
message = Message.from_role_and_content(
Expand Down
5 changes: 5 additions & 0 deletions vllm/entrypoints/openai/parser/harmony_utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -535,6 +535,11 @@ def _parse_browser_tool_call(message: Message, recipient: str) -> ResponseOutput

def _parse_function_call(message: Message, recipient: str) -> list[ResponseOutputItem]:
"""Parse function calls into function tool call items."""
# Sanitize recipient: the model sometimes outputs malformed sequences
# like "to=functions.bash<|channel|>commentary" instead of the correct
# "to=functions.bash <|constrain|>json". Strip the malformed part.
if "<|channel|>" in recipient:
recipient = recipient.split("<|channel|>")[0].strip()
Comment on lines +538 to +542

@gemini-code-assist gemini-code-assist Bot Jan 4, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

This sanitization logic is duplicated in vllm/tool_parsers/openai_tool_parser.py and vllm/entrypoints/openai/serving_chat_stream_harmony.py. To improve maintainability and avoid code duplication, consider extracting this logic into a shared utility function. You could add a sanitize_recipient function in this file and then use it in all three places.

For example, you could add:

def sanitize_recipient(recipient: str) -> str:
    """Sanitizes a malformed tool call recipient by stripping `<|channel|>` and anything after it."""
    if "<|channel|>" in recipient:
        return recipient.split("<|channel|>")[0].strip()
    return recipient

And then replace this block with recipient = sanitize_recipient(recipient).

function_name = recipient.split(".")[-1]
output_items = []
for content in message.content:
Expand Down
9 changes: 8 additions & 1 deletion vllm/entrypoints/openai/serving_chat_stream_harmony.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,13 @@ def extract_harmony_streaming_delta(
and cur_recipient
and cur_recipient.startswith("functions.")
):
# Sanitize recipient: the model sometimes outputs malformed sequences
# like "functions.bash<|channel|>commentary" instead of "functions.bash".
# Strip the malformed part.
sanitized_recipient = cur_recipient
if "<|channel|>" in sanitized_recipient:
sanitized_recipient = sanitized_recipient.split("<|channel|>")[0].strip()
Comment on lines +51 to +56

@gemini-code-assist gemini-code-assist Bot Jan 4, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

This sanitization logic is duplicated in other files. To improve maintainability, it should be centralized into a single utility function. Please see my comment in vllm/entrypoints/openai/parser/harmony_utils.py for a detailed suggestion.


# Count completed tool calls to determine index
base_index = 0
for msg in harmony_parser.messages:
Expand All @@ -59,7 +66,7 @@ def extract_harmony_streaming_delta(
base_index += 1

if prev_recipient != cur_recipient:
tool_name = cur_recipient.split("functions.", 1)[1]
tool_name = sanitized_recipient.split("functions.", 1)[1]
delta_message = DeltaMessage(
tool_calls=[
DeltaToolCall(
Expand Down
8 changes: 7 additions & 1 deletion vllm/tool_parsers/openai_tool_parser.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,11 +65,17 @@ def extract_tool_calls(
tool_args = msg_text
else:
tool_args = msg_text
# Sanitize recipient: the model sometimes outputs malformed
# sequences like "functions.bash<|channel|>commentary"
# instead of "functions.bash". Strip the malformed part.
recipient = msg.recipient
if "<|channel|>" in recipient:
recipient = recipient.split("<|channel|>")[0].strip()
Comment on lines +68 to +73

@gemini-code-assist gemini-code-assist Bot Jan 4, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

This sanitization logic is duplicated in other files. To improve maintainability, it should be centralized into a single utility function. Please see my comment in vllm/entrypoints/openai/parser/harmony_utils.py for a detailed suggestion.

tool_calls.append(
ToolCall(
type="function",
function=FunctionCall(
name=msg.recipient.split("functions.")[1],
name=recipient.split("functions.")[1],
arguments=tool_args,
),
)
Expand Down
Loading