[Bug]: FastIncrementalDetokenizer leaks full prompt text into streamed output when prompt ends with an incomplete UTF-8 token

[spped2000]

Your current environment

Reproduced on vllm/vllm-openai:v0.25.1 (vLLM 0.25.1, tokenizers 0.22.2, aarch64/DGX Spark GB10) — but the bug is platform-independent and CPU-reproducible: it lives in the detokenizer layer. The code on current main (vllm/v1/engine/detokenizer.py, DecodeStream(ids=request.prompt_token_ids, ...)) has the same logic, so main is affected too. Introduced by #34217 (FastIncrementalDetokenizer prefill via tokenizers' new stream, 2026-02-11); every release since is affected.

🐛 Describe the bug

When a request's prompt_token_ids end with an incomplete UTF-8 sequence (byte-fallback / byte-level BPE tokens holding only part of a multi-byte character), FastIncrementalDetokenizer re-emits the entire prompt text inside the first streamed delta.

This is hit by token-array prompts to /v1/completions — a first-class API feature (e.g. lm-eval with tokenized_requests=True, or any client that pre-tokenizes and splits at byte boundaries). Languages whose tokens routinely carry partial UTF-8 bytes (Thai, Burmese, CJK on English-centric vocabs, emoji) make such prompt tails easy to produce.

API-level repro (server: vllm serve facebook/opt-125m); prompt ids decode to "Hello world " + 2 of 3 bytes of U+1F336:

curl -sN http://localhost:8000/v1/completions -H 'Content-Type: application/json' \
  -d '{"model":"facebook/opt-125m","prompt":[31414,232,8103,14285],"stream":true,"max_tokens":8,"temperature":0}'
# first SSE delta: {"text":"Hello world 🌈"}  <-- the entire prompt text leaked

Same with Thai: prompt ids decoding to "สวัสด" + partial bytes of ี produce a first delta of "สวัสดา" — the prompt text streamed back to the client. A control request whose prompt decodes cleanly streams correctly.

Class-level repro (CPU-only):

from transformers import AutoTokenizer
from vllm.sampling_params import SamplingParams
from vllm.v1.engine import EngineCoreRequest
from vllm.v1.engine.detokenizer import FastIncrementalDetokenizer, SlowIncrementalDetokenizer

tok = AutoTokenizer.from_pretrained("openai-community/gpt2")
ids = tok("Hello world \U0001F336", add_special_tokens=False).input_ids  # emoji = 3 byte-level tokens
prompt, gen = ids[:-1], ids[-1:]  # split INSIDE the emoji

def stream(cls):
    req = EngineCoreRequest(request_id="", prompt_token_ids=prompt, mm_features=None,
        sampling_params=SamplingParams(skip_special_tokens=True), pooling_params=None,
        arrival_time=0.0, lora_request=None, cache_salt=None, data_parallel_rank=None)
    d = cls(tok, req); out = ""
    for i, t in enumerate(gen):
        d.update([t], False); out += d.get_next_output_text(i == len(gen) - 1, delta=True)
    return out

print("fast:", repr(stream(FastIncrementalDetokenizer)))  # 'Hello world 🌶'  <-- LEAK
print("slow:", repr(stream(SlowIncrementalDetokenizer)))  # ''               <-- correct

Root cause — tokenizers' step_decode_stream (Rust) primes the prefill lazily and only when the decoded prompt does not end with U+FFFD:

if prefix.is_empty() && !ids.is_empty() {
    let new_prefix = tokenizer.decode(ids, skip_special_tokens)?;
    if !new_prefix.ends_with('�') {   // incomplete prompt tail -> prefix never set
        *prefix = new_prefix;
        *prefix_index = ids.len();
    }
}

With the prefix left empty, the first fully-decodable step returns string[0..] — the whole prompt plus generated text. No exception is raised, so the #19449 InvalidPrefix guard never triggers. Upstream considers this prefill behavior by-design (discussed in huggingface/tokenizers#1856), and the fastokens backend ports the same semantics, so a vLLM-side guard covers both.

SlowIncrementalDetokenizer handles the same input correctly (offset-based bookkeeping; prompt text is structurally unreachable).

Proposed fix (PR follows): in IncrementalDetokenizer.from_new_request, detect a prompt tail that decodes with a trailing U+FFFD (only the last few tokens need decoding) and route the request to SlowIncrementalDetokenizer — same pattern as the existing USE_FAST_DETOKENIZER version gate from #24159. Verified end-to-end: leaking cases become clean and a complete-prompt control request streams byte-identical output before/after.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.