test(tokenizer): BPE UTF-8 regression fixtures for v0.27.0 fix

Adds four HF-verified tokenization assertions to scripts/test_tokenizer.sh:

  café    → [924, 58858]
  naïve   → [3376, 37572, 586]
  日本語   → [101059, 102819]
  привет  → [124436, 26991, 8178]

All four exercise the GPT-2 direct-byte codepoint range (U+00A1-U+00FF)
that v0.27.0 fixed. Before the fix they fell through to byte-fallback
with wrong token IDs; the regression test nails the correct IDs down so
any future refactor of encode_byte_to_bpe_char fails loudly.

8/8 PASS locally.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Author: unamedkr

scripts/test_tokenizer.sh

@@ -75,6 +75,22 @@ check_tokens_single "Qwen3.5-4B-Q4_K_M.gguf"     "Hello" \
 check_tokens_single "Qwen3.6-35B-A3B-UD-IQ4_XS.gguf" "Hello" \
   "TQ_NO_METAL=1 TQ_NO_MLOCK=1"
 
+# BPE UTF-8 direct-byte encode regression — v0.27.0.
+# Before the fix, int'l text was encoded with raw byte bytes ≥ 0x80 that
+# standalone form invalid UTF-8 and failed vocab lookup, so chars dropped
+# or fell back to wrong low-id tokens. Expected IDs verified against HF
+# AutoTokenizer for Qwen/Qwen3-0.6B on 2026-04-21.
+echo ""
+echo "--- BPE UTF-8 direct-byte regression (v0.27.0) ---"
+check_tokens "Qwen3-0.6B-Q4_K_M.gguf"            "café"    "924 58858" \
+  "TQ_NO_METAL=1 TQ_NO_MLOCK=1"
+check_tokens "Qwen3-0.6B-Q4_K_M.gguf"            "naïve"   "3376 37572 586" \
+  "TQ_NO_METAL=1 TQ_NO_MLOCK=1"
+check_tokens "Qwen3-0.6B-Q4_K_M.gguf"            "日本語"  "101059 102819" \
+  "TQ_NO_METAL=1 TQ_NO_MLOCK=1"
+check_tokens "Qwen3-0.6B-Q4_K_M.gguf"            "привет"  "124436 26991 8178" \
+  "TQ_NO_METAL=1 TQ_NO_MLOCK=1"
+
 echo ""
 echo "--- Summary ---  PASS=$PASS  FAIL=$FAIL  SKIP=$SKIP"
 [[ "$FAIL" -eq 0 ]] || exit 1