Skip to content

Commit 4bdbe5f

Browse files
perf(iconv): UTF-8 -> DBCS encode fast path — 2.6-3.7x, now beats glibc
UTF-8 -> DBCS legacy (Shift-JIS / Big5 / GBK / EUC-* / GB2312 / GB18030 / Johab) encode still ran the generic UTF-8-source path: decode one char, then encode_one + encode_char (~100-arm dispatch) per char, plus outer-loop re-entry that re-checks every preceding fast-path gate. Even with the O(1) direct-table encoders (6750bb6), this measured 2.5-3.7x SLOWER than glibc (Shift-JIS 2.51x, Big5 3.65x). The decode side (DBCS -> UTF-8) already had a tight loop; the encode side did not. Add a tight UTF-8 -> DBCS loop: decode each char inline (utf8_decode_step) then call the per-codec encoder directly, skipping encode_char + encode_one and the outer-loop re-entry. Byte-for-byte isomorphic: same decode + the same encode_* the dispatch would pick; a decode error, unrepresentable char, or short room breaks to the generic body for the exact EILSEQ/EINVAL/E2BIG ordering. UTF-8->DBCS encode (ns/B, self / vs glibc): SHIFT_JIS 3.73 -> 1.46 = 2.56x (2.51x -> 0.95x, now FASTER) EUC-JP 5.41 -> 1.46 = 3.71x (0.60x -> 0.15x) BIG5 3.77 -> 1.42 = 2.65x (3.65x -> 0.88x, now FASTER) Isomorphism: iconv_differential_fuzz (7 tests; iconv_cjk_differential_fuzz_vs_glibc covers UTF-8<->all 9 DBCS byte + errno + inbytesleft), 0 divergences vs host glibc. Completes the iconv DBCS encode direction (decode side was done). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1 parent 6fcb2f7 commit 4bdbe5f

1 file changed

Lines changed: 61 additions & 0 deletions

File tree

  • crates/frankenlibc-core/src/iconv

crates/frankenlibc-core/src/iconv/mod.rs

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9971,6 +9971,67 @@ pub fn iconv(
99719971
// the generic body, which reproduces the exact EILSEQ/EINVAL/E2BIG
99729972
// ordering. Placed after the ASCII/sb_translation fast paths so their
99739973
// SIMD bulk-copy still wins for ASCII-transparent pairs.
9974+
// Fast path: UTF-8 -> DBCS legacy codec. Decode each char inline, then
9975+
// call the (direct-table O(1)) per-codec encoder directly in a tight loop,
9976+
// skipping the per-char encode_char (~100-arm match) + encode_one wrapper
9977+
// AND the outer-loop re-entry (re-checking every preceding fast-path gate)
9978+
// the generic UTF-8-source path below pays. Byte-for-byte isomorphic: same
9979+
// utf8_decode_step + char::from_u32 decode and the same encode_* the
9980+
// encode_char dispatch would select; a decode error, an unrepresentable
9981+
// char, or insufficient room leaves in_pos/out_pos untouched and breaks to
9982+
// the generic body for the exact EILSEQ/EINVAL/E2BIG ordering.
9983+
if from_enc == Encoding::Utf8
9984+
&& !cd.emit_bom
9985+
&& matches!(
9986+
cd.to,
9987+
Encoding::Gb18030
9988+
| Encoding::ShiftJis
9989+
| Encoding::Big5
9990+
| Encoding::Gbk
9991+
| Encoding::EucJp
9992+
| Encoding::EucKr
9993+
| Encoding::Cp949
9994+
| Encoding::Gb2312
9995+
| Encoding::Johab
9996+
)
9997+
{
9998+
while in_pos < input.len() {
9999+
let b0 = input[in_pos];
10000+
let (wc, len) = if b0 < 0x80 {
10001+
(u32::from(b0), 1usize)
10002+
} else {
10003+
match crate::string::wchar::utf8_decode_step(&input[in_pos..]) {
10004+
crate::string::wchar::Utf8Step::Char { wc, len } => (wc, len),
10005+
_ => break,
10006+
}
10007+
};
10008+
let Some(ch) = char::from_u32(wc) else {
10009+
break;
10010+
};
10011+
let out = &mut outbuf[out_pos..];
10012+
let r = match cd.to {
10013+
Encoding::Gb18030 => encode_gb18030(ch, out),
10014+
Encoding::ShiftJis => encode_shiftjis(ch, out),
10015+
Encoding::Big5 => encode_big5(ch, out),
10016+
Encoding::Gbk => encode_gbk(ch, out),
10017+
Encoding::EucJp => encode_eucjp(ch, out),
10018+
Encoding::EucKr => encode_euckr(ch, out),
10019+
Encoding::Cp949 => encode_cp949(ch, out),
10020+
Encoding::Gb2312 => encode_gb2312(ch, out),
10021+
Encoding::Johab => encode_johab(ch, out),
10022+
_ => unreachable!(),
10023+
};
10024+
let Ok(w) = r else {
10025+
break;
10026+
};
10027+
in_pos += len;
10028+
out_pos += w;
10029+
}
10030+
if in_pos >= input.len() {
10031+
break;
10032+
}
10033+
}
10034+
997410035
if from_enc == Encoding::Utf8 && !cd.emit_bom {
997510036
let b0 = input[in_pos];
997610037
let decoded = if b0 < 0x80 {

0 commit comments

Comments
 (0)