Skip to content

Add Pinyin romanization with tone handling for Chinese surrogate generation and fuzzy name matching #1470

Description

@maziyarpanahi

Summary

When de-identifying Chinese names OpenMed must generate realistic surrogate names and match name variants, but Chinese has no alphabet, so consistent pseudonymization and matching require Pinyin romanization; the same surname (张) maps to Zhang and a polyphonic character (重 -> chong/zhong) has multiple readings. This is hard because correct Pinyin needs word-level context for polyphones/heteronyms, multiple tone-representation styles (tone marks, TONE3 numbers, toneless NORMAL), and a stable, reproducible mapping so the same Han name always yields the same surrogate.

Scope

  • Add openmed/processing/zh_pinyin.py wrapping pypinyin (MIT) with a to_pinyin function supporting NORMAL, tone-mark, and TONE3 numeric styles, and heteronym output for polyphonic characters, with a clear optional-dependency fallback.
  • Use the Chinese segmenter word tokens (from the segmentation engine) to disambiguate polyphones at the word level rather than per character.
  • Add a deterministic Chinese surrogate-name provider under openmed/core/anonymizer/providers/ that, given a detected Han name span, derives a stable Pinyin key and emits a consistent synthetic replacement Han name keyed through the existing surrogate vault so repeats are consistent within a document.
  • Wire the provider into the anonymizer registry (openmed/core/anonymizer/registry.py) and locale handling (openmed/core/anonymizer/locales.py) for zh, drawing only from a synthetic seeded surname/given-name list.
  • Provide a Pinyin-based fuzzy match key so two surface forms of the same name normalize to the same comparison key for consistency checks.
  • Add tests covering polyphone disambiguation, tone-style outputs, and deterministic surrogate stability.

Acceptance criteria

  • to_pinyin returns the expected reading for a synthetic name (王芳 -> wang fang in NORMAL, wáng fāng with tone marks, wang2 fang1 in TONE3).
  • Polyphone disambiguation: 重庆 romanizes its first character as chong (not zhong) when segmented as a word, on the synthetic probe set.
  • Determinism gate: the same Han input name produces the same surrogate replacement across repeated calls and the same document, verified by the surrogate vault, and offsets of the replaced span remain exact code-point spans.
  • With pypinyin absent, the romanization path degrades gracefully with a clear optional-dependency message and the surrogate provider still returns a deterministic synthetic Han name.
  • Two synthetic surface variants of one name normalize to the same Pinyin fuzzy-match key.

Out of scope

  • Speech/audio or IPA conversion.
  • Cross-document re-identification linkage.
  • Non-name entity surrogates.

Files

  • openmed/processing/zh_pinyin.py
  • openmed/core/anonymizer/providers/clinical_ids.py
  • openmed/core/anonymizer/registry.py
  • openmed/core/anonymizer/locales.py
  • openmed/core/surrogate_vault.py

Task: OM-644 · Milestone: v1.9 · Priority: P2 · Size: L
Depends on: OM-639 · Blocks: —
Roadmap: Chinese & Indian languages initiative
Spec: PLANS/V2/EXECUTION/tasks/OM-644.md

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2MediumfeatureNew capabilityroadmap-v2OpenMed V2 roadmap backlog

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions