Skip to content

Add an Urdu (ur) PII language pack with RTL/bidi-safe spans and extended Arabic-Indic digit handling #1520

Description

@maziyarpanahi

Summary

Urdu has ~51M speakers in India (2011 census; far more worldwide) and is an official language in several Indian states, but OpenMed has no ur pack. Urdu is right-to-left in Perso-Arabic script — the Arabic block plus Urdu-specific letters ٹ ڈ ڑ ں ے ھ and the extended Arabic-Indic digits ۰-۹ (U+06F0-06F9, distinct from Arabic's ٠-٩) — so Aadhaar, phone, and PIN digit runs are LTR islands inside RTL text and naive spans corrupt bidi ordering; OpenMed already ships an Arabic (ar) pack whose RTL and digit-normalization learnings must be reused rather than reinvented.

Scope

  • Register 'ur' in SUPPORTED_LANGUAGES, LANGUAGE_NAMES, LANGUAGE_MODEL_PREFIX, and LANGUAGE_MONTH_NAMES (جنوری..دسمبر) in openmed/core/pii_i18n.py
  • Add _URDU_PII_PATTERNS to LANGUAGE_PII_PATTERNS following the existing _ARABIC_PII_PATTERNS approach: dates with Urdu month names, +91 mobiles, PIN codes with پن کوڈ context (Urdu-belt states: J&K 18xxxx-19xxxx, Telangana 50xxxx, UP 2xxxxx), Aadhaar 4-4-4 reusing validate_aadhaar, and name patterns with honorifics جناب / محترمہ / سید / بیگم / خان and the صاحب/صاحبہ suffix as context
  • Normalize both extended Arabic-Indic (۰-۹) and Arabic-Indic (٠-٩) digits to ASCII before numeric validators, and route ur-vs-ar using Urdu-specific letters (ے ٹ ڈ ڑ ں ھ) so Urdu text never falls into the ar pack
  • Guarantee bidi-safe redaction: spans over LTR digit islands inside RTL lines must not split bidi control characters or reorder logical text, reusing and extending the ar pack's RTL span tests
  • Add DEFAULT_PII_MODELS['ur'] reusing the multilingual privacy-filter model and register the pack via the plugin framework's one-call onboarding; wire LANG_TO_LOCALE['ur'] as a documented approximation (no Urdu Faker locale in Faker 40.x — verified) with an en_IN backend plus a small bundled Urdu name provider (given names + خان/سید/بیگم elements) in openmed/core/anonymizer/locales.py; NATIONAL_ID_PROVIDERS['ur'] = aadhaar
  • Author synthetic golden fixture openmed/eval/golden/fixtures/i18n/ur.jsonl (RTL notes with embedded LTR Aadhaar/phone runs, both digit systems, honorific-suffixed names, Urdu-English code-mixed lines) and extend the leakage gate and unit tests

Acceptance criteria

  • Residual leakage = 0 on the synthetic ur golden fixtures through the release-gate leakage check, including lines using extended Arabic-Indic digits
  • Offsets are grapheme-safe and bidi-safe: no combining mark, ligature, or bidi control is split, and redacted RTL lines round-trip with logical order preserved
  • Urdu fixtures route to the ur pack, never the ar pack, in routing tests keyed on Urdu-specific letters
  • Aadhaar, phone, and PIN values render detection-identical across ASCII, Arabic-Indic, and extended Arabic-Indic digits
  • tests/unit/core/test_locale_coherence.py passes with ur listed as a documented approximate locale emitting exactly one UserWarning

Out of scope

  • Pakistani document formats (CNIC) — Indian documents only in this pack
  • Nastaliq font rendering or PDF layout concerns (text-level only)
  • Training an Urdu-specific NER model
  • Devanagari-written Hindustani (that is the hi pack); real clinical text — synthetic fixtures only

Files

  • openmed/core/pii_i18n.py
  • openmed/core/anonymizer/locales.py
  • openmed/eval/golden/fixtures/i18n/ur.jsonl
  • openmed/eval/release_gates.py
  • tests/unit/core/test_locale_coherence.py
  • tests/unit/eval/test_golden_fixtures.py

Task: OM-694 · Milestone: v1.9 · Priority: P1 · Size: M
Depends on: OM-678 · Blocks: —
Roadmap: Chinese & Indian languages initiative
Spec: PLANS/V2/EXECUTION/tasks/OM-694.md

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1HighfeatureNew capabilityroadmap-v2OpenMed V2 roadmap backlog

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions