feat: automatically OCR pages whose embedded text is unreadable, instead of returning garbled text

[tobocop2]

What is the proposed feature?

Route pages whose text layer is undecodable to OCR, the same way we already route scanned pages.

Today pdf/scan_detect.rs decides scanned-vs-native from raster coverage, invisible-text ratio, glyph count, codec and producer prior. That catches the case where there is no usable text layer. It does not catch the case where there is a text layer, it is visible, it is glyph-rich — and it decodes to garbage.

A PDF hits this when its font gives no path from glyph to character. Concretely, all of:

1. Type0 with /Encoding /Identity-H, CIDSystemInfo ordering Identity — the content-stream codes are CIDs;
2. CIDToGIDMap /Identity — CID == GID, so the codes are raw glyph indices;
3. no /ToUnicode CMap;
4. the embedded CIDFontType2 subset has no cmap table and no post table.

The subset legitimately has no cmap — a CIDFontType2 is addressed by glyph index directly, so rendering never consults one, and subsetters strip both it and post to save bytes. What is left in the file is glyph outlines and glyph indices. Every route to Unicode is severed: no /ToUnicode, no cmap to invert, no post glyph names, and Identity ordering means there is no predefined CMap either.

The page renders perfectly in a viewer. The text simply cannot be extracted — by anyone. On a real failing page, every extractor produces the same mojibake:

extractor	output
pdf_oxide 0.3.74	81,7('67$7(6',675,&7&2857
poppler (pdftotext)	81,7('67$7(6',675,&7&2857
pypdf	81,7('\x0367$7(6\x03',675,&7\x03&2857
pymupdf	81,7('\x0367$7(6\x03',675,&7\x03&2857

Ground truth, as rendered: UNITED STATES DISTRICT COURT.

This is not a pdf_oxide bug — no conformant reader can do better. But the signal that the layer is undecodable is missing, and without it we cannot act.

Why scan_detect misses it. Score these pages against PageScanSignals and they look maximally native:

signal	value on an undecodable page
image_coverage	~0 — there is no raster at all
invisible_text_ratio	~0 — the text is drawn visibly
glyph_count	high — the page is full of glyphs
codec	none
producer_prior	typically not a scanner

So the page scores as confidently born-digital, never goes to OCR, and its mojibake flows straight into extraction output and the index. The heuristic isn't wrong — it has no input that could tell it the glyphs decode to nothing.

Proposed shape. Add "text layer present but undecodable" as a distinct condition alongside scanned, and treat it as an OCR trigger. The check is static and cheap — font subtype, encoding, ordering, /ToUnicode presence, and whether the embedded program carries cmap/post — so it costs nothing per page. Once a page is flagged, the existing OCR path handles it: the glyphs are perfectly legible, so OCR recovers the text cleanly.

Blocked on pdf_oxide#876. We consume pdf_oxide for PDF extraction, and it is the layer that parses the font programs — it already has the font dict and the embedded program in hand, and it currently falls back to CID-as-Unicode and emits the mojibake without telling the caller. I have filed a feature request there to surface the condition (either U+FFFD for the affected runs, or an undecodable_text_layer diagnostic on the result). We should consume whichever signal it lands and wire it into scan_detect.

If pdf_oxide declines, we can detect it ourselves — the font structure is inspectable from our side too — but it would mean re-parsing font programs that pdf_oxide has already parsed, so upstream is the right home for it.

Why would this be a good addition?

Because the failure is silent, and silent extraction failures are the expensive kind.

I hit this running a document-extraction pipeline over a large corpus (~517k born-digital pages). Roughly 1,300 pages — about 0.25% of native pages — land in exactly this condition. Every one of them was indexed as gibberish, and nothing in the output distinguished them from a clean extraction. They are only findable by going looking for them.

Three things make it worth fixing rather than tolerating:

• It is recoverable. These pages are not lost causes. The glyphs render fine, so OCR reads them perfectly. We already ship the OCR backends (xberg-paddle-ocr, xberg-candle-ocr). All that is missing is the trigger. Today we throw away recoverable text.
• It poisons downstream consumers silently. Garbage that looks like text is worse than no text: it defeats search, corrupts embeddings and chunking, and shows up as confident nonsense in any RAG answer built on it. A page that yields nothing at least announces itself.
• It is concentrated, not uniform. 0.25% sounds negligible until you notice it clusters by producer — one subsetting toolchain can render an entire document set unusable, which is exactly what happened in my corpus.

The fix is cheap on our side once the signal exists: one more OCR trigger next to the scanned-page one, reusing machinery we already have.

---

Reproducer

Self-contained — no corpus document needed. Builds a one-page PDF with a Type0/Identity-H font whose embedded subset keeps glyf/loca but has neither cmap nor post, so the page renders as UNITED STATES DISTRICT COURT but carries no recoverable text. Needs pip install fonttools and any DejaVuSans.ttf.

python3 build_undecodable.py <path/to/DejaVuSans.ttf> [outdir]

The generated undecodable.pdf is attached to this issue.

build_undecodable.py

#!/usr/bin/env python3
"""Type0 / Identity-H, no /ToUnicode, CIDToGIDMap /Identity, embedded
CIDFontType2 subset stripped of BOTH `cmap` and `post`. The content stream
carries raw glyph indices; nothing maps a glyph back to a character, so the
page renders correctly but its text is unreadable by any conformant extractor."""
import pathlib, sys
from fontTools.ttLib import TTFont, newTable
from fontTools.pens.ttGlyphPen import TTGlyphPen

DEJAVU = sys.argv[1] if len(sys.argv) > 1 else "DejaVuSans.ttf"
OUT = pathlib.Path(sys.argv[2] if len(sys.argv) > 2 else ".")
PHRASE = "UNITED STATES DISTRICT COURT"

src = TTFont(DEJAVU); cm = src.getBestCmap(); gs = src.getGlyphSet()
# Contiguous ASCII glyph order => GID = ord(c) - 29, the constant offset seen in the wild
used = [chr(x) for x in range(0x20, 0x7F)]
order = [".notdef", "pad1", "pad2"] + [f"u{ord(c):04X}" for c in used]
gid = {c: order.index(f"u{ord(c):04X}") for c in used}

glyf = newTable("glyf"); glyf.glyphOrder = order; glyf.glyphs = {}
hmtx = newTable("hmtx"); hmtx.metrics = {}
for name in order:
    pen = TTGlyphPen(gs)
    if name.startswith("u"):
        ch = chr(int(name[1:], 16))
        if ord(ch) != 0x20 and ord(ch) in cm:
            gs[cm[ord(ch)]].draw(pen)
    glyf.glyphs[name] = pen.glyph(); hmtx.metrics[name] = (600, 0)

for tag in [t for t in src.keys() if t in {"cmap", "post"}]:
    del src[tag]                      # <- the whole point: no cmap, no post
src.setGlyphOrder(order)
src["glyf"] = glyf; src["hmtx"] = hmtx
src["maxp"].numGlyphs = len(order); src["hhea"].numberOfHMetrics = len(order)
ttf = OUT / "undecodable_font.ttf"; src.save(ttf)
fd = ttf.read_bytes()

cids = b"".join(gid[c].to_bytes(2, "big") for c in PHRASE)   # Identity-H: 2-byte CIDs == GIDs
content = b"BT /F1 24 Tf 72 700 Td <" + cids.hex().encode() + b"> Tj ET"
W = " ".join(f"{gid[c]} [600]" for c in sorted(set(PHRASE), key=ord))
objs = [
 b"<< /Type /Catalog /Pages 2 0 R >>",
 b"<< /Type /Pages /Kids [3 0 R] /Count 1 >>",
 b"<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] /Resources << /Font << /F1 4 0 R >> >> /Contents 5 0 R >>",
 b"<< /Type /Font /Subtype /Type0 /BaseFont /AAAAAA+Sub /Encoding /Identity-H /DescendantFonts [8 0 R] >>",  # NO /ToUnicode
 b"<< /Length %d >>\nstream\n%s\nendstream" % (len(content), content),
 b"<< /Type /FontDescriptor /FontName /AAAAAA+Sub /Flags 4 /FontBBox [0 0 1000 1000] /ItalicAngle 0 "
 b"/Ascent 800 /Descent -200 /CapHeight 700 /StemV 80 /FontFile2 7 0 R >>",
 b"<< /Length %d /Length1 %d >>\nstream\n" % (len(fd), len(fd)) + fd + b"\nendstream",
 (f"<< /Type /Font /Subtype /CIDFontType2 /BaseFont /AAAAAA+Sub "
  f"/CIDSystemInfo << /Registry (Adobe) /Ordering (Identity) /Supplement 0 >> "
  f"/FontDescriptor 6 0 R /CIDToGIDMap /Identity /DW 600 /W [{W}] >>").encode(),
]
out = bytearray(b"%PDF-1.5\n%\xe2\xe3\xcf\xd3\n"); offs = []
for i, b in enumerate(objs, 1):
    offs.append(len(out)); out += b"%d 0 obj\n" % i + b + b"\nendobj\n"
x = len(out); out += b"xref\n0 %d\n0000000000 65535 f \n" % (len(objs) + 1)
for o in offs: out += b"%010d 00000 n \n" % o
out += b"trailer\n<< /Size %d /Root 1 0 R >>\nstartxref\n%d\n%%%%EOF\n" % (len(objs) + 1, x)
(OUT / "undecodable.pdf").write_bytes(bytes(out))
print("ground truth (what a viewer renders):", repr(PHRASE))

undecodable.pdf

Expected behaviour today: scan_detect classifies the page as born-digital (no raster, visible text, high glyph count), so it never reaches OCR, and extraction returns 81,7('67$7(6',675,&7&2857 instead of UNITED STATES DISTRICT COURT.

Wanted: the page is recognised as having an unreadable text layer and is sent to OCR, which reads the rendered glyphs correctly.

[tobocop2]

this should hopefully be closed by yfedoseev/pdf_oxide#893