Skip to content

Heal LLM-simplified citations to canonical reference - #3069

Open
John CSA (jluocsa) wants to merge 3 commits into
Azure-Samples:mainfrom
jluocsa:fix/heal-simplified-citations
Open

Heal LLM-simplified citations to canonical reference#3069
John CSA (jluocsa) wants to merge 3 commits into
Azure-Samples:mainfrom
jluocsa:fix/heal-simplified-citations

Conversation

@jluocsa

Copy link
Copy Markdown
Contributor

Purpose

Fixes #2950.

When a document filename contains leading special characters (e.g. a leading
hyphen, as produced for some user-uploaded files: - PyCon US 2025.pdf), the LLM
tends to "simplify" the citation it emits, dropping the leading characters. For
example, given the source - PyCon US 2025.pdf#page=1, the model writes
[PyCon US 2025.pdf#page=1] in its answer.

The frontend citation parser validated the cited text with endsWith, which
correctly recognized it as a valid citation, but then used the simplified text
as the citation reference. The resulting /content/<path> request used
PyCon US 2025.pdf instead of the indexed - PyCon US 2025.pdf, so
check_path_auth() filtered by the wrong sourcefile value and the document
returned 403/404.

This change resolves the cited text back to the canonical citation from
data_points.citations (exact match preferred, otherwise the endsWith match)
and uses that canonical value for the reference and for the
citation_activity_details lookup. Links and lookups now use the indexed value,
so the content route resolves correctly.

Does this introduce a breaking change?

[ ] Yes
[x] No

Citations that already match exactly continue to resolve exactly (the exact match
is preferred). Only previously-broken simplified citations change behavior.

Does this require changes to learn.microsoft.com docs?

[ ] Yes
[x] No

Type of change

[x] Bugfix
[ ] Feature
[ ] Code style update (formatting, local variables)
[ ] Refactoring (no functional changes, no api changes)
[ ] Documentation content changes
[ ] Other... Please describe:

Code quality checklist

  • tsc type-checks cleanly (npx tsc --noEmit).
  • prettier applied to the changed file.
  • The current tests all pass (python -m pytest).

…ples#2950)

Adds backend post-processing of the LLM answer so simplified citation tokens (e.g. [PyCon US 2025.pdf#page=1]) are mapped back to the canonical indexed reference via exact match, else a unique endswith match. Complements the frontend fix so corrected citations are present in the raw response for streaming, API consumers, and exports - matching the maintainer's suggested Solution Azure-Samples#1 (post-process the LLM response).
@jluocsa
John CSA (jluocsa) force-pushed the fix/heal-simplified-citations branch from 3c3acda to 006841c Compare May 31, 2026 01:54
@jluocsa

Copy link
Copy Markdown
Contributor Author

Updated this PR to implement the maintainer's suggested Solution #1 (post-process the LLM response) server-side, in addition to the original frontend safety net.

What changed

  • app/backend/approaches/approach.py — added heal_citation / heal_citations helpers that map a simplified citation token (e.g. [PyCon US 2025.pdf#page=1]) back to the canonical indexed reference via exact match, else a unique endswith match. Ambiguous or no match leaves the token unchanged.
  • app/backend/approaches/chatreadretrieveread.py — applies healing in both the non-streaming and streaming paths (including citations split across token deltas).
  • app/frontend/src/components/Answer/AnswerParser.tsx — original frontend fix retained as a display-layer safety net.
  • tests/test_chatapproach.py — 10 new tests covering exact/unique/ambiguous/no-match cases and a citation split across streaming deltas.

This means corrected citations are now present in the raw response itself (streaming, API consumers, exports), not just the rendered UI. Rebased onto current main; all healing tests pass and ty is clean.

…-citations

# Conflicts:
#	tests/test_chatapproach.py
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

LLM simplifies citations for filenames with special characters, breaking document links

1 participant