Skip to content

fix(recall): widen context tag candidate fetch#215

Merged
jack-arturo merged 1 commit into
mainfrom
fix/201-context-tags-overfetch
Jul 5, 2026
Merged

fix(recall): widen context tag candidate fetch#215
jack-arturo merged 1 commit into
mainfrom
fix/201-context-tags-overfetch

Conversation

@jack-arturo

@jack-arturo jack-arturo commented Jul 4, 2026

Copy link
Copy Markdown
Member

Summary

Closes #201.

Public/API Notes

  • /recall response shape is unchanged.
  • context_tags remains a soft boost; this only widens the pre-scoring vector candidate pool when a semantic query or embedding is present.
  • Lab test-set rows may now include context_tags.

Validation

  • RED confirmed before implementation:
    • tests/test_recall_overfetch.py::test_context_tags_widen_vector_fetch_before_scoring failed with requested vector limit 20 instead of 50
    • tests/lab/test_lab_corpus.py::test_recall_serializes_params failed with unexpected context_tags
    • tests/lab/test_run_recall_test.py::test_run_single_query_forwards_row_context_tags failed because row-level tags were ignored
  • PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 .venv/bin/pytest tests/test_recall_overfetch.py tests/test_context_tag_separator.py -q -> 17 passed
  • PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 .venv/bin/pytest tests/lab/test_lab_corpus.py tests/lab/test_run_recall_test.py -q -> 11 passed
  • PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 .venv/bin/pytest tests/test_api_endpoints.py -q -k recall -> 69 passed, 124 deselected
  • make test -> 631 passed, 1 skipped, 65 deselected
  • make lint -> passed
  • git diff --check -> passed
  • python -m json.tool lab/test_sets/queries_context_tags_20260704.json -> passed
  • make bench-eval BENCH=locomo-mini -> 84.26% overall accuracy (198/235), 367.5s, 69 cat-5 questions skipped by judge-off run; matches latest comparable local LoCoMo-mini artifact
  • make bench-eval BENCH=locomo -> 81.91% overall accuracy (1263/1542), 2072.6s, 444 cat-5 questions skipped by judge-off run; slightly above latest comparable local LoCoMo-full artifact (81.84%, 1262/1542)

Live Lab A/B

Ran two isolated local stacks from the same restored production snapshot, prod-api-20260611-post-repair-001241:

  • baseline: origin/main (7636021) at http://localhost:8011
  • candidate: this PR (a29e9fb) at http://localhost:8012
  • both reported memory_count=9900, vector_count=9900, sync_status=synced

Context-tag probes (lab/test_sets/queries_context_tags_20260704.json, limit=5, current_only=false):

  • baseline: Recall@5/10/20 0.000, MRR 0.000, NDCG@10 0.000, 5 misses
  • candidate: Recall@5/10/20 1.000, MRR 1.000, NDCG@10 1.000, 0 misses

Plain production set (lab/test_sets/queries_prod_20260611.json, limit=20, current_only=true):

  • baseline: Recall@10 0.045, MRR 0.024, NDCG@10 0.029, latency avg 765ms
  • candidate: Recall@10 0.275, MRR 0.220, NDCG@10 0.231, latency avg 972ms
  • observed no regression; latency increased by about 208ms avg on this local restored stack

The context-tag probe is the direct #201 validation signal. The plain-query production lift is observational and likely reflects existing derived context_profile.priority_tags now widening the candidate pool as intended. Ignored result files are under lab/results/issue201/, including issue201_live_ab_summary.json.

@jack-arturo jack-arturo merged commit 01e1082 into main Jul 5, 2026
10 checks passed
@jack-arturo jack-arturo deleted the fix/201-context-tags-overfetch branch July 5, 2026 06:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

recall: context_tags boost dropped at small limit (candidate fetch only widens for hard tag_filters)

1 participant