Emnlp demo publication#376
Draft
vkehfdl1 wants to merge 9 commits into
Draft
Conversation
… original)
Three planning artifacts for the NeurIPS 2026 Evaluations & Datasets Track
submission, branch experiment/paper-publication.
ACTIVE PLAN: docs/plans/neurips-2026-experimentation-plan-FULL.md
This is the primary working document with detailed calculations.
- neurips-2026-experimentation-plan.md
Original draft. Broad scope (20+ pipelines x 9 datasets x all metrics).
- neurips-2026-experimentation-plan-REVISED.md
Critique-driven downscoping for May 2026 deadline (4 retrieval x 4 gen x
3 datasets, single-claim focus).
- neurips-2026-experimentation-plan-FULL.md (PRIMARY, updated)
Full-coverage plan targeting NeurIPS/ICLR/ACL 2027. Includes:
* Verified pipeline/dataset/metric inventory (10 retr + 10 gen + 21 metrics)
* Statistical Power Analysis with MDES per dataset
* Local-vLLM dual-backend design (Qwen 3.6 + Gemma 4 on 2x RTX PRO 6000
Blackwell), eliminating API generation cost
* GPU-time-centric resource model (~41 wall-clock days, ~$3.6K judge API)
* Cross-LLM-backend ranking-stability analysis as new primary finding
* Appendix H with traceable calculation methodology and 12-item assumption
register including Phase 0 critical sensitivity checks
The paper direction now centers on the framework properties that make RAG reimplementation cheaper: unified dataset ingestors, shareable embedded corpus dumps, preimplemented pipelines, and reusable metrics. The NeurIPS starter kit, manuscript, compiled PDF, browser preview, framework-paper pattern notes, and experiment-plan review artifacts are committed together so the paper package is reviewable as a single artifact. Constraint: User requested a NeurIPS-style paper artifact rather than only an experiment plan Constraint: Experimental numeric values remain review markers until real runs populate them Rejected: Keep dataset subset policy as the main dataset table | it hid the broader ingestor/framework coverage claim Rejected: Omit expensive pipelines from full-scale experiments | user explicitly required all implemented compatible pipelines Confidence: high Scope-risk: narrow Directive: Preserve the paper's central four-part reuse claim: dataset abstraction, stored embeddings, preimplemented pipelines, preimplemented metrics Tested: pdflatex + bibtex + pdflatex + pdflatex + pdflatex on paper/neurips2026/main.tex Not-tested: Full AutoRAG benchmark execution and real result replacement
…ions track Retarget the publication from NeurIPS 2026 to the EMNLP 2026 demo track (submission July 10, 2026; 6 content pages + 2 appendix; single-blind). - Rewrite the manuscript as a system demonstration paper using the official ACL style files: system design, novelty vs. existing RAG toolkits (FlashRAG, BERGEN, AutoRAG, Pyserini, RAGAs, DSPy, LangChain/LlamaIndex), target audience, licensing, and CLI workflow. - Include the full demo-track experiment design: Track A/B/C benchmark setup with fixed-model controls, restored-artifact reproducibility measurements, extensibility case study, and key takeaways. Result values are gray \res placeholders pending the scripted runs. - Update coverage to the current codebase: 13 ingestors, 13 retrieval + 17 generation pipelines, 16 reranker backends, retrieval/generation metric suites, Gradio leaderboard. - Document demo-track requirements and remaining TODOs (screencast video, live-demo link, final numbers) in the README. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ntions Analyzed six recent *ACL demo-track RAG system papers (RAGLAB, RAGViz, OpenResearcher, RAGAs, LocalRQA, FlexRAG) and applied their consistent structural and stylistic patterns: - Add repo URL to the abstract so the code link is on page 1. - Add Figure 1: full-width five-stage workflow banner (ingest/restore -> unified DB -> run -> evaluate -> report/share) at the top of page 2, replacing the layered-architecture table; self-contained caption. - Add a roadmap paragraph with section cross-references to the intro. - Add a Demonstration Walkthrough section with a running example (HyDE vs. BM25 vs. dense retrieval on SciFact), restore command, experiment YAML listing, leaderboard drill-down narrative, and a three-step extension recipe. - Extend the comparison table and related-systems prose with the demo-track lineage: RAGLAB, LocalRQA, FlexRAG (with citations), acknowledging FlexRAG's multimodal scope honestly. - Add a user study design (reproduce a row + add a metric, completion/ time/satisfaction placeholders), following RAGLAB's evaluation style. Body still ends within the 6-page content limit; appendix within 2 pages. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
No time to run it before the July 10 deadline; evaluation keeps the benchmark study, restored-artifact measurements, and extensibility case study. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.