Docs: Add Synthetic Control sensitivity checks walkthrough#871
Open
drbenvincent wants to merge 2 commits intomainfrom
Open
Docs: Add Synthetic Control sensitivity checks walkthrough#871drbenvincent wants to merge 2 commits intomainfrom
drbenvincent wants to merge 2 commits intomainfrom
Conversation
Adds a sensitivity-analysis section to sc_pymc.ipynb that walks through the pipeline API (EstimateEffect -> SensitivityAnalysis -> GenerateReport) with PlaceboInTime as the SC default check, and documents the other SC-applicable checks (ConvexHullCheck, LeaveOneOut, PlaceboInSpace, PriorSensitivity) with interpretation guidance. Mirrors the structure introduced for Staggered DiD in #846 and links back to the central guide from #818. Includes a runnable pipeline cell and HTML report iframe so the walkthrough demonstrates an end-to-end placebo-in-time check with concrete pass/fail narrative and a five-step "if this check fails" troubleshooting block. Closes #789 Made-with: Cursor
|
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #871 +/- ##
==========================================
- Coverage 94.60% 94.59% -0.02%
==========================================
Files 80 80
Lines 12764 12764
Branches 770 770
==========================================
- Hits 12076 12074 -2
- Misses 485 486 +1
- Partials 203 204 +1 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Manually rerun so the Synthetic Control sensitivity walkthrough renders correctly in the built docs. Made-with: Cursor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a focused sensitivity-check walkthrough for Synthetic Control to the
sc_pymc.ipynbnotebook, addressing #789 (follow-up to #749 / #785).This continues the docs work started in #818 (central guide) and #846 (Staggered DiD walkthrough), keeping the same structure:
EstimateEffect→SensitivityAnalysis→GenerateReport) and linking back to the central guide andpipeline_workflow.PlaceboInTime,ConvexHullCheck,LeaveOneOut,PlaceboInSpace,PriorSensitivity) with one-line interpretations and noting which checks belong to other estimators.PlaceboInTimedeep-dive with how-it-works narrative grounded in citations (abadie2010synthetic,abadie2021using,reichardt2019quasi).PlaceboInTime(n_folds=2, random_seed=seed)on the existing SC dataset together withGenerateReport(include_plots=True).staggered_did_pymc.ipynb.Notes
NOT SUPPORTEDplacebo-in-time result for the demo dataset (P=0.905, just below the 0.95 default threshold). This ties directly into the "If this check fails" troubleshooting section so readers see the failure path and the next steps.Closes #789
Test plan
prek run --files docs/source/notebooks/sc_pymc.ipynbpassespython scripts/validate_notebooks.py docs/source/notebooks/sc_pymc.ipynbpassesjupyter nbconvert --to notebook --execute --inplace ...so all outputs are populatedMade with Cursor