Skip to content

Docs: Add Synthetic Control sensitivity checks walkthrough#871

Open
drbenvincent wants to merge 2 commits intomainfrom
docs/789-sc-sensitivity-walkthrough
Open

Docs: Add Synthetic Control sensitivity checks walkthrough#871
drbenvincent wants to merge 2 commits intomainfrom
docs/789-sc-sensitivity-walkthrough

Conversation

@drbenvincent
Copy link
Copy Markdown
Collaborator

Summary

Adds a focused sensitivity-check walkthrough for Synthetic Control to the sc_pymc.ipynb notebook, addressing #789 (follow-up to #749 / #785).

This continues the docs work started in #818 (central guide) and #846 (Staggered DiD walkthrough), keeping the same structure:

  • Sensitivity analysis section introducing the pipeline API (EstimateEffectSensitivityAnalysisGenerateReport) and linking back to the central guide and pipeline_workflow.
  • Related checks for synthetic control subsection enumerating the SC-applicable checks (PlaceboInTime, ConvexHullCheck, LeaveOneOut, PlaceboInSpace, PriorSensitivity) with one-line interpretations and noting which checks belong to other estimators.
  • PlaceboInTime deep-dive with how-it-works narrative grounded in citations (abadie2010synthetic, abadie2021using, reichardt2019quasi).
  • Walkthrough with numbered steps and pass/fail interpretation guidance.
  • "If this check fails" subsection with five concrete diagnostic actions: inspect fold summaries, prune the donor pool, add complementary SC checks, sanity-check the convex-hull condition, probe prior sensitivity.
  • Pipeline code cell running PlaceboInTime(n_folds=2, random_seed=seed) on the existing SC dataset together with GenerateReport(include_plots=True).
  • HTML report iframe display cell, matching the pattern used in staggered_did_pymc.ipynb.

Notes

  • The walkthrough deliberately demonstrates a NOT SUPPORTED placebo-in-time result for the demo dataset (P=0.905, just below the 0.95 default threshold). This ties directly into the "If this check fails" troubleshooting section so readers see the failure path and the next steps.
  • Existing cells were left untouched; the new cells were appended after the effect-summary section. The previous trailing empty placeholder cell was removed as part of inserting the new content.

Closes #789

Test plan

  • prek run --files docs/source/notebooks/sc_pymc.ipynb passes
  • python scripts/validate_notebooks.py docs/source/notebooks/sc_pymc.ipynb passes
  • Notebook re-executed end-to-end via jupyter nbconvert --to notebook --execute --inplace ... so all outputs are populated
  • CI notebooks job executes the SC notebook without errors
  • ReadTheDocs preview renders the new sections correctly

Made with Cursor

Adds a sensitivity-analysis section to sc_pymc.ipynb that walks through
the pipeline API (EstimateEffect -> SensitivityAnalysis -> GenerateReport)
with PlaceboInTime as the SC default check, and documents the other
SC-applicable checks (ConvexHullCheck, LeaveOneOut, PlaceboInSpace,
PriorSensitivity) with interpretation guidance. Mirrors the structure
introduced for Staggered DiD in #846 and links back to the central guide
from #818.

Includes a runnable pipeline cell and HTML report iframe so the walkthrough
demonstrates an end-to-end placebo-in-time check with concrete pass/fail
narrative and a five-step "if this check fails" troubleshooting block.

Closes #789

Made-with: Cursor
@review-notebook-app
Copy link
Copy Markdown

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.


Powered by ReviewNB

@drbenvincent drbenvincent added documentation Improvements or additions to documentation OSS_PRODUCT OSS_PRODUCT project priorities. Labs members should get approval before logging hours. labels Apr 28, 2026
@codecov
Copy link
Copy Markdown

codecov Bot commented Apr 28, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.59%. Comparing base (deb8774) to head (741434e).

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #871      +/-   ##
==========================================
- Coverage   94.60%   94.59%   -0.02%     
==========================================
  Files          80       80              
  Lines       12764    12764              
  Branches      770      770              
==========================================
- Hits        12076    12074       -2     
- Misses        485      486       +1     
- Partials      203      204       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Manually rerun so the Synthetic Control sensitivity walkthrough renders
correctly in the built docs.

Made-with: Cursor
@read-the-docs-community
Copy link
Copy Markdown

Documentation build overview

📚 causalpy | 🛠️ Build #32452428 | 📁 Comparing 741434e against latest (deb8774)

  🔍 Preview build  

2 files changed
± 404.html
± notebooks/sc_pymc.html

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation OSS_PRODUCT OSS_PRODUCT project priorities. Labs members should get approval before logging hours.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Docs: Add Synthetic Control sensitivity checks walkthrough (follow-up #749)

1 participant