Sensitivity / Bayesian assurance

[drbenvincent]

Hi @juanitorduz @cetagostini @NathanielF @ErikRingen. On our list of TODO's was to kick off a discussion about the general topic of sensitivity analysis / Bayesian assurance.

Can we start by dropping in any thoughts, ideas, references to papers?

Ideally this will lead to crystallised ideas that people can turn into issues. Once we've got some issues going then I'll create a project board, but there seems little point right now.

Relevant existing issues, which we may want to build upon or replace:

• Feature Request: Bayesian Assurance Framework (aka Bayesian power analysis) #569
• SC: check sensitivity to prior #46

[ErikRingen]

One idea that sprung to mind during the meeting was whether the various forms of sensitivity/assurance tests could be aggregated so that we get one meta-overview of how the estimate might change. Inspired by 'specification curves':

Simonsohn, U., Simmons, J. P., & Nelson, L. D. (2020). Specification curve analysis. Nature human behaviour, 4(11), 1208-1214.

https://aeturrell.github.io/specification_curve/

[NathanielF]

Just noting that we have at least one sensitivity parameter implemented in the IV model. Because the IV model is formulated in a Bayesian way we have to use a joint distribution of treatment and outcome. In the binary case we have: rho here:

CausalPy/causalpy/pymc_models.py

Line 786 in a67408a

rho = pm.Deterministic("rho", pm.math.tanh(rho_unconstr))

In the continuous case it is just the correlation coefficient in the latent covariance:

CausalPy/causalpy/pymc_models.py

Line 825 in a67408a

pm.Deterministic(name="cov", var=pt.dot(l=chol, r=chol.T))

In both cases you can use az.pair_plot(idata, var_names=['rho', 'beta_z'])) to plot the relationship between the sensitivity parameter . This correlation is a proxy for the degree of unmeasured confounding.

Other references i was thinking of is :

This is very nice, but additionally we could look at applied missing data analysis.

This emphasises that the imputation view of causal inference is sensitive to the imputation scheme and generally ties causal inference and missing data view together

[NathanielF]

Also, @cetagostini previously explained to me a experimentation planning routine to estimate the required time to run it based on size of effect.