Feature request: A high level diagnostics function

[seabbs]

The problem

There are lots of different diagnostics, no helper thresholds (I am aware of) and some things are hard to find (i.e. divergent transitions)

A solution

A high-level diagnostics summary function or similar like those in cmdstan (https://systemdynamics101.com/html/cmdstanr) and the higher level wrappers). To work across different algs this would need to be a bit cleverer than what is offered in the stan ecosystem.

Other solutions

Perhaps something like this exists already? Maybe this package is too low level for this kind of feature and it should go elsewhere.

[devmotion]

Maybe PosteriorStats.jl provides what you're looking for? https://julia.arviz.org/PosteriorStats/stable/api/

[seabbs]

I really like that functionality and use it a lot but as far as I am aware it's by parameter and so a little more low-level than what I am getting at here. The stan example I linkied is for diagnostics summarised across the fit, so you can get a high level idea of what is going on/wrong before diving in i.e per divergent transitions, any Rhats above some threshold that kind of thing or a less threshold version would be some summary table. Stan auto omits many of these which I think is a bit far I think (leads to its reputation for being dodgy I think).

I would imagine you could have a simple function on top of the functionality in PosteriorStats.jl to do a fair bit of this with the addition of some alg specific things that I think only live in the model object.

[sethaxen]

There are lots of different diagnostics, no helper thresholds (I am aware of) and some things are hard to find (i.e. divergent transitions)

What do you mean by "helper thresholds"?

as I am aware it's by parameter and so a little more low-level than what I am getting at here.

What do you mean by "by parameter" here?

The stan example I linkied is for diagnostics summarised across the fit

There are a lot of code blocks in the link you provided; which example in particular would you like to emulate?

[seabbs]

Sampler diagnostic warnings and summaries

This section. This is also responsible via a custom print method for the warnings/diagnostics that cmdstan emits I believe.

What do you mean by "helper thresholds"?

Stan (and other PPLs) has optionated thresholds at which it starts giving information i.e ESS is low, Rt is high, etc. These can be useful for new users. However, I can see many reasons for not wanting to do this so in the first instance would really be interested in the high-level diagnostic summary.

What do you mean by "by parameter" here?

As far as I am aware there is a row per parameter in the model summarised across posterior samples.

[sethaxen]

Sampler diagnostic warnings and summaries

This section. This is also responsible via a custom print method for the warnings/diagnostics that cmdstan emits I believe.

Got it, okay. You're referring to diagnostic_summary(), which returns for each chain the num_divergent, num_max_treedepth, and ebfmi. Computing these requires

• num_divergent: either a boolean array divergent for all chains/draws or both the Hamiltonian energy and the energy threshold used by HMC/NUTS to denote a divergent transition
• num_max_treedepth: either a boolean array reached_max_treedepth or both an int array treedepth and the treedepth_threshold used internally by HMC/NUTS.
• ebfmi: same as bfmi, requires energy

I don't think it makes sense for us to provide utility functions for computing the first 2, since these are either a very simple sum over the draws dimension for some representations of outputs or require the user obtains internal parameters of the sampler. That is, these utilities won't really save any user time/effort if we include them generically here; instead, it would make more sense for a package that performs the sampling to implement these utilities.

In the case of ebfmi, we already provide this as bfmi.

Stan (and other PPLs) has optionated thresholds at which it starts giving information i.e ESS is low, Rt is high, etc. These can be useful for new users. However, I can see many reasons for not wanting to do this so in the first instance would really be interested in the high-level diagnostic summary.

I can see a use for this, but it's probably more appropriate in PosteriorStats via summarize. Even so, currently we focus on utilities for computing recommended stats/diagnostics and leave it to the user to interpret them (with recommended values in the docstrings), and I tend to think this is a better way to go for the average user. Of course, any package can build on summarize to raise a warning if some threshold is exceeded.

[seabbs]

Yes though you can easily imagine a version of this that runs multiple of the diagnostics in this package at once and presents them in a nice format.

either a very simple

I found identifying where these actually are and in what format to take reading quite a lot of documentation that is often sparsely signposted so its less about the challenge and more about reducing user friction which currently, on an ecosystem level, feels high to me.

package that performs the sampling to implement these utilities

I don't have a particular view on where in the ecosystem these things should go as I have a poor understanding of how things connect.

we already provide this as bfmi.

As above only as a standalone that users need to hunt out? For the kind of users I would be trying to onboard this isn't something they will realistically do unprompted. For me, this is the core of the use case (reducing friction for users so they check there model) not about enabling a power user who is already wanting to use the low-level fn calls.

in PosteriorStats via summarize.

This seems reasonable though in my experience getting users to read docs is very difficult.

This sounds like this issue isn't well targetted for the package so feel very free to close or move.

[sethaxen]

Python ArviZ recently added a version of this that's based on cmdstan's diagnose function: https://python.arviz.org/projects/stats/en/latest/api/generated/arviz_stats.diagnose.html. It has HMC/NUTS-specific diagnostics (divergences, maximum tree depth, low E-BFMI), and sample-based ones (ESS, R-hat).

We could provide a function diagnose that takes the arguments needed to compute all of these diagnostics (namely, the draws, as well as a boolean array indicating whether a draw is divergent, an integer array with tree depth, and an array with the log-density), but for this to be really useful for users, we'd want to document that packages implementing containers for samples should overload the method for their container type so that it can automatically extract these from the container. This would at least allow the same method to be called on MCMCChains.Chains, FlexiChains.VNChain, and InferenceObjects.InferenceData, if those packages overloaded the method in an extension.