docs: clarify reff is computed from inverse of unsmoothed importance ratios#88
docs: clarify reff is computed from inverse of unsmoothed importance ratios#88sethaxen merged 2 commits intoarviz-devs:mainfrom
reff is computed from inverse of unsmoothed importance ratios#88Conversation
…e ratios Fix arviz-devs#76: `reff` was documented as the relative efficiency of the unsmoothed importance ratios, but it is the relative efficiency of their *inverse* (matching Stan's `loo::psis` definition `r_eff = ess(1/r) / S`). Update the `PSISResult`, `psis`/`psis!`, and `ess_is` docstrings accordingly. Also fix the analogue of arviz-devs/PosteriorStats.jl#54 in the `psis` jldoctest: the example computed `reff` directly from `log_ratios`, which gives a different ESS than the correct computation on the inverse ratios. Switch to `ess(softmax(-log_ratios; dims=(1, 2)); ...)` — equivalent to `ess(1 ./ exp.(log_ratios); ...)` but numerically stable. Add `LogExpFunctions` to the docs environment so `softmax` is available, and regenerate the expected output table. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #88 +/- ##
=======================================
Coverage 97.44% 97.44%
=======================================
Files 6 6
Lines 274 274
=======================================
Hits 267 267
Misses 7 7 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
sethaxen
left a comment
There was a problem hiding this comment.
Thanks for the fix! Indeed, this is what the loo::psis docstring says.
For completeness, this is due to (Section 3.2.8 of the PSIS paper):

FYI in the next breaking release we will switch to using the relative tail-ESS to determine relative efficiency. This is preferable both because the Pareto diagnostic is entirely determined by the tail behavior and because the tail-ESS is invariant to monotone transformations (so whether we use weights, log-weights, or inverse-(log-)weights, we get the same value). posterior now does this by default.
Downgrade docs failure seems unrelated to this PR.
Summary
reffwas documented as the relative efficiency of the unsmoothed importance ratios, but it is the relative efficiency of their inverse (matching Stan'sloo::psisdefinitionr_eff = ess(1/r) / S). Updated thePSISResult,psis/psis!, andess_isdocstrings accordingly.loodocstring PosteriorStats.jl#54 in thepsisjldoctest: the example computedreffdirectly fromlog_ratios, which gives a different ESS than the correct computation on the inverse ratios. Switched toess(softmax(-log_ratios; dims=(1, 2)); ...)— equivalent toess(1 ./ exp.(log_ratios); ...)but numerically stable. AddedLogExpFunctionsto the docs environment sosoftmaxis available, and regenerated the expected output table.Test plan
Documenter.doctest(PSIS; doctestfilters=[r"└.*"])passesdocs/make.jlbuild succeedsPkg.test()passes (511/511)