Skip to content

docs: clarify reff is computed from inverse of unsmoothed importance ratios#88

Merged
sethaxen merged 2 commits intoarviz-devs:mainfrom
devmotion:dmw/docs
Apr 21, 2026
Merged

docs: clarify reff is computed from inverse of unsmoothed importance ratios#88
sethaxen merged 2 commits intoarviz-devs:mainfrom
devmotion:dmw/docs

Conversation

@devmotion
Copy link
Copy Markdown
Contributor

Summary

  • Fix reff incorrectly documented #76: reff was documented as the relative efficiency of the unsmoothed importance ratios, but it is the relative efficiency of their inverse (matching Stan's loo::psis definition r_eff = ess(1/r) / S). Updated the PSISResult, psis/psis!, and ess_is docstrings accordingly.
  • Fix an analogue of Calculation of relative efficiency in loo docstring PosteriorStats.jl#54 in the psis jldoctest: the example computed reff directly from log_ratios, which gives a different ESS than the correct computation on the inverse ratios. Switched to ess(softmax(-log_ratios; dims=(1, 2)); ...) — equivalent to ess(1 ./ exp.(log_ratios); ...) but numerically stable. Added LogExpFunctions to the docs environment so softmax is available, and regenerated the expected output table.
  • Bumped version to 0.9.9.

Test plan

  • Documenter.doctest(PSIS; doctestfilters=[r"└.*"]) passes
  • Full docs/make.jl build succeeds
  • Pkg.test() passes (511/511)

devmotion and others added 2 commits April 21, 2026 11:06
…e ratios

Fix arviz-devs#76: `reff` was documented as the relative
efficiency of the unsmoothed importance ratios, but it is the relative
efficiency of their *inverse* (matching Stan's `loo::psis` definition
`r_eff = ess(1/r) / S`). Update the `PSISResult`, `psis`/`psis!`, and
`ess_is` docstrings accordingly.

Also fix the analogue of arviz-devs/PosteriorStats.jl#54 in the `psis`
jldoctest: the example computed `reff` directly from `log_ratios`, which
gives a different ESS than the correct computation on the inverse
ratios. Switch to `ess(softmax(-log_ratios; dims=(1, 2)); ...)` —
equivalent to `ess(1 ./ exp.(log_ratios); ...)` but numerically stable.
Add `LogExpFunctions` to the docs environment so `softmax` is available,
and regenerate the expected output table.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@codecov
Copy link
Copy Markdown

codecov Bot commented Apr 21, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.44%. Comparing base (ca17ad6) to head (158ae3f).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main      #88   +/-   ##
=======================================
  Coverage   97.44%   97.44%           
=======================================
  Files           6        6           
  Lines         274      274           
=======================================
  Hits          267      267           
  Misses          7        7           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@devmotion devmotion marked this pull request as ready for review April 21, 2026 09:37
Copy link
Copy Markdown
Member

@sethaxen sethaxen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fix! Indeed, this is what the loo::psis docstring says.

For completeness, this is due to (Section 3.2.8 of the PSIS paper):
image

FYI in the next breaking release we will switch to using the relative tail-ESS to determine relative efficiency. This is preferable both because the Pareto diagnostic is entirely determined by the tail behavior and because the tail-ESS is invariant to monotone transformations (so whether we use weights, log-weights, or inverse-(log-)weights, we get the same value). posterior now does this by default.

Downgrade docs failure seems unrelated to this PR.

@sethaxen sethaxen merged commit f9bc0e6 into arviz-devs:main Apr 21, 2026
12 of 13 checks passed
@devmotion devmotion deleted the dmw/docs branch April 21, 2026 10:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

reff incorrectly documented

2 participants