Add CalibrationErrorMetric and CalibrationError handler (#8707)

## Description

Addresses #8505

### Overview
This PR adds calibration error metrics and an Ignite handler to MONAI,
enabling users to evaluate and monitor model calibration for
segmentation and other multi-class probabilistic tasks with shape `(B,
C, spatial...)`.

### What's Included

#### 1. Calibration Metrics (`monai/metrics/calibration.py`)
- **`calibration_binning()`**: Core function to compute calibration bins
with mean predictions, mean ground truths, and bin counts. Exported to
support research workflows where users need per-bin statistics for
plotting reliability diagrams.
- **`CalibrationReduction`**: Enum supporting three reduction methods:
- `EXPECTED` - Expected Calibration Error (ECE): weighted average by bin
count
- `AVERAGE` - Average Calibration Error (ACE): simple average across
bins
- `MAXIMUM` - Maximum Calibration Error (MCE): maximum error across bins
- **`CalibrationErrorMetric`**: A `CumulativeIterationMetric` subclass
supporting:
  - Configurable number of bins
  - Background channel exclusion (`include_background`)
- All standard MONAI metric reductions (`mean`, `sum`, `mean_batch`,
etc.)
  - Batched, per-class computation

#### 2. Ignite Handler (`monai/handlers/calibration.py`)
- **`CalibrationError`**: An `IgniteMetricHandler` wrapper that:
  - Attaches to PyTorch Ignite engines for training/validation loops
- Supports `save_details` for per-sample/per-channel metric details via
the metric buffer
  - Integrates with MONAI's existing handler ecosystem

#### 3. Comprehensive Tests
- **`tests/metrics/test_calibration_metric.py`**: Tests covering:
  - Binning function correctness with NaN handling
  - ECE/ACE/MCE reduction modes
  - Background exclusion
  - Cumulative iteration behavior
  - Input validation (shape mismatch, ndim, num_bins)
- **`tests/handlers/test_handler_calibration_error.py`**: Tests
covering:
  - Handler attachment and computation via `engine.run()`
  - All calibration reduction modes
  - `save_details` functionality
- Optional Ignite dependency handling (tests skip if Ignite not
installed)

### Public API

Exposes the following via `monai.metrics`:
- `CalibrationErrorMetric`
- `CalibrationReduction`
- `calibration_binning`

Exposes via `monai.handlers`:
- `CalibrationError`

### Implementation Notes

- Uses `scatter_add` + counts instead of `scatter_reduce("mean")` for
better PyTorch version compatibility
- Includes input validation with clear error messages
- Clamps bin indices to prevent out-of-range errors with slightly
out-of-bound probabilities
- Uses `torch.nan_to_num` instead of in-place operations for cleaner
code
- Ignite is treated as an optional dependency in tests (skipped if not
installed)

### Related Work
The algorithmic approach follows the calibration metrics from
[Average-Calibration-Losses](https://github.com/cai4cai/Average-Calibration-Losses/),
with related publications:
- [MICCAI 2024
Paper](https://papers.miccai.org/miccai-2024/091-Paper3075.html)
- [arXiv Paper](https://arxiv.org/abs/2506.03942v1)

### Future Work
As discussed in the issue, calibration losses will be added in a
separate PR to keep changes focused and easier to review.

## Checklist
- [x] Code follows MONAI style guidelines (ruff passes)
- [x] All new code has appropriate license headers
- [x] Public API is exported in `__init__.py` files
- [x] Docstrings include examples with proper transforms usage
- [x] Unit tests cover main functionality
- [x] Tests handle optional Ignite dependency gracefully
- [x] No breaking changes to existing API

## Example Usage

```python
from monai.metrics import CalibrationErrorMetric
from monai.transforms import Activations, AsDiscrete

# Setup transforms
softmax = Activations(softmax=True)
to_onehot = AsDiscrete(to_onehot=num_classes)

# Create metric
metric = CalibrationErrorMetric(
    num_bins=15,
    include_background=False,
    calibration_reduction="expected"  # ECE
)

# In evaluation loop
# Note: y_pred should be probabilities in [0,1], y should be one-hot/binarized
for batch_data in dataloader:
    logits, labels = model(batch_data)
    preds = softmax(logits)
    labels_onehot = to_onehot(labels)
    metric(y_pred=preds, y=labels_onehot)

ece = metric.aggregate()
```

### With Ignite Handler
```python
from monai.handlers import CalibrationError, from_engine

calibration_handler = CalibrationError(
    num_bins=15,
    include_background=False,
    calibration_reduction="expected",
    output_transform=from_engine(["pred", "label"]),
    save_details=True,
)
calibration_handler.attach(evaluator, name="calibration_error")
```

---------

Signed-off-by: Theo Barfoot <theo.barfoot@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Eric Kerfoot <17726042+ericspod@users.noreply.github.com>

Author: 3 people

docs/source/handlers.rst

@@ -83,6 +83,12 @@ Panoptic Quality metrics handler
     :members:
 
 
+Calibration Error metrics handler
+---------------------------------
+.. autoclass:: CalibrationError
+    :members:
+
+
 Mean squared error metrics handler
 ----------------------------------
 .. autoclass:: MeanSquaredError

docs/source/metrics.rst

@@ -185,6 +185,15 @@ Metrics
 .. autoclass:: MetricsReloadedCategorical
     :members:
 
+`Calibration Error`
+-------------------
+.. autofunction:: calibration_binning
+
+.. autoclass:: CalibrationReduction
+    :members:
+
+.. autoclass:: CalibrationErrorMetric
+    :members:
 
 
 Utilities

monai/handlers/__init__.py

@@ -12,6 +12,7 @@
 from __future__ import annotations
 
 from .average_precision import AveragePrecision
+from .calibration import CalibrationError
 from .checkpoint_loader import CheckpointLoader
 from .checkpoint_saver import CheckpointSaver
 from .classification_saver import ClassificationSaver

monai/handlers/calibration.py

@@ -0,0 +1,109 @@
+# Copyright (c) MONAI Consortium
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#     http://www.apache.org/licenses/LICENSE-2.0
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from __future__ import annotations
+
+from collections.abc import Callable
+
+from monai.handlers.ignite_metric import IgniteMetricHandler
+from monai.metrics import CalibrationErrorMetric, CalibrationReduction
+from monai.utils import MetricReduction
+
+__all__ = ["CalibrationError"]
+
+
+class CalibrationError(IgniteMetricHandler):
+    """
+    Ignite handler to compute Calibration Error during training or evaluation.
+
+    **Why Calibration Matters:**
+
+    A well-calibrated model produces probability estimates that match the true likelihood of correctness.
+    For example, predictions with 80% confidence should be correct approximately 80% of the time.
+    Modern neural networks often exhibit poor calibration (typically overconfident), which can be
+    problematic in medical imaging where probability estimates may inform clinical decisions.
+
+    This handler wraps :py:class:`~monai.metrics.CalibrationErrorMetric` for use with PyTorch Ignite
+    engines, automatically computing and aggregating calibration errors across iterations.
+
+    **Supported Calibration Metrics:**
+
+    - **Expected Calibration Error (ECE)**: Weighted average of per-bin errors (most common).
+    - **Average Calibration Error (ACE)**: Unweighted average across bins.
+    - **Maximum Calibration Error (MCE)**: Worst-case calibration error.
+
+    Args:
+        num_bins: Number of equally-spaced bins for calibration computation. Defaults to 20.
+        include_background: Whether to include the first channel (index 0) in computation.
+            Set to ``False`` to exclude background in segmentation tasks. Defaults to ``True``.
+        calibration_reduction: Calibration error reduction mode. Options: ``"expected"`` (ECE),
+            ``"average"`` (ACE), ``"maximum"`` (MCE). Defaults to ``"expected"``.
+        metric_reduction: Reduction across batch/channel after computing per-sample errors.
+            Options: ``"none"``, ``"mean"``, ``"sum"``, ``"mean_batch"``, ``"sum_batch"``,
+            ``"mean_channel"``, ``"sum_channel"``. Defaults to ``"mean"``.
+        output_transform: Callable to extract ``(y_pred, y)`` from ``engine.state.output``.
+            See `Ignite concepts <https://pytorch.org/ignite/concepts.html#state>`_ and
+            the batch output transform tutorial in the MONAI tutorials repository.
+        save_details: If ``True``, saves per-sample/per-channel metric values to
+            ``engine.state.metric_details[name]``. Defaults to ``True``.
+
+    References:
+        - Guo, C., et al. "On Calibration of Modern Neural Networks." ICML 2017.
+          https://proceedings.mlr.press/v70/guo17a.html
+        - Barfoot, T., et al. "Average Calibration Losses for Reliable Uncertainty in
+          Medical Image Segmentation." arXiv:2506.03942v3, 2025.
+          https://arxiv.org/abs/2506.03942v3
+
+    See Also:
+        - :py:class:`~monai.metrics.CalibrationErrorMetric`: The underlying metric class.
+        - :py:func:`~monai.metrics.calibration_binning`: Low-level binning for reliability diagrams.
+
+    Example:
+        >>> from monai.handlers import CalibrationError, from_engine
+        >>> from ignite.engine import Engine
+        >>>
+        >>> def evaluation_step(engine, batch):
+        ...     # Returns dict with "pred" (probabilities) and "label" (one-hot)
+        ...     return {"pred": model(batch["image"]), "label": batch["label"]}
+        >>>
+        >>> evaluator = Engine(evaluation_step)
+        >>>
+        >>> # Attach calibration error handler
+        >>> CalibrationError(
+        ...     num_bins=15,
+        ...     include_background=False,
+        ...     calibration_reduction="expected",
+        ...     output_transform=from_engine(["pred", "label"]),
+        ... ).attach(evaluator, name="ECE")
+        >>>
+        >>> # After evaluation, access results
+        >>> evaluator.run(val_loader)
+        >>> ece = evaluator.state.metrics["ECE"]
+        >>> print(f"Expected Calibration Error: {ece:.4f}")
+    """
+
+    def __init__(
+        self,
+        num_bins: int = 20,
+        include_background: bool = True,
+        calibration_reduction: CalibrationReduction | str = CalibrationReduction.EXPECTED,
+        metric_reduction: MetricReduction | str = MetricReduction.MEAN,
+        output_transform: Callable = lambda x: x,
+        save_details: bool = True,
+    ) -> None:
+        metric_fn = CalibrationErrorMetric(
+            num_bins=num_bins,
+            include_background=include_background,
+            calibration_reduction=calibration_reduction,
+            metric_reduction=metric_reduction,
+        )
+
+        super().__init__(metric_fn=metric_fn, output_transform=output_transform, save_details=save_details)

monai/metrics/__init__.py

@@ -13,6 +13,7 @@
 
 from .active_learning_metrics import LabelQualityScore, VarianceMetric, compute_variance, label_quality_score
 from .average_precision import AveragePrecisionMetric, compute_average_precision
+from .calibration import CalibrationErrorMetric, CalibrationReduction, calibration_binning
 from .confusion_matrix import ConfusionMatrixMetric, compute_confusion_matrix_metric, get_confusion_matrix
 from .cumulative_average import CumulativeAverage
 from .f_beta_score import FBetaScore