Skip to content

dataset_schema: add EvaluationResult.summary() returning per-metric mean/count/missingΒ #2893

Description

@Harsh23Kashyap

Problem

src/ragas/dataset_schema.py:412 (EvaluationResult) computes per-metric aggregate values internally for __repr__ (lines 449-454 of __post_init__): it runs safe_nanmean over each metric column. But it doesn't expose that aggregate to callers. The only public way to read the per-metric mean today is repr(result), which returns a string like "{'faithfulness': 0.8920, 'answer_relevancy': 0.7650}" β€” not machine-readable.

For CI gates and regression tracking, callers want a structured per-metric summary (mean, count, missing) that they can assert against:

# Today
mean = float(repr(result).split("'faithfulness': ")[1].split(",")[0])  # ugly

# What they want
summary = result.summary()  # {"faithfulness": {"mean": 0.89, "count": 30, "missing": 0}, ...}
assert summary["faithfulness"]["mean"] >= 0.80

Why it matters

A summary() method is the natural building block for any CI/regression workflow that ingests ragas scores. Without it, callers either (a) parse the __repr__ string (fragile), (b) call to_pandas().mean() (heavier, requires pandas), or (c) hand-roll the loop in __post_init__ (duplicates the existing safe_nanmean path, and gets the NaN handling wrong β€” sum()/n returns NaN, not 0, when any value is NaN).

Proposed solution

Add a summary() method on EvaluationResult that returns a structured per-metric aggregate. Reuse the existing safe_nanmean helper so the NaN semantics match the rest of the file. Return a Dict[str, Dict[str, float | int]] so callers can both summary[name]["mean"] and summary[name]["count"] for downstream assertions.

def summary(self) -> t.Dict[str, t.Dict[str, t.Any]]:
    """Return per-metric aggregate statistics as a structured dict.

    Returns
    -------
    dict of str to dict
        A dict mapping each metric name to a dict with keys ``mean``,
        ``count`` (number of non-null scores), and ``missing`` (number
        of null scores). ``mean`` is ``float("nan")`` when every score
        for the metric is null.
    """
    return {
        name: {
            "mean": safe_nanmean(values),
            "count": sum(1 for v in values if v is not None),
            "missing": sum(1 for v in values if v is None),
        }
        for name, values in self._scores_dict.items()
    }

The method reuses the cached _scores_dict from __post_init__, so it's O(N metrics) at lookup time, not O(N rows).

Acceptance criteria

  • New summary() method on EvaluationResult returns dict[str, dict[str, float | int]].
  • Each per-metric dict has exactly the keys mean, count, missing.
  • mean uses safe_nanmean so a single np.nan in the column does not poison the aggregate.
  • count and missing sum to the total number of scores for the metric.
  • mean is nan when count == 0.
  • Binary columns (e.g. AspectCritic verdicts) work the same as continuous columns β€” mean of a 0/1 column is the fraction of 1s, which is what most callers want.
  • New unit tests cover: 3 metrics, NaN-in-column, all-NaN column, all-present column, mixed metric types.
  • ruff check, ruff format --check, pyright clean on the touched file.
  • Backward compatible: no public API change (additive only).

Out of scope

  • Threshold checks (check_threshold): that's a separate feature with a different shape (caller-supplied thresholds, raise on fail, etc.). The user's issue Add EvaluationResult summary and threshold checksΒ #2760 mixes the two; this PR keeps them separate so the diff stays focused.
  • Per-model cost breakdown: requires a cost_cb and is a different layer.
  • JSON serialization of the summary: callers can json.dumps(result.summary()) themselves; no need to add a summary_json() method.

Environment

  • ragas main (post 0.4.3, commit 298b682)
  • Python 3.12

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions