You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
src/ragas/dataset_schema.py:412 (EvaluationResult) computes per-metric aggregate values internally for __repr__ (lines 449-454 of __post_init__): it runs safe_nanmean over each metric column. But it doesn't expose that aggregate to callers. The only public way to read the per-metric mean today is repr(result), which returns a string like "{'faithfulness': 0.8920, 'answer_relevancy': 0.7650}" β not machine-readable.
For CI gates and regression tracking, callers want a structured per-metric summary (mean, count, missing) that they can assert against:
# Todaymean=float(repr(result).split("'faithfulness': ")[1].split(",")[0]) # ugly# What they wantsummary=result.summary() # {"faithfulness": {"mean": 0.89, "count": 30, "missing": 0}, ...}assertsummary["faithfulness"]["mean"] >=0.80
Why it matters
A summary() method is the natural building block for any CI/regression workflow that ingests ragas scores. Without it, callers either (a) parse the __repr__ string (fragile), (b) call to_pandas().mean() (heavier, requires pandas), or (c) hand-roll the loop in __post_init__ (duplicates the existing safe_nanmean path, and gets the NaN handling wrong β sum()/n returns NaN, not 0, when any value is NaN).
Proposed solution
Add a summary() method on EvaluationResult that returns a structured per-metric aggregate. Reuse the existing safe_nanmean helper so the NaN semantics match the rest of the file. Return a Dict[str, Dict[str, float | int]] so callers can both summary[name]["mean"] and summary[name]["count"] for downstream assertions.
defsummary(self) ->t.Dict[str, t.Dict[str, t.Any]]:
"""Return per-metric aggregate statistics as a structured dict. Returns ------- dict of str to dict A dict mapping each metric name to a dict with keys ``mean``, ``count`` (number of non-null scores), and ``missing`` (number of null scores). ``mean`` is ``float("nan")`` when every score for the metric is null. """return {
name: {
"mean": safe_nanmean(values),
"count": sum(1forvinvaluesifvisnotNone),
"missing": sum(1forvinvaluesifvisNone),
}
forname, valuesinself._scores_dict.items()
}
The method reuses the cached _scores_dict from __post_init__, so it's O(N metrics) at lookup time, not O(N rows).
Acceptance criteria
New summary() method on EvaluationResult returns dict[str, dict[str, float | int]].
Each per-metric dict has exactly the keys mean, count, missing.
mean uses safe_nanmean so a single np.nan in the column does not poison the aggregate.
count and missing sum to the total number of scores for the metric.
mean is nan when count == 0.
Binary columns (e.g. AspectCritic verdicts) work the same as continuous columns β mean of a 0/1 column is the fraction of 1s, which is what most callers want.
New unit tests cover: 3 metrics, NaN-in-column, all-NaN column, all-present column, mixed metric types.
ruff check, ruff format --check, pyright clean on the touched file.
Backward compatible: no public API change (additive only).
Out of scope
Threshold checks (check_threshold): that's a separate feature with a different shape (caller-supplied thresholds, raise on fail, etc.). The user's issue Add EvaluationResult summary and threshold checksΒ #2760 mixes the two; this PR keeps them separate so the diff stays focused.
Per-model cost breakdown: requires a cost_cb and is a different layer.
JSON serialization of the summary: callers can json.dumps(result.summary()) themselves; no need to add a summary_json() method.
Problem
src/ragas/dataset_schema.py:412(EvaluationResult) computes per-metric aggregate values internally for__repr__(lines 449-454 of__post_init__): it runssafe_nanmeanover each metric column. But it doesn't expose that aggregate to callers. The only public way to read the per-metric mean today isrepr(result), which returns a string like"{'faithfulness': 0.8920, 'answer_relevancy': 0.7650}"β not machine-readable.For CI gates and regression tracking, callers want a structured per-metric summary (mean, count, missing) that they can assert against:
Why it matters
A
summary()method is the natural building block for any CI/regression workflow that ingests ragas scores. Without it, callers either (a) parse the__repr__string (fragile), (b) callto_pandas().mean()(heavier, requires pandas), or (c) hand-roll the loop in__post_init__(duplicates the existingsafe_nanmeanpath, and gets the NaN handling wrong βsum()/nreturns NaN, not 0, when any value is NaN).Proposed solution
Add a
summary()method onEvaluationResultthat returns a structured per-metric aggregate. Reuse the existingsafe_nanmeanhelper so the NaN semantics match the rest of the file. Return aDict[str, Dict[str, float | int]]so callers can bothsummary[name]["mean"]andsummary[name]["count"]for downstream assertions.The method reuses the cached
_scores_dictfrom__post_init__, so it's O(N metrics) at lookup time, not O(N rows).Acceptance criteria
summary()method onEvaluationResultreturnsdict[str, dict[str, float | int]].mean,count,missing.meanusessafe_nanmeanso a singlenp.nanin the column does not poison the aggregate.countandmissingsum to the total number of scores for the metric.meanisnanwhencount == 0.meanof a 0/1 column is the fraction of 1s, which is what most callers want.ruff check,ruff format --check,pyrightclean on the touched file.Out of scope
check_threshold): that's a separate feature with a different shape (caller-supplied thresholds, raise on fail, etc.). The user's issue Add EvaluationResult summary and threshold checksΒ #2760 mixes the two; this PR keeps them separate so the diff stays focused.cost_cband is a different layer.json.dumps(result.summary())themselves; no need to add asummary_json()method.Environment
main(post 0.4.3, commit298b682)