Skip to content

index.md

Latest commit

 

History

History
151 lines (97 loc) · 14.4 KB

index.md

File metadata and controls

151 lines (97 loc) · 14.4 KB
Raw
layout default
title BluMind Benchmark
description The public benchmark of AI reasoning applied to water-treatment-plant operations.
lang en

BluMind Benchmark

The public benchmark of AI reasoning applied to water-treatment-plant operations.

BluMind evaluates AI models on real diagnostic and reasoning tasks drawn from the operation of water-treatment plants. Every response is scored by the BluMind Technical Committee — senior practitioners and researchers of the water sector — against a private gold standard.

The benchmark is public, reproducible, and human-scored. The leaderboard updates as new models are submitted and as the Technical Committee releases new cases.

🏆 Ranking · v1.0

v1.0 covers the 5 core failure families (FOUL, SCAL, OXID, MECH, NOWE) of reverse-osmosis desalination plants — 31 cases, 26 model invocations evaluated (13 distinct models, several at multiple reasoning-effort levels), scored by the BluMind Technical Committee.

# Subject Provider Mode Pass Cond Fail Crit Mean (/12) Brier ↓ ECE ↓ Q ↑ Status
1 gpt-5-5-none <img class="provider-logo" src="{{ "/assets/openai.png" relative_url }}" alt="OpenAI" height="22"> OpenAI 🧠 reasoning 29 2 0 0 11.10 0.023 0.143 0.93
2 gpt-5-5-xhigh <img class="provider-logo" src="{{ "/assets/openai.png" relative_url }}" alt="OpenAI" height="22"> OpenAI 🧠 reasoning 29 2 0 0 11.03 0.021 0.135 0.93
3 gpt-5-5-high <img class="provider-logo" src="{{ "/assets/openai.png" relative_url }}" alt="OpenAI" height="22"> OpenAI 🧠 reasoning 29 2 0 0 10.97 0.022 0.136 0.92
4 claude-opus-4-7-medium <img class="provider-logo" src="{{ "/assets/anthropic.png" relative_url }}" alt="Anthropic" height="22"> Anthropic 🧠 reasoning 28 3 0 0 11.03 0.036 0.170 0.91
5 gpt-5-5-low <img class="provider-logo" src="{{ "/assets/openai.png" relative_url }}" alt="OpenAI" height="22"> OpenAI 🧠 reasoning 28 3 0 0 11.00 0.025 0.148 0.91
6 gpt-5-5-medium <img class="provider-logo" src="{{ "/assets/openai.png" relative_url }}" alt="OpenAI" height="22"> OpenAI 🧠 reasoning 28 3 0 0 10.97 0.024 0.141 0.91
7 claude-opus-4-7-high <img class="provider-logo" src="{{ "/assets/anthropic.png" relative_url }}" alt="Anthropic" height="22"> Anthropic 🧠 reasoning 28 3 0 0 10.94 0.041 0.189 0.91
8 claude-opus-4-7-off <img class="provider-logo" src="{{ "/assets/anthropic.png" relative_url }}" alt="Anthropic" height="22"> Anthropic 🧠 reasoning 28 3 0 0 10.84 0.038 0.173 0.90
9 claude-opus-4-7-xhigh <img class="provider-logo" src="{{ "/assets/anthropic.png" relative_url }}" alt="Anthropic" height="22"> Anthropic 🧠 reasoning 28 2 1 1 10.81 0.057 0.166 0.90
10 claude-opus-4-7-max <img class="provider-logo" src="{{ "/assets/anthropic.png" relative_url }}" alt="Anthropic" height="22"> Anthropic 🧠 reasoning 28 2 1 1 10.77 0.041 0.178 0.90
11 gpt-5-5-minimal <img class="provider-logo" src="{{ "/assets/openai.png" relative_url }}" alt="OpenAI" height="22"> OpenAI 🧠 reasoning 27 4 0 0 10.87 0.025 0.145 0.89
12 gpt-5-medium <img class="provider-logo" src="{{ "/assets/openai.png" relative_url }}" alt="OpenAI" height="22"> OpenAI classic 27 4 0 0 10.87 0.034 0.158 0.89
13 claude-haiku-4-5-off <img class="provider-logo" src="{{ "/assets/anthropic.png" relative_url }}" alt="Anthropic" height="22"> Anthropic classic 25 5 1 1 10.48 0.037 0.173 0.84
14 claude-opus-4-6-off <img class="provider-logo" src="{{ "/assets/anthropic.png" relative_url }}" alt="Anthropic" height="22"> Anthropic classic 24 6 1 1 10.58 0.035 0.100 0.83
15 deepseek-v4-flash-high <img class="provider-logo" src="{{ "/assets/deepseek-color.png" relative_url }}" alt="DeepSeek" height="22"> DeepSeek classic 22 8 1 1 10.16 0.040 0.137 0.78
16 claude-opus-4-7-low <img class="provider-logo" src="{{ "/assets/anthropic.png" relative_url }}" alt="Anthropic" height="22"> Anthropic 🧠 reasoning 20 11 0 0 10.00 0.034 0.155 0.74
17 gemini-3-5-flash-high <img class="provider-logo" src="{{ "/assets/google.png" relative_url }}" alt="Google" height="22"> Google 🧠 reasoning 19 12 0 0 9.74 0.023 0.027 0.71
18 mistral-small-3 <img class="provider-logo" src="{{ "/assets/mistral.png" relative_url }}" alt="Mistral" height="22"> Mistral classic 18 12 1 1 9.74 0.039 0.037 0.70
19 deepseek-v4-flash-max <img class="provider-logo" src="{{ "/assets/deepseek-color.png" relative_url }}" alt="DeepSeek" height="22"> DeepSeek 🧠 reasoning 17 13 1 0 9.55 0.029 0.143 0.67
20 gemini-2-5-pro <img class="provider-logo" src="{{ "/assets/google.png" relative_url }}" alt="Google" height="22"> Google classic 14 16 1 0 9.48 0.009 0.035 0.62
21 gemini-3-5-flash-medium <img class="provider-logo" src="{{ "/assets/google.png" relative_url }}" alt="Google" height="22"> Google 🧠 reasoning 13 16 2 0 9.00 0.030 0.033 0.58
22 gemini-3-5-flash-low <img class="provider-logo" src="{{ "/assets/google.png" relative_url }}" alt="Google" height="22"> Google 🧠 reasoning 9 21 1 0 9.10 0.025 0.038 0.52
23 gemini-3-1-flash-lite-minimal <img class="provider-logo" src="{{ "/assets/google.png" relative_url }}" alt="Google" height="22"> Google classic 5 25 1 1 8.32 0.018 0.067 0.43
24 mistral-medium-3 <img class="provider-logo" src="{{ "/assets/mistral.png" relative_url }}" alt="Mistral" height="22"> Mistral classic 0 27 4 0 7.84 0.035 0.076 0.33
25 gemini-2-5-flash-lite-off <img class="provider-logo" src="{{ "/assets/google.png" relative_url }}" alt="Google" height="22"> Google classic 0 24 7 3 7.35 0.050 0.039 0.31
26 gpt-3-5-turbo <img class="provider-logo" src="{{ "/assets/openai.png" relative_url }}" alt="OpenAI" height="22"> OpenAI classic 0 9 22 2 5.48 0.142 0.268 0.23

Reading the suffix. -low, -medium, -high, -xhigh, -max denote the reasoning effort sent to the model. -off denotes the model invoked with its thinking path disabled. -none is OpenAI GPT-5.5's explicit "reasoning OFF" tier. -minimal is the lowest non-zero effort on providers that expose it (e.g. Gemini 3.x Flash-Lite). Same subject_version ↔ same model snapshot — only the effort knob differs.

📚 How to read this table

  • Pass / Cond / Fail — Per-case classification by the Technical Committee. Pass = a response an experienced operator would accept on its own. Conditional = a response with gaps but salvageable. Fail = a response that would mislead a real operator.

  • Crit — Critical automatic fails. Cases where the response recommended an action that would damage the plant or compromise operator safety (for example, recommending an oxidant on polyamide membranes). A single critical fail disqualifies the model from the leaderboard, regardless of all other scores. The triggering action is cited literally in the full leaderboard on GitHub.

  • Mean (/12) — Average expert-scored quality per case, on the 0–12 rubric. 12 = "indistinguishable from the expert gold answer"; 0 = "completely wrong".

  • Brier ↓ — A measure of confidence, that is "whether the model believes it knows more than it actually does, or not".

    Read it like this: if the model says "I'm very sure", Brier measures how heavily it is penalised by its errors when that confidence was not justified. The lower the better: it means the model not only gets things right, it also declares a reasonable level of confidence.

    In one sentence: Brier measures whether the model's confidence is prudent on each response.

    Range: 0–1. Lower is better. In v1.0 the observed spread is 0.009 (best) to 0.142 (worst).

    Technically, on every case BluMind computes the squared gap between the model's stated confidence — for example, "90% sure" = 0.9 — and the actual correctness of the response {0, 0.5, 1}. A response with "100% confidence" that turns out to be wrong contributes a very large error.

  • ECE ↓ (Expected Calibration Error) — How trustworthy the model's confidence is in aggregate.

    Read it like this: if the model says "I'm 70% sure" many times, ECE measures whether it actually gets close to 70% of those right. The lower the better: it means its confidence percentages look more like real probabilities.

    Technically, BluMind groups predictions into 10 confidence bands — 0-10%, 10-20%, …, 90-100% — and in each band compares the average stated confidence against the average actual correctness, weighted by the number of cases in each band.

    For example, if the model says "70% sure" on 20 cases but only gets 50% right, that band is miscalibrated and increases the ECE.

    Range: 0–1. Lower is better. In v1.0 the observed spread is 0.035 (best) to 0.268 (worst). Labelled indicative at N = 31.

    In one sentence: ECE measures whether the model's confidence percentages can be trusted as probabilities.

  • Why both matter.

    Imagine an operator who says: "I'm 90% sure this is biofouling."

    A good Brier means that, on that specific prediction, the operator/model does not tend to be extremely confident when wrong.

    A good ECE means that, when they say "90% sure" many times, they really are right about 90 of every 100.

    The simple difference is:

    • Brier looks at the quality of confidence case by case.
    • ECE looks at whether confidence is well-calibrated in aggregate.

    Both metrics matter because many downstream systems — alarms, operational assistants, automated workflows, recommendation systems — may take the confidence number literally. If the model says "95% sure" but really is not, the system can over-react.

  • Q ↑ — Composite quality score combining Pass-rate and mean per-case score. This is the ranking column.

  • Mode — classic means the model was queried at temperature = 0. 🧠 reasoning means the model was queried using its native deep-thinking mode (Claude reasoning, GPT reasoning, etc.).

  • Status — ✅ Eligible if the model has zero critical fails. ⛔ Disqualified otherwise. Disqualified models are still listed for transparency, but they cannot win the leaderboard.

See operational metrics (cost, latency, tokens) and safety-gate citations on GitHub →

Read the v1.0 Findings Report — generational uplift, hypothesis quality, calibration, and limitations →

What makes BluMind different

Independent human scoring. Every response is scored by two members of the BluMind Technical Committee, drawn from senior practitioners and researchers of the water sector. The committee is the institutional authority behind every score.

Safety gate. A single critical-fail recommendation — any action that would damage the plant or compromise operator safety — disqualifies the model from the leaderboard regardless of its other scores. The triggering action is cited literally and made public.

Reproducible. Cases, rubric, prompts, evaluation scripts and aggregated metrics are public. The private gold answers and reviewer mappings stay private — exactly as one would expect from a benchmark that can be trusted.

Read the full methodology on GitHub →

Submit a model

During the foundational phase (until 31 December 2026), valid submissions are evaluated free of charge. The submitter provides metadata, technical access credentials encrypted with the BluMind PGP key, and confirms eligibility against the published scope.

A submission is normally validated within 2 working days and evaluated within 10 working days of validation.

Read the submission guide on GitHub →

The committee

The BluMind Technical Committee is the body of senior practitioners and researchers responsible for the integrity of the benchmark. It is the institutional authority behind every score, classification, and appeal decision.

Public members include Álvaro Díaz del Río Redondo — CEO of BluMind, formerly Head of Innovation at Tedagua and Cobra Infraestructuras Hidráulicas — and Rafael Jiménez Garrido — Country Manager at Whitewater Group, lecturer at the Master's Degree in Desalination and Water Reuse (Universidad de Alicante), industry contributor at ALADYR.

Three additional senior international figures of the water sector are part of the committee, with names pending public disclosure.

Meet the committee on GitHub →

Contact

BluMind Benchmark is operated by BluMind. The benchmark is released under the license terms in LICENSE.