Skip to content

Commit 0867aac

Browse files
zurfjereluhmiezurfjereluhmie
committed
docs: update README and docs to cover all 9 evaluator factories
1 parent ed91b20 commit 0867aac

5 files changed

Lines changed: 192 additions & 4 deletions

File tree

‎README.md‎

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,10 @@ When iterating on a LangGraph agent, it is hard to know whether a change to a sp
2626
pip install evalwire
2727
# With LangGraph node-isolation helpers:
2828
pip install 'evalwire[langgraph]'
29+
# With LLM-as-a-judge evaluator:
30+
pip install 'evalwire[llm-judge]'
31+
# Everything:
32+
pip install 'evalwire[all]'
2933
```
3034

3135
---
@@ -60,6 +64,59 @@ evalwire run --experiments experiments/
6064

6165
---
6266

67+
## Built-in evaluators
68+
69+
All factories are importable from `evalwire.evaluators` and return a callable with
70+
signature `(output, expected: dict) -> float | bool`.
71+
72+
| Factory | Returns | Use case |
73+
|---|---|---|
74+
| `make_top_k_evaluator(K=20)` | `float` | Position-weighted retrieval scoring |
75+
| `make_membership_evaluator()` | `bool` | Classification / routing label check |
76+
| `make_exact_match_evaluator()` | `bool` | Extractive QA, single ground-truth string |
77+
| `make_contains_evaluator()` | `bool` | Free-text generation, required phrase present |
78+
| `make_regex_evaluator()` | `bool` | Structured format validation (dates, IDs, …) |
79+
| `make_json_match_evaluator(keys)` | `float` | Tool-call / structured-output key matching |
80+
| `make_schema_evaluator(schema)` | `bool` | JSON Schema conformance |
81+
| `make_numeric_tolerance_evaluator(atol, rtol)` | `bool` | Math / calculation tasks with tolerance |
82+
| `make_llm_judge_evaluator(model, prompt, schema)` | `float\|bool` | LLM-as-a-judge with structured output |
83+
84+
### Example
85+
86+
```python
87+
from evalwire.evaluators import make_top_k_evaluator, make_exact_match_evaluator
88+
89+
# Drop the factory return value into your experiment directory as the evaluator
90+
top_k = make_top_k_evaluator(K=5)
91+
exact = make_exact_match_evaluator()
92+
```
93+
94+
### LLM judge
95+
96+
```python
97+
from pydantic import BaseModel
98+
from langchain.chat_models import init_chat_model
99+
from evalwire.evaluators import make_llm_judge_evaluator
100+
101+
class Verdict(BaseModel):
102+
explanation: str
103+
score: bool # True = correct
104+
105+
llm_judge = make_llm_judge_evaluator(
106+
model=init_chat_model("gpt-4o-mini"),
107+
prompt_template=(
108+
"Output: {output}\n"
109+
"Expected: {expected_output}\n"
110+
"Is the output correct? Think step by step, then set score."
111+
),
112+
output_schema=Verdict,
113+
)
114+
```
115+
116+
Requires `pip install 'evalwire[llm-judge]'`.
117+
118+
---
119+
63120
## Node isolation
64121

65122
Use `invoke_node` to call a single LangGraph node without compiling a full graph:

‎docs/api/evaluators.md‎

Lines changed: 68 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,70 @@
11
# evalwire.evaluators
22

3-
::: evalwire.evaluators
3+
Built-in evaluator factories. Each factory returns a callable with the standard
4+
evalwire evaluator signature:
5+
6+
```python
7+
def evaluator(output: Any, expected: dict) -> float | bool: ...
8+
```
9+
10+
The `expected` dict always contains at minimum an `"expected_output"` key whose
11+
value is parsed by the shared `_parse_expected` helper (handles plain strings,
12+
Python-literal strings such as `"['a','b']"`, and lists).
13+
14+
All factories are importable directly from `evalwire.evaluators`:
15+
16+
```python
17+
from evalwire.evaluators import (
18+
make_top_k_evaluator,
19+
make_membership_evaluator,
20+
make_exact_match_evaluator,
21+
make_contains_evaluator,
22+
make_regex_evaluator,
23+
make_json_match_evaluator,
24+
make_schema_evaluator,
25+
make_numeric_tolerance_evaluator,
26+
make_llm_judge_evaluator,
27+
)
28+
```
29+
30+
---
31+
32+
## Retrieval
33+
34+
::: evalwire.evaluators.top_k.make_top_k_evaluator
35+
36+
---
37+
38+
## Classification
39+
40+
::: evalwire.evaluators.membership.make_membership_evaluator
41+
42+
---
43+
44+
## String matching
45+
46+
::: evalwire.evaluators.exact_match.make_exact_match_evaluator
47+
48+
::: evalwire.evaluators.contains.make_contains_evaluator
49+
50+
::: evalwire.evaluators.regex.make_regex_evaluator
51+
52+
---
53+
54+
## Structured output
55+
56+
::: evalwire.evaluators.json_match.make_json_match_evaluator
57+
58+
::: evalwire.evaluators.schema.make_schema_evaluator
59+
60+
---
61+
62+
## Numeric
63+
64+
::: evalwire.evaluators.numeric_tolerance.make_numeric_tolerance_evaluator
65+
66+
---
67+
68+
## LLM judge
69+
70+
::: evalwire.evaluators.llm_judge.make_llm_judge_evaluator

‎docs/api/index.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ Full reference for all public modules in `evalwire`.
44

55
| Module | Description |
66
|---|---|
7-
| [evaluators](evaluators.md) | Built-in evaluator factories (`top_k`, `membership`) |
7+
| [evaluators](evaluators.md) | Built-in evaluator factories (`top_k`, `membership`, `exact_match`, `contains`, `regex`, `json_match`, `schema`, `numeric_tolerance`, `llm_judge`) |
88
| [uploader](uploader.md) | Upload CSV testsets to Phoenix |
99
| [runner](runner.md) | Discover and run experiments |
1010
| [observability](observability.md) | OpenTelemetry tracing setup |

‎docs/index.md‎

Lines changed: 15 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,11 +6,25 @@
66

77
- Upload CSV testsets to Phoenix as named datasets
88
- Run experiments against any LangGraph node with pluggable evaluators
9-
- Built-in `top_k` and `membership` evaluators
9+
- 9 built-in evaluator factories covering retrieval, classification, string matching, structured output, numeric, and LLM-as-a-judge use cases
1010
- OpenTelemetry tracing via `observability.py`
1111
- Config-file driven via `evalwire.toml`
1212
- CLI: `evalwire upload` and `evalwire run`
1313

14+
## Built-in evaluators
15+
16+
| Factory | Returns | Use case |
17+
|---|---|---|
18+
| `make_top_k_evaluator` | `float` | Position-weighted retrieval scoring |
19+
| `make_membership_evaluator` | `bool` | Classification / routing label check |
20+
| `make_exact_match_evaluator` | `bool` | Extractive QA, single ground-truth string |
21+
| `make_contains_evaluator` | `bool` | Free-text generation, required phrase |
22+
| `make_regex_evaluator` | `bool` | Structured format validation (dates, IDs, …) |
23+
| `make_json_match_evaluator` | `float` | Tool-call / structured-output key matching |
24+
| `make_schema_evaluator` | `bool` | JSON Schema conformance |
25+
| `make_numeric_tolerance_evaluator` | `bool` | Math / calculation tasks with tolerance |
26+
| `make_llm_judge_evaluator` | `float \| bool` | LLM-as-a-judge with structured output |
27+
1428
## Navigation
1529

1630
- [Quick Start](quick-start.md) — get up and running in minutes

‎docs/quick-start.md‎

Lines changed: 51 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -55,10 +55,36 @@ async def task(example) -> list[str]:
5555
return result["retrieved_titles"]
5656
```
5757

58-
## Step 4 — Write an evaluator
58+
## Step 4 — Choose an evaluator
59+
60+
You can write a custom evaluator function or use one of the built-in factories.
61+
62+
### Using a built-in evaluator
5963

6064
Create `experiments/rag_pipeline/top_k.py`:
6165

66+
```python
67+
from evalwire.evaluators import make_top_k_evaluator
68+
69+
top_k = make_top_k_evaluator(K=5)
70+
```
71+
72+
All nine built-in factories are available from `evalwire.evaluators`:
73+
74+
| Factory | Returns | When to use |
75+
|---|---|---|
76+
| `make_top_k_evaluator(K)` | `float` | Ranked retrieval — score by position |
77+
| `make_membership_evaluator()` | `bool` | Classification / routing label |
78+
| `make_exact_match_evaluator()` | `bool` | Single correct string answer |
79+
| `make_contains_evaluator()` | `bool` | Output must include a required phrase |
80+
| `make_regex_evaluator()` | `bool` | Output must match a regex pattern |
81+
| `make_json_match_evaluator(keys)` | `float` | Structured output key-value matching |
82+
| `make_schema_evaluator(schema)` | `bool` | JSON Schema conformance |
83+
| `make_numeric_tolerance_evaluator(atol, rtol)` | `bool` | Numeric answer within tolerance |
84+
| `make_llm_judge_evaluator(model, prompt, schema)` | `float\|bool` | LLM-as-a-judge |
85+
86+
### Writing a custom evaluator
87+
6288
```python
6389
def top_k(output: list[str], expected: dict) -> float:
6490
"""Fraction of expected titles present in the top-K retrieved results."""
@@ -69,6 +95,30 @@ def top_k(output: list[str], expected: dict) -> float:
6995
return hits / len(expected_titles)
7096
```
7197

98+
### Using the LLM judge
99+
100+
```python
101+
from pydantic import BaseModel
102+
from langchain.chat_models import init_chat_model
103+
from evalwire.evaluators import make_llm_judge_evaluator
104+
105+
class Verdict(BaseModel):
106+
explanation: str
107+
score: bool # True = correct
108+
109+
llm_judge = make_llm_judge_evaluator(
110+
model=init_chat_model("gpt-4o-mini"),
111+
prompt_template=(
112+
"Output: {output}\n"
113+
"Expected: {expected_output}\n"
114+
"Is the output correct? Think step by step, then set score."
115+
),
116+
output_schema=Verdict,
117+
)
118+
```
119+
120+
Requires `pip install 'evalwire[llm-judge]'`.
121+
72122
## Step 5 — Run experiments
73123

74124
```bash

0 commit comments

Comments
 (0)