|
1 | 1 | # evalwire |
2 | 2 |
|
| 3 | + |
| 4 | + |
3 | 5 | > Systematic, reproducible evaluation of LangGraph nodes and subgraphs against human-curated testsets, tracked in Arize Phoenix. |
4 | 6 |
|
5 | 7 | --- |
@@ -69,17 +71,17 @@ evalwire run --experiments experiments/ |
69 | 71 | All factories are importable from `evalwire.evaluators` and return a callable with |
70 | 72 | signature `(output, expected: dict) -> float | bool`. |
71 | 73 |
|
72 | | -| Factory | Returns | Use case | |
73 | | -|---|---|---| |
74 | | -| `make_top_k_evaluator(K=20)` | `float` | Position-weighted retrieval scoring | |
75 | | -| `make_membership_evaluator()` | `bool` | Classification / routing label check | |
76 | | -| `make_exact_match_evaluator()` | `bool` | Extractive QA, single ground-truth string | |
77 | | -| `make_contains_evaluator()` | `bool` | Free-text generation, required phrase present | |
78 | | -| `make_regex_evaluator()` | `bool` | Structured format validation (dates, IDs, …) | |
79 | | -| `make_json_match_evaluator(keys)` | `float` | Tool-call / structured-output key matching | |
80 | | -| `make_schema_evaluator(schema)` | `bool` | JSON Schema conformance | |
81 | | -| `make_numeric_tolerance_evaluator(atol, rtol)` | `bool` | Math / calculation tasks with tolerance | |
82 | | -| `make_llm_judge_evaluator(model, prompt, schema)` | `float\|bool` | LLM-as-a-judge with structured output | |
| 74 | +| Factory | Returns | Use case | |
| 75 | +| ------------------------------------------------- | ------------- | --------------------------------------------- | |
| 76 | +| `make_top_k_evaluator(K=20)` | `float` | Position-weighted retrieval scoring | |
| 77 | +| `make_membership_evaluator()` | `bool` | Classification / routing label check | |
| 78 | +| `make_exact_match_evaluator()` | `bool` | Extractive QA, single ground-truth string | |
| 79 | +| `make_contains_evaluator()` | `bool` | Free-text generation, required phrase present | |
| 80 | +| `make_regex_evaluator()` | `bool` | Structured format validation (dates, IDs, …) | |
| 81 | +| `make_json_match_evaluator(keys)` | `float` | Tool-call / structured-output key matching | |
| 82 | +| `make_schema_evaluator(schema)` | `bool` | JSON Schema conformance | |
| 83 | +| `make_numeric_tolerance_evaluator(atol, rtol)` | `bool` | Math / calculation tasks with tolerance | |
| 84 | +| `make_llm_judge_evaluator(model, prompt, schema)` | `float\|bool` | LLM-as-a-judge with structured output | |
83 | 85 |
|
84 | 86 | ### Example |
85 | 87 |
|
|
0 commit comments