Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

![evalwire logo](docs/assets/logo.svg)

> Systematic, reproducible evaluation of LangGraph nodes and subgraphs against human-curated testsets, tracked in Arize Phoenix.
> Systematic, reproducible evaluation of any async callable — LangGraph nodes, plain Python functions, REST API endpoints, and more — tracked in Arize Phoenix.

---

Expand All @@ -12,12 +12,12 @@

## What it does

When iterating on a LangGraph agent, it is hard to know whether a change to a specific node improved or degraded its behaviour. Running the full graph end-to-end is expensive and makes it difficult to attribute a score change to a specific component.
When iterating on any LLM-powered callable — a LangGraph node, a plain Python function, or a REST endpoint — it is hard to know whether a change improved or degraded behaviour. Running the full system end-to-end is expensive and makes it difficult to attribute a score change to a specific component.

`evalwire` solves this by:

- Turning a human-curated CSV of queries and expected outputs into versioned [Arize Phoenix](https://phoenix.arize.com/) datasets.
- Letting you define a **task** that isolates and invokes individual LangGraph nodes independently of the rest of the graph.
- Letting you define a **task** — any `async def` callable — that isolates and invokes the component under test.
- Running those tasks against the stored datasets, scoring each output with one or more **evaluators**, and recording results in Phoenix — giving you a reproducible, comparable experiment per run.

---
Expand Down
14 changes: 10 additions & 4 deletions docs/index.md
Original file line number Diff line number Diff line change
@@ -1,15 +1,18 @@
# evalwire

**evalwire** is a Python package for systematic evaluation of [LangGraph](https://github.com/langchain-ai/langgraph) nodes using [Arize Phoenix](https://github.com/Arize-ai/phoenix) experiments.
**evalwire** is a Python package for systematic evaluation of any async callable — including [LangGraph](https://github.com/langchain-ai/langgraph) nodes, plain functions, REST API endpoints, and other LLM frameworks — using [Arize Phoenix](https://github.com/Arize-ai/phoenix) experiments.

## Features

- Upload CSV testsets to Phoenix as named datasets
- Run experiments against any LangGraph node with pluggable evaluators
- 9 built-in evaluator factories covering retrieval, classification, string matching, structured output, numeric, and LLM-as-a-judge use cases
- Run experiments against any async callable with pluggable evaluators
- 12 built-in evaluator factories covering retrieval, classification, string matching, structured output, numeric, LLM-as-a-judge, and evaluator composition
- Export experiment results to CSV or JSON, compare runs, and generate markdown reports
- Validate testsets before upload to catch structural and content issues early
- First-class LangGraph integration via the optional `evalwire[langgraph]` extra
- OpenTelemetry tracing via `observability.py`
- Config-file driven via `evalwire.toml`
- CLI: `evalwire upload` and `evalwire run`
- CLI: `evalwire upload`, `evalwire run`, `evalwire validate`, `evalwire export`, `evalwire compare`, `evalwire report`

## Built-in evaluators

Expand All @@ -24,6 +27,9 @@
| `make_schema_evaluator` | `bool` | JSON Schema conformance |
| `make_numeric_tolerance_evaluator` | `bool` | Math / calculation tasks with tolerance |
| `make_llm_judge_evaluator` | `float \| bool` | LLM-as-a-judge with structured output |
| `make_weighted_evaluator` | `float` | Weighted average of multiple evaluators |
| `make_all_pass_evaluator` | `bool` | AND-composition: all evaluators must pass |
| `make_any_pass_evaluator` | `bool` | OR-composition: at least one evaluator must pass |

## Navigation

Expand Down
2 changes: 1 addition & 1 deletion src/evalwire/__init__.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
"""evalwire — systematic evaluation of LangGraph nodes with Arize Phoenix."""
"""evalwire — evaluate any async callable with Arize Phoenix experiments."""

import logging
from importlib.metadata import version
Expand Down
2 changes: 1 addition & 1 deletion src/evalwire/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ def _make_client():

@click.group()
def main() -> None:
"""evalwire — systematic evaluation of LangGraph nodes with Arize Phoenix."""
"""evalwire — evaluate any async callable with Arize Phoenix."""


@main.command("upload")
Expand Down