From 82aa97fb2221d6f507b53b8cc3b0bc3aa178aa71 Mon Sep 17 00:00:00 2001 From: zurfjereluhmie Date: Thu, 7 May 2026 17:45:16 +0200 Subject: [PATCH] docs: rebrand evalwire as framework-agnostic evaluation tool - Update README, docs/index.md, module docstring, and CLI help to emphasize 'evaluate any async callable' rather than LangGraph-only positioning - LangGraph remains a first-class integration via the optional extra - Also add AGENTS.md and enhancements/ to .gitignore (local knowledge base) --- README.md | 6 +++--- docs/index.md | 14 ++++++++++---- src/evalwire/__init__.py | 2 +- src/evalwire/cli.py | 2 +- 4 files changed, 15 insertions(+), 9 deletions(-) diff --git a/README.md b/README.md index 8cf1234..1677c7e 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ ![evalwire logo](docs/assets/logo.svg) -> Systematic, reproducible evaluation of LangGraph nodes and subgraphs against human-curated testsets, tracked in Arize Phoenix. +> Systematic, reproducible evaluation of any async callable — LangGraph nodes, plain Python functions, REST API endpoints, and more — tracked in Arize Phoenix. --- @@ -12,12 +12,12 @@ ## What it does -When iterating on a LangGraph agent, it is hard to know whether a change to a specific node improved or degraded its behaviour. Running the full graph end-to-end is expensive and makes it difficult to attribute a score change to a specific component. +When iterating on any LLM-powered callable — a LangGraph node, a plain Python function, or a REST endpoint — it is hard to know whether a change improved or degraded behaviour. Running the full system end-to-end is expensive and makes it difficult to attribute a score change to a specific component. `evalwire` solves this by: - Turning a human-curated CSV of queries and expected outputs into versioned [Arize Phoenix](https://phoenix.arize.com/) datasets. -- Letting you define a **task** that isolates and invokes individual LangGraph nodes independently of the rest of the graph. +- Letting you define a **task** — any `async def` callable — that isolates and invokes the component under test. - Running those tasks against the stored datasets, scoring each output with one or more **evaluators**, and recording results in Phoenix — giving you a reproducible, comparable experiment per run. --- diff --git a/docs/index.md b/docs/index.md index 3b22dd4..ad8fe76 100644 --- a/docs/index.md +++ b/docs/index.md @@ -1,15 +1,18 @@ # evalwire -**evalwire** is a Python package for systematic evaluation of [LangGraph](https://github.com/langchain-ai/langgraph) nodes using [Arize Phoenix](https://github.com/Arize-ai/phoenix) experiments. +**evalwire** is a Python package for systematic evaluation of any async callable — including [LangGraph](https://github.com/langchain-ai/langgraph) nodes, plain functions, REST API endpoints, and other LLM frameworks — using [Arize Phoenix](https://github.com/Arize-ai/phoenix) experiments. ## Features - Upload CSV testsets to Phoenix as named datasets -- Run experiments against any LangGraph node with pluggable evaluators -- 9 built-in evaluator factories covering retrieval, classification, string matching, structured output, numeric, and LLM-as-a-judge use cases +- Run experiments against any async callable with pluggable evaluators +- 12 built-in evaluator factories covering retrieval, classification, string matching, structured output, numeric, LLM-as-a-judge, and evaluator composition +- Export experiment results to CSV or JSON, compare runs, and generate markdown reports +- Validate testsets before upload to catch structural and content issues early +- First-class LangGraph integration via the optional `evalwire[langgraph]` extra - OpenTelemetry tracing via `observability.py` - Config-file driven via `evalwire.toml` -- CLI: `evalwire upload` and `evalwire run` +- CLI: `evalwire upload`, `evalwire run`, `evalwire validate`, `evalwire export`, `evalwire compare`, `evalwire report` ## Built-in evaluators @@ -24,6 +27,9 @@ | `make_schema_evaluator` | `bool` | JSON Schema conformance | | `make_numeric_tolerance_evaluator` | `bool` | Math / calculation tasks with tolerance | | `make_llm_judge_evaluator` | `float \| bool` | LLM-as-a-judge with structured output | +| `make_weighted_evaluator` | `float` | Weighted average of multiple evaluators | +| `make_all_pass_evaluator` | `bool` | AND-composition: all evaluators must pass | +| `make_any_pass_evaluator` | `bool` | OR-composition: at least one evaluator must pass | ## Navigation diff --git a/src/evalwire/__init__.py b/src/evalwire/__init__.py index 552d26f..2e02db1 100644 --- a/src/evalwire/__init__.py +++ b/src/evalwire/__init__.py @@ -1,4 +1,4 @@ -"""evalwire — systematic evaluation of LangGraph nodes with Arize Phoenix.""" +"""evalwire — evaluate any async callable with Arize Phoenix experiments.""" import logging from importlib.metadata import version diff --git a/src/evalwire/cli.py b/src/evalwire/cli.py index 52be927..04ebc7b 100644 --- a/src/evalwire/cli.py +++ b/src/evalwire/cli.py @@ -26,7 +26,7 @@ def _make_client(): @click.group() def main() -> None: - """evalwire — systematic evaluation of LangGraph nodes with Arize Phoenix.""" + """evalwire — evaluate any async callable with Arize Phoenix.""" @main.command("upload")