docs: add concepts, guides, configuration, and troubleshooting pages (#48)

Author: zurfjereluhmie

docs/api/evaluators.md

@@ -1,15 +1,12 @@
 # evalwire.evaluators
 
-Built-in evaluator factories. Each factory returns a callable with the standard
-evalwire evaluator signature:
+Built-in evaluator factories. Each factory returns a callable with the standard evalwire evaluator signature:
 
 ```python
-def evaluator(output: Any, expected: dict) -> float | bool: ...
+def evaluator(output, expected: dict) -> float | bool: ...
 ```
 
-The `expected` dict always contains at minimum an `"expected_output"` key whose
-value is parsed by the shared `_parse_expected` helper (handles plain strings,
-Python-literal strings such as `"['a','b']"`, and lists).
+The `expected` dict always contains at minimum an `"expected_output"` key whose value is parsed by the shared `_parse_expected` helper (handles plain strings, Python-literal strings such as `"['a','b']"`, and pipe-delimited strings).
 
 All factories are importable directly from `evalwire.evaluators`:
 
@@ -27,6 +24,8 @@ from evalwire.evaluators import (
 )
 ```
 
+For a guide on writing your own evaluators, see [Writing Custom Evaluators](../guides/custom-evaluators.md).
+
 ---
 
 ## Retrieval

docs/api/runner.md

@@ -1,3 +1,90 @@
 # evalwire.runner
 
+`ExperimentRunner` auto-discovers experiment subdirectories, fetches the matching Phoenix dataset for each one, and runs the task against every example. Results are stored in Phoenix and returned as a list.
+
+## Basic usage
+
+```python
+from phoenix.client import Client
+from evalwire.runner import ExperimentRunner
+
+client = Client()
+
+runner = ExperimentRunner(
+    experiments_dir="experiments",
+    phoenix_client=client,
+    concurrency=2,
+)
+results = runner.run()
+```
+
+## Directory layout
+
+Each subdirectory of `experiments_dir` that contains a `task.py` is treated as one experiment. The directory name must exactly match a Phoenix dataset name:
+
+```
+experiments/
+  es_search/           <- matches dataset named "es_search"
+    task.py            <- must define async def task(example)
+    top_k.py           <- evaluator: must define top_k = ...
+    exact_match.py     <- evaluator: must define exact_match = ...
+  source_router/       <- matches dataset named "source_router"
+    task.py
+    is_in.py
+```
+
+Subdirectories without `task.py` are silently skipped. Files (non-directories) at the top level of `experiments_dir` are also skipped.
+
+## Experiment naming
+
+Each run in Phoenix is named `{prefix}_{dataset_name}_{iso_timestamp}`. The default prefix is `"eval"`. Override it with `experiment_name_prefix`:
+
+```python
+results = runner.run(experiment_name_prefix="nightly")
+# produces e.g. "nightly_es_search_2025-01-15T09:30:00"
+```
+
+## Running a subset
+
+Pass `names` to run only specific experiments:
+
+```python
+results = runner.run(names=["es_search"])
+```
+
+## Dry run
+
+Set `dry_run=True` (or `dry_run=N` to limit to N examples) to execute tasks without uploading results to Phoenix. Useful for smoke-testing your task code:
+
+```python
+runner = ExperimentRunner(
+    experiments_dir="experiments",
+    phoenix_client=client,
+    dry_run=3,
+)
+runner.run()
+```
+
+## Async tasks
+
+Tasks are `async` functions. evalwire wraps them in a per-thread event loop so Phoenix's synchronous runner can call them. The loop is kept open between calls (unlike `asyncio.run()`) so that async I/O libraries which reuse connections across calls work correctly.
+
+## Error behaviour
+
+If any experiment fails (dataset not found, task raises, evaluator raises), `runner.run()` raises `SystemExit(1)` after all experiments complete so that CI pipelines fail loudly. Successful experiments are still returned.
+
+## Pitfalls
+
+- The dataset name must match the directory name exactly (case-sensitive).
+- At least one evaluator file is required per experiment. Phoenix raises an error if `run_experiment` is called with an empty evaluator list.
+- Relative imports inside experiment modules work because evalwire adds the parent of `experiments_dir` to `sys.path` during discovery. It is removed again afterwards.
+
+## See also
+
+- [Concepts](../concepts.md) for the experiment lifecycle
+- [Configuration Reference](../configuration.md) for `evalwire.toml` keys
+- [CLI Reference](cli.md) for `evalwire run`
+
+---
+
 ::: evalwire.runner

docs/api/uploader.md

@@ -1,3 +1,54 @@
 # evalwire.uploader
 
+`DatasetUploader` reads a CSV testset and uploads it to Arize Phoenix as one named dataset per unique tag value. It handles three conflict modes (`skip`, `overwrite`, `append`) and supports multi-tag rows via a configurable delimiter.
+
+## Basic usage
+
+```python
+from phoenix.client import Client
+from evalwire.uploader import DatasetUploader
+
+client = Client()
+
+uploader = DatasetUploader(
+    csv_path="data/testset.csv",
+    phoenix_client=client,
+)
+datasets = uploader.upload(on_exist="skip")
+print(datasets)  # {"es_search": <Dataset>, "source_router": <Dataset>}
+```
+
+## CSV format
+
+The CSV must contain at least a tag column, one input column, and one expected-output column:
+
+```csv
+user_query,expected_output,tags
+"find cycling paths","url-a | url-b","es_search | source_router"
+"find parks","url-c","es_search"
+```
+
+Pipe-delimited values in any column are split into lists. A row with `tags = "es_search | source_router"` is added to both datasets.
+
+## Conflict modes
+
+| `on_exist` | Behaviour |
+|---|---|
+| `"skip"` | Do nothing if the dataset already exists. |
+| `"overwrite"` | Delete the existing dataset and re-create it. |
+| `"append"` | Call `add_examples_to_dataset` on the existing dataset. If not found, create it. |
+
+## Pitfalls
+
+- Phoenix raises `ValueError` (not a Phoenix-specific exception) when `get_dataset` is called for a non-existent dataset. evalwire catches this and creates the dataset instead.
+- There is no official delete method in the Phoenix Python client. evalwire calls the REST endpoint `DELETE /v1/datasets/{id}` directly for the `overwrite` mode.
+- Creating a dataset with a name that already exists returns a 409 Conflict error, not a new version. Use `on_exist="overwrite"` to replace it.
+
+## See also
+
+- [Configuration Reference](../configuration.md) for `evalwire.toml` keys
+- [CLI Reference](cli.md) for `evalwire upload`
+
+---
+
 ::: evalwire.uploader

docs/changelog.md

@@ -0,0 +1,3 @@
+{%
+  include-markdown "../CHANGELOG.md"
+%}

docs/concepts.md

@@ -0,0 +1,102 @@
+# Concepts
+
+This page explains the mental model behind evalwire before you dive into the quick start or API reference.
+
+## The four building blocks
+
+### Datasets
+
+A **dataset** is a named collection of test examples stored in Arize Phoenix. Each example has input fields (e.g. `user_query`) and expected-output fields (e.g. `expected_output`). evalwire creates datasets by uploading a CSV testset and splitting it by a tag column.
+
+One CSV row can belong to multiple datasets. Just pipe-delimit the tag value:
+
+```
+user_query,expected_output,tags
+"find cycling paths","url-a | url-b","es_search | source_router"
+```
+
+This row appears in both the `es_search` and `source_router` datasets.
+
+### Experiments
+
+An **experiment** is the result of running a task against a dataset and scoring each output with one or more evaluators. Phoenix records every experiment run with a timestamp so you can compare results across code changes.
+
+evalwire discovers experiments by scanning a directory. Each subdirectory that contains a `task.py` is treated as one experiment. The directory name must match a Phoenix dataset name:
+
+```
+experiments/
+  es_search/         ← matched to the "es_search" Phoenix dataset
+    task.py
+    top_k.py         ← evaluator
+  source_router/     ← matched to the "source_router" Phoenix dataset
+    task.py
+    is_in.py         ← evaluator
+```
+
+### Tasks
+
+A **task** is an `async` function that receives a Phoenix example object and returns the output to be scored. Its job is to call your system under test and return a result in whatever form your evaluators expect.
+
+```python
+# experiments/es_search/task.py
+async def task(example):
+    user_query = example.input["user_query"]
+    return await my_retrieval_function(user_query)
+```
+
+The function must be named `task`.
+
+### Evaluators
+
+An **evaluator** is a plain callable with the signature:
+
+```python
+def evaluator(output, expected: dict) -> float | bool: ...
+```
+
+Each evaluator file in an experiment directory is auto-loaded. The callable must share its name with the file:
+
+```python
+# experiments/es_search/top_k.py
+from evalwire.evaluators import make_top_k_evaluator
+
+top_k = make_top_k_evaluator(K=5)
+```
+
+Return `float` (0.0–1.0) for graded scoring or `bool` for pass/fail. Phoenix displays both.
+
+## The experiment lifecycle
+
+```
+CSV testset
+    │
+    │  evalwire upload
+    ▼
+Phoenix Datasets  (one per unique tag)
+    │
+    │  evalwire run
+    ▼
+Task function  (called once per example)
+    │
+    ▼
+Output
+    │
+    │  evaluator(output, expected)
+    ▼
+Experiment Results  (stored in Phoenix, visible in UI)
+```
+
+1. **Upload**: `evalwire upload` reads your CSV, groups rows by the tag column, and creates or updates one Phoenix dataset per unique tag value.
+2. **Run**: `evalwire run` scans the experiments directory, matches each subdirectory to a Phoenix dataset by name, calls `task` on every example, and scores the output with each evaluator file found in that directory.
+3. **Compare**: open the Phoenix UI, navigate to the dataset, and switch to the **Experiments** tab to compare runs side-by-side.
+
+## How `expected` is structured
+
+Inside every evaluator, the `expected` parameter is a dict containing all output columns from the original CSV row. The most important key is `"expected_output"`, which evalwire parses with `_parse_expected`:
+
+- Plain string `"answer"` → `["answer"]`
+- Pipe-delimited string `"a | b"` → `["a", "b"]`
+- Python-literal string `"['a', 'b']"` → `["a", "b"]`
+- Already a list → returned as-is
+
+Most built-in evaluators read `expected["expected_output"]` for you. When writing a custom evaluator you can also access any other column directly: `expected["my_column"]`.

docs/configuration.md

@@ -0,0 +1,90 @@
+# Configuration Reference
+
+evalwire can be driven entirely from a `evalwire.toml` file at the root of your project. CLI flags always take precedence over config file values, which take precedence over hardcoded defaults.
+
+## Precedence
+
+```
+CLI flag  >  evalwire.toml  >  hardcoded default
+```
+
+## Example file
+
+```toml
+[dataset]
+csv_path   = "data/testset.csv"
+on_exist   = "skip"
+input_keys = ["user_query"]
+output_keys = ["expected_output"]
+tag_column = "tags"
+delimiter  = "|"
+
+[experiments]
+dir         = "experiments"
+prefix      = "eval"
+concurrency = 4
+```
+
+## `[dataset]` section
+
+Used by `evalwire upload`.
+
+| Key | Type | Default | Description |
+|---|---|---|---|
+| `csv_path` | string | none (required) | Path to the CSV testset file. |
+| `on_exist` | `"skip"` / `"overwrite"` / `"append"` | `"skip"` | How to handle a dataset that already exists in Phoenix. |
+| `input_keys` | list of strings | `["user_query"]` | CSV column names treated as example inputs. |
+| `output_keys` | list of strings | `["expected_output"]` | CSV column names treated as expected outputs. |
+| `tag_column` | string | `"tags"` | CSV column used to split rows into separate datasets. |
+| `delimiter` | string | `"\|"` | Character used to assign a row to multiple datasets. |
+
+### `on_exist` modes
+
+| Value | Behaviour |
+|---|---|
+| `skip` | Leave the existing dataset untouched. |
+| `overwrite` | Delete the existing dataset and re-create it from the CSV. |
+| `append` | Add the new rows to the existing dataset. |
+
+## `[experiments]` section
+
+Used by `evalwire run`.
+
+| Key | Type | Default | Description |
+|---|---|---|---|
+| `dir` | string | `"experiments"` | Path to the experiments directory. |
+| `prefix` | string | `"eval"` | Prefix prepended to every experiment name in Phoenix. |
+| `concurrency` | integer | `1` | Number of experiments to run in parallel. |
+
+## `[phoenix]` section
+
+Reserved for future use. Currently read by `get_phoenix_config()` but not consumed by the CLI.
+
+## CLI flag reference
+
+### `evalwire upload`
+
+| Flag | Config key | Default |
+|---|---|---|
+| `--csv PATH` | `dataset.csv_path` | none |
+| `--on-exist MODE` | `dataset.on_exist` | `skip` |
+| `--input-keys COLS` | `dataset.input_keys` | `user_query` |
+| `--output-keys COLS` | `dataset.output_keys` | `expected_output` |
+| `--tag-column COL` | `dataset.tag_column` | `tags` |
+| `--delimiter CHAR` | `dataset.delimiter` | `\|` |
+| `--config PATH` | n/a | `./evalwire.toml` |
+
+### `evalwire run`
+
+| Flag | Config key | Default |
+|---|---|---|
+| `--experiments PATH` | `experiments.dir` | `experiments` |
+| `--name NAME` | n/a | all experiments |
+| `--prefix PREFIX` | `experiments.prefix` | `eval` |
+| `--concurrency N` | `experiments.concurrency` | `1` |
+| `--dry-run [N]` | n/a | off |
+| `--config PATH` | n/a | `./evalwire.toml` |
+
+`--name` is repeatable: `evalwire run --name es_search --name source_router`.
+
+`--dry-run` runs the task but does not upload results to Phoenix. Optionally accepts a number to limit the run to the first N examples per dataset.