docs: add concepts, guides, configuration, and troubleshooting pages

- Add Concepts page explaining datasets, experiments, tasks, evaluators,
  and the upload/run/compare lifecycle
- Add Writing Custom Evaluators guide covering the evaluator contract,
  expected dict structure, float vs bool, file naming, composing multiple
  evaluators, and error handling
- Add Configuration Reference page documenting all evalwire.toml keys,
  on_exist modes, and a full CLI flag reference for both commands
- Add Troubleshooting page for common errors: event loop closed, Phoenix
  connection issues, missing CSV path, dataset not found, optional extras,
  CSV formatting, and empty evaluator list
- Expand uploader.md and runner.md API pages with prose introductions,
  usage examples, pitfalls, and cross-links
- Add cross-link to custom evaluators guide from evaluators.md
- Add Changelog page (copy of CHANGELOG.md) to docs site
- Update mkdocs.yml nav with all new pages and a Guides section
- Update index.md navigation links

Author: zurfjereluhmie

docs/api/evaluators.md

@@ -1,15 +1,12 @@
 # evalwire.evaluators
 
-Built-in evaluator factories. Each factory returns a callable with the standard
-evalwire evaluator signature:
+Built-in evaluator factories. Each factory returns a callable with the standard evalwire evaluator signature:
 
 ```python
-def evaluator(output: Any, expected: dict) -> float | bool: ...
+def evaluator(output, expected: dict) -> float | bool: ...
 ```
 
-The `expected` dict always contains at minimum an `"expected_output"` key whose
-value is parsed by the shared `_parse_expected` helper (handles plain strings,
-Python-literal strings such as `"['a','b']"`, and lists).
+The `expected` dict always contains at minimum an `"expected_output"` key whose value is parsed by the shared `_parse_expected` helper (handles plain strings, Python-literal strings such as `"['a','b']"`, and pipe-delimited strings).
 
 All factories are importable directly from `evalwire.evaluators`:
 
@@ -27,6 +24,8 @@ from evalwire.evaluators import (
 )
 ```
 
+For a guide on writing your own evaluators, see [Writing Custom Evaluators](../guides/custom-evaluators.md).
+
 ---
 
 ## Retrieval

docs/api/runner.md

@@ -1,3 +1,90 @@
 # evalwire.runner
 
+`ExperimentRunner` auto-discovers experiment subdirectories, fetches the matching Phoenix dataset for each one, and runs the task against every example. Results are stored in Phoenix and returned as a list.
+
+## Basic usage
+
+```python
+from phoenix.client import Client
+from evalwire.runner import ExperimentRunner
+
+client = Client()
+
+runner = ExperimentRunner(
+    experiments_dir="experiments",
+    phoenix_client=client,
+    concurrency=2,
+)
+results = runner.run()
+```
+
+## Directory layout
+
+Each subdirectory of `experiments_dir` that contains a `task.py` is treated as one experiment. The directory name must exactly match a Phoenix dataset name:
+
+```
+experiments/
+  es_search/           <- matches dataset named "es_search"
+    task.py            <- must define async def task(example)
+    top_k.py           <- evaluator: must define top_k = ...
+    exact_match.py     <- evaluator: must define exact_match = ...
+  source_router/       <- matches dataset named "source_router"
+    task.py
+    is_in.py
+```
+
+Subdirectories without `task.py` are silently skipped. Files (non-directories) at the top level of `experiments_dir` are also skipped.
+
+## Experiment naming
+
+Each run in Phoenix is named `{prefix}_{dataset_name}_{iso_timestamp}`. The default prefix is `"eval"`. Override it with `experiment_name_prefix`:
+
+```python
+results = runner.run(experiment_name_prefix="nightly")
+# produces e.g. "nightly_es_search_2025-01-15T09:30:00"
+```
+
+## Running a subset
+
+Pass `names` to run only specific experiments:
+
+```python
+results = runner.run(names=["es_search"])
+```
+
+## Dry run
+
+Set `dry_run=True` (or `dry_run=N` to limit to N examples) to execute tasks without uploading results to Phoenix. Useful for smoke-testing your task code:
+
+```python
+runner = ExperimentRunner(
+    experiments_dir="experiments",
+    phoenix_client=client,
+    dry_run=3,
+)
+runner.run()
+```
+
+## Async tasks
+
+Tasks are `async` functions. evalwire wraps them in a per-thread event loop so Phoenix's synchronous runner can call them. The loop is kept open between calls (unlike `asyncio.run()`) so that async I/O libraries which reuse connections across calls work correctly.
+
+## Error behaviour
+
+If any experiment fails (dataset not found, task raises, evaluator raises), `runner.run()` raises `SystemExit(1)` after all experiments complete so that CI pipelines fail loudly. Successful experiments are still returned.
+
+## Pitfalls
+
+- The dataset name must match the directory name exactly (case-sensitive).
+- At least one evaluator file is required per experiment. Phoenix raises an error if `run_experiment` is called with an empty evaluator list.
+- Relative imports inside experiment modules work because evalwire adds the parent of `experiments_dir` to `sys.path` during discovery. It is removed again afterwards.
+
+## See also
+
+- [Concepts](../concepts.md) for the experiment lifecycle
+- [Configuration Reference](../configuration.md) for `evalwire.toml` keys
+- [CLI Reference](cli.md) for `evalwire run`
+
+---
+
 ::: evalwire.runner

docs/api/uploader.md

@@ -1,3 +1,54 @@
 # evalwire.uploader
 
+`DatasetUploader` reads a CSV testset and uploads it to Arize Phoenix as one named dataset per unique tag value. It handles three conflict modes (`skip`, `overwrite`, `append`) and supports multi-tag rows via a configurable delimiter.
+
+## Basic usage
+
+```python
+from phoenix.client import Client
+from evalwire.uploader import DatasetUploader
+
+client = Client()
+
+uploader = DatasetUploader(
+    csv_path="data/testset.csv",
+    phoenix_client=client,
+)
+datasets = uploader.upload(on_exist="skip")
+print(datasets)  # {"es_search": <Dataset>, "source_router": <Dataset>}
+```
+
+## CSV format
+
+The CSV must contain at least a tag column, one input column, and one expected-output column:
+
+```csv
+user_query,expected_output,tags
+"find cycling paths","url-a | url-b","es_search | source_router"
+"find parks","url-c","es_search"
+```
+
+Pipe-delimited values in any column are split into lists. A row with `tags = "es_search | source_router"` is added to both datasets.
+
+## Conflict modes
+
+| `on_exist` | Behaviour |
+|---|---|
+| `"skip"` | Do nothing if the dataset already exists. |
+| `"overwrite"` | Delete the existing dataset and re-create it. |
+| `"append"` | Call `add_examples_to_dataset` on the existing dataset. If not found, create it. |
+
+## Pitfalls
+
+- Phoenix raises `ValueError` (not a Phoenix-specific exception) when `get_dataset` is called for a non-existent dataset. evalwire catches this and creates the dataset instead.
+- There is no official delete method in the Phoenix Python client. evalwire calls the REST endpoint `DELETE /v1/datasets/{id}` directly for the `overwrite` mode.
+- Creating a dataset with a name that already exists returns a 409 Conflict error, not a new version. Use `on_exist="overwrite"` to replace it.
+
+## See also
+
+- [Configuration Reference](../configuration.md) for `evalwire.toml` keys
+- [CLI Reference](cli.md) for `evalwire upload`
+
+---
+
 ::: evalwire.uploader

docs/changelog.md

@@ -0,0 +1,93 @@
+# Changelog
+
+## [0.4.0](https://github.com/zurfjereluhmie/evalwire/compare/v0.3.1...v0.4.0) (2026-04-23)
+
+
+### Features
+
+* add logo SVG and update theme colors in mkdocs configuration ([#22](https://github.com/zurfjereluhmie/evalwire/issues/22)) ([67f44a9](https://github.com/zurfjereluhmie/evalwire/commit/67f44a93a9bc3f3aa51c463ea32a96264b62a04b))
+
+
+### Bug Fixes
+
+* enhance package typing, logging, and schema evaluation ([#28](https://github.com/zurfjereluhmie/evalwire/issues/28)) ([2a73529](https://github.com/zurfjereluhmie/evalwire/commit/2a735296a4f4eabdd7f82581667987f513360ff2))
+* loosen langgraph version pin from ==1.1.6 to >=1.1,<2 ([#32](https://github.com/zurfjereluhmie/evalwire/issues/32)) ([ebab441](https://github.com/zurfjereluhmie/evalwire/commit/ebab441141c01ccd07cc9d533f76f95f8d7718d9))
+* overwrite mode now deletes existing dataset before re-creating ([#39](https://github.com/zurfjereluhmie/evalwire/issues/39)) ([eab726c](https://github.com/zurfjereluhmie/evalwire/commit/eab726c873a6a3fa8d314376e9ca00f7ba9b4b4c))
+* resolve mutable defaults, narrow exception handling, and sys.modules leak ([#33](https://github.com/zurfjereluhmie/evalwire/issues/33)) ([788c642](https://github.com/zurfjereluhmie/evalwire/commit/788c642bbde795291f947110be1c653d89f322b9))
+
+## [0.3.1](https://github.com/zurfjereluhmie/evalwire/compare/v0.3.0...v0.3.1) (2026-03-31)
+
+
+### Documentation
+
+* update PyPI version badge to include cache parameter ([29791dd](https://github.com/zurfjereluhmie/evalwire/commit/29791dd70013b31ece21a7ed3aad534d5a6df748))
+
+## [0.3.0](https://github.com/zurfjereluhmie/evalwire/compare/v0.2.2...v0.3.0) (2026-03-31)
+
+
+### Features
+
+* add 7 evaluator factories with tests and updated dependencies ([#13](https://github.com/zurfjereluhmie/evalwire/issues/13)) ([91b493e](https://github.com/zurfjereluhmie/evalwire/commit/91b493e244d8c83e55f7bb88b53e9770867cfaef))
+* add verified PyPI project URLs ([#17](https://github.com/zurfjereluhmie/evalwire/issues/17)) ([950f3e6](https://github.com/zurfjereluhmie/evalwire/commit/950f3e6c5f3751bf7fc4c65114241f5e1607d999))
+
+
+### Documentation
+
+* update README and docs to cover all 9 evaluator factories ([#16](https://github.com/zurfjereluhmie/evalwire/issues/16)) ([5a82c1d](https://github.com/zurfjereluhmie/evalwire/commit/5a82c1d3c7d2d1ba38fa7a46cfd600a47151949e))
+
+## [0.2.2](https://github.com/zurfjereluhmie/evalwire/compare/v0.2.1...v0.2.2) (2026-03-30)
+
+
+### Bug Fixes
+
+* use per-thread persistent event loop to prevent 'Event loop is closed' ([86f97ba](https://github.com/zurfjereluhmie/evalwire/commit/86f97ba73e080f5c95dceaa93c6d8c33b875ee34))
+
+## [0.2.1](https://github.com/zurfjereluhmie/evalwire/compare/v0.2.0...v0.2.1) (2026-03-30)
+
+
+### Bug Fixes
+
+* wrap async tasks for sync Phoenix client ([#5](https://github.com/zurfjereluhmie/evalwire/issues/5)) ([9d370cc](https://github.com/zurfjereluhmie/evalwire/commit/9d370cc1c4a24a7721ddfc1274be0e123553594e))
+
+## [0.2.0](https://github.com/zurfjereluhmie/evalwire/compare/v0.1.0...v0.2.0) (2026-03-30)
+
+
+### Features
+
+* add pytest and pytest-mock to dev dependencies ([6e55a38](https://github.com/zurfjereluhmie/evalwire/commit/6e55a38365324bce8937a5470d99ad5978e38cdb))
+* add runtime dependencies, extras, and CLI entry point ([a97d4f6](https://github.com/zurfjereluhmie/evalwire/commit/a97d4f6420213cc63b24a241e8d586b2b8887334))
+* **demo:** auto-load .env via python-dotenv in run.py ([2e996b1](https://github.com/zurfjereluhmie/evalwire/commit/2e996b1594dbcc300905dafd9ae1ae07a5f693f0))
+* expose public API in package __init__ ([c8828ce](https://github.com/zurfjereluhmie/evalwire/commit/c8828ce6487b94787b567dff23fe700b28179dd8))
+* implement built-in evaluators (top_k and membership) ([d819763](https://github.com/zurfjereluhmie/evalwire/commit/d8197634cdfb1ed2b511147bf6972af957e23e3a))
+* implement CLI upload and run commands ([241d614](https://github.com/zurfjereluhmie/evalwire/commit/241d614e172598e4aa5cc3337d8062a07434e081))
+* implement DatasetUploader ([c2f587f](https://github.com/zurfjereluhmie/evalwire/commit/c2f587fbbc34aaccc2a1717852f9ac7c59866278))
+* implement ExperimentRunner with auto-discovery ([b2cba1d](https://github.com/zurfjereluhmie/evalwire/commit/b2cba1dd61c9a6491d183ae15af4f9cbcca7db05))
+* implement LangGraph node isolation helpers ([2f7c62a](https://github.com/zurfjereluhmie/evalwire/commit/2f7c62a68980d5048618668d90978e02e052d708))
+* implement setup_observability ([3b9d67a](https://github.com/zurfjereluhmie/evalwire/commit/3b9d67a3bb29472111a6bca5423fb1a3701553e7))
+* implement TOML config loader ([b877f2f](https://github.com/zurfjereluhmie/evalwire/commit/b877f2f8e0a0b288d8f63828b08eb296383ab36b))
+* initialize project structure with essential files and configurations ([668cbff](https://github.com/zurfjereluhmie/evalwire/commit/668cbfff7f4a6a689f9a57233dbfca3541a26c9b))
+* replace demo/requirements.txt with demo dependency-group in pyproject.toml ([3c3b67e](https://github.com/zurfjereluhmie/evalwire/commit/3c3b67ef7be2ac9483546a20954552e4942889e9))
+* **runner:** implement concurrency via ThreadPoolExecutor and auto-create __init__.py ([763c53c](https://github.com/zurfjereluhmie/evalwire/commit/763c53c29caf9e3693661878622b70ac7f3bca4d))
+
+
+### Bug Fixes
+
+* align uploader and runner to phoenix.Client flat API (>=13) ([b768011](https://github.com/zurfjereluhmie/evalwire/commit/b768011e10732b838f8ff4a4ad5e299468dedde9))
+* **ci:** suppress unresolved-import for test_task_async.py in ty.toml ([1ae7cbc](https://github.com/zurfjereluhmie/evalwire/commit/1ae7cbce388f2c56b37e2ab49bab601011c77ad8))
+* **error-handling:** log exc_info on swallowed exceptions in uploader and runner ([b82475a](https://github.com/zurfjereluhmie/evalwire/commit/b82475a86d3741c64bd241e84a4b49b51c4a9c8b))
+* **evaluators:** guard top_k against None output when task failed ([a4b96d8](https://github.com/zurfjereluhmie/evalwire/commit/a4b96d854da0cf0da19939fe8323136aaec6da6d))
+* **langgraph:** annotate build_subgraph return type as CompiledStateGraph via TYPE_CHECKING ([f512d54](https://github.com/zurfjereluhmie/evalwire/commit/f512d54fe8e92728cb8130b409425a6461ff8002))
+* **runner:** switch to client.experiments.run_experiment namespaced API ([5eb8e63](https://github.com/zurfjereluhmie/evalwire/commit/5eb8e637df7d2f5d3b1b5c06ae437e864a7c07cb))
+* **types:** resolve all ty type errors across package and tests ([c0314df](https://github.com/zurfjereluhmie/evalwire/commit/c0314df476efb80b04dceeafec24b11f8dc9d509))
+* **typing:** annotate setup_observability return as TracerProvider and tighten dict types ([0bc67a3](https://github.com/zurfjereluhmie/evalwire/commit/0bc67a3226bebd04298889b2d658c17b98350d17))
+* **uploader:** switch to client.datasets.* namespaced API and fix overwrite delete step ([f51d6d8](https://github.com/zurfjereluhmie/evalwire/commit/f51d6d8153b94a2f2717ae6788bc70e86d921dd0))
+* **uploader:** use explicit list defaults for input_keys and output_keys ([b71280d](https://github.com/zurfjereluhmie/evalwire/commit/b71280d205c3e0d35c81be08264252457b0da7ee))
+* **uploader:** use real Phoenix 13.x API and type phoenix_client as Client ([7b7b465](https://github.com/zurfjereluhmie/evalwire/commit/7b7b4658c21a68f47bc1561aa4b3ec1fe7824764))
+* use is_string_dtype to support pandas 3.x StringDtype in _load_csv ([01c17ed](https://github.com/zurfjereluhmie/evalwire/commit/01c17ed398be86fe832cf42b09c2e038fe2ab563))
+
+
+### Documentation
+
+* add lazy-import comment to _make_client and export build_subgraph/invoke_node ([519a7b4](https://github.com/zurfjereluhmie/evalwire/commit/519a7b4a72263db51e615fe45fbf80fdebe96c45))
+* add MkDocs setup with Material theme, mkdocstrings, and make targets ([5bfd1a2](https://github.com/zurfjereluhmie/evalwire/commit/5bfd1a235fd2ce581ad58698ca129b97c9b76da8))
+* write README and quick-start guide ([f21a259](https://github.com/zurfjereluhmie/evalwire/commit/f21a25990eb09ac3b6d4c3b6390bedf13e49a27a))

docs/concepts.md

@@ -0,0 +1,102 @@
+# Concepts
+
+This page explains the mental model behind evalwire before you dive into the quick start or API reference.
+
+## The four building blocks
+
+### Datasets
+
+A **dataset** is a named collection of test examples stored in Arize Phoenix. Each example has input fields (e.g. `user_query`) and expected-output fields (e.g. `expected_output`). evalwire creates datasets by uploading a CSV testset and splitting it by a tag column.
+
+One CSV row can belong to multiple datasets. Just pipe-delimit the tag value:
+
+```
+user_query,expected_output,tags
+"find cycling paths","url-a | url-b","es_search | source_router"
+```
+
+This row appears in both the `es_search` and `source_router` datasets.
+
+### Experiments
+
+An **experiment** is the result of running a task against a dataset and scoring each output with one or more evaluators. Phoenix records every experiment run with a timestamp so you can compare results across code changes.
+
+evalwire discovers experiments by scanning a directory. Each subdirectory that contains a `task.py` is treated as one experiment. The directory name must match a Phoenix dataset name:
+
+```
+experiments/
+  es_search/         ← matched to the "es_search" Phoenix dataset
+    task.py
+    top_k.py         ← evaluator
+  source_router/     ← matched to the "source_router" Phoenix dataset
+    task.py
+    is_in.py         ← evaluator
+```
+
+### Tasks
+
+A **task** is an `async` function that receives a Phoenix example object and returns the output to be scored. Its job is to call your system under test and return a result in whatever form your evaluators expect.
+
+```python
+# experiments/es_search/task.py
+async def task(example):
+    user_query = example.input["user_query"]
+    return await my_retrieval_function(user_query)
+```
+
+The function must be named `task`.
+
+### Evaluators
+
+An **evaluator** is a plain callable with the signature:
+
+```python
+def evaluator(output, expected: dict) -> float | bool: ...
+```
+
+Each evaluator file in an experiment directory is auto-loaded. The callable must share its name with the file:
+
+```python
+# experiments/es_search/top_k.py
+from evalwire.evaluators import make_top_k_evaluator
+
+top_k = make_top_k_evaluator(K=5)
+```
+
+Return `float` (0.0–1.0) for graded scoring or `bool` for pass/fail. Phoenix displays both.
+
+## The experiment lifecycle
+
+```
+CSV testset
+    │
+    │  evalwire upload
+    ▼
+Phoenix Datasets  (one per unique tag)
+    │
+    │  evalwire run
+    ▼
+Task function  (called once per example)
+    │
+    ▼
+Output
+    │
+    │  evaluator(output, expected)
+    ▼
+Experiment Results  (stored in Phoenix, visible in UI)
+```
+
+1. **Upload**: `evalwire upload` reads your CSV, groups rows by the tag column, and creates or updates one Phoenix dataset per unique tag value.
+2. **Run**: `evalwire run` scans the experiments directory, matches each subdirectory to a Phoenix dataset by name, calls `task` on every example, and scores the output with each evaluator file found in that directory.
+3. **Compare**: open the Phoenix UI, navigate to the dataset, and switch to the **Experiments** tab to compare runs side-by-side.
+
+## How `expected` is structured
+
+Inside every evaluator, the `expected` parameter is a dict containing all output columns from the original CSV row. The most important key is `"expected_output"`, which evalwire parses with `_parse_expected`:
+
+- Plain string `"answer"` → `["answer"]`
+- Pipe-delimited string `"a | b"` → `["a", "b"]`
+- Python-literal string `"['a', 'b']"` → `["a", "b"]`
+- Already a list → returned as-is
+
+Most built-in evaluators read `expected["expected_output"]` for you. When writing a custom evaluator you can also access any other column directly: `expected["my_column"]`.