diff --git a/docs/api/evaluators.md b/docs/api/evaluators.md index 453e9a1..26e7f79 100644 --- a/docs/api/evaluators.md +++ b/docs/api/evaluators.md @@ -1,15 +1,12 @@ # evalwire.evaluators -Built-in evaluator factories. Each factory returns a callable with the standard -evalwire evaluator signature: +Built-in evaluator factories. Each factory returns a callable with the standard evalwire evaluator signature: ```python -def evaluator(output: Any, expected: dict) -> float | bool: ... +def evaluator(output, expected: dict) -> float | bool: ... ``` -The `expected` dict always contains at minimum an `"expected_output"` key whose -value is parsed by the shared `_parse_expected` helper (handles plain strings, -Python-literal strings such as `"['a','b']"`, and lists). +The `expected` dict always contains at minimum an `"expected_output"` key whose value is parsed by the shared `_parse_expected` helper (handles plain strings, Python-literal strings such as `"['a','b']"`, and pipe-delimited strings). All factories are importable directly from `evalwire.evaluators`: @@ -27,6 +24,8 @@ from evalwire.evaluators import ( ) ``` +For a guide on writing your own evaluators, see [Writing Custom Evaluators](../guides/custom-evaluators.md). + --- ## Retrieval diff --git a/docs/api/runner.md b/docs/api/runner.md index 3d2a079..dc6be42 100644 --- a/docs/api/runner.md +++ b/docs/api/runner.md @@ -1,3 +1,90 @@ # evalwire.runner +`ExperimentRunner` auto-discovers experiment subdirectories, fetches the matching Phoenix dataset for each one, and runs the task against every example. Results are stored in Phoenix and returned as a list. + +## Basic usage + +```python +from phoenix.client import Client +from evalwire.runner import ExperimentRunner + +client = Client() + +runner = ExperimentRunner( + experiments_dir="experiments", + phoenix_client=client, + concurrency=2, +) +results = runner.run() +``` + +## Directory layout + +Each subdirectory of `experiments_dir` that contains a `task.py` is treated as one experiment. The directory name must exactly match a Phoenix dataset name: + +``` +experiments/ + es_search/ <- matches dataset named "es_search" + task.py <- must define async def task(example) + top_k.py <- evaluator: must define top_k = ... + exact_match.py <- evaluator: must define exact_match = ... + source_router/ <- matches dataset named "source_router" + task.py + is_in.py +``` + +Subdirectories without `task.py` are silently skipped. Files (non-directories) at the top level of `experiments_dir` are also skipped. + +## Experiment naming + +Each run in Phoenix is named `{prefix}_{dataset_name}_{iso_timestamp}`. The default prefix is `"eval"`. Override it with `experiment_name_prefix`: + +```python +results = runner.run(experiment_name_prefix="nightly") +# produces e.g. "nightly_es_search_2025-01-15T09:30:00" +``` + +## Running a subset + +Pass `names` to run only specific experiments: + +```python +results = runner.run(names=["es_search"]) +``` + +## Dry run + +Set `dry_run=True` (or `dry_run=N` to limit to N examples) to execute tasks without uploading results to Phoenix. Useful for smoke-testing your task code: + +```python +runner = ExperimentRunner( + experiments_dir="experiments", + phoenix_client=client, + dry_run=3, +) +runner.run() +``` + +## Async tasks + +Tasks are `async` functions. evalwire wraps them in a per-thread event loop so Phoenix's synchronous runner can call them. The loop is kept open between calls (unlike `asyncio.run()`) so that async I/O libraries which reuse connections across calls work correctly. + +## Error behaviour + +If any experiment fails (dataset not found, task raises, evaluator raises), `runner.run()` raises `SystemExit(1)` after all experiments complete so that CI pipelines fail loudly. Successful experiments are still returned. + +## Pitfalls + +- The dataset name must match the directory name exactly (case-sensitive). +- At least one evaluator file is required per experiment. Phoenix raises an error if `run_experiment` is called with an empty evaluator list. +- Relative imports inside experiment modules work because evalwire adds the parent of `experiments_dir` to `sys.path` during discovery. It is removed again afterwards. + +## See also + +- [Concepts](../concepts.md) for the experiment lifecycle +- [Configuration Reference](../configuration.md) for `evalwire.toml` keys +- [CLI Reference](cli.md) for `evalwire run` + +--- + ::: evalwire.runner diff --git a/docs/api/uploader.md b/docs/api/uploader.md index 39aa494..527aaa9 100644 --- a/docs/api/uploader.md +++ b/docs/api/uploader.md @@ -1,3 +1,54 @@ # evalwire.uploader +`DatasetUploader` reads a CSV testset and uploads it to Arize Phoenix as one named dataset per unique tag value. It handles three conflict modes (`skip`, `overwrite`, `append`) and supports multi-tag rows via a configurable delimiter. + +## Basic usage + +```python +from phoenix.client import Client +from evalwire.uploader import DatasetUploader + +client = Client() + +uploader = DatasetUploader( + csv_path="data/testset.csv", + phoenix_client=client, +) +datasets = uploader.upload(on_exist="skip") +print(datasets) # {"es_search": , "source_router": } +``` + +## CSV format + +The CSV must contain at least a tag column, one input column, and one expected-output column: + +```csv +user_query,expected_output,tags +"find cycling paths","url-a | url-b","es_search | source_router" +"find parks","url-c","es_search" +``` + +Pipe-delimited values in any column are split into lists. A row with `tags = "es_search | source_router"` is added to both datasets. + +## Conflict modes + +| `on_exist` | Behaviour | +|---|---| +| `"skip"` | Do nothing if the dataset already exists. | +| `"overwrite"` | Delete the existing dataset and re-create it. | +| `"append"` | Call `add_examples_to_dataset` on the existing dataset. If not found, create it. | + +## Pitfalls + +- Phoenix raises `ValueError` (not a Phoenix-specific exception) when `get_dataset` is called for a non-existent dataset. evalwire catches this and creates the dataset instead. +- There is no official delete method in the Phoenix Python client. evalwire calls the REST endpoint `DELETE /v1/datasets/{id}` directly for the `overwrite` mode. +- Creating a dataset with a name that already exists returns a 409 Conflict error, not a new version. Use `on_exist="overwrite"` to replace it. + +## See also + +- [Configuration Reference](../configuration.md) for `evalwire.toml` keys +- [CLI Reference](cli.md) for `evalwire upload` + +--- + ::: evalwire.uploader diff --git a/docs/changelog.md b/docs/changelog.md new file mode 100644 index 0000000..67259da --- /dev/null +++ b/docs/changelog.md @@ -0,0 +1,3 @@ +{% + include-markdown "../CHANGELOG.md" +%} diff --git a/docs/concepts.md b/docs/concepts.md new file mode 100644 index 0000000..2a42d10 --- /dev/null +++ b/docs/concepts.md @@ -0,0 +1,102 @@ +# Concepts + +This page explains the mental model behind evalwire before you dive into the quick start or API reference. + +## The four building blocks + +### Datasets + +A **dataset** is a named collection of test examples stored in Arize Phoenix. Each example has input fields (e.g. `user_query`) and expected-output fields (e.g. `expected_output`). evalwire creates datasets by uploading a CSV testset and splitting it by a tag column. + +One CSV row can belong to multiple datasets. Just pipe-delimit the tag value: + +``` +user_query,expected_output,tags +"find cycling paths","url-a | url-b","es_search | source_router" +``` + +This row appears in both the `es_search` and `source_router` datasets. + +### Experiments + +An **experiment** is the result of running a task against a dataset and scoring each output with one or more evaluators. Phoenix records every experiment run with a timestamp so you can compare results across code changes. + +evalwire discovers experiments by scanning a directory. Each subdirectory that contains a `task.py` is treated as one experiment. The directory name must match a Phoenix dataset name: + +``` +experiments/ + es_search/ ← matched to the "es_search" Phoenix dataset + task.py + top_k.py ← evaluator + source_router/ ← matched to the "source_router" Phoenix dataset + task.py + is_in.py ← evaluator +``` + +### Tasks + +A **task** is an `async` function that receives a Phoenix example object and returns the output to be scored. Its job is to call your system under test and return a result in whatever form your evaluators expect. + +```python +# experiments/es_search/task.py +async def task(example): + user_query = example.input["user_query"] + return await my_retrieval_function(user_query) +``` + +The function must be named `task`. + +### Evaluators + +An **evaluator** is a plain callable with the signature: + +```python +def evaluator(output, expected: dict) -> float | bool: ... +``` + +Each evaluator file in an experiment directory is auto-loaded. The callable must share its name with the file: + +```python +# experiments/es_search/top_k.py +from evalwire.evaluators import make_top_k_evaluator + +top_k = make_top_k_evaluator(K=5) +``` + +Return `float` (0.0–1.0) for graded scoring or `bool` for pass/fail. Phoenix displays both. + +## The experiment lifecycle + +``` +CSV testset + │ + │ evalwire upload + ▼ +Phoenix Datasets (one per unique tag) + │ + │ evalwire run + ▼ +Task function (called once per example) + │ + ▼ +Output + │ + │ evaluator(output, expected) + ▼ +Experiment Results (stored in Phoenix, visible in UI) +``` + +1. **Upload**: `evalwire upload` reads your CSV, groups rows by the tag column, and creates or updates one Phoenix dataset per unique tag value. +2. **Run**: `evalwire run` scans the experiments directory, matches each subdirectory to a Phoenix dataset by name, calls `task` on every example, and scores the output with each evaluator file found in that directory. +3. **Compare**: open the Phoenix UI, navigate to the dataset, and switch to the **Experiments** tab to compare runs side-by-side. + +## How `expected` is structured + +Inside every evaluator, the `expected` parameter is a dict containing all output columns from the original CSV row. The most important key is `"expected_output"`, which evalwire parses with `_parse_expected`: + +- Plain string `"answer"` → `["answer"]` +- Pipe-delimited string `"a | b"` → `["a", "b"]` +- Python-literal string `"['a', 'b']"` → `["a", "b"]` +- Already a list → returned as-is + +Most built-in evaluators read `expected["expected_output"]` for you. When writing a custom evaluator you can also access any other column directly: `expected["my_column"]`. diff --git a/docs/configuration.md b/docs/configuration.md new file mode 100644 index 0000000..6d57ce7 --- /dev/null +++ b/docs/configuration.md @@ -0,0 +1,90 @@ +# Configuration Reference + +evalwire can be driven entirely from a `evalwire.toml` file at the root of your project. CLI flags always take precedence over config file values, which take precedence over hardcoded defaults. + +## Precedence + +``` +CLI flag > evalwire.toml > hardcoded default +``` + +## Example file + +```toml +[dataset] +csv_path = "data/testset.csv" +on_exist = "skip" +input_keys = ["user_query"] +output_keys = ["expected_output"] +tag_column = "tags" +delimiter = "|" + +[experiments] +dir = "experiments" +prefix = "eval" +concurrency = 4 +``` + +## `[dataset]` section + +Used by `evalwire upload`. + +| Key | Type | Default | Description | +|---|---|---|---| +| `csv_path` | string | none (required) | Path to the CSV testset file. | +| `on_exist` | `"skip"` / `"overwrite"` / `"append"` | `"skip"` | How to handle a dataset that already exists in Phoenix. | +| `input_keys` | list of strings | `["user_query"]` | CSV column names treated as example inputs. | +| `output_keys` | list of strings | `["expected_output"]` | CSV column names treated as expected outputs. | +| `tag_column` | string | `"tags"` | CSV column used to split rows into separate datasets. | +| `delimiter` | string | `"\|"` | Character used to assign a row to multiple datasets. | + +### `on_exist` modes + +| Value | Behaviour | +|---|---| +| `skip` | Leave the existing dataset untouched. | +| `overwrite` | Delete the existing dataset and re-create it from the CSV. | +| `append` | Add the new rows to the existing dataset. | + +## `[experiments]` section + +Used by `evalwire run`. + +| Key | Type | Default | Description | +|---|---|---|---| +| `dir` | string | `"experiments"` | Path to the experiments directory. | +| `prefix` | string | `"eval"` | Prefix prepended to every experiment name in Phoenix. | +| `concurrency` | integer | `1` | Number of experiments to run in parallel. | + +## `[phoenix]` section + +Reserved for future use. Currently read by `get_phoenix_config()` but not consumed by the CLI. + +## CLI flag reference + +### `evalwire upload` + +| Flag | Config key | Default | +|---|---|---| +| `--csv PATH` | `dataset.csv_path` | none | +| `--on-exist MODE` | `dataset.on_exist` | `skip` | +| `--input-keys COLS` | `dataset.input_keys` | `user_query` | +| `--output-keys COLS` | `dataset.output_keys` | `expected_output` | +| `--tag-column COL` | `dataset.tag_column` | `tags` | +| `--delimiter CHAR` | `dataset.delimiter` | `\|` | +| `--config PATH` | n/a | `./evalwire.toml` | + +### `evalwire run` + +| Flag | Config key | Default | +|---|---|---| +| `--experiments PATH` | `experiments.dir` | `experiments` | +| `--name NAME` | n/a | all experiments | +| `--prefix PREFIX` | `experiments.prefix` | `eval` | +| `--concurrency N` | `experiments.concurrency` | `1` | +| `--dry-run [N]` | n/a | off | +| `--config PATH` | n/a | `./evalwire.toml` | + +`--name` is repeatable: `evalwire run --name es_search --name source_router`. + +`--dry-run` runs the task but does not upload results to Phoenix. Optionally accepts a number to limit the run to the first N examples per dataset. diff --git a/docs/guides/custom-evaluators.md b/docs/guides/custom-evaluators.md new file mode 100644 index 0000000..dbeb501 --- /dev/null +++ b/docs/guides/custom-evaluators.md @@ -0,0 +1,135 @@ +# Writing Custom Evaluators + +evalwire evaluators are plain Python callables. This guide covers the full contract, common patterns, and best practices. + +## The evaluator contract + +Every evaluator must follow this signature: + +```python +def evaluator_name(output, expected: dict) -> float | bool: ... +``` + +- `output` is whatever your `task` function returned for a given example. +- `expected` is a dict of all output columns from the original CSV row. The key `"expected_output"` is always present. +- Return `float` (0.0 to 1.0) for graded scoring or `bool` for pass/fail. + +## The `expected` dict + +`expected` contains every output column from the CSV row. For a CSV like: + +```csv +user_query,expected_output,tags +"find cycling paths","url-a | url-b","es_search" +``` + +`expected` will be: + +```python +{"expected_output": ["url-a", "url-b"]} +``` + +evalwire parses `expected_output` through `_parse_expected` before passing it to evaluators: + +| CSV value | Parsed result | +|---|---| +| `"answer"` | `["answer"]` | +| `"a \| b"` | `["a", "b"]` | +| `"['a', 'b']"` | `["a", "b"]` | +| already a list | unchanged | + +You can also add extra columns to the CSV and read them directly from `expected`: + +```python +def my_evaluator(output, expected: dict) -> bool: + threshold = float(expected.get("score_threshold", 0.5)) + return float(output) >= threshold +``` + +## When to return `float` vs `bool` + +Use `float` when the quality of the output is graded (e.g. how many correct items were retrieved). Use `bool` when the output is simply correct or incorrect. + +Phoenix displays both types. Float scores are averaged across examples; bool scores are shown as a pass rate. + +## File and name conventions + +Each evaluator lives in its own file inside the experiment directory. The callable must share the name of the file (without `.py`): + +``` +experiments/ + es_search/ + task.py + top_k.py <- must define a callable named `top_k` + exact_match.py <- must define a callable named `exact_match` +``` + +Multiple evaluators per experiment are supported: just add more files. + +## Using a built-in factory + +The simplest approach is to assign the factory's return value at module level: + +```python +# experiments/es_search/top_k.py +from evalwire.evaluators import make_top_k_evaluator + +top_k = make_top_k_evaluator(K=5) +``` + +All nine factories are importable from `evalwire.evaluators`. See the [Evaluators API reference](../api/evaluators.md) for full signatures. + +## Writing a custom function + +For use cases not covered by the built-ins, write a plain function: + +```python +# experiments/es_search/recall.py + +def recall(output: list[str], expected: dict) -> float: + expected_items = set(expected.get("expected_output", [])) + if not expected_items: + return 0.0 + hits = sum(1 for item in output if item in expected_items) + return hits / len(expected_items) +``` + +## Composing multiple evaluators + +Run several evaluators on the same experiment by adding one file per evaluator: + +``` +experiments/ + es_search/ + task.py + recall.py + precision.py + exact_match.py +``` + +All three will appear as separate score columns in the Phoenix experiment view. + +## Error handling + +If your evaluator raises an exception for a single example, the experiment run fails for that example. Guard against bad output types to keep a run from aborting mid-way: + +```python +def recall(output, expected: dict) -> float: + if not isinstance(output, list): + return 0.0 + expected_items = set(expected.get("expected_output", [])) + if not expected_items: + return 0.0 + return sum(1 for item in output if item in expected_items) / len(expected_items) +``` + +For the LLM judge specifically, the `on_error` parameter controls behaviour when the model call fails: + +```python +llm_judge = make_llm_judge_evaluator( + model=model, + prompt_template=PROMPT, + output_schema=Verdict, + on_error="silent", # returns 0.0 / False on failure instead of raising +) +``` diff --git a/docs/index.md b/docs/index.md index e984829..3b22dd4 100644 --- a/docs/index.md +++ b/docs/index.md @@ -27,5 +27,10 @@ ## Navigation -- [Quick Start](quick-start.md) — get up and running in minutes -- [API Reference](api/index.md) — full module documentation +- [Quick Start](quick-start.md): get up and running in minutes +- [Concepts](concepts.md): understand datasets, experiments, tasks, and evaluators +- [Guides: Writing Custom Evaluators](guides/custom-evaluators.md): evaluator contract, patterns, and best practices +- [Configuration](configuration.md): full `evalwire.toml` reference +- [Troubleshooting](troubleshooting.md): common errors and fixes +- [API Reference](api/index.md): full module documentation +- [Changelog](changelog.md): version history diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md new file mode 100644 index 0000000..fe28e3e --- /dev/null +++ b/docs/troubleshooting.md @@ -0,0 +1,86 @@ +# Troubleshooting + +## "RuntimeError: Event loop is closed" + +This happens when an async task is called multiple times and the event loop created by the first call was closed before the next call. evalwire uses a per-thread persistent event loop (since v0.2.2) to prevent this. If you see it: + +- Make sure you are on evalwire `>=0.2.2`. +- Do not call `asyncio.run()` inside your task function. Use `await` directly. + +## Phoenix connection errors + +**Symptom:** `ConnectionRefusedError` or `httpx.ConnectError` when running `evalwire upload` or `evalwire run`. + +**Causes and fixes:** + +1. Phoenix is not running. Start it with `docker compose up -d` or follow the [Phoenix docs](https://docs.arize.com/phoenix). +2. The wrong endpoint is set. evalwire uses the `PHOENIX_COLLECTOR_ENDPOINT` environment variable. Check it points to your Phoenix instance: + ```bash + export PHOENIX_COLLECTOR_ENDPOINT=http://localhost:6006 + ``` +3. A firewall or Docker network is blocking the port. + +## "No CSV path provided" + +``` +Error: No CSV path provided. Use --csv or set csv_path in evalwire.toml. +``` + +Either pass `--csv data/testset.csv` on the command line or add this to `evalwire.toml`: + +```toml +[dataset] +csv_path = "data/testset.csv" +``` + +## Dataset not found during `evalwire run` + +``` +SystemExit: 1 +``` + +evalwire looks up each experiment directory name as a Phoenix dataset. If the dataset does not exist, the run fails. Make sure you have run `evalwire upload` first and that the directory name matches the dataset tag exactly (case-sensitive). + +Run a subset to narrow down which experiment is failing: + +```bash +evalwire run --name my_experiment +``` + +## Import errors for optional extras + +**`ImportError: langgraph is required`** + +Install the langgraph extra: + +```bash +pip install 'evalwire[langgraph]' +``` + +**`ImportError: langchain-core is required to use make_llm_judge_evaluator`** + +Install the llm-judge extra: + +```bash +pip install 'evalwire[llm-judge]' +``` + +## CSV formatting issues + +**Rows appear in the wrong dataset or not at all.** + +- Check that the tag column name matches `tag_column` (default: `"tags"`). Use `--tag-column` or set it in `evalwire.toml`. +- Check that the delimiter matches `delimiter` (default: `"|"`). If your values contain pipes for other reasons, choose a different delimiter and set `--delimiter`. +- Rows with an empty tag cell are skipped silently. + +**`expected_output` is not parsed correctly.** + +The `_parse_expected` helper uses `ast.literal_eval` on string values. This means numeric-looking strings like `"0"` or `"1.5"` are converted to Python numeric types (`0`, `1.5`), which can cause type mismatches in string-based evaluators. Use distinct string values for expected outputs when possible. + +## "At least one evaluator is required" + +Phoenix raises this when `run_experiment` is called with an empty evaluator list. Make sure each experiment directory contains at least one `.py` file (other than `task.py`) that defines a callable matching the filename. + +## Experiment name conflicts + +Phoenix experiment names include a timestamp, so duplicate names are unlikely. If you need to re-run the same experiment, the old results are preserved and the new run appears alongside it in the UI. diff --git a/mkdocs.yml b/mkdocs.yml index 3847768..79f5bce 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -32,6 +32,7 @@ theme: plugins: - search + - include-markdown - mkdocstrings: handlers: python: @@ -47,6 +48,11 @@ plugins: nav: - Home: index.md - Quick Start: quick-start.md + - Concepts: concepts.md + - Guides: + - Writing Custom Evaluators: guides/custom-evaluators.md + - Configuration: configuration.md + - Troubleshooting: troubleshooting.md - API Reference: - evalwire: api/index.md - Evaluators: api/evaluators.md @@ -56,6 +62,7 @@ nav: - Config: api/config.md - CLI: api/cli.md - LangGraph: api/langgraph.md + - Changelog: changelog.md markdown_extensions: - pymdownx.highlight: diff --git a/pyproject.toml b/pyproject.toml index a38b92c..177193b 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -80,6 +80,7 @@ do_not_mutate = ["*__init__.py", "*cli.py"] dev = [ "hypothesis>=6.152.1", "mkdocs>=1.6.1", + "mkdocs-include-markdown-plugin>=7.2.2", "mkdocs-material>=9.0", "mkdocstrings[python]>=0.25", "mutmut>=3.5", diff --git a/uv.lock b/uv.lock index dae98d5..b22789a 100644 --- a/uv.lock +++ b/uv.lock @@ -501,6 +501,15 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/bb/52/f57ded73f1527a18e0712281eb49c4ae240038bb4dc7083fd288b4adc811/botocore-1.43.2-py3-none-any.whl", hash = "sha256:b823454d751a1c24bb403b5b07ab65007689654abb21787df923684e0743976c", size = 14982693, upload-time = "2026-05-01T19:42:54.602Z" }, ] +[[package]] +name = "bracex" +version = "2.6" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/63/9a/fec38644694abfaaeca2798b58e276a8e61de49e2e37494ace423395febc/bracex-2.6.tar.gz", hash = "sha256:98f1347cd77e22ee8d967a30ad4e310b233f7754dbf31ff3fceb76145ba47dc7", size = 26642, upload-time = "2025-06-22T19:12:31.254Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/9d/2a/9186535ce58db529927f6cf5990a849aa9e052eea3e2cfefe20b9e1802da/bracex-2.6-py3-none-any.whl", hash = "sha256:0b0049264e7340b3ec782b5cb99beb325f36c3782a32e36e876452fd49a09952", size = 11508, upload-time = "2025-06-22T19:12:29.781Z" }, +] + [[package]] name = "cachetools" version = "7.0.5" @@ -1106,6 +1115,7 @@ demo = [ dev = [ { name = "hypothesis" }, { name = "mkdocs" }, + { name = "mkdocs-include-markdown-plugin" }, { name = "mkdocs-material" }, { name = "mkdocstrings", extra = ["python"] }, { name = "mutmut" }, @@ -1148,6 +1158,7 @@ demo = [ dev = [ { name = "hypothesis", specifier = ">=6.152.1" }, { name = "mkdocs", specifier = ">=1.6.1" }, + { name = "mkdocs-include-markdown-plugin", specifier = ">=7.2.2" }, { name = "mkdocs-material", specifier = ">=9.0" }, { name = "mkdocstrings", extras = ["python"], specifier = ">=0.25" }, { name = "mutmut", specifier = ">=3.5" }, @@ -2617,6 +2628,19 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/88/29/744136411e785c4b0b744d5413e56555265939ab3a104c6a4b719dad33fd/mkdocs_get_deps-0.2.2-py3-none-any.whl", hash = "sha256:e7878cbeac04860b8b5e0ca31d3abad3df9411a75a32cde82f8e44b6c16ff650", size = 9555, upload-time = "2026-03-10T02:46:32.256Z" }, ] +[[package]] +name = "mkdocs-include-markdown-plugin" +version = "7.2.2" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "mkdocs" }, + { name = "wcmatch" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/ad/2d/bdf1aee3f4f7b34148b0f62298b62f03415160cb2707f09503c99a0a7cd5/mkdocs_include_markdown_plugin-7.2.2.tar.gz", hash = "sha256:f052ccb741eccf498116b826c1d78a2d761c56747372594709441cee0963fbc9", size = 25415, upload-time = "2026-03-29T15:15:14.2Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/38/a5/f6b2f0aa805dbda52f6265e9aff1450c8643195442facf29d475bdeba15d/mkdocs_include_markdown_plugin-7.2.2-py3-none-any.whl", hash = "sha256:f2ec4487cf32d3e33ca528f9366f20fb9280ded9c8d1630eb2bbda244962dcd1", size = 29528, upload-time = "2026-03-29T15:15:13.079Z" }, +] + [[package]] name = "mkdocs-material" version = "9.7.6" @@ -5925,6 +5949,18 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/6e/d4/ed38dd3b1767193de971e694aa544356e63353c33a85d948166b5ff58b9e/watchfiles-1.1.1-pp311-pypy311_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:3e6f39af2eab0118338902798b5aa6664f46ff66bc0280de76fca67a7f262a49", size = 457546, upload-time = "2025-10-14T15:06:13.372Z" }, ] +[[package]] +name = "wcmatch" +version = "10.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "bracex" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/79/3e/c0bdc27cf06f4e47680bd5803a07cb3dfd17de84cde92dd217dcb9e05253/wcmatch-10.1.tar.gz", hash = "sha256:f11f94208c8c8484a16f4f48638a85d771d9513f4ab3f37595978801cb9465af", size = 117421, upload-time = "2025-06-22T19:14:02.49Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/eb/d8/0d1d2e9d3fabcf5d6840362adcf05f8cf3cd06a73358140c3a97189238ae/wcmatch-10.1-py3-none-any.whl", hash = "sha256:5848ace7dbb0476e5e55ab63c6bbd529745089343427caa5537f230cc01beb8a", size = 39854, upload-time = "2025-06-22T19:14:00.978Z" }, +] + [[package]] name = "wcwidth" version = "0.7.0"