Enhancement: Standardize example structure and add tests

[teilomillet]

Goal

Ensure all examples follow consistent structure and have tests.

Motivation

• Inconsistent structure makes examples hard to navigate
• Examples without tests can break silently
• Need standard way to verify examples work

Standard Example Structure

Minimal Examples (10-20 lines)

examples/basics/
├── minimal_budget_clamp.py
├── minimal_budget_exceeded.py
└── minimal_p95_logging.py

• Single file
• Short docstring
• Runnable as-is
• No tests needed (too simple)

Concept Examples (50-100 lines)

examples/concepts/budget_hardstop/
├── demo.py              # Main example
├── README.md            # What it demonstrates
└── test_demo.py         # Basic smoke test

• One concept, well explained
• README with learning goals
• Smoke test verifies it runs

Production Examples (100+ lines)

examples/production/invoice_processing/
├── README.md            # Setup, quick start
├── architecture.md      # Design decisions
├── pipeline.py          # Main logic
├── models.py            # Data models
├── config.py            # Configuration
├── requirements.txt     # Dependencies
├── tests/
│   ├── test_pipeline.py
│   ├── test_models.py
│   └── fixtures/        # Test data
└── docker/              # If applicable
    └── docker-compose.yml

• Production-ready code
• Full test coverage
• Deployable

Tasks

• Define templates for each example type
• Create example template generator script

  ./scripts/new_example.py --type production --name my_example

• Add tests to existing examples
  • budget_hardstop_demo.py
  • typed_outcomes_demo.py
  • job_delegation_demo.py
  • report_service/*
• Add CI job to run all example tests
• Document testing approach in examples/CONTRIBUTING.md
• Migrate examples to new structure (preserve git history)

Testing Strategy

Mock Provider

All examples should run with MockProvider for testing:

# Allow both real and mock
if os.getenv("USE_MOCK"):
    provider = MockProvider(...)
else:
    provider = "openai"

Test Coverage

• Smoke test: example runs without error
• Output validation: result has expected structure
• Budget enforcement: budget limits are respected
• Error cases: graceful handling of failures

CI Integration

# .github/workflows/examples.yml
name: Examples
on: [push, pull_request]
jobs:
  test-examples:
    runs-on: ubuntu-latest
    steps:
      - run: pytest examples/ --mock

Acceptance Criteria

• Clear structure guidelines documented
• All production examples have tests
• CI runs example tests
• Template generator works
• Migration to new structure complete

Estimated Effort

Large (3-4 days)