docs(oss): onboarding audit — first run is blocked by wrong install commands, a nonexistent model id, and fabricated API

[davidberenstein1957]

An onboarding-readiness audit of src/content/docs/oss/** against Giskard-AI/giskard-oss@main (giskard-checks 1.0.2rc1, giskard-scan 1.0.0rc1), optimized for a developer who has never used Giskard and wants a first check running in 10 minutes.

Every claim below was verified by reading the library source or executing against it.

The five things that most hurt a new developer

1. The checks quickstart cannot run, and never says why. It configures Generator(model="openai/gpt-5.4-nano") — a model that appears nowhere in giskard-oss (the library default is openai/gpt-4o-mini, libs/giskard-checks/src/giskard/checks/settings.py:10). The setup cells are stripped from the rendered page, so the reader never learns an API key is needed; the first visible cell dies with WorkflowError: Step processing failed.
2. Five install variants across the tree, two of them broken. oss/solutions/scan-vulnerabilities.mdx omitted --pre, so it installed Giskard 2.19.2 and every sample on the page failed. The notebooks used pip install giskard-checks, which has no candidate because everything shipped is a pre-release.
3. Documented API that does not exist. how-to/simulate-users.ipynb built a whole section on a simulator_output metadata key with zero occurrences in giskard-oss — the notebook's own recorded output proves the branch never fired. Also a phantom Persona class, a phantom Equals(key=...) parameter, and a phantom answer_key.
4. Suite was documented nowhere in how-to or use-cases, so four pages hand-rolled ~400 lines reimplementing it, including a 134-line RAGTestSuite. The CI/CD guide shipped no CI report despite SuiteResult.to_junit_xml existing.
5. No single golden path, and no failure ever shown. Four peer sidebar groups offered four plausible starts, three pages nominated three different first destinations, and /oss linked neither Scan nor Solutions. Every recorded output across 13 notebooks was a pass, and no page documented a single error message.

Also found

• The execution model in explanation/core-concepts.md was wrong in three ways: checks run per step (not per interaction), a spec is an async generator that can yield many interactions, and a failing step marks every later step SKIP.
• MISSING vs None was documented backwards across every static-value field in the checks reference. Passing None is a value and silently disables the *_key fallback.
• The scan install page pointed LiteLLM users at GiskardLLMGenerator, which raises ValueError: Provider 'mistral' is not configured and not in the registry mid-scan. LiteLLM needs LiteLLMGenerator.
• 10 public symbols in giskard.checks.__all__ were documented nowhere; Scenario was missing six of its eleven builder methods, including with_target().
• No troubleshooting page, no migration page, no versioning/stability policy, no security or issue-reporting path, and no cost or latency figure on either golden path.
• scan/how-to/index.mdx omitted three of its eight pages, including wrap-your-agent — the step between the toy tutorial and a real agent.

Not a problem

The mechanical API delta is clean: scripts/snapshot-api.py + scripts/diff-api.py against docs/api-baseline/ show zero real deltas for both packages. Nothing documented has been removed or re-signed. (The 74 raw deltas were X | Y vs Union[X, Y] interpreter noise — fixed separately so the differ stops crying wolf.)

The scan reference and explanation pages were checked symbol by symbol and contained no wrong signature, default, tag, or constant.

Full report with every finding as file:line — problem — fix, and a worklist of 52 PR-sized items ranked P0/P1/P2, is attached to the PR.

[davidberenstein1957]

Correction to finding 1 / §1.2.1 (the gpt-5.4-nano model id).

The audit called this "a model that appears nowhere in giskard-oss" and treated it as a P0 typo to be replaced with openai/gpt-4o-mini. The first half is true; the conclusion was wrong.

CI proved it: replacing it broke six notebooks with DeploymentNotFound. gpt-5.4-nano is the deployment name on the Azure AI proxy that .github/workflows/ci.yml points the notebooks at (OPENAI_BASE_URL: secrets.AZURE_AI_API_BASE), not a public model id. It was never a typo.

Neither id is right on its own — the proxy's name is meaningless to a reader, and a public id does not resolve on the proxy. The resolution in #207: the notebooks name no model at all. giskard-checks already reads GISKARD_CHECKS_DEFAULT_MODEL itself (pydantic env_prefix, settings.py:23) and otherwise uses openai/gpt-4o-mini, so a reader gets a model they can actually run and CI overrides it through a repository variable.

Recording this here because the audit report is not being committed to the repo — it would have shipped with this finding stated wrongly at the top.