Is your feature request related to a problem? Please describe.
Right now RAG failures and weird behaviours are often visible in logs and traces, but they are not mapped to a consistent failure taxonomy. Different teams use different ad-hoc names like “hallucination”, “bad chunks”, “index bug”, “startup race”, etc., so it is hard to compare incidents or to build repeatable playbooks on top of the same patterns.
RagaAI Catalyst is already strong on observability, monitoring and evaluation. What seems to be missing (from a user point of view) is a simple, shared “failure map” that engineers can point to when they say “this incident is of type X”, especially for long-horizon RAG pipelines in production.
Describe the solution you'd like
I would like to suggest adding WFGY 16 Problem Map as an optional RAG failure taxonomy in the docs or examples.
WFGY 16 Problem Map is an open source list of sixteen common failure modes in RAG and LLM infra, each with a short description and concrete structural fix. It covers things like:
- retrieval hallucination even when the retrieved context is correct
- vector store ingestion / schema mistakes
- bad chunking vs embedding contracts
- bootstrap / startup ordering issues between API and vector DB
- secrets / config drift around first deploy
- memory / multi-agent / reasoning collapse patterns
Repo: https://github.com/onestardao/WFGY
Problem Map: https://github.com/onestardao/WFGY/tree/main/ProblemMap/README.md
In Catalyst, this could be as simple as:
- a short “RAG failure taxonomy” or “Troubleshooting” section mentioning WFGY 16 Problem Map, and/or
- an example notebook where a run is annotated with one of the 16 Problem Map numbers (No.1–No.16) when an incident is investigated.
This gives teams a shared vocabulary on top of the existing observability / evaluation features.
Describe alternatives you've considered
Alternatives are:
- every team invents its own internal taxonomy for RAG failures, which makes it harder to share knowledge or reuse playbooks across projects, or
- relying only on metrics / traces without a named failure map, which helps with debugging a single incident but not with building a long-term library of patterns.
I also considered proposing a brand new taxonomy just for RagaAI, but WFGY 16 Problem Map already exists as an independent MIT-licensed project with its own users and stars, so reusing that vocabulary seems simpler and more interoperable.
Additional context
WFGY 16 Problem Map is MIT-licensed, text-only, and lives entirely in a public GitHub repo (no external services). The intention is not to push a tight integration or any vendor lock-in, only to offer a compact vocabulary for “what went wrong” in RAG systems that can sit next to Catalyst’s existing observability and evaluation features.
If you feel this is in scope, I am happy to open a very small PR adding a one-line reference in whichever doc section you consider appropriate. If it is out of scope for this project, no worries at all – I mainly wanted to ask before sending a PR.
Is your feature request related to a problem? Please describe.
Right now RAG failures and weird behaviours are often visible in logs and traces, but they are not mapped to a consistent failure taxonomy. Different teams use different ad-hoc names like “hallucination”, “bad chunks”, “index bug”, “startup race”, etc., so it is hard to compare incidents or to build repeatable playbooks on top of the same patterns.
RagaAI Catalyst is already strong on observability, monitoring and evaluation. What seems to be missing (from a user point of view) is a simple, shared “failure map” that engineers can point to when they say “this incident is of type X”, especially for long-horizon RAG pipelines in production.
Describe the solution you'd like
I would like to suggest adding WFGY 16 Problem Map as an optional RAG failure taxonomy in the docs or examples.
WFGY 16 Problem Map is an open source list of sixteen common failure modes in RAG and LLM infra, each with a short description and concrete structural fix. It covers things like:
Repo: https://github.com/onestardao/WFGY
Problem Map: https://github.com/onestardao/WFGY/tree/main/ProblemMap/README.md
In Catalyst, this could be as simple as:
This gives teams a shared vocabulary on top of the existing observability / evaluation features.
Describe alternatives you've considered
Alternatives are:
I also considered proposing a brand new taxonomy just for RagaAI, but WFGY 16 Problem Map already exists as an independent MIT-licensed project with its own users and stars, so reusing that vocabulary seems simpler and more interoperable.
Additional context
WFGY 16 Problem Map is MIT-licensed, text-only, and lives entirely in a public GitHub repo (no external services). The intention is not to push a tight integration or any vendor lock-in, only to offer a compact vocabulary for “what went wrong” in RAG systems that can sit next to Catalyst’s existing observability and evaluation features.
If you feel this is in scope, I am happy to open a very small PR adding a one-line reference in whichever doc section you consider appropriate. If it is out of scope for this project, no worries at all – I mainly wanted to ask before sending a PR.