Skip to content

Commit e0e96fe

Browse files
lesnik512claude
andauthored
docs: fix the improvement tail (P1–P18) (#80)
Close out the convenience/readability/gap tail from the 2026-06-13 docs audit; signatures and defaults verified against source. - P1/P2/P3 subscriber.md: document propagate_inbound_headers, the standard passthrough kwargs, and a retry-strategy params/defaults table. - P4/P5 publisher.md: show the broker.publisher() signature; add a FastAPI-specific callout to the OutboxResponse chained-publish example (the session must outlive the handler return — vanilla apps call broker.publish directly). - P6 index.md: signpost the three observability entries (concept / reference / step-by-step). - P7 instrumentation-seams.md: state the recorder "must not block" constraint; make the event count self-documenting. - P8 observability.md: add a bundled-adapter wiring snippet. - P9/P10/P11 comparison.md: bullet the feature wall, add a decision matrix, give the CDC verdict a falsifiable detail. - P12/P13 how-it-works.md: flesh out the relay stub; dedupe the idempotency note. - P14/P15 testing.md: runnable loop-mode example; drop the inconsistent pytest.mark.asyncio marker. - P16 troubleshooting.md: name the missing-timer_id DLQ migration cause. - P17 basic.md: drop the unexplained max_workers=4. - P18 relay.md: setup note + OutboxResponse import in the anti-pattern. All findings (B1–B5, C1–C3, I1–I15, P1–P18) now resolved. mkdocs build --strict clean. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent e19b07d commit e0e96fe

12 files changed

Lines changed: 199 additions & 63 deletions

File tree

docs/concepts/comparison.md

Lines changed: 37 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -5,28 +5,42 @@ trade-offs. This page names the alternatives, says when each is the better
55
choice, and ends each comparison with a one-line verdict so a scanning
66
reader can lift the answer without reading the discussion.
77

8+
| Alternative | Pick *it* when… | Pick `faststream-outbox` when… |
9+
|---|---|---|
10+
| [Writing your own](#vs-writing-your-own-outbox-table-and-worker) | You only ever need the MVP shape | You expect the system to live for years |
11+
| [CDC / Debezium](#vs-cdc-debezium-logical-replication) | You already need WAL capture, or producers are outside your control | You own the producer and want retry / DLQ / timers in-process |
12+
| [Kafka txns / Rabbit confirms](#vs-kafka-transactions-or-rabbitmq-publisher-confirms) | The bus is already running and you need bus-scale throughput | Postgres is your only durable store |
13+
| [Plain `LISTEN/NOTIFY`](#vs-plain-listennotify) | You only need a wake-up, not a delivery guarantee | You need durability, replay, and retry |
14+
| [Celery / RQ / Dramatiq](#vs-celery-or-rq-dramatiq-with-a-db-backend) | You're modelling ad-hoc background jobs | You're modelling events that must commit with a DB write |
15+
| [FastStream foreign broker](#vs-faststream-kafkabroker-rabbitbroker-directly) | You have no DB write to commit alongside the publish | You need atomicity with a DB write (use *both*, via [Relay](../usage/relay.md)) |
16+
817
## vs. writing your own outbox table and worker
918

1019
A bespoke outbox is the most common starting point — the pattern itself is
1120
straightforward, and an MVP can ship in an afternoon. What `faststream-outbox`
1221
buys you, in concrete terms, is the pile of pieces that turn that MVP into a
13-
production system: per-row **lease tokens** with a load-bearing invariant
14-
that any new fetch / terminal path must preserve; the **partial-index
15-
design** the fetch CTE depends on (without those, the disjunctive WHERE
16-
clause falls back to seq-scan as the table grows); the **fetch-and-claim
17-
CTE shape** with `FOR UPDATE SKIP LOCKED` that reclaims both unleased rows
18-
and expired leases without a separate reaper; the
19-
retry-strategy hierarchy (`ExponentialRetry`, `NoRetry`, …) enforcing
20-
`max_attempts` and `max_total_delay_seconds` uniformly; `validate_schema()` via
21-
Alembic's `autogenerate.compare_metadata`; **drain semantics** on stop with
22-
the `running` / `_stopping` two-flag dance and parallel-gathered subscriber
23-
shutdown; **`LISTEN/NOTIFY`** short-circuit on top of polling, with NOTIFY
24-
suppression on future-dated rows and `timer_id` conflict no-ops;
25-
**`timer_id` dedup** via a partial unique index plus
26-
`on_conflict_do_nothing`; the **DLQ atomicity CTE** that rolls back the
27-
DELETE when the DLQ insert fails. None of these is hard individually; in
28-
aggregate they decide whether the outbox survives the second year of
29-
production load.
22+
production system:
23+
24+
- per-row **lease tokens** with a load-bearing invariant that any new
25+
fetch / terminal path must preserve;
26+
- the **partial-index design** the fetch CTE depends on (without those, the
27+
disjunctive WHERE clause falls back to seq-scan as the table grows);
28+
- the **fetch-and-claim CTE shape** with `FOR UPDATE SKIP LOCKED` that
29+
reclaims both unleased rows and expired leases without a separate reaper;
30+
- the **retry-strategy hierarchy** (`ExponentialRetry`, `NoRetry`, …)
31+
enforcing `max_attempts` and `max_total_delay_seconds` uniformly;
32+
- **`validate_schema()`** via Alembic's `autogenerate.compare_metadata`;
33+
- **drain semantics** on stop, with the `running` / `_stopping` two-flag
34+
dance and parallel-gathered subscriber shutdown;
35+
- **`LISTEN/NOTIFY`** short-circuit on top of polling, with NOTIFY
36+
suppression on future-dated rows and `timer_id` conflict no-ops;
37+
- **`timer_id` dedup** via a partial unique index plus
38+
`on_conflict_do_nothing`;
39+
- the **DLQ atomicity CTE** that rolls back the DELETE when the DLQ insert
40+
fails.
41+
42+
None of these is hard individually; in aggregate they decide whether the
43+
outbox survives the second year of production load.
3044

3145
You also pick up the [Subscriber](../usage/subscriber.md),
3246
[Publisher](../usage/publisher.md), [Dead-letter queue](../usage/dlq.md), and
@@ -57,9 +71,12 @@ outbox row is cheap to write inline with the domain write), when you
5771
need **handler-level retry, DLQ, and scheduled-delivery semantics
5872
inline** (CDC pushes those concerns to a separate consumer layer), and
5973
when the **async-Python logical-replication tooling gap** is too thin
60-
to lean on. The last point is the load-bearing one for this project — a
61-
2026-05-07 reassessment confirmed the gap had not closed sufficiently to
62-
make CDC the recommended path here.
74+
to lean on: there is no async-native logical-decoding client comparable
75+
to Debezium's JVM connectors, so a Python CDC path means either running
76+
the JVM stack alongside your app or driving `pg_recvlogical` / a thin
77+
`psycopg` replication-protocol wrapper yourself. That is the load-bearing
78+
point for this project — a 2026-05-07 reassessment confirmed the gap had
79+
not closed sufficiently to make CDC the recommended path here.
6380

6481
**TL;DR.** Pick CDC when you already need WAL capture or have producers
6582
outside your control. Pick this when you own the producer and want

docs/concepts/instrumentation-seams.md

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -61,16 +61,22 @@ Three events fire **outside** the handler invocation, with no
6161
## What the recorder seam observes naturally
6262

6363
The recorder is a `Callable[[str, Mapping[str, Any]], None]` invoked at
64-
six subscriber events and one producer event. Plus `dlq_written` when
65-
the DLQ is configured. It fires whether or not a handler is in scope:
64+
six core subscriber events (`fetched`, `dispatched`, `acked`,
65+
`nacked_retried`, `nacked_terminal`, `lease_lost`), a conditional
66+
`dlq_written` when the DLQ is configured, and one producer event
67+
(`published`). It fires whether or not a handler is in scope:
6668

6769
- All three bus-invisible events above.
6870
- Plus `acked` / `nacked_retried` / `nacked_terminal` / `dispatched` /
6971
`published` from inside the handler-execution paths, with explicit
7072
`subscriber` and `queue` tags.
7173

7274
The recorder cannot bracket span lifecycles (it's a callable, not a
73-
context manager), so tracing belongs to the middleware seam.
75+
context manager), so tracing belongs to the middleware seam. It also
76+
runs **on the dispatch event loop and must not block** — a synchronous
77+
`Counter.inc()` is fine; an HTTP / StatsD push is not. See
78+
[Observability § Recorder must not block](../usage/observability.md#recorder-must-not-block)
79+
for the full contract.
7480

7581
## Layering: middleware seam vs. recorder seam
7682

docs/index.md

Lines changed: 7 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -65,8 +65,9 @@ process, no Kafka.
6565
- [Comparison](concepts/comparison.md) — vs writing your own, vs CDC,
6666
vs Kafka transactions, vs `LISTEN/NOTIFY`, vs Celery, vs FastStream
6767
foreign-broker direct.
68-
- [Instrumentation seams](concepts/instrumentation-seams.md) — the
69-
recorder seam vs native middleware, and why both exist.
68+
- [Instrumentation seams](concepts/instrumentation-seams.md)*concept:*
69+
the recorder seam vs native middleware, and why both exist. **Read this
70+
first** if you're deciding what to wire.
7071

7172
### Guides
7273

@@ -81,7 +82,8 @@ process, no Kafka.
8182
- [Schema validation](usage/schema-validation.md) — opt-in
8283
Alembic-driven check for `/health` and CI.
8384
- [Setup Prometheus and OpenTelemetry](usage/setup-prometheus-opentelemetry.md)
84-
— wire the native middleware and recorder adapters end-to-end.
85+
*step-by-step:* wire the native middleware and recorder adapters
86+
end-to-end.
8587

8688
### Reference
8789

@@ -93,8 +95,8 @@ process, no Kafka.
9395
walking every subscriber via `broker.subscribers`.
9496
- [Dead-letter queue](usage/dlq.md) — opt-in audit table, atomicity
9597
via a single CTE, `dlq_written` metric, retention patterns.
96-
- [Observability](usage/observability.md)recorder seam plus
97-
native Prometheus / OpenTelemetry middleware.
98+
- [Observability](usage/observability.md)*reference:* the recorder-seam
99+
API, the full event/tag catalog, and the operator PromQL playbook.
98100

99101
### Operations
100102

docs/introduction/how-it-works.md

Lines changed: 11 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -178,13 +178,22 @@ you add).
178178

179179
## Failure modes
180180

181-
- **Handlers must be idempotent.** Crash between commit-of-handler-side-effects and the broker's `DELETE` re-delivers the message.
181+
- **Handlers must be idempotent.** A crash between the handler's side effect and the broker's `DELETE` re-delivers the message — see [At-least-once delivery](#at-least-once-delivery) above.
182182
- **Best-effort ordering only.** `FOR UPDATE SKIP LOCKED` does not preserve strict order under concurrent workers. If you need strict per-aggregate ordering, route to a single subscriber and run a single worker.
183183
- **DLQ is opt-in.** Without `dlq_table=`, terminal failures `DELETE` the row.
184184

185185
## Relay to Kafka / RabbitMQ / NATS / Redis
186186

187-
> **Relay outbox rows to Kafka / RabbitMQ / NATS / Redis with a single decorator → [Relay tutorial](../tutorials/add-kafka-relay.md).**
187+
An `OutboxSubscriber` can source a FastStream-native cross-broker chain:
188+
stack a foreign-broker publisher decorator on the subscriber
189+
(`@kafka_pub @broker_outbox.subscriber("q")`) and the handler's return
190+
value is forwarded to the real bus. The outbox row stays the durability
191+
boundary — the row commits with the domain write, and the relay carries
192+
at-least-once end to end. If the foreign publish fails, the row is **not**
193+
nacked through the `retry_strategy`; its lease simply expires and a later
194+
fetch retries the relay, so a transient bus outage never loses the row.
195+
196+
> **Worked end-to-end example → [Relay tutorial](../tutorials/add-kafka-relay.md).**
188197
189198
## Acknowledgements
190199

docs/operations/troubleshooting.md

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -54,7 +54,11 @@ pattern compounds.
5454

5555
**Diagnose.** Run `await broker.validate_schema()` against the live
5656
DB (the `[validate]` extra is required). It will surface missing
57-
columns / indexes on the DLQ table.
57+
columns / indexes on the DLQ table. A frequent cause on older
58+
deployments is a hand-written DLQ migration missing the `timer_id`
59+
column — `validate_schema()` reports it as a missing column on the DLQ
60+
table. (The [Alembic guide](../operations/alembic.md#adding-the-dlq-after-the-fact)
61+
now includes it; pre-fix migrations may not.)
5862

5963
**Fix.** Bring the DLQ schema up to spec (apply the missing migration,
6064
or rename / drop the drifted column / index). After the schema is

docs/usage/basic.md

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -39,13 +39,13 @@ Subscribers work like any FastStream subscriber. Decorate a handler with
3939
`@broker.subscriber(queue, ...)`:
4040

4141
```python
42-
@broker.subscriber("orders", max_workers=4)
42+
@broker.subscriber("orders")
4343
async def handle(order_id: int) -> None:
4444
print(f"order {order_id}")
4545
```
4646

47-
See [Subscriber](./subscriber.md) for the full options list, tuning guide,
48-
and retry strategies.
47+
See [Subscriber](./subscriber.md) for the full options list (concurrency
48+
via `max_workers`, tuning, retry strategies).
4949

5050
## 4. Publish a message
5151

@@ -87,7 +87,7 @@ broker = OutboxBroker(engine, outbox_table=outbox_table)
8787
app = FastStream(broker)
8888

8989

90-
@broker.subscriber("orders", max_workers=4)
90+
@broker.subscriber("orders")
9191
async def handle(order_id: int) -> None:
9292
print(f"order {order_id}")
9393

docs/usage/observability.md

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,29 @@ def recorder(event: str, tags: dict) -> None:
3636
broker = OutboxBroker(engine, outbox_table=outbox_table, metrics_recorder=recorder)
3737
```
3838

39+
### Bundled adapters
40+
41+
You rarely need to hand-write the callable. Two ready-made recorders ship
42+
as optional extras and emit the `faststream_outbox_*` series the PromQL
43+
playbook below keys off:
44+
45+
```python
46+
from prometheus_client import CollectorRegistry
47+
from faststream_outbox.metrics.prometheus import PrometheusRecorder
48+
49+
registry = CollectorRegistry()
50+
broker = OutboxBroker(
51+
engine,
52+
outbox_table=outbox_table,
53+
metrics_recorder=PrometheusRecorder(app_name="checkout", registry=registry),
54+
)
55+
```
56+
57+
`OpenTelemetryRecorder` (`faststream_outbox.metrics.opentelemetry`) is the
58+
OTel equivalent. Full wiring — including running the recorder seam and the
59+
native middleware together — is in
60+
[Setup Prometheus and OpenTelemetry](./setup-prometheus-opentelemetry.md).
61+
3962
### Recorder must not block
4063

4164
The recorder is called from the event loop. **Do not block in it.**

docs/usage/publisher.md

Lines changed: 26 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -102,8 +102,23 @@ async def checkout(order: Order, session: AsyncSession) -> None:
102102
Per-call `headers` are merged with the publisher's static headers
103103
(per-call wins).
104104

105+
Constructor signature:
106+
107+
```python
108+
broker.publisher(
109+
queue: str,
110+
*,
111+
headers: dict[str, str] | None = None,
112+
title: str | None = None, # AsyncAPI operation title
113+
description: str | None = None, # AsyncAPI operation description
114+
schema: Any | None = None, # AsyncAPI payload schema override
115+
include_in_schema: bool = True, # exclude from the AsyncAPI spec when False
116+
) -> OutboxPublisher
117+
```
118+
105119
The publisher exists primarily for AsyncAPI spec coverage and to
106-
encapsulate per-queue config (static headers, etc.).
120+
encapsulate per-queue config — hence the `title` / `description` / `schema`
121+
/ `include_in_schema` knobs above, alongside the static `headers`.
107122

108123
### Not a relay decorator
109124

@@ -129,6 +144,16 @@ flow routes the returned value through the producer; the same
129144
transactional contract applies (you provide the session, the row commits
130145
with your domain writes):
131146

147+
!!! note "This pattern is FastAPI-specific"
148+
The returned `OutboxResponse` is published **after** the handler
149+
returns, so its `session` must outlive the handler call. FastAPI's
150+
`Depends(get_session)` provides exactly that — a session torn down by
151+
the dependency after the response flow. Opening your own `async with
152+
session_factory() as session:` inside the handler does **not** work
153+
here: the session closes on `return`, before the row is inserted.
154+
Outside FastAPI, call `broker.publish(..., session=session)` directly
155+
inside your handler instead (see [§ `broker.publisher`](#not-a-relay-decorator)).
156+
132157
```python
133158
from fastapi import Depends
134159
from sqlalchemy.ext.asyncio import AsyncSession

docs/usage/relay.md

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,10 @@ boundary, and the relay carries an at-least-once guarantee end to end.
2424

2525
## Minimal relay
2626

27+
The snippets below assume `engine` and `outbox_table` from the
28+
[Basic usage](./basic.md) setup (`make_outbox_table(metadata)` +
29+
`create_async_engine(...)`).
30+
2731
```python
2832
from faststream.kafka import KafkaBroker
2933
from faststream_outbox import OutboxBroker
@@ -167,6 +171,9 @@ broker_outbox.include_router(outbox_router)
167171
**Do not** combine `OutboxResponse(...)` and a foreign-publisher decorator.
168172

169173
```python
174+
from faststream_outbox import OutboxResponse
175+
176+
170177
@publisher_kafka
171178
@broker_outbox.subscriber("outbox_queue")
172179
async def relay(body: dict) -> OutboxResponse:

docs/usage/subscriber.md

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -89,6 +89,7 @@ Per-subscriber knobs, passed to `@broker.subscriber("…", …)`:
8989
| `max_deliveries` | `None` (unbounded) | Total claims (including lease-expiry re-claims) after which the row is dropped without invoking the handler. Defends against handlers that consistently wedge. |
9090
| `ack_policy` | `AckPolicy.NACK_ON_ERROR` | See [Ack policy](#ack-policy) |
9191
| `retry_strategy` | `ExponentialRetry(...)` | See [Retry strategies](#retry-strategies) |
92+
| `propagate_inbound_headers` | `False` | Relay-only. When `True`, fills `Response.headers` from the inbound message *if* the handler returned a `Response` with empty headers (user-set headers always win). See [Relay](./relay.md). |
9293

9394
```python
9495
@broker.subscriber(
@@ -106,6 +107,10 @@ The factory in `subscriber/factory.py` warns or raises on likely-wrong
106107
combinations (`lease_ttl_seconds <= max_fetch_interval`, `max_deliveries`
107108
without retry, `min_fetch_interval > max_fetch_interval`, etc.).
108109

110+
The table above lists the outbox-specific knobs. The standard FastStream
111+
subscriber kwargs pass through unchanged too: `dependencies`, `parser`,
112+
`decoder`, and the AsyncAPI `title_` / `description_` / `include_in_schema`.
113+
109114
## Slow handlers — dedicated queue
110115

111116
When a handler's tail latency exceeds the subscriber's `lease_ttl_seconds`,
@@ -206,6 +211,19 @@ when non-zero, the computed delay is multiplied by `1 +
206211
U(-jitter_factor/2, +jitter_factor/2)` to spread out retries, matching
207212
`ExponentialRetry`'s shape.
208213

214+
### Strategy parameters
215+
216+
| Strategy | Required | Optional (default) |
217+
|---|---|---|
218+
| `NoRetry` |||
219+
| `ConstantRetry` | `delay_seconds` | `jitter_factor` (`0.0`) |
220+
| `LinearRetry` | `initial_delay_seconds`, `step_seconds` | `jitter_factor` (`0.0`) |
221+
| `ExponentialRetry` | `initial_delay_seconds` | `multiplier` (`2.0`), `max_delay_seconds` (`None`), `jitter_factor` (`0.0`) |
222+
223+
Every strategy except `NoRetry` also accepts the shared caps `max_attempts`
224+
(default `None`) and `max_total_delay_seconds` (default `None`); reaching
225+
either returns `None`, which is terminal (the row is deleted, or DLQ'd).
226+
209227
### Retry only on transient errors
210228

211229
Strategies receive the raised `exception` so users may subclass for

0 commit comments

Comments
 (0)