feat(conductor): add per-provider inflight blocking queue

[francomascareloai]

Summary

Adds a per-provider inflight request limiter with a bounded blocking queue. When concurrent requests to a single provider exceed the configured limit, additional requests wait for a slot (up to a configurable timeout) instead of being immediately rejected with HTTP 503 provider_backpressure.

This prevents burst-driven upstream rate-limiting (e.g. 403 "access paused" / 429) when many concurrent subagents hit the same provider simultaneously.

Problem

When N concurrent requests target the same upstream provider and exceed the provider's real concurrency tolerance, the proxy forwards all of them and the provider responds with 403 access paused / 429 — which can force credential revocation and disrupt all users of that provider. There is currently no backpressure mechanism that absorbs short bursts.

Solution

A per-provider counting semaphore (chan struct{} of capacity MaxInflightPerProvider):

• Fast path (slot available): non-blocking channel send, returns immediately. Negligible overhead when disabled (MaxInflightPerProvider == 0 → no-op).
• Slow path (slots full): blocking select with a bounded context.WithTimeout (MaxQueueWaitMS). On timeout, returns a retryable *auth.Error (HTTP 503, Retryable: true).
• Head-of-line safe: the limiter-map mutex is released before the blocking select, so a saturated provider A does not block provider B.
• Permit lifecycle: acquire-late + release-per-iteration (NOT function-scoped defer, which would leak across continue in the failover loop). Panic-recover is iteration-scoped via an IIFE closure that releases the permit and re-panics.
• Streaming: the permit lifetime is bound to the stream via a wrapper goroutine around Chunks. Both the inner and outer ctx.Done branches drain originalChunks to avoid blocking the upstream producer on client disconnect mid-stream.
• Idempotent release: releaseFunc uses sync.Once so a double-release never steals a token from a different acquirer.

New config keys

Key	Default	Description
provider-resilience.max-inflight-per-provider	4	Max concurrent upstream calls per provider key. 0 disables the limiter (no-op fast path). Clamped to [0, 10000].
provider-resilience.max-queue-wait-ms	30000	Max time a request waits for a slot before returning 503 retryable. 0 = wait forever (not recommended). Clamped to [0, 300000].

provider-resilience:
  max-inflight-per-provider: 4
  max-queue-wait-ms: 30000

Metrics

Exposed via Manager.ResilienceMetricsSnapshot():

Metric	Meaning
cliproxy_provider_backpressure_reject_total	Requests rejected with 503 after queue wait timeout
cliproxy_provider_queue_wait_ns_sum	Total nanoseconds spent waiting in the queue
cliproxy_conductor_permit_attempts_total	Acquire attempts (includes backpressure rejects, not only successes)
cliproxy_conductor_permit_ns_sum	Acquire overhead (latency of the acquire call itself)

Tests

New file sdk/cliproxy/auth/conductor_resilience_test.go (645 lines, 9 tests, all pass with -race):

• TestBlockingQueuePerProvider — N goroutines, limit K: K acquire immediately, N-K block, release propagates.
• TestQueueWaitTimeout — 2nd request gets 503 after max-queue-wait-ms.
• TestCtxCancelUnblocksWaiter — cancelled waiter returns ctx.Err(), no goroutine leak.
• TestNoHeadOfLineBlocking — saturated provider A does not block provider B (<50ms).
• TestStreamingReleaseOnChannelClose — permit released when Chunks closes.
• TestStreamingWrapperDrainsOnInnerCancel — inner ctx.Done drains originalChunks (no producer deadlock on mid-stream cancel).
• TestReleaseFuncIdempotent — double-release does not steal another acquirer's token (len(limiter) stays 1).
• TestAcquireProviderPermitDisabled / nil-safety — no-op when MaxInflightPerProvider == 0 or nil manager/cfg.
• Throughput benchmark.

Scope (intentionally narrow)

This PR contains only the inflight blocking queue. It does not include:

• Circuit breaker (per-credential or per-provider) — see adjacent PR feat(auth): per-credential circuit breaker for consecutive transient failures #3598.
• Token-refresh semaphore — see adjacent PR feat(auth): stable jitter and per-provider semaphore for token refresh #3597 (different concern: refresh, not request inflight).
• Any auth/management changes.

Validation

• go vet ./internal/config/... ./sdk/cliproxy/auth/... → clean.
• go test ./sdk/cliproxy/auth/ -race → 9/9 PASS.
• go build ./cmd/server → success (56MB binary).
• Reviewed by trading-systems-reviewer → FEEDBACK_CONSENSUS: PASS (0 HIGH/CRITICAL, 1 LOW naming nit resolved: metric renamed conductor_permit_total → conductor_permit_attempts_total to reflect it counts attempts).

Files

• internal/config/config.go (+37) — ProviderResilienceConfig, constants, defaults, clamping.
• sdk/cliproxy/auth/conductor.go (+314/-5) — acquireProviderPermit, releaseFunc, ResilienceMetricsSnapshot, 5 acquire sites, 2 streaming wrappers.
• sdk/cliproxy/auth/conductor_resilience_test.go (new, 645) — 9 tests.

Backward compatibility

Default behavior is unchanged: MaxInflightPerProvider defaults to 4, but operators who do not set provider-resilience in config get the code default. Setting max-inflight-per-provider: 0 fully disables the limiter (no-op fast path, zero channel ops).

[github-actions]

This pull request targeted main.

The base branch has been automatically changed to dev.

[gemini-code-assist]

Code Review

This pull request implements per-provider inflight request limiting (Provider Resilience) using a bounded blocking queue based on buffered channels. It introduces new configuration parameters, integrates permit acquisition and release across execution and streaming paths, and adds comprehensive tests to verify the resilience behavior. The feedback suggests optimizing the permit acquisition logic by avoiding unnecessary context allocations when no queue timeout is configured.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

When MaxQueueWaitMS is 0 (wait forever), creating a cancelable child context via context.WithCancel(ctx) is unnecessary and introduces allocation and parent-child registration overhead. Instead, you can use the parent ctx directly when no timeout is configured.

Suggested change

	var waitCtx context.Context
	var cancel context.CancelFunc
	if pr.MaxQueueWaitMS > 0 {
	waitCtx, cancel = context.WithTimeout(ctx, time.Duration(pr.MaxQueueWaitMS)*time.Millisecond)
	} else {
	waitCtx, cancel = context.WithCancel(ctx)
	}
	defer cancel()
	var waitCtx context.Context
	if pr.MaxQueueWaitMS > 0 {
	var cancel context.CancelFunc
	waitCtx, cancel = context.WithTimeout(ctx, time.Duration(pr.MaxQueueWaitMS)*time.Millisecond)
	defer cancel()
	} else {
	waitCtx = ctx
	}