Skip to content

GatewayClient RPCs have no response timeout: a silent gateway hangs the chat forever and deadlocks all subsequent turns #1507

Description

@lihongguang0014

Summary

Every RPC sent by the CLI's GatewayClient awaits its response with no timeout. If the gateway stays transport-alive but never answers one request (a stalled handler, a response frame dropped by a flaky proxy, or a server-side bug), the chat hangs forever with no error — and one specific path (sessions.messages.unsubscribe) awaits its untimed RPC while holding a lock, permanently deadlocking every subsequent turn even after the gateway recovers.

Reproduction

  1. Start opensquilla chat connected to the gateway and begin a turn.
  2. Make the gateway stop answering requests while keeping the WebSocket open (e.g. kill/block the gateway's request handler, or drop a single response frame at a proxy).
  3. Observe: the turn never completes and no error is ever surfaced. Pressing Ctrl+C also hangs (the cancellation handler's abort_session is itself an untimed _call).
  4. Alternative trigger at turn end: after a successful turn, if the gateway loses the sessions.messages.unsubscribe response, the client holds _subscription_lock forever — the next turn's subscribe_session_events blocks permanently, so the whole chat session is bricked.

Root cause

src/opensquilla/cli/gateway_client.py_call registers a future and awaits it with no deadline:

# gateway_client.py:632-655 (excerpt)
req_id = str(uuid.uuid4())
loop = asyncio.get_event_loop()
fut: asyncio.Future[dict] = loop.create_future()
self._pending[req_id] = fut
try:
    await self._ws.send(...)
except Exception as exc:
    ...
res = await fut          # <-- no asyncio.wait_for / deadline

The future is resolved by exactly two things: a matching res frame in _listen (:592-594) or _mark_connection_failed failing all pending futures (:1270-1273). A transport-healthy but response-silent gateway triggers neither, so await fut blocks forever.

Lock amplification (deadlock):

# gateway_client.py:1098-1107 — _remove_event_subscription
async with self._subscription_lock:
    ...
    try:
        await self._call("sessions.messages.unsubscribe", {"key": session_key})
    except (ConnectionError, GatewayRPCError):
        return

send_message always calls subscription.close() in its finally (:1225-1231), which reaches this path holding self._subscription_lock. subscribe_session_events (:997) needs the same lock, so one lost unsubscribe response permanently deadlocks every later turn — the lock is never released because the untimed _call never returns.

The codebase already applies timeouts everywhere else — chat/gateway_runtime.py:1118,1149 wrap exit-receipt RPCs in asyncio.wait_for(..., timeout=2.0), and the server-side client has recv_event(timeout=...) — but GatewayClient._call and the send_message stream wait have none.

Suggested fix

  • Give _call a response deadline (e.g. asyncio.wait_for(fut, timeout=...) with a generous default) that fails pending futures with a clear ConnectionError/TimeoutError when no response arrives.
  • Do not hold _subscription_lock across the network call in _remove_event_subscription (pop the subscription from the registry under the lock, then send the untimed/wait_for-bounded unsubscribe outside it), so one lost response can never block future subscribes.
  • Optionally bound the send_message stream wait (subscription.get()) with an idle timeout so a silent stream surfaces an error instead of hanging the turn.

Impact

The active turn hangs forever at "generating" with zero diagnostics; the Ctrl+C escape hatch is also blocked (abort is itself untimed); and the unsubscribe deadlock variant bricks the entire chat session with no recovery short of restarting the process.

Test coverage

None: tests/test_cli/test_gateway_client_keepalive.py covers send failures that raise ConnectionError and manually-resolved futures, but no test feeds a "response never arrives" future to _call, and no test exercises the unsubscribe-under-lock path with a silent gateway.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    has-linked-prAn open pull request is linked to this issue

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions