Summary
Every RPC sent by the CLI's GatewayClient awaits its response with no timeout. If the gateway stays transport-alive but never answers one request (a stalled handler, a response frame dropped by a flaky proxy, or a server-side bug), the chat hangs forever with no error — and one specific path (sessions.messages.unsubscribe) awaits its untimed RPC while holding a lock, permanently deadlocking every subsequent turn even after the gateway recovers.
Reproduction
- Start
opensquilla chat connected to the gateway and begin a turn.
- Make the gateway stop answering requests while keeping the WebSocket open (e.g. kill/block the gateway's request handler, or drop a single response frame at a proxy).
- Observe: the turn never completes and no error is ever surfaced. Pressing Ctrl+C also hangs (the cancellation handler's
abort_session is itself an untimed _call).
- Alternative trigger at turn end: after a successful turn, if the gateway loses the
sessions.messages.unsubscribe response, the client holds _subscription_lock forever — the next turn's subscribe_session_events blocks permanently, so the whole chat session is bricked.
Root cause
src/opensquilla/cli/gateway_client.py — _call registers a future and awaits it with no deadline:
# gateway_client.py:632-655 (excerpt)
req_id = str(uuid.uuid4())
loop = asyncio.get_event_loop()
fut: asyncio.Future[dict] = loop.create_future()
self._pending[req_id] = fut
try:
await self._ws.send(...)
except Exception as exc:
...
res = await fut # <-- no asyncio.wait_for / deadline
The future is resolved by exactly two things: a matching res frame in _listen (:592-594) or _mark_connection_failed failing all pending futures (:1270-1273). A transport-healthy but response-silent gateway triggers neither, so await fut blocks forever.
Lock amplification (deadlock):
# gateway_client.py:1098-1107 — _remove_event_subscription
async with self._subscription_lock:
...
try:
await self._call("sessions.messages.unsubscribe", {"key": session_key})
except (ConnectionError, GatewayRPCError):
return
send_message always calls subscription.close() in its finally (:1225-1231), which reaches this path holding self._subscription_lock. subscribe_session_events (:997) needs the same lock, so one lost unsubscribe response permanently deadlocks every later turn — the lock is never released because the untimed _call never returns.
The codebase already applies timeouts everywhere else — chat/gateway_runtime.py:1118,1149 wrap exit-receipt RPCs in asyncio.wait_for(..., timeout=2.0), and the server-side client has recv_event(timeout=...) — but GatewayClient._call and the send_message stream wait have none.
Suggested fix
- Give
_call a response deadline (e.g. asyncio.wait_for(fut, timeout=...) with a generous default) that fails pending futures with a clear ConnectionError/TimeoutError when no response arrives.
- Do not hold
_subscription_lock across the network call in _remove_event_subscription (pop the subscription from the registry under the lock, then send the untimed/wait_for-bounded unsubscribe outside it), so one lost response can never block future subscribes.
- Optionally bound the
send_message stream wait (subscription.get()) with an idle timeout so a silent stream surfaces an error instead of hanging the turn.
Impact
The active turn hangs forever at "generating" with zero diagnostics; the Ctrl+C escape hatch is also blocked (abort is itself untimed); and the unsubscribe deadlock variant bricks the entire chat session with no recovery short of restarting the process.
Test coverage
None: tests/test_cli/test_gateway_client_keepalive.py covers send failures that raise ConnectionError and manually-resolved futures, but no test feeds a "response never arrives" future to _call, and no test exercises the unsubscribe-under-lock path with a silent gateway.
Summary
Every RPC sent by the CLI's
GatewayClientawaits its response with no timeout. If the gateway stays transport-alive but never answers one request (a stalled handler, a response frame dropped by a flaky proxy, or a server-side bug), the chat hangs forever with no error — and one specific path (sessions.messages.unsubscribe) awaits its untimed RPC while holding a lock, permanently deadlocking every subsequent turn even after the gateway recovers.Reproduction
opensquilla chatconnected to the gateway and begin a turn.abort_sessionis itself an untimed_call).sessions.messages.unsubscriberesponse, the client holds_subscription_lockforever — the next turn'ssubscribe_session_eventsblocks permanently, so the whole chat session is bricked.Root cause
src/opensquilla/cli/gateway_client.py—_callregisters a future and awaits it with no deadline:The future is resolved by exactly two things: a matching
resframe in_listen(:592-594) or_mark_connection_failedfailing all pending futures (:1270-1273). A transport-healthy but response-silent gateway triggers neither, soawait futblocks forever.Lock amplification (deadlock):
send_messagealways callssubscription.close()in itsfinally(:1225-1231), which reaches this path holdingself._subscription_lock.subscribe_session_events(:997) needs the same lock, so one lost unsubscribe response permanently deadlocks every later turn — the lock is never released because the untimed_callnever returns.The codebase already applies timeouts everywhere else —
chat/gateway_runtime.py:1118,1149wrap exit-receipt RPCs inasyncio.wait_for(..., timeout=2.0), and the server-side client hasrecv_event(timeout=...)— butGatewayClient._calland thesend_messagestream wait have none.Suggested fix
_calla response deadline (e.g.asyncio.wait_for(fut, timeout=...)with a generous default) that fails pending futures with a clearConnectionError/TimeoutErrorwhen no response arrives._subscription_lockacross the network call in_remove_event_subscription(pop the subscription from the registry under the lock, then send the untimed/wait_for-bounded unsubscribe outside it), so one lost response can never block future subscribes.send_messagestream wait (subscription.get()) with an idle timeout so a silent stream surfaces an error instead of hanging the turn.Impact
The active turn hangs forever at "generating" with zero diagnostics; the Ctrl+C escape hatch is also blocked (abort is itself untimed); and the unsubscribe deadlock variant bricks the entire chat session with no recovery short of restarting the process.
Test coverage
None:
tests/test_cli/test_gateway_client_keepalive.pycovers send failures that raiseConnectionErrorand manually-resolved futures, but no test feeds a "response never arrives" future to_call, and no test exercises the unsubscribe-under-lock path with a silent gateway.