fix(wsrouter): ensure no message reply is lost

[FrogTheFrog]

Please tick as appropriate:

• I have tested this code on a steam deck or on a PC
• My changes generate no new errors/warnings
• This is a bugfix/hotfix
• This is a new feature

If you're wanting to update a translation or add a new one, please use the weblate page: https://weblate.werwolv.net/projects/decky/

Description

Motivation

My plugin MoonDeck is experiencing issues once on a blue moon, where the UI gets stuck. This happens, because I queue promises to optimize some calls to backend python script. Sometimes after SteamDeck wakes up, the promise queue gets stuck, because the call to the backend is never resolved and the plugin must then be reloaded.

When Decky loses WebSocket connection, all of the running calls are left in an unresolved state, because the backend never sends the reply to the frontend anymore. I do not know if this solves my issue completely (was not able to confirm that this is the root cause since it so rare), however Decky should not just leave promises in a "forever-unresolved" state while the plugin is still active.

New call-reply protocol

The existing protocol has been extended with new message types:

• DISCARD
• RECEIVED_RESPONSE
• FULL_SYNC

When the connection is first established (or re-established), a FULL_SYNC message is sent with ALL of previously submitted calls, but which did not get a reply yet. When backend receives such a message it will:

• Try to send the call to the plugin if this call is still unknown - frontend either tried to send it while the connection was down or lost.
• If the call was already made before (the connection was lost while waiting for reply), the backend will continue waiting for the reply from plugin or send a cached response:
  • Even if the connection is down, the backend will cache all of the responses instead of retrying the same call once the connection is up again in order not to surprise the plugin with multiple unexpected calls.
• If the frontend no longer contains the call message, that backend STILL knows about (technically possible due to dropped RECEIVED_RESPONSE message, more on the later), the backend will simply cleanup the cache.
  • Theoretically, it is possible for this to happen while the backend is still waiting for a reply from plugin. Once the reply is received, it will still be sent to frontend were it's simply ignored.

When the frontend receives a response (either ERROR, REPLY or DISCARD), it will always reply with RECEIVED_RESPONSE, so that the backend may cleanup its response cache. This way:

• In case the response is lost from backend -> frontend, we will still get it after FULL_SYNC, because the response cache would still contain it.
• In case the RECEIVED_RESPONSE is lost from frontend -> backend, the FULL_SYNC will cleanup the cache on backend.

Finally, the DISCARD message is used to notify the frontend to just cleanup its cache without resolving the promise (for garbage-collection). At the moment it is only done for pending plugin calls whose python backend is being unloaded and they would never be resolved. We could send an ERROR, but then not every plugin is gracefully handling such errors (if errors are handled at all).

Testing

I've manually made the Decky close the websocket every 30 seconds and confirmed that before the fix, the promises would get stuck when plugin backend is still generating a reply when the socked closes.

With this fix, it never happens again and cached replies are sent back once the connection is restored.

Additionally, I disabled/enabled and reloaded the plugin while this was happening and have confirmed that the DISCARD message is being sent accordingly for plugin replies where plugin is still "thinking". Verified that such replied do not throw any exception in the frontend.