Checkpoint Mars

Table of Contents

• Quickstart
• The Game
• The Loop
• Mechanics
• Architecture
• Impressions

Quickstart

1. From the project root, go to the backend:
   • cd backend
2. Create a .env file in the backend directory
   • fill OPENAI_API_KEY with your own API key
3. Start the backend services with Docker:
   • docker compose up (or docker-compose up on older setups)
4. Wait until all containers are up and healthy.
5. Download the newest build for your architecture, e.g. from the github releases on this repository and extract the .zip file.
6. Run the game executable from the extracted folder.

Important: ALWAYS make sure all containers are running before starting the game. If you stop either the game or the containers, you MUST stop everything and do a clean restart, again starting the containers first and then the game.

Notes:

• Currently, you can only exit the game via the Operating System, e.g. Alt+F4 on Windows.
• During development, we experienced high latency on the openai API between around 4 p.m. and 12 a.m. (UTC+1), so around the time americans are working. Best play & test the game before this. Response times are logged in the backend containers. TTS should take 2-5 secs, and STT 1-2 secs.

The Game

You are travelling through Mars space and are stopped at a customs checkpoint. You are smuggling legal and illegal goods in your boxes. A customs officer enters your ship and interrogates you about your cargo. You can interact with him via voice commands. Don't let him find the wrong items! Outsmart him and make sure he does not look into boxes containing illegal goods.

The Loop

1. Fill your boxes with your items and remember the contents.
2. Proceed to the inspection.
3. Answer the officer‘s questions as truthfully as possible, but hide illegal items. He knows what you legally carry, but not what is in which box.
4. Suspicious answers like uncertainty about box contents cause the agent to open boxes.
5. You lose if you misrepresented content of two opened boxes or when an illegal item is found. Otherwise, you win.

But be aware: The officer will open at least one box. It may be advisable to lure him onto a false trail.

Shaping the interaction

Audio

• Thinking sounds - After saying something to the officer, he plays after a short cooldown a random thinking sound like ("hmm", "aha", "really...", "If you say so...") and more. Thinks reduces the perceived latency.
• Push-to-talk - The player can hold the push-to-talk button to talk to the officer. When the player releases the button, the officer will stop talking and the mic will be disabled. This makes playing easier, e.g. when other people are nearby. A UI element notifies the player about when they can talk.
• Agent voice
  • Uses spatial audio with the officer as the audio source.
  • Applies style instructions for immersive voice interactions.
  • Audio filters: high dry-mix, low pitch, and high distortion are applied for distinctive sound.

Visual

• Proximity based action triggers - E.g. animation on "continue to inspection" is triggered when the player is close to the inspection area.
• Color & symbol coded boxes to talk about them with the officer.
• Responsive officer
  • Walks to the box he talks about.
  • Actually opens boxes in the game.
  • Turns his head to the player when he talks.

Mechanics

• Move: WASD or Arrow Keys
• Jump: Space
• Look around: Mouse
• Interact: Press E
• Talk: Hold F while talking, then release
• Restart the game: Press Enter

Architecture

The game is built with Unity and talks to two backend services:

• GameAgentSubscriberService (gRPC): Unity connects to this service to send game state and events and to receive commands for the officer’s actions (e.g. “go to box”, “open box”). It is the bridge between the game and gRPC-based control of the world.
• AudioService (WebSocket): Receives microphone audio from Unity, runs speech-to-text, and streams TTS audio responses back into the game. It also sends and receives control messages to (a) trigger the initial inspection greeting, (b) disable the mic while the agent is thinking, and (c) notify the audio service when the player releases the push‑to‑talk button.

Behind these services lives a single MCP-based Agent Service (HTTP + MCP + LLM) that impersonates the customs officer. It:

• Exposes a REST /api/chat endpoint for text requests.
• Receives the player’s utterances as text from AudioService.
• Calls the selected LLM (OpenAI, Gemini, or local Ollama).
• Uses MCP tools to send gRPC commands via GameAgentSubscriberService to actually move and animate the officer in the game.

For detailed information about the services and the AI agent, see the backend README.

High-level system overview

Flow of a spoken answer to a question from the officer

Impressions

• Filling boxes – Prepare your cargo by placing items into boxes. Don't forget what you put where!

• Heading to border patrol – Once packed, proceed to the checkpoint to start the inspection.

• Box inspection – The officer opens a box to inspect it. Hopefully, it does not contain any illegal goods!