Skip to content

feat: audio input (browser → device, mic injection) for simulator/emulator #334

Description

@jo-duchan

Goal

Inject audio input (browser → device, mic injection) into sim/emulator sessions over the existing browser ↔ relay ↔ agent path, so voice/mic features can be exercised during manual QA. Hard constraint: must not degrade the current video/render (streaming) pipeline.

Split out from #259, which now scopes the output direction only (device → browser, phase-4). Input is the lower-priority, harder direction.

Why separate / why later

  • Smaller use case — only matters for testing voice/mic-driven features, not general QA.
  • Harder — needs a virtual input device on the device side plus a browser capture surface (getUserMedia), which adds a privacy/permission surface the output path doesn't have.
  • Prerequisite: output ships first — the transport channel, the optional audio agent interface (ISP), and the per-platform capture plumbing land with feat: audio output (device → browser) for simulator/emulator #259. Input reuses that foundation.

Open questions (investigate at planning time)

  • Android: how to feed PCM into the emulator mic on the default gRPC emulator host-encode path (not scrcpy, which is MCP/CI only).
  • iOS: simulator mic injection path — simctl has no audio API; likely needs a virtual input device.
  • Browser side: getUserMedia consent UX and how mic capture rides the separate audio WS channel without touching video backpressure.
  • A/V sync: loose sync acceptable (same as output) — do not over-engineer.

Design notes

Related: #259 (output-first, the foundation this builds on).

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestphase-5-plusPhase 5+ — Not yet scheduled

    Projects

    Status
    No status

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions