Goal: add a full-duplex voice interface — talk to the Stack Tutor and hear it respond in real time using NVIDIA's own speech-to-speech model.
PersonaPlex is a 7B full-duplex speech-to-speech model from NVIDIA ADLR (January 2026). Full-duplex means it simultaneously listens and speaks — true conversational overlap, not a cascaded ASR to LLM to TTS pipeline. Around 170ms turn-taking latency. Built on the Moshi architecture with a Mimi speech codec.
- Persona control: conditioned on a voice prompt (16 built-in voice embeddings NATF0-3, NATM0-3, VARF0-4, VARM0-4) and a text prompt (role, scenario, background)
- WebSocket: browser captures mic via AudioWorklet, Opus codec, WebSocket to model, audio back over the same WebSocket
- Built-in web UI at
localhost:8998 - License: code is MIT, weights are NVIDIA Open Model License (commercial use permitted)
PersonaPlex is self-hosted only (not on build.nvidia.com). It requires an A100 80GB or H100. Combine this phase with Phase 10 — rent one GPU and run both in the same session.
Cost: Lambda Labs H100 SXM ~$3.29/hr, budget 4-6 hours for Phase 10 + Phase 11 = ~$15-20. RunPod A100 80GB ~$1.64/hr = ~$8-10 for PersonaPlex alone.
sudo apt-get install -y libopus-devgit clone https://github.com/NVIDIA/personaplex.git
cd personaplex
pip install moshi/.# Accept the model license on HuggingFace:
# https://huggingface.co/nvidia/personaplex-7b-v1
export HF_TOKEN=hf_your_token_here
SSL_DIR=$(mktemp -d)
python -m moshi.server --ssl "$SSL_DIR" \
--voice-prompt voices/NATM0.pt \
--text-prompt "You are NVIDIA Stack Tutor, a friendly technical expert on the NVIDIA AI software stack. You help developers understand NIM, NeMo, Triton, TensorRT-LLM, and the broader NVIDIA ecosystem. Keep answers concise and conversational."Open https://<gpu-vm-ip>:8998 (accept the self-signed cert). Click the mic button. Ask "What is NIM?" and hear the model respond in real time, full-duplex, in the NATM0 voice.
Path A — Standalone demo (1-2 hours): run PersonaPlex alongside the text Stack Tutor. Use the text prompt to give it the same persona and domain expertise. No code changes to the existing app — voice in addition to text, not voice integrated with the RAG pipeline.
Path B — Full integration (4-6 hours): build a bridge service that captures PersonaPlex's text-token output (the byproduct transcription), pipes the transcript into the existing agent loop (RAG retrieval, guardrails, tool use), and feeds the agent's text response back to PersonaPlex as a continuation prompt. More complex because PersonaPlex is end-to-end and does not natively expose a hook between "I understood you" and "here's my response" — the Moshi codebase supports it but it is deep integration.
Path A covers ~90% of the demo value at ~20% of the effort. Path B is the right next step if you want the voice interface to speak the actual RAG-backed answers.
| Option | What it is | Pros | Cons |
|---|---|---|---|
| Nemotron 3 VoiceChat (12B) | Full-duplex speech-to-speech on NIM, hosted | No GPU required, on build.nvidia.com (early access) | May not be available to all users yet |
| Nemotron Voice Agent Blueprint | Cascaded pipeline: Riva ASR to Nemotron LLM to Riva TTS | GitHub repo, uses NIM, well-documented | Not full-duplex, more services to wire |
| Riva ASR NIM + Riva TTS NIM | Individual speech-to-text and text-to-speech hosted endpoints | Modular, hosted, can wrap the existing agent | Not full-duplex, more plumbing, latency from multiple hops |
If Nemotron VoiceChat is available on build.nvidia.com by the time you do this phase, it is the better choice — same full-duplex concept, hosted, no GPU rental.
Screen-record a voice interaction and put it in your repo README. A 60-second video of a natural conversation with an agent, running on NVIDIA's own voice model, is a strong portfolio artefact.
NVIDIA's PersonaPlex 7B full-duplex speech-to-speech model integrated into a working agent. Moshi architecture with a Mimi speech codec for ~170ms turn-taking — genuinely full-duplex, not a cascaded ASR-LLM-TTS pipeline. Self-hosted on a single A100 80GB.
Back to README