Add ElevenLabs-based STT and TTS implementations as alternatives to OpenAITranscriber and OpenAITTSConfig

[Lancetnik]

Summary

AG2 currently ships built-in Speech-to-Text (STT) and Text-to-Speech (TTS) support via OpenAI, as documented in the STT & TTS guide. The two OpenAI-specific building blocks are:

• OpenAITranscriber — implements the STTConfig protocol and exposes .pipe(agent) to wrap an Agent in a VoicePipeline.
• OpenAITTSConfig — the config consumed by TTSObserver, which listens to streamed ModelMessageChunk events, batches them into sentence-sized chunks, calls the TTS provider, and emits SynthesizedAudioEvents.

This issue proposes adding ElevenLabs as an alternative provider for both halves of the voice pipeline, so users can choose ElevenLabs for STT and/or TTS without changing the rest of the agent/pipeline architecture.

Motivation

ElevenLabs is widely regarded for its high-quality, low-latency, and highly natural-sounding voices, plus a large library of voices and voice-cloning capabilities. Offering it as a drop-in alternative gives users:

• Provider choice / avoiding lock-in to a single vendor.
• Higher voice quality and broader voice/language selection for TTS.
• A competitive ASR option for STT (ElevenLabs Scribe).
• The ability to mix providers (e.g., OpenAI for STT + ElevenLabs for TTS) since both sides are wired independently.

Because the existing design already abstracts STT behind the STTConfig protocol and TTS behind a pluggable config consumed by TTSObserver, adding ElevenLabs should require no changes to Agent, VoicePipeline, TTSObserver, or the audio I/O primitives — only two new provider implementations.

Proposed implementation

1. STT — ElevenLabsTranscriber

A new class mirroring the OpenAITranscriber API so it is a true drop-in replacement:

• Implements the same STTConfig protocol and exposes .pipe(agent) returning a VoicePipeline.
• pipeline.ask(voice_input) transcribes the VoiceInput (16-bit PCM bytes + sample rate + channel count) and forwards the resulting text to the agent's normal ask() flow, returning a VoiceReply.
• Emits the same TranscriptionChunkEvent and TranscriptionCompletedEvent on the agent's stream so existing live-caption subscribers keep working unchanged.
• Backed by the ElevenLabs Speech-to-Text API (Scribe). Accepts a model id and standard provider options (e.g., language hint, diarization where supported).

Target usage (identical shape to today's docs):

from autogen.beta import Agent, config
from autogen.beta.live import ElevenLabsTranscriber, SoundDeviceRecorder

agent = Agent("assistant", config=config.OpenAIConfig("gpt-5", streaming=True))

pipeline = ElevenLabsTranscriber("scribe_v1").pipe(agent)

recorder = SoundDeviceRecorder()
voice_input = recorder.record(duration=5)
reply = await pipeline.ask(voice_input)
print(reply.body)

2. TTS — ElevenLabsTTSConfig

A new config class mirroring OpenAITTSConfig so it plugs straight into TTSObserver:

• Consumed by TTSObserver exactly like OpenAITTSConfig, synthesizing the sentence-sized chunks TTSObserver produces and emitting SynthesizedAudioEvents onto the stream for SoundDevicePlayer to play.
• Exposes ElevenLabs-native parameters: model (e.g., eleven_turbo_v2_5 / eleven_multilingual_v2), voice / voice_id, and voice settings such as stability, similarity_boost, style, and speed.
• Should request a PCM output format compatible with SoundDevicePlayer (sample rate / 16-bit PCM) so the existing player works without modification.
• Streaming-friendly: leverage ElevenLabs streaming synthesis so audio starts playing per sentence chunk, preserving the low-latency sentence-level pipelining the docs describe.

Target usage (identical shape to today's docs):

from autogen.beta import Agent, config
from autogen.beta.live import ElevenLabsTTSConfig, SoundDevicePlayer, TTSObserver

agent = Agent(
    name="assistant",
    prompt="You are a helpful voice assistant.",
    config=config.OpenAIResponsesConfig(model="gpt-5", streaming=True),
    observers=[
        TTSObserver(config=ElevenLabsTTSConfig(
            model="eleven_turbo_v2_5",
            voice_id="<voice-id>",
            stability=0.5,
            similarity_boost=0.75,
            speed=1.0,
        )),
    ],
)

async with SoundDevicePlayer() as player:
    await agent.ask("Hello, agent!", stream=player.stream)

Mixing providers

Because STT and TTS are wired independently, users could combine providers, e.g. OpenAI STT with ElevenLabs TTS, or vice versa, without further changes.

Configuration & auth

• Read the API key from an environment variable (e.g., ELEVENLABS_API_KEY) consistent with how other providers source credentials.
• Add elevenlabs as an optional dependency (e.g., an [elevenlabs] extra) so it is not required for the core install.

Acceptance criteria

• ElevenLabsTranscriber implements STTConfig and works as a drop-in for OpenAITranscriber (same .pipe() / .ask() surface and the same transcription events).
• ElevenLabsTTSConfig plugs into TTSObserver as a drop-in for OpenAITTSConfig, emitting SynthesizedAudioEvents playable by SoundDevicePlayer.
• Streaming/sentence-level synthesis is supported to keep latency low.
• Credentials sourced from an environment variable; elevenlabs added as an optional extra.
• Unit tests for both classes (mocking the ElevenLabs API).
• Docs updated on the STT & TTS page showing the ElevenLabs alternatives.

References

• AG2 STT & TTS docs: https://docs.ag2.ai/latest/docs/beta/live/stt_tts/
• ElevenLabs: https://elevenlabs.io/

[stepacool]

Hello

[chiruu12]

Hi @stepacool are you still actively working on this? I don't see any linked PRs or updates in the past couple of weeks, so just checking in.

If it's stalled, I'd be glad to take it on the proposed ElevenLabsTranscriber / ElevenLabsTTSConfig drop-in design maps cleanly onto the existing STTConfig protocol and TTSObserver plumbing, and I'd be happy to implement both providers with mocked unit tests and docs updates per the acceptance criteria. If you've already made progress, I'm equally happy to help with tests, docs, or review instead.

@Lancetnik if there's no objection from @stepacool, could you reassign this to me?

[stepacool]

Good afternoon, @chiruu12 , I'm actually on it and I tihnk there was an issue with the PR. Let me get it open, you can take over/add if it looks unfinished.