Overview
ZoomAIRealtimeLLMService provides speech-to-speech conversation with the Voice Agent from Zoom AI Services. The Voice Agent runs the whole voice loop on Zoom’s side: turn detection, speech recognition, the model, and speech synthesis. The service replaces STT, the LLM, and TTS in a pipeline, and supports function calling, interruptions, and transcripts like OpenAI Realtime. The Voice Agent detects turns and handles barge-in itself, so the pipeline needs no VAD analyzer.
Zoom AI Voice Agent API Reference
Pipecat’s API methods for Zoom AI Voice Agent integration
Example Implementation
Complete Zoom AI Voice Agent conversation example
Voice Agent Documentation
Official Zoom Voice Agent API documentation
Zoom AI Services
Get credentials and manage your Zoom AI Services plan
Installation
Prerequisites
Zoom AI Account Setup
- Zoom AI Services access: Sign up through Zoom AI Services.
- API key: Create a Zoom AI Services API key. See Get credentials.
Required Environment Variables
ZOOM_AI_API_KEY: Your Zoom AI Services API key
Configuration
ZoomAIRealtimeLLMService
Zoom AI Services API key. It is sent on every connection and reconnect, so it
must stay valid for as long as the service runs.
Runtime-configurable settings. See Settings below.
Settings
Runtime-configurable settings passed via thesettings constructor argument using ZoomAIRealtimeLLMService.Settings(...). They map to the Voice Agent’s session settings and can be updated mid-conversation with LLMUpdateSettingsFrame. See Service Settings for details.
provider, listen, and speak also accept plain dicts in the same shape, such as {"voice": "ava"}.
Provider
See Bring your own key for the providers and models the Voice Agent supports.
Listen
Speak
Usage
Basic Setup
LLMContext, as for other LLM services. See Function Calling.
Choosing a Voice and Vocabulary
Using Your Own OpenAI or Anthropic Model
Updating Settings Mid-Conversation
Send anLLMUpdateSettingsFrame to change the instructions, tools, or voice during a conversation:
Notes
- Turn detection: The Voice Agent detects turns, noise, and barge-in itself. None of these is configurable.
- Settings fixed at connection:
model,provider, andlistenare fixed once the session is configured. Updating them mid-conversation logs a warning and applies to the next session. Updates to instructions, tools, and voice take effect immediately. - Audio: Input audio is sent at the pipeline’s input sample rate. The agent speaks 16 kHz audio, played at 24 kHz. Audio that arrives while the session is connecting or reconnecting is held (up to the latest three seconds) and sent once the session is ready.
- Transcripts: The caller’s speech is pushed upstream as
TranscriptionFrames, and the agent’s speech downstream asTTSTextFrames. - Text-only replies: When the audio output modality is disabled, the agent replies in text only.
- Reconnection: A session lasts up to 60 minutes. If the session ends unexpectedly (the connection closing, or a session reaching its time limit), the service reconnects, up to 3 times in a row, and restores the conversation. If reconnecting fails, it pushes a permanent error.
- Errors: If Zoom refuses the connection (an invalid API key, for example), the error includes Zoom’s reason. Errors the Voice Agent reports without ending the session are pushed as non-fatal errors. If it rejects the session’s settings (an unknown voice, say) or the provider rejects your provider key, the error is permanent. Errors from the Voice Agent include the session’s tracking ID (
x-zm-trackingid), which Zoom support uses to find a session. See Error Handling. - Function call output: A function call’s output is created right after the call in the conversation, where the Voice Agent’s model reads it.
- System instruction precedence: The
system_instructionfrom service settings takes precedence over an initial system message in the LLM context. A warning is logged when both are set.