Skip to main content

Overview

ZoomAIRealtimeLLMService provides speech-to-speech conversation with the Voice Agent from Zoom AI Services. The Voice Agent runs the whole voice loop on Zoom’s side: turn detection, speech recognition, the model, and speech synthesis. The service replaces STT, the LLM, and TTS in a pipeline, and supports function calling, interruptions, and transcripts like OpenAI Realtime. The Voice Agent detects turns and handles barge-in itself, so the pipeline needs no VAD analyzer.

Zoom AI Voice Agent API Reference

Pipecat’s API methods for Zoom AI Voice Agent integration

Example Implementation

Complete Zoom AI Voice Agent conversation example

Voice Agent Documentation

Official Zoom Voice Agent API documentation

Zoom AI Services

Get credentials and manage your Zoom AI Services plan

Installation

Prerequisites

Zoom AI Account Setup

  1. Zoom AI Services access: Sign up through Zoom AI Services.
  2. API key: Create a Zoom AI Services API key. See Get credentials.

Required Environment Variables

  • ZOOM_AI_API_KEY: Your Zoom AI Services API key

Configuration

ZoomAIRealtimeLLMService

str
required
Zoom AI Services API key. It is sent on every connection and reconnect, so it must stay valid for as long as the service runs.
str
Voice Agent WebSocket URL.
ZoomAIRealtimeLLMService.Settings
default:"None"
Runtime-configurable settings. See Settings below.
bool
default:"False"
Whether to start with audio input paused.

Settings

Runtime-configurable settings passed via the settings constructor argument using ZoomAIRealtimeLLMService.Settings(...). They map to the Voice Agent’s session settings and can be updated mid-conversation with LLMUpdateSettingsFrame. See Service Settings for details. provider, listen, and speak also accept plain dicts in the same shape, such as {"voice": "ava"}.

Provider

See Bring your own key for the providers and models the Voice Agent supports.

Listen

Speak

Usage

Basic Setup

Register tools on the LLMContext, as for other LLM services. See Function Calling.

Choosing a Voice and Vocabulary

Using Your Own OpenAI or Anthropic Model

The Voice Agent still transcribes the caller and speaks the reply; only the model’s turn uses your provider account.

Updating Settings Mid-Conversation

Send an LLMUpdateSettingsFrame to change the instructions, tools, or voice during a conversation:

Notes

  • Turn detection: The Voice Agent detects turns, noise, and barge-in itself. None of these is configurable.
  • Settings fixed at connection: model, provider, and listen are fixed once the session is configured. Updating them mid-conversation logs a warning and applies to the next session. Updates to instructions, tools, and voice take effect immediately.
  • Audio: Input audio is sent at the pipeline’s input sample rate. The agent speaks 16 kHz audio, played at 24 kHz. Audio that arrives while the session is connecting or reconnecting is held (up to the latest three seconds) and sent once the session is ready.
  • Transcripts: The caller’s speech is pushed upstream as TranscriptionFrames, and the agent’s speech downstream as TTSTextFrames.
  • Text-only replies: When the audio output modality is disabled, the agent replies in text only.
  • Reconnection: A session lasts up to 60 minutes. If the session ends unexpectedly (the connection closing, or a session reaching its time limit), the service reconnects, up to 3 times in a row, and restores the conversation. If reconnecting fails, it pushes a permanent error.
  • Errors: If Zoom refuses the connection (an invalid API key, for example), the error includes Zoom’s reason. Errors the Voice Agent reports without ending the session are pushed as non-fatal errors. If it rejects the session’s settings (an unknown voice, say) or the provider rejects your provider key, the error is permanent. Errors from the Voice Agent include the session’s tracking ID (x-zm-trackingid), which Zoom support uses to find a session. See Error Handling.
  • Function call output: A function call’s output is created right after the call in the conversation, where the Voice Agent’s model reads it.
  • System instruction precedence: The system_instruction from service settings takes precedence over an initial system message in the LLM context. A warning is logged when both are set.

Event Handlers