Pipeline Lifecycle
StartFrame
The first frame pushed into a pipeline, marking the point at which frames start flowing. Every processor receives this before any DataFrames or ControlFrames arrive; a processor holds anything that reaches it earlier and delivers it, in arrival order, once theStartFrame has passed.
Pipeline configuration reaches processors through FrameProcessorSetup in setup(), which runs before this frame.
CancelFrame
Stops the pipeline immediately, skipping any queued non-SystemFrames. Use this when you need to abort without waiting for pending work to drain. For example, when the user has left the session.Errors
ErrorFrame
Carries an error notification, typically pushed upstream so earlier processors can react.Human-readable error message. Credentials it quotes (API keys, tokens, signed
URL signatures, auth headers, URL passwords) are replaced with
[REDACTED].What kind of failure this was — a rejected API key (
AUTHENTICATION) versus a
provider outage (SERVER), for example. See Error
Handling.The processor that raised the error. Read
processor.is_usable to tell an
error the processor can carry on from apart from one that has finished it.Deprecated since 1.8.0, removed in 2.0.0. Check
processor.is_usable
instead, and set the pipeline’s
processor_unusable_policy
to decide what an unusable processor does to the run.FatalErrorFrame
An unrecoverable error requiring the bot to shut down. Thefatal field is always True.
Inherits from ErrorFrame.
Processor Pause/Resume
Resuming always travels the high-priority input queue, so a paused processor is reached regardless of what has queued up behind the pause. Pausing comes in two forms:FrameProcessorPauseFrame is a ControlFrame that takes effect in order, after the frames ahead of it, while FrameProcessorPauseUrgentFrame takes effect immediately.
FrameProcessorResumeFrame
Resumes a previously paused processor, releasing all buffered frames for processing in the order they arrived.FrameProcessorPauseUrgentFrame
Pauses a processor immediately, without waiting for queued frames to drain first.FrameProcessorResumeUrgentFrame
Equivalent toFrameProcessorResumeFrame. Either resumes a paused processor and releases its buffered frames.
Interruptions
InterruptionFrame
Interrupts the pipeline, discarding pending DataFrames and ControlFrames except uninterruptible ones (interruptible is False), such as EndFrame. Typically triggered when the user starts speaking during a bot response.
User Speaking State
UserStartedSpeakingFrame
Indicates that a user turn has begun. By this point, transcriptions are usually already flowing through the pipeline.UserStoppedSpeakingFrame
Marks the end of a user turn. ASpeculationGate releases whatever speculative response it is holding on this frame, since the turn that response answers has now ended.
UserTurnResponseCancelFrame
Cancels the speculative response for the current user turn. Emitted when an eager end of turn is withdrawn, because the user resumed speaking or the committed transcript doesn’t match the eager one. When the user speaks again after a turn’s answer has started but before the turn ended, the user aggregator interrupts the bot, or broadcasts this frame when the turn doesn’t interrupt the bot. The LLM service stops generating, and itsSpeculationGate discards what it was holding — which is everything the response produced, so nothing further down the pipeline has anything to undo.
A system frame so it overtakes the speculative output it cancels.
EagerEndOfTurnCancelFrame is a deprecated alias, deprecated since 1.15.0 and removed in 2.0.0. Use UserTurnResponseCancelFrame instead. Frames pushed under the old name still cancel.
ProposedUserStartedSpeakingFrame
Proposes that a user turn has started. Emitted by a component with its own turn detection, typically an STT or realtime LLM service whose provider reports speech boundaries. It is a proposal rather than a decision: anExternalUserTurnStartStrategy resolves it into a UserStartedSpeakingFrame and broadcasts the interruption.
Its end-of-turn counterpart, ProposedUserStoppedSpeakingFrame, is a ControlFrame. Resolving a start proposal broadcasts an interruption, which has to reach the pipeline ahead of whatever is queued; resolving a stop proposal has to stay ordered against the final TranscriptionFrame, since the strategy needs that text in hand to close the turn on.
UserSpeakingFrame
Emitted by the VAD processor while the user is actively speaking. Useful for UI feedback or suppressing idle timeouts.UserTurnInferenceCompletedFrame
Indicates that the user turn is semantically complete. Emitted by any component that can judge conversational turn completeness — for example an LLM with turn-completion markers, an STT service with built-in turn detection, or a dedicated end-of-turn classifier.ExternalUserTurnCompletionStopStrategy, which is deprecated, finalizes the turn on this frame. An absence of this frame means the turn is not yet considered complete.
UserTurnCompletionFrame
The LLM’s judgment of whether the user’s turn is complete. An LLM service that uses turn-completion markers pushes it upstream once per inference."complete", "short" (the user will continue shortly), or "long" (the
user needs more time). None when the response has no marker. A tool call
before any marker counts as "complete".The id of the
LLMContextFrame that started the inference, or None when it
isn’t known.UserMuteStartedFrame
Broadcast when one or more user mute strategies activate. User mute temporarily suppresses user input while the bot is speaking to prevent interruptions. While muted, theLLMUserAggregator drops incoming user frames (InputAudioRawFrame, TranscriptionFrame, InterimTranscriptionFrame, EagerTranscriptionFrame, UserBackchannelFrame, UserStartedSpeakingFrame, UserStoppedSpeakingFrame, ProposedUserStartedSpeakingFrame, ProposedUserStoppedSpeakingFrame, VAD signals, and InterruptionFrame). Lifecycle frames (StartFrame, EndFrame, CancelFrame) are never muted.
UserMuteStoppedFrame
Broadcast when all active user mute strategies deactivate, allowing user input to be processed again.VAD Events
These frames are emitted directly by the Voice Activity Detection (VAD) processor and carry timing metadata. Higher-level speaking-state frames (UserStartedSpeakingFrame, UserStoppedSpeakingFrame) are derived from these.
VADUserStartedSpeakingFrame
VAD confirmed that speech has started.The VAD’s
start_secs setting: how many seconds of speech it took to confirm
the start. Not a timestamp.VADUserStoppedSpeakingFrame
VAD confirmed that speech has ended.The VAD’s
stop_secs setting: how many seconds of silence it took to confirm
the stop. Not a timestamp.SpeechControlParamsFrame
Broadcast with the current VAD or turn detection parameters at pipeline start and whenever they change at runtime.Bot Speaking State
BotStartedSpeakingFrame
Emitted by the output transport when the bot begins speaking. Broadcast in both directions so processors on either side of the transport can react.BotStoppedSpeakingFrame
Emitted by the output transport when the bot finishes speaking. Also broadcast in both directions.BotSpeakingFrame
Emitted continuously while the bot is speaking. Processors can use this to suppress idle timeouts or drive visual indicators.Connection Status
BotConnectedFrame
The bot has joined the transport room. Only relevant for SFU-based transports: Daily, LiveKit, HeyGen, and Tavus.ClientConnectedFrame
A client or participant has connected to the transport.Input Frames
Input frames carry raw data from transport sources into the pipeline. AsSystemFrames, they are never discarded during interruptions. Incoming user data must always be processed.
InputAudioRawFrame
Raw audio received from the transport. Inherits theaudio, sample_rate, num_channels, and num_frames fields from the AudioRawFrame mixin.
Inherits from AudioRawFrame.
UserAudioRawFrame
Audio from a specific user in a multi-participant session. Inherits fromInputAudioRawFrame.
InputImageRawFrame
Raw image received from the transport. Inheritsimage, size, and format from the ImageRawFrame mixin.
Inherits from ImageRawFrame.
UserImageRawFrame
An image from a specific user, optionally tied to a pending image request. Inherits fromInputImageRawFrame.
Whether appending this image should trigger an LLM completion.
None
preserves the default behavior of triggering one; only an explicit False
suppresses it.The original request frame that triggered this image capture.
InputFileRawFrame
Raw file received from the transport, usually from an RTVI client. Inheritsfile, type, filename, and format from the FileRawFrame mixin.
Inherits from FileRawFrame.
UserFileRawFrame
A file from a specific user. Whenappend_to_context is True, the LLM assistant aggregator adds it to the context as a universal file_base64 or file_url message, which each LLM service converts to its provider format. Unsupported MIME types or source combinations raise an error during conversion rather than being dropped; the offending file message is then removed from the context so later requests are not blocked.
Inherits from InputFileRawFrame.
Custom LLM-specific options for processing this file, such as
detail for
OpenAI or citations for Bedrock. Not passed to the LLM service, so it has no
effect.Whether appending this file should trigger an LLM completion.
None preserves
the default behavior of triggering one; only an explicit False suppresses
it.InputTextRawFrame
Text received from the transport, such as a user typing in a chat interface. Inherits thetext field from TextFrame.
Inherits from TextFrame.
DTMF Input
InputDTMFFrame
A DTMF keypress received from the transport. Inherits fromDTMFFrame.
OutputDTMFUrgentFrame
A DTMF keypress for immediate output, bypassing the normal frame queue. Setbuttons or button; the frame raises ValueError when neither is set.
Inherits from DTMFFrame.
A single DTMF keypad entry to send, equivalent to
buttons=[button]. Ignored
when buttons is set.One or more DTMF keypad entries to send.
OutputDTMFUrgentFrame.from_string("123#") builds this from a dial string.Transport Messages
InputTransportMessageFrame
A message received from an external transport. The message format is transport-specific.OutputTransportMessageUrgentFrame
An outbound transport message that bypasses the normal queue for immediate delivery.Function Calling
FunctionCallsStartedFrame
Signals that one or more function calls are about to begin executing.FunctionCallCancelFrame
Signals that a function call was cancelled, typically due to user interruption when the function’scancel_on_interruption flag is set.
Whether to run the LLM after the cancellation. Set only when the call is
cancelled by its own timeout.
ExternalFunctionCallsStartedFrame
Reports function calls made outside this pipeline, such as by a backend thatOpenAILiveLLMService delegates to. Nothing in the pipeline acts on the ExternalFunctionCall*Frame family; observers such as RTVIObserver report them as they report the matching FunctionCall*Frame. Each call’s later phases arrive as ExternalFunctionCallInProgressFrame, ExternalFunctionCallResultFrame, and ExternalFunctionCallCancelFrame.
The calls made. Each
ExternalFunctionCall has a function_name and a
tool_call_id.ExternalFunctionCallInProgressFrame
An external function call is running.ExternalFunctionCallResultFrame
An external function call produced a result.Whether this result completes the call.
False marks an intermediate result
in a stream that ends with a final one.ExternalFunctionCallCancelFrame
An external function call was cancelled.User Interaction
UserImageRequestFrame
Requests an image from a specific user, typically to capture a camera frame for vision processing.STTMuteFrame
Mutes or unmutes the STT service. While muted, incoming audio is not sent to the STT provider.UserIdleTimeoutUpdateFrame
Updates the user idle timeout at runtime. Setting the timeout to0 disables idle detection, and setting a positive value enables it. Updates apply immediately: a running idle timer restarts with the new duration, and if the bot is waiting for the user to speak, the timer is armed right away.
Diagnostics
MetricsFrame
Performance metrics collected from processors. Emitted when metrics reporting is enabled in the pipeline configuration.Service Metadata
ServiceMetadataFrame
Base metadata frame broadcast by services at startup, providing information about service capabilities and configuration.The turn strategies the service recommends, when it has an opinion — a service
with its own turn detection uses this to ask for
ExternalUserTurnStrategies.STTMetadataFrame
Metadata from an STT service, including latency characteristics used for turn detection tuning. Inherits fromServiceMetadataFrame.
P99 latency in seconds for time-to-final-segment. Used by turn detectors to
calibrate wait times.
LLMServiceMetadataFrame
Broadcast in both directions at pipeline start by an LLM service. Processors on either side, notablyLLMContextAggregatorPair, read is_realtime_service to tell whether a realtime (speech-to-speech) service is in the pipeline and configure themselves accordingly. See Realtime (Speech-to-Speech) Services.
Inherits from ServiceMetadataFrame.
RTVI
Frames for the Real-Time Voice Interface (RTVI) protocol, which bridges clients and the pipeline. These frames handle custom messaging between the client and server.RTVIServerMessageFrame
Sends a server message to the connected client.RTVIClientMessageFrame
A message received from the client, expecting a server response viaRTVIServerResponseFrame.
RTVIServerResponseFrame
Responds to anRTVIClientMessageFrame. Include the original client message frame to ensure the response is properly correlated. Set the error field to respond with an error instead of a normal response.
Error message. When set, the client receives an
error-response instead of a
server-response.Pipeline Worker Frames
Pipeline worker frames provide a system-priority mechanism for requesting pipeline actions from outside the normal frame flow. They are converted into their corresponding standard frames when processed.WorkerSystemFrame
Base class for system-priority worker frames.CancelWorkerFrame
Requests immediate pipeline cancellation. Converted to aCancelFrame when processed by the pipeline.
Inherits from WorkerSystemFrame.
InterruptionWorkerFrame
Requests a pipeline interruption. Converted to anInterruptionFrame when processed.
Inherits from WorkerSystemFrame.