Skip to main content

Overview

A classifier answers typed questions about some state. The state is plain text or structured data, such as a transcript with speaker labels or a screen snapshot. A question is a YesNoQuestion, a ChoiceQuestion among options, or a ScoreQuestion on a scale. Questions are asked by name, several about one state at once, and each gets a typed result with probabilities. A classifier is a plain object, not a frame processor. Whoever needs answers creates one, keeps it, and calls it. Nothing is added to a pipeline and no frames flow into it. Pipecat components that make small decisions take one as a classifier argument, such as VoicemailDetector, ClassifierUserTurnStrategies, UIWorker, and the eval judge. For a walkthrough, see Pipecat Classifiers.

JevClassifier

Answers through Jev, TypeSafe’s hosted classification model, with calibrated probabilities.

ClefClassifier

Answers through Cloudflare’s Clef model on Workers AI.

GoogleDecisionsClassifier

Answers through Google’s Decisions API, about images, audio and files too.

HumeAudioClassifier

Answers questions about a clip of audio through Hume’s Audio Decisions API.

OpenAIDecisionsClassifier

Answers through OpenAI’s Decisions API.

LLMClassifier

Answers through any Pipecat LLM service that supports run_inference().

PipecatToneClassifier

Answers Pipecat Tone’s own emotion, move, sarcasm and backchannel questions on the CPU.

PipecatVoiceClassifier

Answers Pipecat Voice Reader’s own tone, sarcasm, laughter and speech rate questions about a clip of audio on the CPU.

Questions

Every field that describes something (instructions, yes, no, an option, a level) takes text, or structured data such as a dict holding the question in one field and what it refers to in others.

YesNoQuestion

Whether the state meets a condition.
str | dict[str, Any] | list[Any]
required
What is being checked for, as a yes or no question.
str | dict[str, Any] | list[Any] | None
default:"None"
What counts as a yes, when the question alone leaves it open.
str | dict[str, Any] | list[Any] | None
default:"None"
What counts as a no.

ChoiceQuestion

Which of several options fits the state.
str | dict[str, Any] | list[Any]
required
What is being decided.
dict[str, str | dict[str, Any] | list[Any] | None]
required
The options to choose from, each mapped to a description of when it applies, or None when the option’s name says enough.

ScoreQuestion

Where the state falls on an ordered scale.
str | dict[str, Any] | list[Any]
required
What is being rated.
list[str | dict[str, Any] | list[Any]]
required
The levels of the scale in order, lowest first, each described in a few words or as structured data. At least two.

Images

A classifier that can see images takes them alongside the state, in the inputs argument of ask(), yes_no(), choice() and score(). Check supports_images to find out whether a classifier can: ClefClassifier, GoogleDecisionsClassifier, JevClassifier, OpenAIDecisionsClassifier and LLMClassifier can.

ClassifierImage

An encoded image the questions can look at.
bytes
required
The encoded image: PNG, JPEG or WebP.
str
default:"read from data"
The image’s MIME type: image/png, image/jpeg or image/webp. Read from data when not given. A ValueError is raised if data is not one of these image types.
await ClassifierImage.from_frame(frame) builds an image from an ImageRawFrame, such as a frame from the user’s camera. Raw pixels are encoded as a JPEG, and a frame whose format is already a MIME type keeps its bytes.

Audio

A classifier that can hear takes clips of audio alongside the state, in the inputs argument of ask(), yes_no(), choice() and score(). Check supports_audio to find out whether a classifier can. Each classifier declares the clips it accepts with min_audio_secs, max_audio_secs and audio_sample_rate, and the length it answers best with, preferred_audio_secs. A clip outside the accepted length raises ClassifierError, and a clip is resampled to audio_sample_rate before the classifier gets it.

ClassifierAudio

A clip of audio the questions can hear.
bytes
required
The clip, as 16-bit mono PCM.
int
required
The clip’s sample rate in Hz.
duration_secs returns how long the clip is, in seconds.

Files

A classifier that takes files, such as a PDF or a video, accepts them alongside the state in inputs. Its file_types property lists the MIME types it takes, and a file of any other type raises ClassifierError. GoogleDecisionsClassifier takes PDFs, videos, images and audio; none of the other classifiers Pipecat includes takes files.

ClassifierFile

An encoded file the questions can look at. One of data and url is required.
str
required
The file’s MIME type, such as application/pdf or video/mp4.
bytes | None
default:"None"
The file’s bytes.
str | None
default:"None"
Where the file can be fetched from, for a classifier whose provider fetches files itself.
A ValueError is raised if both or neither of data and url is given. ClassifierFile.from_frame(frame) builds a file from a FileRawFrame, such as a file the user sent, from its base64 data or its URL.

Results

YesNoResult

The answer to a YesNoQuestion.

ChoiceResult

The answer to a ChoiceQuestion.

ScoreResult

The answer to a ScoreQuestion. ScoreResult.probability(level) returns the probability of one level, given as the question gave it, and raises KeyError if the scale has no such level:

ClassifierError

Raised when a classifier cannot answer: the backend failed or did not reply in time, its reply could not be turned into results, it was given an input of a kind it does not take (images it cannot see, audio it cannot hear, or a file of a type not in file_types), or a clip of audio whose length it does not accept. Every classifier answers or raises within a time bound of its own, so a caller waiting on one is never left hanging.

BaseClassifier

The base class every classifier extends. It is a BaseObject, so it takes an optional name and supports event handlers.

Methods

ask

Answers several questions about one state, in one call. questions can mix kinds. Returns one result per question, under the same names, each of the type its question calls for. inputs are the images, clips of audio and files that come with the state, of the kinds the classifier takes: images, audio and files. A classifier raises ClassifierError when given an input of a kind it does not take, or a clip whose length it does not accept.
Sequence[ClassifierImage] | None
default:"None"
Images the questions can look at.Deprecated since 1.16.0, removed in 2.0.0. Pass the images in inputs instead.

yes_no, choice, score

Typed versions of ask() for questions of one kind, taking the same inputs and images. They return results already typed, so no isinstance check is needed. All four raise ClassifierError when the answers could not be produced.

setup

An owner that runs inside an agent calls this with its task manager before the first question. JevClassifier, ClefClassifier, GoogleDecisionsClassifier, HumeAudioClassifier, and OpenAIDecisionsClassifier also open their connection here, so the first question does not pay for it.

cleanup

Releases the classifier’s resources. Call it when you are done asking.

Properties

Event Handlers

on_metrics

Called after every call with its metrics: a ProcessingMetricsData with the time the call took and, when the classifier knows it, an LLMUsageMetricsData with the tokens it used.
A classifier cannot push frames. When its owner is a frame processor, the owner puts the data in a MetricsFrame so it reaches the pipeline’s observers like any other metrics:

Writing a Classifier

To back classifiers with another model or service, subclass BaseClassifier and implement _ask(). It receives the state, the questions and inputs, which holds only the kinds the classifier takes, and returns the results by name, plus the tokens the call used when that is known (or None). A classifier overrides supports_images, supports_audio or file_types to take those kinds. A classifier that hears also declares the clips it accepts, and the clips in inputs are of an accepted length at audio_sample_rate. Raise ClassifierError when the backend fails. The public methods, typing and metrics are handled by the base class.