Overview
NVIDIA Nemotron Speech provides three STT service implementations:NvidiaSTTService— Real-time streaming transcription using Nemotron ASR Streaming models with interim results and continuous audio processing.NvidiaSegmentedSTTService— Segmented transcription using Canary models with advanced language support, word boosting, and enterprise-grade accuracy.NvidiaSageMakerSTTService— Streaming transcription using NVIDIA Nemotron ASR via an AWS SageMaker bidirectional-stream endpoint with automatic reconnection on error.
NVIDIA Nemotron Speech STT API Reference
Pipecat’s API methods for NVIDIA Nemotron Speech STT integration
Example Implementation
Complete example with NVIDIA services integration
NVIDIA ASR NIM Documentation
Official NVIDIA ASR NIM documentation
NVIDIA Developer Portal
Access API keys and Nemotron Speech services
Installation
To use NVIDIA Nemotron Speech services, install the required dependency:Prerequisites
NVIDIA Nemotron Speech Setup
Before using NVIDIA Nemotron Speech STT services, you need:- NVIDIA Developer Account (for cloud deployments): Sign up at NVIDIA Developer Portal
- API Key (for cloud deployments): Generate an NVIDIA API key for Nemotron Speech services
- Model Selection: Choose between Nemotron ASR Streaming (streaming) and Canary (segmented) models
Environment Variables
NVIDIA_API_KEY: Your NVIDIA API key for authentication (required for cloud endpoint, not needed for local deployments)
NvidiaSTTService
Real-time streaming transcription using NVIDIA Nemotron Speech’s streaming ASR models.NVIDIA API key for authentication. Required when using the cloud endpoint. Not
needed for local deployments.
NVIDIA Nemotron Speech server address. For local deployments, pass the local
address (e.g.
localhost:50051).Deprecated since 0.0.105, removed in 2.0.0. Use
settings=NvidiaSTTService.Settings(...) instead.Runtime-configurable settings. See Settings below.
Whether to use SSL for the gRPC connection. Defaults to
True for the NVIDIA
cloud endpoint. Set to False for local deployments.Custom Nemotron Speech configuration string (e.g.
"enable_vad_endpointing:true,neural_vad.onset:0.65").P99 latency from speech end to final transcript in seconds. Override for your
deployment. See stt-benchmark.
Settings
Runtime-configurable settings passed via thesettings constructor argument using NvidiaSTTService.Settings(...). These can be updated mid-conversation with STTUpdateSettingsFrame. See Service Settings for details.
Usage
Notes
- Model cannot be changed after initialization: Use the
model_function_mapparameter in the constructor to specify the model and function ID. - Streaming: Provides real-time interim and final results through continuous audio streaming.
- Automatic reconnection: The service automatically reconnects if the gRPC stream drops unexpectedly, maintaining continuous transcription.
- Speaker diarization: With
speaker_diarization=True, each contiguous run of words from one speaker in a final result is pushed as its own transcription frame, with the NVIDIA speaker tag (numbered from1) inspeaker.label. Usespeaker_format, for example"Speaker {speaker_number}: {text}", to label the text. A final result with an untagged word, no word information, or words that can’t be located in its transcript is pushed unsplit and unformatted, with nospeaker. Interim results are always pushed that way. Changingspeaker_formatat runtime doesn’t reconnect the stream. - Metrics support: This service supports metrics generation (
can_generate_metrics()returnsTrue).
NvidiaSegmentedSTTService
Batch/segmented transcription using NVIDIA Nemotron Speech’s Canary models. Processes complete audio segments after VAD detects speech boundaries.NVIDIA API key for authentication. Required when using the cloud endpoint. Not
needed for local deployments.
NVIDIA Nemotron Speech server address. For local deployments, pass the local
address (e.g.
localhost:50051).Deprecated since 0.0.105, removed in 2.0.0. Use
settings=NvidiaSegmentedSTTService.Settings(...) instead.Runtime-configurable settings. See Settings below.
Whether to use SSL for the gRPC connection. Defaults to
True for the NVIDIA
cloud endpoint. Set to False for local deployments.Custom Nemotron Speech configuration string (e.g.
"enable_vad_endpointing:true,neural_vad.onset:0.65").P99 latency from speech end to final transcript in seconds. Override for your
deployment. See stt-benchmark.
Settings
Runtime-configurable settings passed via thesettings constructor argument using NvidiaSegmentedSTTService.Settings(...). These can be updated mid-conversation with STTUpdateSettingsFrame. See Service Settings for details.
Usage
Notes
- Model cannot be changed after initialization: Use the
model_function_mapparameter in the constructor to specify the model and function ID. - Segmented processing: Processes complete audio segments for higher accuracy compared to streaming.
- Language support: See the NVIDIA ASR NIM documentation for the languages each model supports.
- Word boosting: Use
boosted_lm_wordsandboosted_lm_scoreto improve recognition of domain-specific terms.
NvidiaSageMakerSTTService
Streaming speech recognition using NVIDIA Nemotron ASR via an AWS SageMaker bidirectional-stream endpoint. This service maintains a persistent HTTP/2 bidi-stream connection to a deployed SageMaker endpoint that proxies to NVIDIA NIM’s realtime WebSocket API.Runtime-configurable settings. See Settings below.
P99 latency from speech end to final transcript in seconds. Override for your
deployment.
Settings
Runtime-configurable settings passed via thesettings constructor argument using NvidiaSageMakerSTTService.Settings(...). Changes sent with STTUpdateSettingsFrame are stored but not applied to the open session. See Service Settings for details.
Usage
Notes
- AWS SageMaker deployment required: This service requires a deployed SageMaker endpoint running NVIDIA Nemotron ASR NIM. See the deployment example for setup instructions.
- AWS credentials: Requires
AWS_ACCESS_KEY_IDandAWS_SECRET_ACCESS_KEYenvironment variables for SageMaker authentication. - Environment variables:
SAGEMAKER_ASR_ENDPOINT_NAMEfor the endpoint name. - Automatic reconnection: The service automatically reconnects on error to maintain continuous transcription.
- Session start failures: If the SageMaker session fails to start, the service pushes an
ErrorFramecategorized asRATE_LIMIT,SERVER,CONNECTIVITY, orINVALID_REQUESTand is marked unusable (is_usableisFalse). - NIM-driven finalization: Pushes interim transcripts as NIM returns them and a final transcript when NIM completes an utterance. VAD events don’t trigger finalization.
- Metrics support: This service supports metrics generation (
can_generate_metrics()returnsTrue).