Discover Amazon Nova Sonic, a voice model delivering lifelike conversations for next-gen generative AI applications across industries.
Voice interfaces are crucial for improving the customer experience in a variety of contexts, including interactive education, gaming, customer service call automation, and language acquisition. However, developing voice-enabled applications has certain difficulties.
Conventional methods for developing voice-enabled apps necessitate the intricate coordination of several models, including text-to-speech to convert text back to audio, language models to comprehend and provide responses, and speech recognition to turn speech to text.
In addition to making development more difficult, this disjointed method loses important language context like tone, prosody, and speaking style that is necessary for organic dialogues. Conversational AI applications that require low latency and sophisticated comprehension of both verbal and nonverbal cues for smooth dialogue handling and organic turn-taking may be impacted by this.
Today, launching Amazon Nova Sonic, the newest member of the Amazon Nova family of foundation models (FMs) accessible in Amazon Bedrock, to simplify the deployment of speech-enabled apps.
With low latency and industry-leading pricing performance, Amazon Nova Sonic combines speech understanding and generation into a single model that developers can utilize to create conversational AI experiences that are natural and human-like. When creating conversational applications, this integrated method simplifies development and lowers complexity.
Without the need for a separate model, its unified model architecture provides real-time text transcription and expressive speech production. The end product is an adaptive speech response that dynamically modifies its delivery according to the input speech’s prosody, including timbre and tempo.
In order to engage with external services and APIs and carry out tasks in the customer’s environment, including knowledge grounding with enterprise data using Retrieval-Augmented Generation (RAG), developers can use function calling, also referred to as tool use, and agentic workflows when using Amazon Nova Sonic.
With more languages on the horizon, Amazon Nova Sonic offers strong speech recognition for American and British English at launch, regardless of speaking patterns or acoustic settings.
With integrated safeguards for watermarking and content filtering, Amazon Nova Sonic was created with ethical AI at the forefront of innovation.
Amazon Nova Sonic in action
A contact center in the telecom sector serves as the demo’s setting. Amazon Nova Sonic responds to a customer’s request to enhance their subscription plan.
Through the use of tools, the model may communicate with other systems and leverage agentic RAG with Amazon Bedrock Knowledge Bases to collect up-to-date, customer-specific data, including price information, subscription plans, and account details.
In the demo, streaming speech input is transcribed, and streaming speech responses are shown as text. The conversation’s sentiment is shown in two different ways: a pie chart that shows the general distribution and a time chart that shows how it changes over time. Additionally, there is a section on AI insights that offers call center agents contextual advice. The average response time and the overall chat time distribution between the customer and the agent are two more intriguing metrics displayed in the web interface.
You may see how customer sentiment increases during the chat with the support person by looking at the stats and listening to the voices.
An example of how Amazon Nova Sonic responds to disruptions is shown in the video, pausing to listen before naturally carrying on the discussion.
Let’s now examine how to incorporate speech functionality into your apps.
Using Amazon Nova Sonic
Similar to how you would enable other FMs, you must first toggle model access in the Amazon Bedrock console before you can begin using Amazon Nova Sonic. Locate the Amazon Nova Sonic under the Amazon models in the Model access area of the navigation window, then enable it for your account.
Invoke Model With Bidirectional Stream is a new bidirectional streaming API offered by Amazon Bedrock that enables you to build low-latency, real-time conversational experiences on top of the HTTP/2 protocol. To ensure that the dialog flows naturally, you can use this API to transmit audio input to the model and receive audio output in real time.
With this model ID: amazon.nova-sonic-v1:0, you can utilize the new API to access Amazon Nova Sonic.
The model operates using an event-driven architecture on both the input and output streams following session initialization, where you can set up inference parameters.
The input stream contains three main event types:
- System prompt: To establish the conversation’s general system prompt
- Real-time processing of continuous audio input through streaming
- Tool result handling: After tool use is requested in the output events, the tool returns the results of the tool use calls to the model.
Likewise, the output streams contain three sets of events:
- Automatic speech recognition (ASR) streaming: The output of real-time speech recognition is converted into a speech-to-text transcript.
- Tool use handling: In the case that a tool use event occurs, it must be handled with the data supplied here, with the output being returned as input events.
- Audio output streaming: Because the Amazon Nova Sonic model produces audio more quickly than real-time playback, a buffer is required in order to play output audio in real-time.
The Amazon Nova model cookbook repository contains examples of how to use Amazon Nova Sonic.
Prompt engineering for speech
When creating prompts for the Amazon Nova Sonic, you should focus on conversational flow and intelligibility when heard rather than viewed, and optimize the text for auditory comprehension rather than visual reading.
When assigning your assistant a role, emphasize conversational qualities (such being kind, understanding, and succinct) above text-oriented qualities (like being thorough, methodical, and detailed). The following could be a decent baseline system prompt:
You're a friend. The transcripts of a natural, real-time discussion will be exchanged verbally between you and the user. For conversational situations, keep your answers brief usually two or three sentences.
In general, while developing speech model prompts, refrain from asking for sound effects, vocal characteristic changes (e.g., singing, age, or accent), or visual formatting (e.g., tables, bullet points, or code blocks).
Things to know
The US East (N. Virginia) AWS Region now offers Amazon Nova Sonic. To view the pricing models, go to Amazon Bedrock pricing.
The Amazon Nova Sonic is capable of comprehending speech in a variety of speaking styles and producing speech in expressive voices that sound both feminine and masculine in a range of English dialects, including American and British. More language support will be available soon.
Amazon Nova Sonic is resistant to background noise and seamlessly manages user interruptions without losing the content of the discussion. The model has a default session limit of 8 minutes and supports a context window of 32K audio tokens with a rolling window to accommodate longer chats.
The new bidirectional streaming API is supported by the AWS SDKs listed below:
- C++ AWS SDK
- Java AWS SDK
- AWS JavaScript SDK
- Kotlin’s AWS SDK
- AWS Ruby SDK
- AWS SDK for Rust
- Swift AWS SDK
This new experimental SDK is available to Python developers and facilitates the use of Amazon Nova Sonic’s bidirectional streaming features.
Amazon Nova Sonic offers the framework for organic, captivating voice interactions, whether you’re developing conversational experiences, language learning apps, or customer support solutions.
