The NVIDIA Dynamo Open-Source Library accelerates and scales AI reasoning models.
Inference Optimizations by NVIDIA Blackwell and Dynamo Reduce Costs and Improve Performance for Scaling Test-Time Compute Increase DeepSeek-R1 throughput 30x. NVIDIA Dynamo, an open-source inference tool, has been announced today to expand and accelerate AI reasoning models in AI factories rapidly and affordably.
Orchestrating and coordinating AI inference requests over a large fleet of GPUs is crucial to running AI factories at the lowest cost and maximizing token income.
As AI reasoning spreads, each AI model will generate tens of thousands of tokens to “think” with each prompt. Improved inference performance and lower inference costs boost service provider growth and revenue.
AI factories that use reasoning AI models can optimize token revenue using NVIDIA Dynamo, which replaces NVIDIA Triton Inference Server. Disaggregated serving divides large language model (LLM) processing and production on GPUs and coordinates and speeds up inference communication among hundreds of GPUs. This optimizes GPU resource use and lets each step be optimized for its needs.
NVIDIA is helping industries worldwide train AI models to think and learn in new ways, making them more complex. “NVIDIA Dynamo helps serve these models at scale, driving cost savings and efficiencies across AI factories, enabling a future of custom reasoning AI.”
Dynamo boosts performance and income for Llama model AI factories on NVIDIA Hopper with the same GPUs. When operated on a large cluster of GB200 NVL72 racks, NVIDIA Dynamo’s intelligent inference optimizations boost token creation by over 30x per GPU for the DeepSeek-R1 model.
For these inference performance improvements, NVIDIA Dynamo leverages technologies that cut costs and accelerate throughput. It can find GPUs in large clusters that minimize response computations and route queries and dynamically add, delete, and reallocate GPUs to meet changing request volumes and types. It can offload inference data to cheaper memory and storage devices and quickly recover them to cut inference costs.
NVIDIA Dynamo is open source and supports PyTorch, SGLang, NVIDIA TensorRT-LLM, and vLLM, allowing researchers, startups, and enterprises to optimize AI model serving across disaggregated inference. AWS, Cohere, CoreWeave, Dell, Fireworks, Google Cloud, Lambda, Meta, Microsoft Azure, Nebius, NetApp, OCI, Perplexity, Together AI, and VAST users can utilize it to accelerate AI inference adoption.
Inference Supercharged
NVIDIA Dynamo spreads the KV cache, which stores inference system knowledge from prior requests, across thousands of GPUs.
It then sends new inference requests to GPUs with the best knowledge match to avoid expensive recomputations and free up GPUs to respond to new requests.
NVIDIA GPUs and inference software provide Perplexity AI the performance, dependability, and scalability businesses and users desire, handling hundreds of millions of requests per month. “With its improved distributed serving capabilities, companies are excited to use Dynamo to meet the compute demands of new AI reasoning models and drive even more inference-serving efficiencies.”
Agentic AI
AI supplier Cohere plans to integrate NVIDIA Dynamo to enable agentic AI in its Command models.
Scaling sophisticated AI models requires complex multi-GPU scheduling, smooth coordination, and low-latency communication libraries that transport reasoning contexts between memory and storage. “As an anticipate that NVIDIA Dynamo will enable us to provide the business clients with an exceptional user experience.”
Disaggregated Serving
The NVIDIA Dynamo inference platform also uses disaggregated serving to distribute LLM computing phases like understanding user queries and delivering the best response to multiple GPUs. This strategy is ideal for reasoning models like the recently launched NVIDIA Llama Nemotron model family, which use advanced inference methods to improve contextual comprehension and response formulation. Disaggregated serving allows resource allocation and phase fine-tuning, improving throughput and response time.
Together AI, the AI Acceleration Cloud, wants to integrate its Together Inference Engine with NVIDIA Dynamo to scale inference workloads across GPU nodes. Therefore, Together AI can dynamically handle traffic blockages at different model pipeline stages.
“Together AI‘s patented inference engine delivers industry-leading performance. NVIDIA Dynamo’s openness and versatility allow one to quickly incorporate its components into the engine to handle more requests while maximizing accelerated processing costs and resource use. It’s excited to employ the platform’s new capabilities to offer affordable open-source reasoning models.
NVIDIA Dynamo Unpacked
Four major NVIDIA Dynamo updates reduce inference serving costs and improve user experience:
- GPU Planner: Dynamically adds and removes GPUs to meet user demand, preventing over- or under-provisioning.
- LLM-aware smart routers distribute requests among large GPU fleets to prevent expensive GPU recomputations of repetitive or overlapping requests and free up GPUs to react to new requests.
- The inference-optimized Low-Latency Communication Library abstracts heterogeneous data sharing and speeds up GPU-to-GPU communication.
- Memory Manager: An engine that automatically offloads and reloads inference data to cheaper memory and storage without compromising user experience.
NVIDIA AI Enterprise will provide production-grade security, support, and stability to NVIDIA Dynamo in the future. NVIDIA NIM microservices will offer it.
NVIDIA Dynamo
Low Latency Distributed Inference for Generative AI. Distributed Inference with Low Latency.
An open-source modular inference system called NVIDIA Dynamo is used to support generative AI models in distributed settings. With dynamic resource scheduling, intelligent request routing, optimized memory management, and expedited data transfer, it makes it possible to scale inference workloads over huge GPU fleets with ease.
NVIDIA Dynamo is the best option for AI factories seeking to operate at the lowest possible cost in order to maximize token income creation because it boosted the number of requests handled by up to 30x when serving the open-source DeepSeek-R1 671B reasoning model on NVIDIA GB200 NVL72.
All of the main AI inference backends are supported by NVIDIA Dynamo, which also offers large language model (LLM)-specific optimizations including disaggregated serving, speeding, and scaling AI reasoning models as efficiently and cheaply as possible. It will be supported in a later release of NVIDIA AI Enterprise.
NVIDIA Dynamo Features
Examine the Features of NVIDIA Dynamo Disaggregated Serving, which divides the LLM context (prefill) and generation (decode) phases among different GPUs. This allows for autonomous GPU allocation and customized model parallelism, which increases the number of requests serviced per GPU.
In order to eliminate bottlenecks and maximize efficiency, GPU Planner dynamically distributes GPU workers between context and generation phases while monitoring GPU capacity in distributed inference systems.
In order to protect compute resources and maintain balanced load distribution across massive GPU fleets, smart routers efficiently infer traffic, reducing the need for expensive recompilation of repetitive or overlapping requests.
Facilitates data transmission across a variety of devices, including as GPUs, CPUs, networks, and storage, while speeding up data flow in distributed inference situations.
NVIDIA Dynamo Advantages
Seamlessly Scale From One GPU to Thousands of GPUs
Avoid over- or under-provisioning of GPU resources by streamlining and automating GPU cluster deployment with prebuilt, simple-to-deploy tools and enabling dynamic autoscaling with real-time LLM-specific data.
Increase Inference Serving Capacity While Reducing Costs
Utilize sophisticated LLM inference serving optimizations, such as disaggregated serving, to service more inference requests while maintaining user experience.
Future-Proof Your AI Infrastructure and Avoid Costly Migrations
By eliminating expensive migration efforts and guaranteeing compatibility with your current AI stack, the open and modular design makes it simple to select the inference-serving components that best meet your specific requirements.
Accelerate Time to Deploy New AI Models in Production
Regardless of the backend, NVIDIA Dynamo‘s support for all key frameworks, such as TensorRT-LLM, vLLM, SGLang, PyTorch, and others, guarantees that you can swiftly implement new generative AI models.
AI Implementation Using NVIDIA Dynamo Use cases
Supporting Models of Reasoning
In order to tackle complicated issues, reasoning models produce more tokens, which raises the cost of inference. These models are optimized by NVIDIA Dynamo using features like disaggregated serving. By separating the prefill and decode computational stages onto different GPUs, this method enables AI inference teams to optimize each stage separately. Better use of resources, more queries answered by each GPU, and reduced inference expenses are the outcomes.
Distributed Inference
It becomes difficult to serve AI models effectively as they are too big to fit on a single node. Because distributed inference necessitates dividing models among several nodes, coordination, scalability, and communication become more difficult. Careful management is required to make sure these nodes operate as a coherent unit, particularly when workloads are dynamic. NVIDIA Dynamo makes this easier by offering pre-built Kubernetes capabilities that handle scheduling, scaling, and serving with ease, allowing you to concentrate on implementing AI rather than infrastructure management.
Scalable AI Agents
Multiple model LLMs, retrieval systems, and specialised tools that operate in real-time synchronization are all necessary for AI agents. Scaling these agents is a difficult task that necessitates ultra-low-latency communication to preserve responsiveness, effective KV cache management, and clever GPU scheduling.
With its integrated intelligent GPU planner, smart router, and low-latency communication library, NVIDIA Dynamo simplifies this procedure and makes AI agent scaling effective and smooth.
Code Generation
Iterative refinement is frequently necessary for code generation in order to modify prompts, define requirements, or debug outputs in response to the model’s replies. Every time a user turns, this back-and-forth requires recalculating the context, which raises the cost of inference. By allowing context reuse and offloading to economical memory, minimizing costly re-computation, and lowering total inference costs, NVIDIA Dynamo optimizes this process.
See What Industry Leaders Have to Say About NVIDIA Dynamo
Cohere
Complex multi-GPU scheduling, smooth coordination, and low-latency communication libraries that smoothly move reasoning contexts between memory and storage are necessary for scaling sophisticated AI models. It anticipate that NVIDIA Dynamo will enable us to provide the enterprise clients with an exceptional user experience.
Perplexity AI
It looking forward to leveraging NVIDIA Dynamo with its enhanced distributed serving capabilities to drive even more inference serving efficiencies and meet the compute demands of new AI reasoning models.” To handle hundreds of millions of requests every month and depend on NVIDIA’s GPUs and inference software to deliver the performance, reliability, and scale to business and users demand.
Together AI
Disaggregated serving and context-aware routing are two examples of new advanced inference approaches that are necessary for scaling reasoning models economically. Together, AI uses the proprietary inference engine to deliver performance that leads the industry. Because of NVIDIA Dynamo‘s openness and modularity, it can easily integrate its components into the engine to handle more requests while maximizing the investment in accelerated computing and optimizing resource usage.
Take a Close Look
Learn how to deploy, run, and scale AI models for inference in computer vision, recommender systems, generative AI, LLMs, and other areas.
