New Features in AMD ROCm 6.4: Plug-and-Play Containers, Modular Deployment, and Groundbreaking Inference for Scalable AI on AMD Instinct GPUs
As the size and complexity of contemporary AI workloads increase, so do the demands for deployment simplicity and performance. For companies developing the AI and HPC of the future on AMD Instinct GPUs, AMD ROCm 6.4 represents a significant advancement.
ROCm software keeps gaining traction with expanding support across top AI frameworks, optimised containers, and modular infrastructure tools, enabling users to stay in control of their AI infrastructure, innovate more quickly, and work more intelligently.
The AMD ROCm 6.4 software provides a smooth route to high performance with AMD Instinct GPUs, regardless of whether you’re managing big GPU clusters, training multi-billion parameter models, or distributing inference over multi-node clusters.
In order to make AI development quick, easy, and scalable, this article highlights five significant advancements in AMD ROCm 6.4 that directly address typical problems encountered by infrastructure teams, model developers, and AI researchers.
ROCm Containers for Training and Inference: Plug-and-Play AI on Instinct GPUs
It takes a lot of time, is prone to mistakes, and slows down iteration cycles to set up and maintain optimal conditions for training and inference. The AMD ROCm 6.4 program presents a robust collection of pre-optimized, ready-to-run training and inference containers made especially for AMD Instinct GPUs.
- Built for low-latency LLM inference, vLLM (Inference Container) supports plug-and-play open models including the most recent Gemma 3 (day-0), Llama, Mistral, Cohere, and others.
- With DeepGEMM, FP8 support, and simultaneous multi-head attention, SGLang (Inference Container) offers excellent throughput and efficiency for DeepSeek R1 and agentic processes.
- LLM training on AMD Instinct MI300X GPUs is made easier with the help of PyTorch (Training Container), which comes with performance-tuned variants of PyTorch that support sophisticated attention techniques. Now optimised for FLUX, Llama 2 (70B), and Llama 3.1 (8B, 70B).1-dev.
- Training Container Megatron-LM Llama 3.1, Llama 2, and DeepSeek-V2-Lite are examples of large-scale language models that can be effectively trained using this specially ROCm-tuned fork of Megatron-LM.
These containers give AI researchers quicker access to turnkey environments so they may conduct experiments and assess new models. Model developers can benefit from pre-tuned support for the most sophisticated LLMs available today, including as DeepSeek, Gemma 3, and Llama 3.1, without having to invest time in intricate configuration. Additionally, these containers facilitate easier maintenance and a more seamless scale-out for infrastructure teams by providing uniform, reproducible deployment across development, testing, and production environments.
PyTorch for ROCm Receives a Significant Improvement: Quicker Focus for Quicker Training
Ineffective attention techniques can quickly become a significant bottleneck, slowing iteration and raising infrastructure costs as training large language models (LLMs) continues to push the boundaries of computation and memory. Within the PyTorch framework, the AMD ROCm 6.4 software offers significant performance improvements, such as enhanced Flex Attention, TopK, and Scaled Dot-Product Attention (SDPA).
- Flex Attention: Provides a notable improvement in performance over ROCm 6.3, significantly cutting down on memory overhead and training time, particularly in LLM workloads that depend on sophisticated attention methods.
- TopK: TopK processes now operate up to three times quicker, speeding up inference reaction times without sacrificing output quality (source).
- SDPA: longer-context, smoother inference.
These enhancements result in quicker training times, lower memory overhead, and better hardware use. As a result, model developers can more effectively refine larger models, AI researchers can conduct more experiments in less time, and Instinct GPU users eventually enjoy reduced time-to-train and increased return on infrastructure expenditures.
The ROCm PyTorch container comes with these upgrades pre-installed.
Performance of Next-Gen Inference on AMD Instinct GPUs Using vLLM and SGLang
It is always difficult to provide low-latency, high-throughput inference for large language models, particularly when new models appear and demands for deployment speed increase. Inference-optimized builds of vLLM and SGLang, tailored for AMD Instinct GPUs, are ROCm 6.4’s direct response to this issue. This release enables AI researchers to get faster time-to-results on large-scale benchmarks, while model developers may deploy real-world inference pipelines with minimal modification or rewrite, to its strong support for popular models like Grok, DeepSeek R1, Gemma 3, and Llama 3.1 (8B, 70B, and 405B). Stable, production-ready containers that receive weekly upgrades enable infrastructure teams guarantee performance, consistency, and dependability at scale.
- SGLang with DeepSeek R1: Set a new throughput record on the Instinct MI300X
- vLLM with Gemma 3: Day-0 compatibility for smooth Instinct GPU deployment
When combined, these tools offer a full-stack inference environment, with weekly and bi-weekly updates for the development and stable containers, respectively.
AMD GPU Operator for Smooth Instinct GPU Cluster Management
Performance and dependability may be hampered by the manual driver upgrades, operational outages, and restricted visibility into GPU health that are frequently associated with scaling and managing GPU workloads across Kubernetes clusters. The AMD GPU Operator streamlines cluster operations from start to finish with AMD ROCm 6.4 by automating GPU scheduling, driver lifecycle management, and real-time telemetry. This means that AI and HPC administrators can confidently deploy AMD Instinct GPUs in air-gapped and secure environments with full observability, infrastructure teams can carry out upgrades with little interruption, and Instinct customers can take advantage of increased uptime, lower operational risk, and more robust AI infrastructure.
Among the new features are:
- Cordon, drain, and reboot automatically for rolling upgrades.
- Increased support for Ubuntu 22.04/24.04 and Red Hat OpenShift 4.16–4.17, which helps guarantee compatibility with contemporary cloud and business environments.
- Device Metrics Exporter for measuring health in real time, based on Prometheus.
Using the New Instinct GPU Driver, Software Modularity
Coupled driver stacks decrease interoperability across environments, slow down upgrade cycles, and raise maintenance risk. The Instinct GPU Driver, a modular driver design that isolates the kernel driver from ROCm user space, is introduced by the AMD ROCm 6.4 program.
principal advantages,
- Now, infra teams can independently update ROCm libraries or drivers.
- Extended compatibility period of 12 months (compared to 6 months in previous editions)
- Greater adaptability in deploying ISV software, bare metal, and containers
This makes fleet-wide updates easier and lowers the chance of breaking changes, which is particularly helpful for cloud providers, government agencies, and businesses with stringent SLAs.
Bonus Point: AITER for Accelerated Inference
AMD ROCm 6.4 includes AITER, a high-performance inference library with drop-in, pre-optimized kernels that eliminates laborious tweaking.
Provides (source):
- The decoder can execute up to 17 times faster.
- 14X increases in multi-head focus
- 2X LLM throughput for inference
