Optimising AI and deep learning on AWS Trainium and AWS Inferentia using AWS Neuron SDK
What is AWS Neuron?
The AWS Neuron SDK performs generative AI and deep learning tasks on Amazon EC2 instances powered by AWS Trainium and AWS Inferentia. A compiler, runtime, training and inference libraries, and developer tools for debugging, profiling, and monitoring are included. Building and implementing deep learning and AI models, optimising for maximum performance and minimal cost, and gaining a deeper understanding of model behaviour are all part of Neuron’s end-to-end machine learning (ML) development process.
Native compatibility with widely used machine learning frameworks and libraries
PyTorch, JAX, and key machine learning libraries like Hugging Face Optimum Neurone, PyTorch Lightning, and AXLearn are all fully integrated with Neurone. Additionally, PyTorch, XLA, and JAX developers can utilise Neuron’s compiler optimisations for Inferentia and Trainium because it supports OpenXLA, which includes StableHLO and GSPMD. In addition to third-party services like Ray (Anyscale), Domino Data Lab, Datadog, and Weights & Biases, Neurone allows you to leverage instances based on Trainium and Inferentia with services like Amazon SageMaker, Amazon EKS, Amazon ECS, AWS ParallelCluster, and AWS Batch.
Libraries for distributed training and inference
Using the free source PyTorch libraries NxD Training and NxD Inference, Neurone incorporates pre-built optimisations for distributed training and inference. NxD Training enables several model architectures, parallelism techniques, and training procedures while streamlining and optimising large-scale distributed training. With essential features including on-device sampling, QKV weight fusion, continuous batching, speculative decoding, dynamic bucketing, and distributed inference, NxD Inference offers a complete solution for optimised model inference. Hugging Face TGI and vLLM are two serving solutions that NxD Inference integrates with. A model hub for various model architectures is included in both of them.
Superior competencies in applied science
To enable scientists and researchers to push the limits of open source AI research and innovation on Trainium and Inferentia, Neurone offers a number of applied science capabilities. Researchers can construct and fine-tune compute kernels for best performance by using the Neurone Kernel Interface (NKI), which gives direct access to hardware primitives and instructions available on Trainium and Inferentia. It is a Python-based programming environment that uses tile-level semantics and the widely used Triton-like vocabulary. NKI can be used by researchers to improve deep learning models by adding new features, improvements, and scientific breakthroughs. By building their own operators that are tailored for Inferentia and Trainium, developers can expand the capability of the SDK with Neuron’s proprietary C++ operators.
Strong developer tools
An extensive toolkit for monitoring, managing, and optimising deep learning models on EC2 instances powered by AWS Inferentia and Trainium is provided by the AWS Neuron SDK. It offers tools such as Neuron-top, Neuron-monitor, and Neurone Sysfs to keep an eye on system details, model execution, and hardware resources. Through connection with Amazon CloudWatch and other well-known observability tools like Data Dog and Weights & Biases, Neurone makes monitoring easier for containerised applications running on Kubernetes and EKS. Furthermore, the neuron-profile tool offers native profiling capabilities for well-known machine learning frameworks and aids in locating and resolving performance issues in distributed and single-node applications.
Getting started
Using deep learning Amazon Machine Images
With the Neurone SDK, well-known frameworks, and useful libraries pre-configured, Neurone Deep Learning Amazon Machine Images (Neurone DLAMIs) let you start training and doing inference on AWS Inferentia right away. Neurone DLAMIs simplify your process and maximise efficiency, removing setup hassles so you can concentrate on creating and implementing AI models.
Using Deep Learning Containers
Use pre-configured AWS Neuron Deep Learning Containers (Neurone DLCs) with Trainium and Inferentia frameworks optimised to quickly deploy models. Build your own containers and take advantage of Kubernetes capabilities like Helm Charts, the Neurone Device Plugin, and the Neurone Scheduler Extension for unique solutions. For scalable deployments, easily interface with AWS services such as Amazon EKS, AWS Batch, and Amazon ECS.
Using Hugging Face
Standard Hugging Face APIs for Trainium and Inferentia are provided by Optimum Neurone, which connects Hugging Face Transformers and the AWS Neuron SDK. Both training and inference solutions are provided, along with assistance for large-scale model deployment and training for AI processes. Optimum Neurone makes using Trainium and Inferentia for machine learning easier by supporting Amazon SageMaker and pre-built Deep Learning Containers. Through this integration, developers can use Trainium and Inferentia for transformer-based projects while still working with the well-known Hugging Face interfaces.
Using Amazon SageMaker JumpStart
Neurone models can be trained and deployed with Amazon SageMaker JumpStart. JumpStart facilitates the deployment and fine-tuning of well-known models, including Meta’s Llama family.
