Cloud HPC Toolkit for AI and Machine Learning

Applying Cloud HPC Toolkit to AI and ML workloads

The way we address complicated issues is changing due to the convergence of workloads related to AI and machine learning with high performance computing (HPC) platforms. Because HPC systems provide the AI-enabled computing infrastructure and parallel processing capabilities required to train ML workloads like large language models (LLMs), which are AI models trained on enormous amounts of textual data to produce pertinent responses to requests in natural language, they are well-suited for AI and machine learning workloads.

In the meanwhile, by improving their methods and setups, AI and machine learning workloads may be leveraged to increase the performance of HPC systems. Additionally, AI is being used to either accelerate or replace conventional HPC approaches for solving issues, like AlphaFold’s work on protein folding.

The communities of AI, machine learning, and HPC are finding new ways to collaborate and innovate together as a result of this combination. In order to create and deploy models at scale, companies are employing AI and machine learning frameworks, such as the NVIDIA NeMo framework, on traditional HPC clusters. These systems may be set up on Google Cloud NVIDIA GPU supercomputers, including the A3 virtual machines (VMs) that are equipped with NVIDIA H100 Tensor Core GPUs.

Enhancements to the Google Cloud HPC Toolkit for AI and ML

With the Google Cloud HPC Toolkit, you can quickly and easily create your own simple YAML file or use an existing blueprint to quickly and easily get a cluster up and running in a matter of minutes for your HPC, AI, and machine learning workloads. This set of open-source tools and resources helps you create repeatable, turnkey HPC environments.

We are thrilled to provide today’s Cloud HPC Toolkit enhancements, which make it possible for workloads including AI and machine learning on Google Cloud. To guarantee the greatest performance for your AI and machine learning requirements, we worked with our partner NVIDIA to design the AI and machine learning blueprint. There are blueprints with predefined partitions that support the G2, A2, and A3 NVIDIA GPU VM types.

Furthermore, the systems may use the most recent NCCL Fast Socket improvements and be based on our Ubuntu Deep Learning VM Image. You can now easily interact with unprivileged containers and specify the container in a Slurm task thanks to robust tools like the Pyxis plugin for Slurm Workload Manager included in the blueprint and the enroot container utility. You can quickly create up an HPC environment on Google Cloud that will let you to train your LLMs on NVIDIA GPUs.

Employee the Cloud HPC Toolkit to Implement AI and ML

Using the HPC Toolkit to install clusters is a straightforward procedure that usually requires three steps:

• To configure the Cloud HPC Toolkit for your Google Cloud project, follow the instructions.
• Make a deployment folder for HPC. After customizing one of the github repository’s examples, users often execute ghpc create <path-to-deployment-configuration.yaml>. This saves hundreds of lines of custom setup by creating a deployment folder including all of the auto-generated Packer build scripts, VM starting scripts, and Terraform settings required to deploy your cluster.
• Using ghpc deploy <path-to-deployment-folder/>, deploy the infrastructure. This sets off the deployment of unique virtual machine images, networks, firewalls, and instance templates in addition to the controller nodes and Slurm login that are used to start workloads.

You may train and implement AI models just like any other conventional HPC workload when the HPC cluster has been deployed. Start by logging onto the Slurm login node and doing preliminary verification tests, including

gpus-per-node=8 nvidia-smi srun -N 1

After that, you may install and use the NVIDIA NeMo framework by following these steps. Likewise, this article may also be used to set up LLaMA, which is a basic LLM created by Meta.