Gemma 3 Quantization Aware Training(QAT) Models: Introducing AI to GPUs for consumers.
Google released Gemma 3, its latest open model, last month. With its inherent BFloat16 (BF16) precision, Gemma 3 swiftly became a top model that could run on a single high-end GPU, such as the NVIDIA H100, while still delivering state-of-the-art performance.
Google is releasing new versions of Gemma 3 that are optimized with Quantization Aware Training(QAT), which significantly lowers memory needs without sacrificing quality, to make it even more accessible. This lets you run Gemma 3 27B locally on consumer-grade GPUs like the NVIDIA RTX 3090.
Understanding performance, precision, and quantization
The performance (Elo score) of newly published large language models is displayed in the above chart. Compare two anonymous models’ responses; greater bars imply better performance. The projected number of NVIDIA H100 GPUs required to run that model using the BF16 data format is shown beneath each bar.
For this comparison, why BFloat16? A popular numerical format for inferring numerous large models is BF16. It indicates that 16 bits of precision are used to express the model parameters. It is easier to compare models in a shared inference setup when using BF16 for all models. By excluding factors like disparate hardware or optimization strategies like quantization, which shall be discussed next, this enables us to compare the intrinsic capabilities of the models themselves.
It’s crucial to remember that, although BF16 is used in this chart for fair comparison, deploying the largest models frequently necessitates the use of lower-precision formats, such as FP8, in order to reduce enormous hardware requirements (such as the number of GPUs), possibly sacrificing performance in the process.
The Need for Accessibility
Also heard you clearly: you want the power of Gemma 3 on the hardware you already own, even though top performance on top-tier hardware is fantastic for cloud deployments and research. Enabling effective performance on consumer-grade GPUs found in PCs, laptops, and even phones is part of the effort to make strong AI accessible.
Performance Meets Accessibility with Quantization Aware Training(QAT) in Gemma 3
Quantization is useful in this situation. Quantization lowers the accuracy of the numbers (the model’s parameters) that AI models store and use to compute answers. Quantization can be compared to condensing an image by using fewer colors. It can use less bits, such as 8 (int8) or even 4 (int4), rather than 16 bits per integer (BFloat16).
When utilising int4, each integer is represented by 4 bits, reducing data size by 4x compared to BF16. Google introduces Gemma 3 models that are robust to quantization because quantization frequently results in performance reduction. For each Gemma 3 model, one can release some quantized variations that allow inference using your preferred inference engine, including MLX, llama.cpp, and Q4_0 (a popular quantization format) for Ollama.
How do can keep things high-quality? Quantization Aware Training(QAT) is what the developers utilize. QAT integrates the quantization process during training rather than only after the model has been fully trained. In order to enable quantization for smaller, faster models with less degradation later on while preserving accuracy, Quantization Aware Training(QAT) simulates low-precision operations during training.
Going further, the team utilized probabilities from the non-quantized checkpoint as goals when using Quantization Aware Training(QAT) on about 5,000 steps. Using the llama.cpp perplexity evaluation, the quantize down to Q4_0 and reduce the perplexity loss by 54%.
See the Difference: Massive VRAM Savings
Int4 quantization has a profound effect. Examine how much VRAM (GPU memory) is needed simply to load the model weights.
- Gemma 3 27B: down from 54 GB (BF16) to 14.1 GB (int4).
- Gemma 3 12B: BF16: 24 GB, int4: 6.6 GB.
- 8 GB (BF16) to 2.6 GB (int4) for Gemma 3 4B.
- Gemma 3 1B: drops from 2 GB (BF16) to 0.5 GB (int4).
Run Gemma 3 on Your Device
These significant cuts make it possible to operate more potent, larger models on generally accessible consumer hardware:
- Gemma 3 27B (int4): You may now operate the largest Gemma 3 variant locally on a single desktop NVIDIA RTX 3090 (24GB VRAM) or comparable card.
- Gemma 3 12B (int4): Provides potent AI capabilities to portable computers by operating effectively on laptop GPUs such as the NVIDIA RTX 4060 Laptop GPU (8GB VRAM).
- Models Smaller (4B, 1B): Provide even more accessibility for devices with little resources, such as toasters (if you have a good one) and phones.
Easy Integration with Popular Tools
There want these models to be simple for you to use in the workflow of your choice. Hugging Face and Kaggle offer the official int4 and Q4_0 unquantized Quantization Aware Training(QAT) models. It have teamed up with well-known development tools that make it easy to test the quantized checkpoints based on QAT:
- Ollama: Start up fast with just a single command, all of Google Gemma 3 QAT models are natively supported as of right now.
- LM Studio: With its intuitive interface, you may quickly download and run Gemma 3 Quantization Aware Training(QAT) models on your desktop.
- MLX: Use MLX to infer Gemma 3 QAT models on Apple Silicon in an effective and optimized manner.
- Gemma.cpp: For extremely effective inference straight on the CPU, use the specialised C++ implementation.
- Llama.cpp: Native support of the GGUF-formatted Quantization Aware Training(QAT) models makes it simple to integrate into current workflows.
More Quantization in the Gemma verse
Although the active Gemmaverse offers a wealth of options, a approved Quantisation Aware Training(QAT) models offer a high-quality baseline. These frequently make use of Post-Training Quantization (PTQ), and people like Bartowski, Unsloth, and GGML have made substantial contributions that are easily accessible on Hugging Face. Examining various community alternatives offers a greater range of trade-offs between size, speed, and quality to meet particular requirements.
Start Now
One important step in democratizing AI development is bringing cutting-edge AI performance to technology that is within reach. You may now take advantage of state-of-the-art features on your personal desktop or laptop to Gemma 3 models that have been optimized by Quantization Aware Training(QAT).
Investigate the quantised models and begin construction:
- Utilise Ollama on your computer.
- Locate the Models on Kaggle and Hugging Face
- Utilize Google AI Edge on your mobile device.
