At SIGGRAPH, NVIDIA unveiled fVDB, a new deep-learning framework designed to produce virtual worlds that are AI-ready.
fVDB
It is an open-source deep learning framework for sparse, large-scale, high-performance spatial intelligence that was created by NVIDIA. On top of OpenVDB, it constructs NVIDIA-accelerated AI operators to enable 3D generative AI, neural radiance fields, digital twins at the scale of reality, and other features.
The fVDB PyTorch extension can be accessed through the fVDB Early Access program.
Generative Artificial Intelligence with Spatial Perception
It offers high resolution and large datasets for 3D deep learning infrastructure. Based on the VDB format, it integrates essential AI operators into a unified, cohesive framework. The foundation for generative physical AI with spatial intelligence is provided by fVDB.
Superior Capabilities, Elevated Resolution
On top of NanoVDB, which offers OpenVDB GPU acceleration, are created fVDB AI operators. Real-time optimised ray tracing and sparse convolution are among the functions supported by the framework. Throughput of data processing is maximised and memory footprint is minimised with fVDB, enabling faster training and real-time inference.
Smooth Integration
It has the ability to read and write VDB datasets right out of the box if you’re already using the VDB format. It is compatible with various tools and libraries, including the Kaolin Library for 3D deep learning and Warp for Pythonic spatial computation. It’s easy to integrate fVDB into your current AI workflow.
Examine Important Elements
Entire Operator Set
Differentiable operators such as convolution, pooling, attention, and meshing are available in fVDB and are specifically tailored for high-performance 3D deep learning applications. With the help of these operators, you may create intricate neural networks for spatial intelligence applications, such as 3D generative modelling and large-scale point cloud reconstruction.
Quicker Ray Tracing
To provide quick and precise ray tracing, it makes use of the Hierarchical Digital Differential Analyser (HDDA) technique included in OpenVDB. Neural radiance fields (NeRFs) can be quickly trained at the city-scale to produce ray-traced visualisations.
Sparse Convolution Optimisation
Sparse convolution operators in fVDB are capable of handling large 3D datasets. For activities like volumetric data analysis and physics simulation, fVDB provides quick and high-accuracy spatial data processing by optimising memory access patterns and computing burden.
Coming Soon: fVDB NIMs
Soon, NVIDIA NIM inference microservices with fVDB capability will be available, allowing developers to integrate the fVDB core architecture into USD processes. In NVIDIA Omniverse, fVDB NIMs produce geometry based on OpenUSD.
It is built on top of OpenVDB, the industry standard framework for modelling and visualising sparse volumetric data, like smoke, water, fire, and clouds.
Physical AI that is generated through generative means, like self-driving cars and real-world robots, must possess “spatial intelligence,” or the capacity to perceive, comprehend, and act in three dimensions.
It is crucial to capture both the vast scope and incredibly minute details of the environment we live in. However, it is challenging to turn reality into a virtual representation for AI training.
There are numerous ways to gather raw data for real-world settings, including lidar and neural radiance fields (NeRFs). This data is translated into huge, AI-ready settings that are displayed in real time by fVDB.
The debut of fVDB at SIGGRAPH marks a significant advancement in the ways that sectors can profit from digital twins of the real world, building on a decade of progress in the OpenVDB standard.
Agents are trained in virtual worlds that are realistically scaled. Drones are used to collect city-scale 3D models for disaster preparedness and climate science. These days, smart cities and metropolitan areas are even planned using 3D generative AI.
By utilising this, industries may leverage spatial intelligence at a greater scale and resolution than previously possible, resulting in even more intelligent physical AI.
Based on NanoVDB, a GPU-accelerated data structure for effective 3D simulations, the framework constructs NVIDIA-accelerated AI operators. Convolution, pooling, attention, and meshing are some of the operators in this group; they are all intended for use in high-performance 3D deep learning applications.
Businesses can create sophisticated neural networks for spatial intelligence, such as 3D generative modelling and large-scale point cloud reconstruction, by using AI operators.
The outcome of a protracted endeavour by NVIDIA’s research team, It is currently utilised to assist projects under NVIDIA Research, NVIDIA DRIVE, and NVIDIA Omniverse that necessitate high-fidelity representations of expansive, intricate real-world environments.
Principal Benefits of fVDB
- Larger: Four times the geographical dimension of earlier frameworks
- Quicker: 3.5 times quicker than earlier frameworks
- Interoperable: Companies have complete access to enormous real-world datasets. VDB datasets are read into full-sized 3D environments with fVDB. Real-time rendered and AI-ready for developing spatially intelligent physical AI.
- Greater power: Ten times as many operators as previous frameworks. By merging features that previously needed several deep-learning libraries, fVDB streamlines procedures.
It will soon be offered as microservices for NVIDIA NIM inference. Three of the microservices will let companies integrate fVDB into OpenUSD workflows and produce AI-ready OpenUSD geometry in NVIDIA Omniverse, a platform for developing generative physical AI applications for industrial digitalisation. They are as follows:
- fVDB Mesh Generation NIM: Creates virtual 3D worlds based on reality
- fVDB NeRF-XL NIM: Utilising Omniverse Cloud APIs, it creates extensive NeRFs in OpenUSD.
- fVDB Physics Super-Res NIM Produces an OpenUSD-based, high-resolution physics simulation by performing super-resolution.
OpenVDB, a key technology utilised by the visual effects industry, has won many Academy Awards in the last ten years while being headquartered in the Academy Software Foundation. Since then, its applications in industry and science have expanded to include robots and industrial design in addition to entertainment.
NVIDIA keeps improving the OpenVDB library, which is available for free. The startup released NanoVDB four years ago, which gave OpenVDB GPU capability. This resulted in an order-of-magnitude speed increase, making real-time simulation and rendering possible as well as quicker performance and simpler programming.
NeuralVDB
A large-scale volume representation using AI-enabled data compression technology is called NeuralVDB. Compared to OpenVDB, the industry-standard library for modelling and rendering sparse volumetric data, like water, fire, smoke, and clouds, it offers a noticeable increase in efficiency.
With the release of NeuralVDB two years ago, NVIDIA expanded its machine learning capabilities beyond NanoVDB to compress VDB volumes’ memory footprint by up to 100 times. This has made it possible for researchers, developers, and producers to work with incredibly complicated and huge datasets.
On top of NanoVDB, fVDB creates AI operators to enable spatial intelligence at the scale of reality. Submit an application to the fVDB PyTorch extension early-access program. Additionally, fVDB will be accessible through the OpenVDB GitHub repository.
