The DeanBeat: Nvidia CEO Jensen Huang says AI will automatically populate 3D images for metaverses

Interested in knowing what’s next for the gaming industry? Join gaming CEOs to discuss emerging parts of the industry in October at GamesBeat Summit Next. Register today.


It takes kinds of artificial intelligence to create a virtual world. nvidia Executive Director Jensen Huang He said this week during a Q&A at a GTC22 online event that the AI ​​will automatically populate 3D images of metaverses.

He believes AI will take the first step in creating 3D objects that populate the vast virtual worlds of the metaverse – and then human creators will take the reins and polish them to their liking. And while that’s a pretty big claim about how smart an AI is, Nvidia has research to back it up.

This morning, Nvidia Research is announcing a new AI model that can help contribute to massive virtual worlds created by ever-increasing numbers of companies and creators that can be easily inhabited with a variety of 3D buildings, vehicles, characters, and more.

These kind of casual photos represent a tremendous amount of hard work. Nvidia said the real world is full of diversity: the streets are lined with unique buildings, with different cars passing and diverse crowds running through them. Handcrafting a 3D virtual world that mirrors this is incredibly time-consuming, making it difficult to fill out a detailed digital environment.

This kind of task is what Nvidia wants to make it easier with it Omniverse Cloud tools and services. He hopes to make developers’ lives easier when it comes to creating metaverse apps. And auto-generated art — as we’ve seen with the likes of DALL-E and other AI models this year — is one way to lighten the burden of building a world of virtual worlds like in snow crash or Ready player one.

Jensen Huang, CEO of Nvidia, speaking at the GTC22 keynote.

In a press Q&A earlier this week, Huang asked what could make the metaverse come faster. He hinted at Nvidia Research’s work, though the company hasn’t spilled the beans until today.

“First of all, you know, the metaverse is created by users. We either made it manually or created it with the help of artificial intelligence.” Huang said, “In the future, it is very likely that we will describe some properties of a house or a city property or something like that. And it’s like this city, or like Toronto, or like New York City, and it creates a new city for us. We may not like him. We can give it additional claims. Or we can keep pressing ‘enter’ until one we want to start from is automatically created. And then, out of that world, we’ll modify it. And so I think artificial intelligence to create virtual worlds is coming true as we speak.”

GET3D details

Trained using only 2D images, Nvidia GET3D generates 3D shapes with high-resolution textures and intricate geometric details. These 3D objects are created in the same format used by popular graphics software applications, allowing users to instantly import their shapes into 3D monitors and game engines for further editing.

The created objects can be used in 3D representations of buildings, outdoor spaces or entire cities, designed for industries including gaming, robotics, architecture and social media.

GET3D can generate an almost unlimited number of 3D shapes based on the trained data. Like an artist turning a piece of clay into a detailed sculpture, the model transforms figures into intricate 3D shapes.

“The gist of it is exactly the technology I was talking about just a second ago called Big Language Models,” he said. “To be able to learn from all the creations of mankind, to be able to imagine a three-dimensional world. And so from words, through a large linguistic model, one day you will come out, triangles, geometry, textures, materials. And then, we will modify it. And because whatever None of them are pre-prepared, none are pre-rendered, all physics simulation and all light simulation must be performed in real time.This is why our latest technology is so important for RTX neural rendering.Because we can’t do it by brute force.We We need artificial intelligence help for us to do that.”

With a training dataset of 2D images of cars, for example, it creates a collection of sedans, trucks, racing cars, and pickups. When trained on animal images, he comes up with creatures such as foxes, rhinos, horses, and bears. Due to the chairs, the model generates various swivel chairs, dining chairs and ergonomic chairs.

“GET3D brings us one step closer to democratizing AI-powered 3D content creation,” said Sanja Fidler, vice president of AI research at Nvidia and head of the AI ​​lab that created the tool. “Its ability to instantly create 3D shapes can be a game-changer for developers, helping them quickly fill virtual worlds with diverse and interesting things.”

GET3D is one of more than 20 research papers and workshops authored by Nvidia that has been accepted at the NeurIPS AI Conference, taking place in New Orleans around the world, from November 26 to December. 4.

Nvidia said that although it is faster than manual methods, previous 3D AI models were limited in the level of detail they could produce. Even modern inverse rendering methods can only create 3D objects based on 2D images taken from different angles, requiring developers to create one 3D shape at a time.

GET3D can instead produce about 20 shapes per second when inference runs on a single Nvidia graphics processing unit (GPU) – it acts like a 2D image-generating adversarial network, while creating 3D objects. The larger and more diverse the set of training data learned from it, the more diverse and diverse it is
Detailed output.

Nvidia researchers trained GET3D on synthetic data consisting of 2D images of 3D shapes taken from different camera angles. The team only took two days to train the model on about 1 million images using Nvidia A100 Tensor Core GPUs.

GET3D gets its name from its ability to create 3D nets with explicit texture – meaning that the shapes you create are in a triangular grid shape, like a papier-mâché model, covered with textured material. This allows users to easily import objects into game engines, 3D modelers and movie viewers – and edit them.

Once creators export the shapes generated by GET3D to a graphics application, they can apply realistic lighting effects as the object moves or rotates in a scene. By integrating another AI tool from NVIDIA Research, StyleGAN-NADA, developers can use text prompts to add a specific style to an image, such as modifying a car into a burning car or taxi, or turning an ordinary house into a single haunted house.

The researchers note that a future version of GET3D could use camera position estimation techniques to allow developers to train the model on real-world data rather than on synthetic data sets. It can also be optimized to support global generation – which means developers can train GET3D on all kinds of 3D shapes at once, rather than having to train it on one object class at a time.

Prologue is Brendan Greene's next project.
Prologue is Brendan Greene’s next project.

Huang said that AI will generate worlds. Those worlds will be simulated, not just animated. To do all this, Huang anticipates the need to create a “new type of data center around the world.” It’s called a GDN, not a CDN. It’s a graphics delivery network, tested by Nvidia’s GeForce Now cloud gaming service. Nvidia took this service and used it to create Omniverse Cloud, a suite of tools that can be used to create Omniverse applications, anytime, anywhere. GDN will host cloud gaming as well as Omniverse Cloud’s metaverse tools.

This type of network can provide the necessary real-time computing for the metaverse.

“This is the interaction that is basically instantaneous,” Huang said.

Any game developers asking for this? Well, actually, I know one. Brendan Greene, creator of Battle Royale PlayerUnknown’s Productions, requested this type of technology this year when he announced and then revealed Prologue. Artemis projectAn attempt to create a virtual world the size of the Earth. He said it can only be built with a combination of game design, user-generated content, and artificial intelligence.

Well, holy shit.

GamesBeat creed When gaming industry coverage is “where passion meets work”. What does this mean? We want to tell you how the news matters to you – not only as a decision maker in a game studio, but also as a game fan. Whether you’re reading our articles, listening to our podcasts, or watching our videos, GamesBeat will help you learn about the industry and enjoy interacting with it. Discover our briefings.

Leave a Comment