Deep Dive: How Synthetic Data Can Enhance AR/VR and the Metaverse

Deep Dive: How Synthetic Data Can Enhance AR/VR and the Metaverse

Were you unable to attend Transform 2022? Check out all the Summit sessions in our on-demand library now! Watch here.


The metaverses have captured our collective imagination. The exponential development of internet-connected devices and virtual content is preparing the metaverse for public acceptance, requiring companies to go beyond traditional methods of metaverse content creation. However, next-generation technologies such as the metaverse, which use artificial intelligence (AI) and machine learning (ML), rely on massive data sets to operate effectively.

This reliance on large data sets brings new challenges. Technology users are becoming more aware of how their sensitive personal data is obtained, stored and used, which has led to regulations designed to prevent organizations from using personal data without express permission.

Without large amounts of accurate data, it is impossible to train or develop AI/ML models, which severely limits metaverse development. As this predicament becomes more pressing, synthetic data is gaining traction as a solution.

In fact, According to GartnerBy 2024, 60% of the data required to create AI and analytics projects will be generated industrially.

Machine learning algorithms generate synthetic data by ingesting real data for training behavior patterns and creating simulated dummy data that retains the statistical properties of the original data set. Such data can replicate real-world conditions and, unlike standard anonymized data sets, is not prone to the same flaws as real data.

Reimagining digital worlds using synthetic data

As AR/VR and metaverse developments progress toward more nuanced digital environments, they now require new capabilities for humans to seamlessly interact with the digital world. This includes the ability to interact with virtual objects, enhance on-device rendering with accurate eye-gaze estimation, visualize a realistic user avatar and create a solid 3D digital overlay over the actual environment. ML models learn 3D objects such as networks, transformable models, and surface rules from photographs and getting such visual data to train these AI models is difficult.

Training a 3D model requires a large amount of face and whole body data, including accurate 3D annotations. The model must also be taught to perform tasks such as hand position and grid estimation, body position estimation, gaze analysis, 3D environment reconstruction, and avatar synthesis for codec.

“The metaverse will be powered by powerful new computer vision machine learning models that can understand the 3D space around a user, accurately capture movement, understand gestures and interactions, and translate emotions, speech, and facial details into realistic avatars,” Yashar BehzadiCEO and founder of Synthesis AI, for VentureBeat.

“To build these basic models, you will require large amounts of data with rich 3D labels,” Behzadi said.

An example of presenting gesture appreciation for digital photos. Source: Synthesis AI

For these reasons, the metaverse is undergoing a paradigm shift – moving away from modeling and towards a data-centric approach to development. Rather than making incremental improvements to an algorithm or model, researchers can improve the performance of an AI model in the metaverse more effectively by improving the quality of the training data.

Traditional approaches to building computer vision rely on human commentators who cannot provide the required labels. However, synthetic data or computer-generated data that simulates reality has proven to be a promising new approach.

By using synthetic data, companies can create customizable data that can make projects run more efficiently as it can be easily distributed among creative teams without worrying about privacy laws compliance. This provides greater autonomy, allowing developers to be more efficient and focus on tasks that generate revenue.

Behzadi says he believes that coupling CVI technologies with synthetic AI models will allow synthetic data technologies to provide massive amounts of diverse and optimally labeled data to power the metaverse.

To improve the user experience, the devices used to enter the metaverse play an equally important role. However, the hardware must be supported by software that makes the transition between the real and virtual worlds seamless, and this would be impossible without computer vision.

To work properly, AR/VR devices need to understand their location in the real world to augment users with a detailed and accurate 3D map of the virtual environment. Therefore, gaze estimation (that is, knowing where a person is looking through an image of their face and eyes), is a critical problem for current AR and VR devices. In particular, virtual reality relies heavily on elaborate rendering, a technique in which the image in the center of the field of view is produced with high resolution and excellent detail, but the image on the edges gradually deteriorates.

The tracking architecture and eye gaze estimation for VR devices spreads an elaborate display. That is, the image in the center of the field of view is produced in high resolution but the image on the edges gradually deteriorates for more efficient performance. Source: Synthesis AI

according to Richard KerrisD., vice president of development platform Omniverse at NVIDIA, synthetic data generation can act as a remedy for such cases, as it can provide visually accurate examples of use cases when interacting with objects or creating environments for training.

“Synthetic data generated with simulation accelerates the development of AR/VR applications by providing integration of continuous development and testing workflows,” Kerris told VentureBeat. “Moreover, when generated from the digital twin of the actual world, this data can help train AI systems on many near-field sensors that are invisible to the human eye, as well as improve the tracking accuracy of location sensors.”

Entering virtual reality, one needs to be an avatar for an immersive virtual social experience. Future intractable environments may need realistic virtual images that represent real people and can capture their poses. However, creating such an avatar is a difficult computer vision problem, which is now addressed through the use of synthetic data.

Kiris explained that the biggest challenges are faced while creating a variety of avatars in high definition, along with accessories such as related clothes, hairstyles and emotes, without compromising privacy.

“Procedural generation of widely diverse digital human characters can create infinitely Different human poses Animating characters for specific use cases. “Creating actions with synthetic data helps process these many types of glyphs,” Kerris said.

Recognizing things with computer vision

To estimate the position and physical properties of 3D objects in digital worlds such as the metaverse, light must interact with the object and its environment to generate an effect similar to the real world. Therefore, AI-based computer vision models of the metallic line require an understanding of object surfaces to accurately render them within a 3D environment.

according to Swapnil SrivastavaGlobal Head of Data and Analytics at Evalueserve, Using synthetic data, AI models can predict and make tracking more realistic based on body types, lighting/lighting, backgrounds, environments, and more.

“Metaverse/omniverse or similar ecosystems will rely heavily on realistic expressive and behavioral humans, and this can now be achieved using synthetic data. It is humanly impossible to annotate 2D and 3D images at a pixel perfect scale. Through synthetic data, this technological and physical barrier is bridged. , allowing for accurate commentary, versatility and customization while ensuring realism,” Srivastava told VentureBeat.

Gesture recognition is another essential mechanism for interacting with virtual worlds. However, building models for accurate hand tracking is complex, given the complexity of hands and the need for 3D positional tracking. Further complicating the task is the need to capture data that accurately represents the diversity of users, from skin tone to the presence of rings, watches, shirt sleeves, and more.

Behzadi says the industry is now using synthetic data to train hand-tracking systems to overcome such challenges.

“By taking advantage of 3D manual models, companies can create massive amounts of precisely 3D tagged data across demographics, confounders, camera perspectives, and environments,” Behzadi said.

“The data can then be produced across environments and camera locations/types for unprecedented diversity since the data generated has no fundamental privacy concerns. This level of detail is much greater than what humans can provide and enables a greater level of realism to operate the metaverse.” , he added.

Srivastava said that compared to the current process, the metaverse will collect more personal data such as facial features, body gestures, health, financial and social preference and biometrics, among many other things.

The protection of these personal data points should be the highest priority. Organizations need effective data management and security policies, as well as an approval governance process. Ensuring ethics in AI will be critical to scaling effectiveness in the metaverse while creating responsible data for training, storage, and deployment of models in production.”

Likewise, Behzadi said synthetic data technologies will allow more inclusive models to be built in ethical and privacy-compliant ways. However, because the concept is new, wide adoption will require education.

“metaverse is a broad and evolving term, but I think we can expect very new and immersive experiences – whether it’s for social interactions, reimagining consumer and shopping experiences, new media types, or applications we haven’t yet imagined,” Behzadi said. com is a step in the right direction to help build a community of researchers and industry partners for technology development.

Creating simulation-ready data sets is a challenge for companies that want to use synthetic data generation to build and operate virtual worlds in the metaverse. Kerris says that off-the-shelf 3D assets are not enough to implement accurate training models.

These data sets must contain the information and characteristics that make them useful. For example, weight, friction, and other factors must be included in the assets in order for them to be useful in training,” Kerris said. “We can expect a growing set of sim-ready libraries from companies, which will help accelerate use cases for synthetic data generation in metaverse applications, for cases industrial use such as robotics and digital twins.”

GamesBeat creed When gaming industry coverage is “where passion meets work”. What does this mean? We want to tell you how important the news is to you – not only as a decision maker in a game studio, but also as a game fan. Whether you’re reading our articles, listening to our podcasts, or watching our videos, GamesBeat will help you learn about the industry and enjoy interacting with it. Discover our briefings.

#Deep #Dive #Synthetic #Data #Enhance #ARVR #Metaverse

Leave a Comment

Your email address will not be published. Required fields are marked *