Some day in the not-so-distant future, an independent pottery store owner will instantly and effortlessly display their hand-made vase—straight from the kiln—right in front of their customers’ eyes at a level of detail visually indistinguishable from reality. They’ll use their phone or their smart glasses to take a few pictures of the vase from various angles, upload those images to their ecommerce platform with an app, and moments later they will see, spin, and zoom into their vase in stunning detail directly in their web store. That same day, just moments later, a shopper might search the internet for vases in an augmented reality storefront using their Quest headset or smart glasses, find a pottery store, and click on the “new product series” tab to enter an interactive virtual space where they’ll see a stunning vase right in front of them. They’ll focus their gaze on the ornately crafted handle to see its details. They’ll pick it up with their hands and project it into the corner of the room they’re standing in with a natural gesture, aiming it right at the location they’ve been considering. Seeing that it’s a perfect fit, they’ll be able to complete their purchase with ease.

In e-commerce and immersive reality applications, visually-realistic 3D objects that might one day enable these kinds of experiences are called “digital twins.” The term “twin” is used because the digital representation of the object is indistinguishable from the actual physical counterpart in ways that matter for the application at hand. For most applications, the twin needs to be indistinguishable in its visual characteristics. This entails having extremely accurate geometry down to the sub-millimeter level and having materials with textures and light reflectivity that look just like the real object. Creating this level of realism for a porous and organically shaped hand-made vase with a shiny gloss is no small feat. The large majority of digital twins in e-commerce and immersive applications today are painstakingly handcrafted by teams of technical artists at costs that can in some cases rival those of creating the physical object itself, making the barrier to entry for all but the highest-end products impossible to climb.

3D reconstruction is a central research topic in the field of computer vision. Its goal is the creation of a 3D representation of an object or a scene from 2D images. Making 3D reconstruction robust and efficient is the most important step that the computer vision community can take towards the democratization of digital twin creation for businesses like the pottery shop described above. In recent years, there have been enormous strides in the field with the application of neural networks and deep learning. Some of the most successful methods have focused on creating an implicit representation of the object that is rendered at the time of viewing and re-rendered with each change of view angle. The algorithms for training these models are constantly evolving with the goal of creating digital twins that are increasingly more detailed, robust to various object types and lighting conditions, faster to process for real-time applications on a variety of devices or compute architectures, and ultimately more realistic for end users. But in most of these dimensions, we haven’t arrived at solutions that are usable for the market yet.

We’ve been making significant investments in the development of reconstruction algorithms for digital twin creation for years. At Connect 2022, Meta Founder & CEO Mark Zuckerberg presented some of our research results and described our goal of being able to easily create digital twins from mobile capture for immersive experiences in augmented reality. We’ve made significant progress on improving state-of-the-art dense reconstruction techniques with Neural Radiance Fields and Gaussian splatting. At the same time, we’re also investigating sparse-view 3D reconstruction techniques that will enable the digitization of objects and scenes from casual observations with future smart glasses similar to our research glasses prototype, Project Aria.

As with all machine learning-based methods, data will be a key to further progress in 3D reconstruction for the entire computer vision community. 3D object datasets have grown in size in recent years but are still lacking in geometrical and material realism. Moreover, few of these datasets contain objects that are associated with actual physical-world counterparts. Just as creating digital twins is too costly for small businesses, the cost of creating such a training dataset has been prohibitively high for the academic community.

DTC is a large and highly detailed set of over 2,400 3D object models that are sub-millimeter-level accurate with respect to their physical counterparts and highly photorealistic. We selected common objects found in the home, like toys and kitchen utensils. We used a state-of-the-art scanning rig from Covision Media to capture the geometry and material properties of these objects and then fine-tuned them to achieve unparalleled realism.

The dataset also contains accompanying 2D recordings for a subset of the objects using both a traditional DSLR camera and our Project Aria research glasses to enable research to improve reconstruction for both camera and smart glasses capture applications. Most of the 3D objects have a corresponding physical-world counterpart that can be purchased from common vendors if researchers want to acquire the source objects and do their own source data capture experimentation.

DTC will enable researchers to accelerate the development of new reconstruction techniques with the aim of making reconstruction ready for real-world applications. They can start using the Digital Twin Catalog today to train, fine-tune, or evaluate machine learning-based reconstruction techniques. Visit the DTC page on the Project Aria website for further details on how to access the dataset and get started.

We’re thrilled to announce our partnership with Shopify, a leading global commerce company that provides essential internet infrastructure for commerce, to further accelerate object reconstruction research. Recognized for empowering businesses with its comprehensive tools and services, Shopify understands the critical role of high-quality product media in building a successful online storefront. Given the complexities and substantial costs associated with creating detailed 3D digital models of products, our collaboration aims to bridge this gap.

Through a collaborative partnership between Shopify and participating merchants, we’re leveraging an array of 3D assets provided by these merchants to significantly enhance AI’s ability to generate accurate and detailed digital twins. Over the next few months, we’ll partner to make over 7,000 digital twins available through an academic competition aimed at improving reconstruction algorithms.

“This partnership with Meta is a natural extension of our commitment to innovation,” says Matt Koenig, a product lead working on 3D support at Shopify. “By teaming up, we aim to make high-quality 3D digital assets more accessible and easier to create for entrepreneurs. This initiative is one of many collaborations designed to enhance the capabilities of our platform and empower our merchants to better engage their customers.”

Shopify’s high-end furniture merchant Arhaus Furniture was excited to participate in our research efforts with the hopes of seeing a commercial solution in the future.

“In an industry where experiencing products in person can be prohibitive, this collaboration allows us to bridge the gap with cutting-edge technology, bringing our physical pieces into the digital realm,” says Steve Bauer, senior vice president of e-commerce and digital at Arhaus. “We believe 3D technology has a valuable place in the future of the home furnishings industry, which is why we’re excited to be partnering with Meta.”

Visit the DTC page on the Project Aria website to download the dataset and begin improving your reconstruction algorithms today.

Stay tuned to the Project Aria website blog for more details on the forthcoming competition and how these additional objects can be downloaded and used.