The technology needed to create the holodeck
We consider it an essential change.
Recently, while visiting my elderly mother in Germany, I realized that it might be one of the last times I see her in the cozy house she has called home for over twenty years. At that moment, I decided to make the most of the situation and took many photos of the place to preserve the memories: the warm fireplace, the bookshelves filled with family books, and the old garden bench out front, which had been signed by everyone during a special birthday celebration many years ago.
Then, I decided to try something new. I opened the Scaniverse app, a 3D scanner developed by Niantic, the company behind Pokémon Go, and captured some of those objects as if they were 3D. I crouched down and carefully walked around, moving my phone to capture every angle. Although the results were not perfect, the experience was quite profound. When I visualized the captures later, both on my phone and with a virtual reality headset, I was able to view the worn garden bench from all angles, as if I were standing in front of it. The emotion I felt was unexpected.
This type of experience is made possible by Gaussian splatting, an innovative 3D capture method that has emerged in the last two years and is revolutionizing the tech industry. Companies like Niantic and Google are using it to improve their mapping products; Snap has integrated support for "splats," which is what these captured objects are called, into its Lens Studio platform, and Meta is looking to employ Gaussian splatting to create a metaverse resembling the real world.
Tech companies are very excited about Gaussian splatting due to its ability to capture and digitally recreate three-dimensional objects in a photorealistic manner. It is anticipated that soon it will allow anyone to scan entire rooms, potentially changing how creatives in Hollywood and other fields record 3D videos. When combined with generative AI, this technology has the potential not only to preserve existing spaces but also to transport us to new worlds in 3D.
Tipatat Chennavasin, an AR/VR expert and investor, describes this technology as a radical change in the field of 3D graphics in over 30 years. Chennavasin has expressed his fascination with Gaussian splatting, comparing it to the holodeck in Star Trek, which allowed the crew to enter holographic three-dimensional simulations of real and imagined spaces. "We are starting to achieve photorealism similar to that of the holodeck."
Capturing objects in 3D is not new, although previous methods relied on polygons, which limited the visual quality of the objects, often making them look unrealistic. However, in the summer of 2023, a group of European scientists presented a new approach called "3D Gaussian splatting," which replaces meshes with a series of translucent bubbles known as Gaussians. This method allows for capturing objects with much richer details and accuracy in color and transparency.
Experts in the field have received this innovation with enthusiasm, seeing the potential to create true photorealistic 3D. According to Brian McClendon, Vice President of Engineering at Niantic, this represents a fundamental change in 3D capture. After acquiring the Scaniverse app in 2021, Niantic has integrated Gaussian splatting into the platform and allows users to scan monuments and other points of interest for the construction of a 3D map of the world, which is essential for their games and AR experiences.
Moreover, Gaussian splatting is not limited to static captures. The startup Gracia AI has been using this technology to record volumetric 3D videos, which can be viewed on Meta Quest headsets. This includes clips that immortalize action from multiple angles, something that was previously challenging to achieve due to restrictions in the clothing of the recorded subjects.
Meta's ambitious proposal with Gaussian splats is manifested in the Hyperscape application, introduced at its recent Meta Connect conference, which allows users to explore photorealistic 3D representations of spaces, though it is still considered a technological demonstration and faces scalability challenges.
As Gaussian splatting technology rapidly advances, a future opens up where people will be able to generate photorealistic 3D spaces through a combination of AI and 3D capture, offering immersive experiences with visual quality almost indistinguishable from reality.
With all these possibilities, one might reflect on which space to visit first if one had access to a holodeck. For me, it might simply be my mother's cozy house and that old garden bench.
