Varjo wants you to create photorealistic VR ‘scenes’ with your phone

Varjo has unveiled an app that lets users scan physical spaces with their smartphone to create photorealistic 3D “scenes” for virtual reality (VR) devices. 

The VR headset maker on Tuesday announced the preview of its Teleport app, which it said will lower the barrier for 3D content creation — a time-consuming and costly process that typically involves high-endequipment and know-how. “One thing holding back VR and 3D applications is just how hard is to create content,” said Patrick Wyatt, chief product officer at Varjo. 

He described the Teleport app as “a self-serve way that anyone with a smartphone can start creating their own 3D scenes,” allowing them to share their surroundings with others. 

To create a 3D scene, users scan a physical space with their smartphone camera (an iPhone Pro 12 is the minimum requirement for Teleport) — a process that takes several minutes. It’s possible to film indoor or outdoor scenes  (anything up to the size of a small town square will work), though more dynamic environments with crowds of people or lots cars could result in blurred footage. 

The footage is uploaded to Varjo’s cloud servers to build a high-resolution 3D scene.  When accessed via a VR headset, users can then move around the virtual space and view a reproduction of the environment that was recorded. 

Given Varjo’s focus on enterprise mixed reality and VR, Wyatt said Teleport can be used for training, planning, and remote assistance. But he sees Teleport as “foundational tech” that could have broad applications. “We’re not too prescriptive on use cases,” he said. “We want to see all the cool things people will do with it.”

While VR environments are often created with computer graphics, photorealism is preferred for certain enterprise purposes. “Much of what businesses want and need has to be as close to the real thing as possible to use those assets for engineering, sales and marketing purposes,” said Anshel Sag, principal analyst at Moor Insights and Strategy. “Without photorealism, it becomes a lot less valuable and powerful.”

Until recently, photorealistic 3D content has been produced either with expensive Lidar scanners that can cost several thousand dollars, or photogrammetry techniques that are more accessible, but still require specialist skills. In both cases, there are limitations on quality, too, said Wyatt.

A key advantage of Teleport is the use of a machine learning technique called Gaussian splatting. This enables full 3D scenes to be produced from a set of photos, with more realistic lighting, textures, and reflections – ideal for immersive applications such as Teleport, said Wyatt.

Gaussian splatting simplifies the creation of photorealistic 3D environments, though the technology has its own limitations, said Sag. “The biggest challenge for creating 3D content has primarily been the cost and time it takes to generate the assets,” said Sag. “Gaussian splatting is a way to take some shortcuts in the creation of content to make it cheaper and faster with minimal tradeoffs in terms of quality.

“That said, it isn’t without its problems, as the Gaussian splats don’t always come out right or need very specific capture techniques to work right.”

Varjo isn’t the only company to use Gaussian splatting for 3D content creation. Others include Luma and Polycam. Wyatt said Teleport differs in its focus on the creation of 3D environments rather than smaller objects, as well as a need for a higher image resolution so that content can be viewed effectively on a VR headsets. 

Varjo plans to make Teleport commercially available towards the end of 2024. A waitlist for early access is available here.