Apple Vision Pro Users Can Soon Create Their Own Environments

Key Takeaways

1. Apple announced a new feature for the Apple Vision Pro that will allow users to create their own environments using panoramic photos they have taken.
2. The company also introduced “spatial reframing” technology, which uses generative AI to fill in uncaptured parts of flat photos to create deeper, more immersive 3D images.
3. Both the user-created environments feature and the enhanced spatial photo immersion are scheduled to release this fall.

---

At WWDC, Apple announced a new feature for its Apple Vision Pro VR/AR headsets: users will soon be able to create their own environments from panoramic photos they take. The company also teased greater immersion for spatial photos. Both improvements are coming this fall.

Apple Vision Pro’s environments — immersive, detailed 3D backdrops that can be toggled on and off from inside the Vision Pro — are one of the stand-out features of the headset. You can control the immersion level like you would a dimmer switch on a light, letting you blend reality and virtuality as you see fit, and you can use your Vision avatar in chats with other Vision Pro users in shared environments. Presumably, all these features will be available in user-created environments.

But environments “work” because they’re more than photographic backdrops. They’re vision-encompassing 3D scenes that feature sound elements, too. Apple didn’t offer specific details, but it seems that transforming panoramic photos into immersive environments goes hand-in-hand with Apple improving its spatial photos. Transforming snapshots into 3D pics is already a Vision Pro feature, but they’re limited. The 3D isn’t very “deep,” and they can only be viewed from the angle they were taken. The gap between those photos and fully immersive environments — where you can see what’s behind you, what’s on the ground, what’s in the sky, etc. — is wide. Even panoramic iOS photos, those landscape pictures you take by spinning in place, leave a lot of blanks to fill in, like the sky and the ground. That’s where the AI component comes in.

Spatial Reframing Could Be Apple’s Step Towards Realistic 3D

While you can’t create “real” 3D unless you capture objects and places from different angles, Apple showed off its upcoming “spatial reframing” technology, which gives a hint as to how it’s planning to pull off more immersive 3D from flat photos and how panoramic photos can become environments. During the presentation, Apple’s selling point for spatial reframing was the ability to improve your snapshots — but using generative AI to fill in the parts of the photo that you didn’t capture is no doubt how the company plans to create deeper 3D photos and 360° environments.

The Vision Pro’s instant 3D photos are compelling enough that some users have gotten into creating Gaussian Splats — scans that let you experience real 3D objects you can see from any angle in virtual space. Experiencing immersive, realistic 3D versions of places you’ve been and people you know is the coolest thing you can do with a VR headset. But there are mixed feelings about letting AI “improve” the process. The technology is impressive, but the power of Gaussian Splats comes from how accurately they re-create the scenes you capture. An AI-filled creation simply can’t replicate that, and there’s something unsettling about letting AI alter the record of your personal memories. Still, it will be worth trying out when it arrives this fall.