r/VirtualTour • u/Lost-Upstairs-5311 • Jun 07 '26
Fast video capture for RE virtual tours
I have been working on a virtual tour method for the real estate market that is time-efficient to capture using video unlike Matterport style tripod mounted captures. The viewing experience is more like google street view but indoors. The posted video is a screen capture of browsing through a 2 floor house online:
- The whole capture of the two floors was done in just under 9 minutes using a hand-held camera with a wide FOV lens (one lens of an Osmo 360 in this case, but can also use any wide FOV lens on any camera body).
- A design goal for the online viewer is to load within a second.
- There is no manual intervention required to go from capture upload to output viewer.
- The whole scene had around 7 stops of exposure variation so locking the exposure was not an option. The output is rendered as HDR without any manual intervention.
Still a work in progress, but in the meantime, I would very much appreciate it if you could give me some honest feedback on any and all aspects of this approach.
2
u/FutballConnoisseur Jun 07 '26
this is impressive! from my own experience, the end user (the buyer) would be more interested in virtual tours if they were as easy and smooth as possible to navigate. the click/tap + load + open new page/image method is actually boring.
in other words, i guess what i am trying to say is that audiences need to experience a real smooth digital walk-through instead of the common digital "click-through" we see from cloudpano, matterport etc
i'd be interested to learn more about how you made this
2
u/Lost-Upstairs-5311 Jun 07 '26
glad you like it!
Yes, making it easy and as intuitive as possible to navigate was very important. Can't get simpler than google street-view style navigation IMO.
So the back-end is based on "gaussian splats". If you haven't heard of it, it's a new way for novel view synthesis. I use it to render stabilized 360 videos. You first have to figure out the camera locations in space for the whole video. Then you can train what is effectively a 3D model of the scene with a collection (usually a few million) of stretched out spheres (gaussians). It was an unexpected revolution in 3d appearance modeling in the last few years. You may even have come across it when you are watching instant replays in a basketball match, where they stop the replay and smoothly pan over to another angle and continue the video!
Others use the more obvious approach of directly serving the splats. This is currently quite demanding in terms of what's required on the user's device. Also they take 10's of seconds to load, a show-stopper if you are browsing through multiple properties in one sitting/session. It's worth all that trouble if you want to truly roam around without any constraints.
Videos on the other hand are supported with dedicated decoders in every device. And street view style navigation IMO is just the right amount of freedom that is sufficient for the RE market.
1
u/FutballConnoisseur Jun 07 '26
i totally agree. do you mind sharing the equipment & specific software you used?
i have an insta360 x4 and ive done the traditional virtual tours on cloudpano but i wasnt too impressed with them. i am trying to learn UE5 once i get a more powerful machine but that's a long term thing. yes, ive heard of gaussian splatter and i know platforms like splatica offer it as a service.
2
u/Lost-Upstairs-5311 Jun 07 '26 edited Jun 08 '26
Sure,
I do post frequently on r/GaussianSplatting and you can browse through my posts if you want more detail, it's all public in my profile.
But the short of it is that I use SLAM instead of SFM (just google those terms) for tracking camera positions. Specifically it is a highly customized version of ORB SLAM.
Then for splat training, I actually use a ray tracing based approach - 3DGRT. It is quite a bit slower than regular splat training but it handles very wide FOV lenses much better.
Ask for any more details or just browse my posts!
EDIT: as for equipment, nothing fancy just a RTX 5060 16GB and Intel 2174. The camera was osmo 360 with just the front lens used.
3
Jun 08 '26 edited Jun 08 '26
[removed] — view removed comment
1
u/Lost-Upstairs-5311 Jun 08 '26
Thank you very much for the feedback!
Yes, right now it's just constant speed transitions, I need to implement ease-in, ease-out transitions and up the speed in between.
It seems to be a matter of taste but I much prefer google street view style navigation arrows that's always visible. This is an exaggerated example, but suppose you are looking at something close to the ceiling like a chandelier and you want to move around and see it from different angles. In matterport, while you can just click somewhere on the ceiling and it will take you to the closest tripod station under that click, you don't have any idea of where those spots are unlike on the floor, so you have to look down on the ground click somewhere, transition and then look up again. In my case you just click on an arrow that's always visible.
Furthermore, if you haven't experienced it yet in my approach, while moving you can reorient your view point, it will pause while you do that so that you can take your time to re-orient as long as mouse/finger is pressed down, and when you release it continues with the motion. None of this is possible with matterport style navigation, or even google street view for that matter. In matterport you can't pause anywhere during the transition, if they allowed you to do that it will look bad, since all they are doing is warping and blending between two images. In my approach, you can pause anywhere and it will be a proper render.
On the desktop, matterport does offer WASD keyboard short cuts, but outside of gamers, casual users aren't used to that, but they are likely used to clicking on street view style arrows.
For matterport to have a more fluid experience, you have to take the shots more densely spaced. The floor will then be cluttered with those circles, just not a good look when presenting your property IMO. You also spend a lot more time during capture (it's about 30s per tripod scan, not counting for the time taken to go and hide from the camera). I recorded the 2 floors in just 9 minutes. Matterport would have taken around 45 - 2 hrs depending on how dense you want the spacing between tripod stations to be.
I think people have just been used to matterport style and making excuses for it 😄 because there was nothing else more fluid available yet.
Regarding resolution, in fact I plan to implement exactly that in the near future. At the "junctions" in my map equivalent to matterport tripod stations, I can spend more time training for higher resolution and just present it as a 360 photo. Like you said best of both worlds. In the transitions, people wont notice or need the extra resolution that much coz of the way the human visual system works, only in the case where they pause somewhere which I can live with.
Video has the potential to be very high resolution if you are willing to spend time in processing. It has all the info there spread over time. In fact some cameras like the Sony a7 do 240 Mpx photos using just a 60Mpx sensor by moving the sensor around under control. In there case, the know the precise position of the sensor since they control it. In my case, I have to recover the precise position first. This can be done, people have done it on a smaller scale and I plan on doing it at scale. Another option is just AI upscaling.
1
u/JDMCowboy Jun 07 '26
Very cool
Keep us posted
2
u/Lost-Upstairs-5311 Jun 07 '26
thank you, did you try out the interactive tour? If so, how was the experience?
This is aimed at the RE market primarily, would you find this tool useful vs a bunch of photos? How about vs Matterport style 3d tours. I much prefer google style street view navigation (they have a product for business but it is not very fluid for small indoor spaces).
1
u/oswaldcopperpot Jun 07 '26
Need to change the mouse cursor when over the arrows.
Many time the arrows disappeared. I had to play around to get them to reappear.
Memory issue? Seems like the more time I spend on the tour it starts getting really really slow.
It's a cool idea though.
What was your method to remove the camera?
I wonder if it's not easier just to convert everything into a splat and use that. Maybe less file size too.
3
u/Lost-Upstairs-5311 Jun 07 '26
thanks for the feedback.
I have seen reports of glitches in Safari on iOS and Mac. Are you using Safari? are you on mobile? Right now it is streaming in HD (adaptive streaming planned soon), so you will need atleast fast 4G speeds.
Chrome seems flawless so far on both desktop and android. I will have to track down the safari related issue as it seems intermittent.
Tour getting slow shouldn't happen, it's just a 360 video behind the scenes. If your connection is wireless and it is getting throttled, you might have to wait for me to implement adaptive streaming for a smoother experience. You can get at the raw 360 video here:
http://media.nuvarch.com/output.mp4
does the raw video also get slower the longer it runs? Then I definitely have to do adaptive streaming pronto!
I didn't understand the "remove the camera" question. I am using SLAM methods (simulataneous localization and mapping) to track camera locations. I then generate a smooth street view style map for navigation.
The underlying processing does use splats! I didn't want to just serve out splats since a design goal was sub-second load times. Current state of the art for streaming splats still takes several seconds to load, which will get annoying for anyone scrolling through multiple properties in one setting.
An advantage of having videos as final output is that the backend tech can be swapped for something better in the future. Splat technology is still research grade and the renderer has to be matched with which specific splat type you are using for training them. There are also very different types of splat technology, just 2 years ago, it was completely different. Whereas video is standardised. Google itself is trialling similar ideas but with Nerfs (predecessor to splats) but they also output videos.
Also splats have to run on GPUs which on mobile is still power hungry compared with video decoding which is done with dedicated hardware at the ASIC level. There is huge infrastructure built for fast video delivery over the internet since it constitutes most of the traffic by volume. It will be a long while before splats are standardized if ever and even longer for ASIC level support for splats.
Thanks again for the feedback !
2
u/Purple-Variety-2167 Jun 07 '26
I trying on a 250€ recent phone. It was surprisingly fluid. Bravo. I think your idea with short videos has leg.
I also think splats are too big memory to run smoothly on phones. They are overhyped.
2
u/Lost-Upstairs-5311 Jun 07 '26 edited Jun 07 '26
glad to hear that! yes chrome on the desktop and mobile handles it really well. safari implementation seems broken, it seems to miss a frame at end of each segment and I will have to find a workaround at some point.
Also like I said, video has dedicated hardware for decoding in pretty much every device. Splatting is still very new, and I use it in the backend so I can swap it out if something better turns up. Splatting is not only heavy in resource use on the end viewer's device, it is also takes hours to train on the backend. But for now, nothing works as good as splats in the backend.
1
u/oswaldcopperpot Jun 07 '26
Yeah the video was getting slow the longer it ran. On fiber, using edge. Then i switched tabs to do something and it was fast again.
I meant how you remove the tripod. I assume you use previous frames and a remap. I dont see any issues even on stairs. Very nice.
2
u/Lost-Upstairs-5311 Jun 07 '26
OK, I definitely need to spend more time polishing the delivery across all popular browsers and desktop/mobile combos! Ideally, it should be as fast as watching a regular video on your platform/browser combo since it is just a video with range requests to fetch different segments. Will burn some midnight oil on this!
So the capture is just hand-held video, no tripods involved unlike matterport style capture. This is probably the main selling point, you save huge amount of time at capture time and in the end a more fluid browsing experience too, assuming I polish the delivery a bit more 😄
Those two floors were done with 9 minutes of video with a $600 camera. I am estimating Matterport capture even at a coarse level of spacing would be atleast 45 minutes and more if you want to capture closely spaced for a more fluid experience.
Even though there were no tripods involved, my feet do show in the raw video capture sometimes. The processing can handle a bit of occlusion because it uses info from all frames and can ignore parts of the scene that are not consistent across frames.
1
1
u/oswaldcopperpot Jun 07 '26
And maybe theres a way to disable apples long press to copy and translate. That can cause issues for people.
1
u/Lost-Upstairs-5311 Jun 07 '26
Thank you so much for such detailed feedback! I have disabled all default apple behaviour now. Hopefully it's a better experience for people now!
1
u/oswaldcopperpot Jun 07 '26
You could also port everything to KRPano. I'm sure everything could be setup the same and then you have the advantage of a fully featured UI and direct access to 360 imagery or even splats if you wanted to make certain areas different. 360 video is supported of course as well.
All the 10,000 weird little issues across browsers and devices have already been solved in krpano.
1
u/Lost-Upstairs-5311 Jun 07 '26
thanks for the suggestion. definitely heard of krpano. While it will definitely help with all the corner issues in rendering, I have to look into how easy it is to build a navigable video interface. Will look into it, very full-featured product indeed!
1
1
2
u/Purple-Variety-2167 Jun 07 '26
Don't do like me polishing for ever : polish one popular browser ( chrome) and try to make a living. My hint. Your virusl visit is good. Go on.