Real time volumetric video

In 2018, we were contacted by Ericsson, who wanted to explore what could be done using 5G as an enabler for Augmented Reality experiences. The end result turned out pretty well! Here’s a brief write up of the projects and what we were able to achieve.

1. Real time viewing of remote band members

Ericsson already had a setup for low latency audio, called the “MusiConnect”. By adding volumetric video to the mix, the idea was to showcase the possibility for a band to perform together, even though some members of the band would be somewhere else.

The setup included a couple of Kinect v2 cameras, beefy PC:s for processing, a prototype 5G link provided by Ericsson’s lab in Aachen, and of course Microsoft HoloLens for viewing. We also added an iPad “spectator view” that would capture and optionally display the experience for bystanders on a large screen TV.

2. Real time volumetric video calls

Inspired by the success, it was time to build something a bit more challenging. By creating a fully bi-directional experience for MWC 2019, we gave visitors a way to experience a not too distant future where video calls will be in 3D and you will be able to see the person on the other end, as if he or she was standing right in front of you. In addition to the bi-directional setup, the new application also included real time meshing instead of point clouds, further adding to the image quality.

Next steps

At some point, it would be fun to continue exploring this project and for example upgrade it to use Kinect for Azure and HoloLens 2. There have been significant improvements both in the hardware and software so we expect to be able to create a “high definition” experience pretty soon. Stay tuned!

Leave a Reply

Your email address will not be published. Required fields are marked *