Metaverse Getting a Workout With Holographic Communication
The introduction of 5G networks combined with advances in holographic communication is presenting a vision of the metaverse without the head hardware.
By John P. Desmond, Editor, AI in Business
The metaverse and holographic communication are getting a boost from the introduction of 5G networks, which make the idea of lifelike digital avatars able to be seen without klugey headsets much more possible.
Ericsson, the Swedish multinational telecom company, has been working to commercialize 5G in new ways. “Once confined to the realm of science fiction, holographic communication now ranks as one of the most wanted 5G-enabled applications by both consumers and enterprise users, according to recent research,” stated a recent account in Ericsson Technology Review, written by multiple authors including Sara Thorson, head of concept development, Ericsson Consumer & Industry Lab.
Holographic communication is defined by Ericsson as real-time capturing, encoding, transporting and rendering of 3D representations, anchored in space, of remote persons shown as stereoscopic images or as 3D video in extended reality (XR) headsets that deliver a visual effect similar to a hologram.
“After several years of experience with video calls on smartphones and tablets, many users report that they are eagerly awaiting the chance to meet others digitally using immersive communication services such as 3D holographic augmented reality (AR) calls over 5G.” the Ericsson authors state. More than 50 percent of smartphone users expect this capability to be available within a few years.
While the beachhead for holographic communication might be in consumer entertainment, the opportunity for office workers is intriguing. The Ericsson researchers cited a barrier to remote work as being the need for social interaction, with “more immersive forms of digital interaction” seen as a potential solution.
Off With the Headsets
Meanwhile, the metaverse lurks. (See Reality-Checking the Metaverse, AI in Business, March 10, 2022) But to gain access to it today, requires a klugey headset.
“The idea that our virtual Zoom meetings will move to the metaverse or 3D space within two to three years, is a complete fantasy,” stated Brian Sheehan, professor of advertising, Syracuse University, in a recent account in AdWeek. He adds, “In order to have a work meeting in the metaverse today, every member of your team must have a VR headset. That’s a big investment for companies. Every member of your team must feel comfortable wearing them and using them.”
This is unlikely. “We know for a fact that many people find them cumbersome and awkward. Many get a feeling akin to seasickness when wearing them too long due to latency effects,” Sheehan stated. “While 5G helps a bit, not everyone has access to 5G. The minute one person in your meeting either doesn’t have a headset or doesn’t feel comfortable with the headset, you’re all back on Zoom.
He called out the metaverse on another key point. “One of the great fallacies of the metaverse ‘movement,’ driven by marketing hype, is that 3D is better than 2D. That is not always the case.” As a virtual meeting experience, people like Zoom.
He acknowledges “a realistic chance that many people would spend a lot of time in the metaverse in the not-so-distant future,” as long as “one big thing happens. We get rid of the headsets.”
Enter holographic communication, 5G and AI. “The real promise for the metaverse is not in headsets,” Sheehan stated. “It is in holograms or, as it is being called, ‘Holoportation.’ ' He cited ongoing work by Microsoft in this area, and from a company named Proto (recently renamed from Portl). The company’s product, recently demonstrated at SXSW, is a large box that projects a hologram-like 3D image from one space into another space with no glasses or headset.
Tensor Holography Showing Promise to MIT Researchers
MIT researchers have explored similar techniques. Calling it “tensor holography,” the team used deep learning techniques to accelerate computer-generated holography. The team designed a convolutional neural network that used a chain of trainable tensors to mimic how humans process visual information. To train the system, the team built a custom database of 4,000 pairs of computer-generated images. Each pair matched a picture with its corresponding holograms, resulting in photorealistic training data. Moreover, the compact tensor network required less than 1MB of memory.
“People previously thought that with existing consumer-grade hardware, it was impossible to do real-time 3D holography computations,” stated Liang Shi, the study’s lead author and a PhD student in MIT’s Department of Electrical Engineering and Computer Science (EECS), in an account in MIT News.
The results surprised the team. “We are amazed at how well it performs,” stated advisor and co-author Wojciech Matusik. Taking milliseconds, the tensor holography system was able to craft holograms from images with depth information, paving the way for real-time 3D holography. The memory requirements are “negligible, considering the tens and hundreds of gigabytes available on the latest cell phone,” he stated.
Applications of 3D holograms are anticipated to include, according to a recent account in Interesting Engineering:
Making learning more interactive and accessible to students and teachers, especially to students in remote areas or those without access to costly lab equipment;
Helping students of architecture and design and medical science, where students could gain practical experience;
Detailed, full-color medical imagery of what’s going on inside a patient’s body, going beyond the flat imagery of MRIs and ultrasound scans.
Ericsson Holographic Test Using 5G Network Encouraging
At Ericsson, researchers divide the area of holographic communication into three paradigms: avatars; professional-quality digital representations of users; and consumer-friendly digital representations of users. “State-of-the-art avatar technologies use AI to generate a photorealistic representation of a user,” the authors state, adding that consumer-friendly digital representations are generated with the help of an AI-enabled 3D-capturing setup on consumer grade phones or tablets.
The Ericsson researchers conducted a test using existing 5G technology to build a pipeline that enabled high-quality holographic communication; they conducted the test on a live 5G New Radio (NR) network, the new global standard for a 5G wireless air interface.
The 3D frames were captured by a single 3D camera connected to a computer, which was connected by Ethernet to the 5G network. The captured stream was compressed and sent over the 5G network to a 5G mobile phone with connected AR glasses. The decoding and rendering were executed on the mobile phone and displayed in the AR glasses. SLAM functionality, for Simultaneous Localization and Mapping, made it possible to walk around the 3D representation
The researchers were optimistic about the results. “The emergence of lightweight AR glasses and powerful 3D compression algorithms have made it possible to start deploying AR use cases using existing 5G technology,” the authors stated.
Beam me up or over, might be coming.
Read the source articles and information in Ericsson Technology Review, AdWeek, in MIT News and in Interesting Engineering.
(Write to the editor here.)