by Evangelos Vlachos (University of Edinburgh) and Aris S. Lalos (ISI/ATHENA R.C.)
5G mobile networks are expected to support applications with very high-bandwidth and ultra-low latency requirements. Augmented reality (AR) and virtual reality (VR) are among the most attractive use cases. Of course, processing and communication of the huge amount of the four dimensional (4D) data introduce challenging requirements to the network design. To meet these requirements a novel framework that optimizes jointly multiple objectives is required. In this article we provide a paradigm where the reconstruction quality and the transmission efficiency of 4D data are jointly optimized.
The augmented reality (AR) and virtual reality (VR) revolution is still in its early phase. These technologies have the potential to revolutionise multiple industries over the next five to ten years. They will transform the way we interact with the surrounding world, unlock new experiences and increase productivity. However, the real-time interactivity of the extended reality (AR/VR) applications requires very high bandwidth and ultra-low latency supported by the 5G networks. Indeed, the scale of acquired data in real-time operation is growing very quickly, making the communication and processing a very challenging task. To meet these requirements a multi-objective framework for the network design is required. Specifically, objectives from different layers of the network stack should to be optimized jointly. As such, reconstruction quality, transmission efficiency, energy efficiency of the mobile device, and communication bandwidth are some indicative objectives.
Low latency is a critical requirement for delivering the best AR and VR experience because even small delays can have a disorientating effect. For instance, when a user turns and the landscape does not move simultaneously, the user may experience motion sickness. VR requires less than 1ms latency and currently the global average latency is 36ms on fixed and 81ms on mobile. Coincidentally, these requirements comprise some of the main targets to be addressed by modern 5G networks. To achieve the low-latency requirement, efficient and low-complexity processing is necessary. AR and VR applications require the acquisition and processing of complex and highly deformable 3D objects represented by four dimensional (4D) data. Modern signal processing and mathematical optimisation tools play a crucial role for the acquisition and the reconstruction of this massive amount of data [1].
Figure 1: Simple paradigm of the proposed design. Next-generation network technologies (i.e.,massive MIMO) can compensate the losses due to low-end hardware (i.e., analog-to-digital and digital-to-analog components (ADC/DAC). To achive real-time high-quality reconstruction at the user’s XR equipment, the parameters which define the perceptual quality and the transmission efficiency have to be optimized jointly.
One crucial processing step for increasing quality of experience in extended reality applications is 4D models enhancement. The acquired 4D models are represented by a sequence of time varying and unstructured point clouds. These data are usually corrupted by several forms of imperfections, like scanning noise, outliers and missing parts of surfaces. Hence, enhancement techniques are an essential pre-processing step before rendering the models to AR and VR headsets. The exploitation of the spatial and temporal coherence of the generated time varying point-clouds (TVPC) may provide low-complexity techniques for real-time operation. For instance, in [2] modern mathematical optimization tools (matrix completion theory) have been used for enhancement and reconstruction of the 4D data from a reduced number of points.
Compression is another important process for streaming the 4D data. Compression of th 4D acquired geometry represented by TVPCs is very resource demanding operation. This is attributed to the fact that the raw geometry data are usually represented using floating point precision. As a result, the compression of geometry information introduces tough challenges to enable aggressive compression ratios, without significantly reducing the visual quality. To overcome these issues, compression of the 4D models should also exploit the spatio-temporal correlations of the data. In [3], a sparse coding technique is proposed, to generate compact representations of static geometries.
Ultra-low latency requirement also poses demanding specifications for the wireless communication processing. Mobile devices with low-end hardware and strict power constraints have to communicate and process the 4D data in real-time. To this end, promising designs for energy efficient receivers have been proposed recently. These designs exploit the benefits of 5G, where a large number of antennas (massive MIMO) could be employed at the mobile terminals. Specifically, massive MIMO designs can compensate the losses due to low-end hardware. For instance, since analog-to-digital components (ADC) have exponential power consumption, lowering their specifications significantly reduces the overall system power consumption. However this is at the expense of the introduced distortion to the transmitted/received signal. Therefore, novel techniques which will optimise overall user experiences through a distortions aware design are required [4]. These examples call for novel designs of operations and architectures for 5G mobile networks.
From our perspective, the successful realisation of the real-time extended reality applications over 5G mobile networks, must address several challenges (e.g., quality-of user-experience, energy-efficiency, limited hardware resources) in a joint and combined manner. The development of novel multi-objective mathematical optimization tools is a key enabler to accomplish this challenging endeavour.
Link: Augmented and Virtual Reality: the First Wave of 5G Killer Apps, ABI Research, 2017
References:
[1] A. S. Lalos, et al.: “Signal Processing on Static and Dynamic 3D Meshes: Sparse Representations and Applications”, in IEEE Access, vol. 7, pp. 15779-15803, 2019.
[2] E. Vlachos, et al.: “Distributed Consolidation of Highly Incomplete Dynamic Point Clouds Based on Rank Minimization”, in IEEE Transactions on Multimedia, vol. 20, no. 12, pp. 3276-3288, Dec. 2018.
[3] A. S. Lalos, et al.: “Compressed Sensing for Efficient Encoding of Dense 3D Meshes Using Model-Based Bayesian Learning”, in IEEE Transactions on Multimedia, vol. 19, no. 1, pp. 41-53, Jan. 2017.
[4] A. Lalos, et al.:: “Energy Efficient Transmission of 3D Meshes over mmWave-based Massive MIMO Systems”, IEEE International Conference on Multimedia and Expo, 2019.
Please contact:
Evangelos Vlachos
University of Edinburgh, Scotland, UK
