by Muhannad Ismael, Roderick McCall and Joan Baixauli (Luxembourg Institute of Science and Technology)
Using mixed reality (MR) head-mounted displays (HMDs) for fluid simulations (FS) is challenging. This is due to the capacity in MR-HMDs to perform FS in real time with reasonable frames per second. FSMR (fluid simulation in mixed reality) models position-based fluids via an MR-HMD using remote particles system computing.
MR applications have recently gained traction owing to the rapid development of HMDs such as HoloLens version1/2 [L1], Apple Vision Pro and Meta Quest Pro. For instance, application areas include training, research studies in computer vision, remote collaboration and assistance, and entertainment. However, current MR-HMDs often lack the processing or graphical capacity, for example the frames per second rate is often quite low. A low frame rate can have adverse effects on simulations of complex models such as those including FS.
FS in MR can be an effective tool for training and education, especially in fields such as engineering, medicine, and chemistry. It permits the simulation of the behaviour of liquids, such as water, smoke, or other fluids, within a mix of real and virtual objects. These simulations consider the physics of fluid flow, viscosity, and interactions with both virtual and real-world objects.
For example, in engineering [1], FS in MR can be used to train engineers on fluid dynamics and fluid-structure interactions. By visualising the flow of fluids in real time, engineers can better understand the behaviour of fluids in various situations and make informed decisions on the design and optimisation. In medicine [2], FS in MR can be used to train medical professionals on the behaviour of blood flow in the human body. Medical professionals can better understand the flow of blood in real time and the effects of various medical interventions on blood flow. As a result, they can make informed decisions on patient care. FS is also important for the applications which provide laboratory safety training based on MR for those working in hazardous chemical laboratories without the risk of real-world hazards [L2]. It permits the simulation of hazardous situations, such as chemical spills or leaks, in a safe and controlled manner. This is important since it allows trainees to develop the necessary skills and knowledge to handle dangerous situations safely. However, where the realism of the simulation is too low, there is risk that some of the potential benefits of MR may be undermined.
However, FS is computationally expensive due to Navier-Stokes equations. These are a set of partial differential equations that describe the behaviour of fluids in motion, including liquids and gases. There are various techniques for FS using Navier-Stokes equations, including Finite Difference Method, Finite Volume Method, and Smoothed Particle Hydrodynamics. These techniques differ in the way they discretise the equations and simulate the fluid behaviour, but they all rely on solving the Navier-Stokes equations numerically. Navier-Stokes equations can be computationally intensive, especially for complex scenes and high resolutions.
Achieving FS real-time performance is even harder considering the limited capacity of hardware resources of MR-HMD. In MR applications, real-time performance is essential to maintain a smooth and realistic experience for the user.
In the FSMR project, we employed Position Based Fluids (PBF) [3], which is defined as an algorithm using Smoothed Particle Hydrodynamics solvers but inheriting the stability of the geometric position-based dynamics algorithm. Hence, one advantage of PBF is that it is relatively fast and can be implemented efficiently on both CPUs and GPUs. This makes it a popular technique for real-time applications such as video games and interactive simulations. PBF is applied remotely on a local server and displayed on Microsoft HoloLens v2.
Figure 1: Schema general of the architecture of project.
We have evaluated the performance of PBF using HoloLens v2 and desktop PC as illustrated in Figure 1. The average number of Frame Per Second (FPS) was 8 FPS and 60 FPS on HoloLens and PC respectively (see Figure 2). PBF which consisted of computing density, pressures and total force acting on each particle is applied on desktop PC used as local server. The particle system is then transferred via Wi-Fi connection to MR-HMDs where a rendering process is applied. Rendering a particle system remotely on a local server could affect the real-time performance due to the size of rendering volume that is transferred via Wi-Fi connection. Therefore, in this project, we decide to compute only the particle system on the local server and then transfer this to the MR-HMD as a set of 3D points. The points are then rendered on the MR-HMD. This technique resulted in 25fps. (see Figure 3).
Figure 2: Position Based Fluids (PBF). Left: 60 FPS, implementation on desktop PC (number of particles 29K, RAM 32 GB, CPU–Intel Core I9 GPU– NVIDIA GeForce RTX 3080).
Right: 8 FPS, Implementation on HoloLens v2 with same number of particles.
Figure 3: Left: Remote PBF computing with 24 fps in average. Rigth: Local PBF computing with 8 fps in average.
The results are encouraging; however, higher fps could be achieved by optimisation of the date being transferred from the local server to the MR-HMD via Wi-Fi. Therefore, for future activities, we propose (1) streaming the particle system using TCP sockets which permits the establishment of a connection and the exchange of data reliably and in the correct order (2) to reduce network bandwidth requirements and achieve real-time data streaming. For (2) LZ4, Zstandard or Brotli compression methods could be applied to compress the particle system. Finally, (3) we plan to use a little-endian network format which is a way of storing binary data in which the least significant byte comes first. The latter will ensure that the compressed particle system data can be easily transmitted and received over a network.
In conclusion, we have presented an overview of an approach for fluid simulation within MR-HMDs and suggested future areas of work. Such an approach has the potential to be relevant in a large number of domains.
The project was funded internally by Luxembourg Institute of Science and Technology.
Links:
[L1] https://www.microsoft.com/fr-fr/hololens
[L2] https://pubs.acs.org/doi/10.1021/acs.jchemed.1c00979
References:
[1] Y. Zhu, T. Fukuda, and N. Yabuki, “Integrating animated computational fluid Dynamics into mixed reality for building-renovation design,” Technologies, vol. 8, no. 1, p. 4, Dec. 2019. https://doi.org/10.3390/technologies8010004.
[2] A. J. Lungu, et al., “A review on the applications of virtual reality, augmented reality and mixed reality in surgical simulation: an extension to different kinds of surgery,” Expert Rev. Med. Devices, vol. 18, no. 1, pp. 47–62, Jan. 2021. https://doi.org/10.1080/17434440.2021.1860750.
[3] M. Macklin and M. Müller, “Position based fluids,” ACM Trans. Graph., vol. 32, no. 4, pp. 1–12, Jul. 2013. https://doi.org/10.1145/2461912.2461984
Please contact:
Muhannad Ismael, Luxembourg Institute of science and Technology, Luxembourg
