Desktop Virtual Reality for 3D and 4D Medical and Biological Data Analysis

by Jurriaan D. Mulder

The Personal Space Station (PSS) brings Virtual Reality (VR) to the desktop of the medical and scientific professional. Its purpose is to make VR more useful and accessible for the effective analysis of 3D and 4D data in medical and biological research. To this end, PS-Tech in the Netherlands and CWI are developing and improving new techniques and methods for the application of VR in 3D and 4D data analysis.

Three-dimensional (3D) and time-dependent (4D) datasets are becoming increasingly important in medicine, microscopy, and biology. Such a vast amount of information implies a need for fast, accurate and cost-effective analysis. Visualization - the ability to present complex data as multidimensional images - combined with direct control over that data in VR provides a tool to satisfy that need. In a VR environment the data is presented truly in 3D and users can interact with the data directly in the 3D space. However, traditional VR systems tend to be bulky, difficult to use, and expensive, and their use has therefore been mainly limited to dedicated VR centres. In other words, the use of VR remained beyond the scope of most medical and biological scientists.

The Personal Space Station was developed at CWI, and is now also commercially available from the CWI spin-off company Personal Space Technologies (PS-Tech) in Amsterdam. The PSS is a desktop interface that allows the researcher to interact with 3D and 4D images in a natural and intuitive manner, under normal office working conditions. About the size of a child's school desk, the PSS is portable, yet still large enough to create a virtual environment in the user's personal space. The images are presented to the researcher using a head-tracked, stereoscopic display. In addition, the researcher can control, explore and interact with the data directly in 3D and 4D. Therefore, both the viewing and the interaction with the data are achieved in a transparent and intuitive manner, allowing the researcher to focus on the analysis instead of the user interface.

Figure 1: The PSS explained. In the PSS the user holds the image in his hands and manipulates it with tangible interaction devices. The user looks at the monitor via a mirror. This enables the user to bring his hands into the same environment as the virtual 3D-objects without interrupting the visual image. With the use of a custom optical tracking system, the user can interact with the 3D images using wireless, tangible interaction devices. Therefore, interaction with virtual objects can take place in a direct, natural and intuitive way. Hand-eye coordination and "proprioception" (the subconscious perception of the position of muscles and joints) are used to the full. The user is "attracted" by the objects he sees; his natural reaction is usually to grab and manipulate them.

The PSS is now progressing from a scientific concept to a device for medical and biological data analysis. 3D and 4D datasets from ultrasound, MRI, MRA, CTA and confocal microscopy can be analysed and explored in VR. The user holds a dataset - like a 4D ultra sound dataset of a heart - in one hand and can orientate and position it. The other hand is used to interact with the dataset for manipulation, like slicing, dicing, and pointing. Next to the 3D interaction tools the user has standard 2D tools like a mouse and keyboard. Depending on the task the user selects the appropriate tool (eg menu operations using 2D tools, positioning a data set using 3D tools). The ability to select the right kind of tool for the task at hand not only simplifies the process of working with 3D data, but it also simplifies application development.

Scientists at research institutes, universities and academic hospitals are working with CWI and Personal Space Technologies to create the PSS applications needed for the analysis of complex medical and biological data. At the Academic Medical Center (AMC) in Amsterdam, the PSS will be used to analyse 4D ultrasound data of quantitatively complex (patho)physiological processes and the effects of genetic background and therapeutic intervention. Processes such as tumour development, atherosclerotic plaque formation in blood vessels, the development of the heart and the effects of genetic background and therapeutic strategies on these processes can be studied effectively in 4D.

Figure 2: Some examples of interaction and visualization in the PSS. From left: 3D and 4D microscopy; Cardiology, 3D and 4D Ultrasound; Brain Research, segmented CT data.

At the University of Amsterdam, the PSS is used in live-cell imaging to facilitate visualization and analysis of large 3D and 4D data sets obtained with the microscope unit. A controlled light-exposure microscopy (CLEM) microscopy unit combined with the PSS is the chosen method that enables time-resolved study of fluorescently labelled cells to gain insight into cell dynamic processes. This methodology will allow delicate intracellular processes such as DNA repair, protein aggregation and transport to be more easily studied.

The PSS is also used for the analysis and modelling of 3D vascular data, ultrasound analysis in cardiology, brain research, neurosurgery and medical education. With the development of these applications, and the close cooperation between CWI, Personal Space Technologies and the medical and biological research institutes and universities, VR is being brought into the world of the medical and biological professional. The ultimate goal of the project is to bring 3D visualization and interaction within reach of all users of complex 3D and 4D data sets. To achieve this, further research and development is needed in several fields, including human-computer interaction, tracking and display, large dataset management, and automated and interactive image segmentation methods. CWI and PS-Tech will continue to cooperate with scientists from different fields and application areas to continuously improve and develop new techniques and methods for the application of VR in 3D and 4D data analysis.

Link:
http://www.ps-tech.com

Please contact:
Jurriaan Mulder
Personal Space Technologies
Tel: +31 20 3311 214
E-mail: mullieps-tech.com