by Maria di Summa (CNR-STIIMA), Nicola Mosca (CNR-STIIMA), and Angelo Cardellicchio (CNR-STIIMA)
The railway sector has always played a key role in communication systems and today, safety standards within this sector has reached very high levels. However, despite these high standards, the industry's goal remains achieving a completely accident-free status by leveraging resources and technologies.
In the current landscape, companies across various industries are showing keen interest in new technologies such as virtual reality (VR) and augmented reality (AR), which were once predominantly associated with the the gaming industry. The key to understanding this renewed interest lies in the integrated use of these technologies [1]. This trend towards the synergistic use of different technologies to optimise results is documented in the literature [2].
In response to the imperative to enhance safety levels in the railway sector, Mermec, an Italian company specialising in rail transport technologies and industrial applications, collaborated with the CNR-STIIMA institute and a small company from southern of Italy, Advantech srl, to introduce the VRail project - Virtual and Augmented Reality for Railways. The project aims to develop a complex monitoring system for railway infrastructure based on the most innovative and emerging technologies available on the market today.
The project involves the creation of an integrated monitoring system that enhances operator awareness and planning capabilities within the railway sector. The system comprises three components: data collection from the field, processing to identify anomalies, and immersive presentation through virtual and augmented reality scenarios.
Point clouds representing sections of railway infrastructure are acquired and transformed into interactive virtual environments, that can be navigated interactively by an operator responsible for planning maintenance interventions (Figure 1). Within this virtual environment, the operaters have at their disposal a series of additional information obtained thanks to the use of deep-learning algorithms, and an enjoying a realistic vision of the state of the places. This additional information increases the operators’ awareness and makes it easier for them to plan maintenance operations.
Figure 1: Stretch of railway infrastructure in virtual reality.
Maintenance instructions prepared by planning operators within the virtual environment are presented in augmented reality to support maintenance workers in identifying intervention areas and executing tasks effecively.
The proposed system represents an interesting and innovative planning tool that allows multiple users to interact remotely, plan maintenance interventions, and carry them out using systems to support decisions and maintenance operations.
The VR environment was created by implementing field-acquired point clouds in Unity using a 3D laser scanner called the BLK360 [L1]. The virtual reality environment can be viewed using the HTC Vive Pro2 viewer [L2]. The deep-learning system analyses the state of the bolts that fix the sleepers to the tracks and categorises them as intact, damaged, or requiring inspection. The status identified by the intelligent system for each bolt is displayed in the virtual environment. Within the virtual environment, the operator in charge of planning interventions can accept or reject the suggestions of the intelligent system. Finally, the choices made by the design operator are transmitted to the augmented reality system, which will support the maintenance operators in carrying out their task.
Developing a complex system to support maintenance and planning activities is fundamental in the industrial and transport sectors. This method connects and verifies different tasks to improve security using mixed reality and artificial intelligence. Possible applications range from the industrial sector to the transport sector.
Future studies may explore similar systems in different scenarios and investigate alternative hardware devices to solve technical problems. It is fascinating to understand how to interact better in a virtual environment.
Links:
[L1] https://www.vive.com/us/product/vive-pro2/overview/
[L2] https://shop.leica-geosystems.com/leica-blk/blk360
References:
[1] M. di Summa, et al., “Extended reality and artificial intelligence: synergic approaches in real world applications,” Roadmapping Extended Reality: Fundamentals and Applications, M. Alcañiz, M. Sacco, J.G. Tromp, Eds., Wiley, pp. 183-192, 2022.
[2] P. Cassará, et al., “E-navigation: a distributed decision support system with extended reality for bridge and ashore seafarers”, in IEEE Transactions on Intelligent Transportation Systems, vol. 24, no. 11, pp. 13384–13395, 2023. https://doi.org/10.1109/TITS.2023.3311817
Please contact:
Maria di Summa, CNR-STIIMA, Italy
