by Michael Hubner and Jan Nausner (AIT Austrian Institute of Technology)
In this article, we introduce our Multimodal Fusion Architecture for Sensor Applications - MuFASA, which our research group has developed at the Austrian Institute of Technology. It offers a robust fusion architecture for real-time sensor applications, providing situational awareness and precise decision support.
The Austrian Institute of Technology (AIT) has developed an innovative multimodal fusion architecture for sensor applications, known as MuFASA. This technology is designed for real-time sensor applications, aiming to enhance the safety and security of end-users such as firefighters, paramedics, and other practitioners operating in challenging and dangerous conditions.
Technological Overview
MuFASA introduces a robust fusion architecture that integrates heterogeneous data from various sensors and external sources to establish situational awareness in real-time. This system ensures precise decision support by offering modular fusion modules that exploit sensor data on different levels. The levels are categorised as Signal Level Fusion, Feature Level Fusion and Decision Level Fusion. At each level, specific challenges are addressed to ensure the full potential of the available sensor information is exploited.
To illustrate, at the Signal level, raw sensor data is employed to enhance or even create new features based on homogeneous and heterogeneous sensor modalities. Signal level fusion occurs in real-time fusion scenarios or may be an additional step in the pre-processing of signals to yield features of interest. A representative example would be image fusion of RGB images and infrared images for the purpose of detecting persons under difficult lighting conditions.
At the feature level, the objective is to fuse the sensor features with the intention of establishing spatio-temporal coincidence across all sensor modalities. This is a crucial step in providing situational awareness, as demonstrated by the generation of heat maps. The fusion modules dedicated to this level are also designed with the aim of improving detection accuracy and reducing false alarms compared to a system that does not utilise any fusion modules. At the decision level, fusion models operate with the refined features of the previous level. Thus far, only sensor data has been used in the fusion process. A primary objective of the decision level is to incorporate external information, such as end-user experience and contextual or tactical information, into the fusion process. This output represents the highest level of fusion and is essential for precise decision support.
MuFASA selects its methodologies with a focus on real-time capability and ease of interpretation by those responsible for operational management. This provides immediate response capabilities to the acting personnel. The system is based on well-known fusion architectures described by JDL and Lou and Kay [1]. The software modules employ state-of-the-art statistical methodologies in the field of soft computing, such as Probabilistic Occupancy Mapping, Bayesian Inference, and first-order logic. Figure 1 illustrates these methodologies. These techniques enable the system to process and interpret vast amounts of sensor data efficiently.
Figure 1: Illustration of MuFASA´s employed feature level methodologies utilizing Bayesian Inference and heat maps. The result of the statistical fusion is represented as an alert (red marker).
MuFASA is being developed by the Austrian Institute of Technology (AIT), which is drawing on insights from previous and ongoing research projects. AIT’s dedication to developing technology for public safety and security has been a key factor in the success of this project. The primary objective of the research group at AIT is to enhance the safety and security of end-users, including firefighters, paramedics, and other practitioners. In search and rescue operations, unmanned systems are increasingly used to assess situations and locate victims under complex and dangerous conditions. These tasks require numerous sensors, which generate a vast amount of data that can be difficult to interpret. MuFASA addresses this challenge by providing a robust fusion architecture that integrates data from different sensors and external sources, ensuring real-time situational awareness and precise decision support. The work is the culmination of years of research, which commenced in 2016. The project’s objective is to provide a comprehensive approach to multimodal sensor data fusion, ensuring the utilisation of technologies is not limited to specific use cases. This flexibility and generalised approach enables MuFASA to be employed in a range of real-time sensor applications, enhancing its versatility and effectiveness. The technology is currently being evaluated in use cases across various sectors, including border security, surveillance of critical infrastructure, search, and rescue operations, and CBRN reconnaissance.
Future activities
MuFASA represents a significant advancement in sensor data fusion technology, offering real-time situational awareness and precise decision support. Developed by the Austrian Institute of Technology, this innovative system is poised to enhance the safety and security of end-users across various sectors, ensuring that practitioners can operate more effectively in challenging and dangerous conditions.
Its capabilities will continue to evolve with new use cases in these sectors. The main goal is to ensure precise communication of results and minimise false alarms. To achieve this, ongoing research focuses on optimization procedures to find optimal fusion configurations necessary for designing effective fusion systems. This will result in high-quality, interpretable data that can be easily understood by practitioners, ultimately increasing the safety and security of personnel in dangerous missions.
MuFASA's capabilities have been evaluated in the context of railway safety in a national research project (completed in 2023) called MOBILIZE [L1]. This project evaluated the performance of MuFASA in three typical critical railway security events.
Link:
[L1] https://projekte.ffg.at/projekt/4105746
References:
[1] Meng et al., “A Survey on Machine Learning for Data Fusion”. Elsevier, Volume 57, p115-129, https://doi.org/10.1016/j.inffus.2019.12.001
[2] M. Hubner et al., “A Bayesian Approach - Data fusion for robust detection of vandalism and trespassing related events in the context of railway security,” 27th Int. Conf. on Information Fusion (FUSION), Venice, Italy, 2024, pp. 1-7, DOI: 10.23919/FUSION59988.2024.10706430.
Please contact:
Michael Hubner
AIT Austrian Institute of Technology, Austria
