by Christian Kollmitzer (AIT Austrian Institute of Technology GmbH), Melanie Schranz (Lakeside Labs GmbH) and Manuel Warum (AIT Austrian Institute of Technology GmbH)
Current methods for collecting spatial information from, e.g. crime scenes, historical excavation sites or construction sites, sometimes have significant limitations. Our approach is based on a modular, adaptable system that can be equipped with multiple sensors, an intuitive presentation and processing of collected, heterogeneous data. The generated model can be stored in a structured manner and supplemented with additional information such as place and time, whereby a continuous signature procedure is used.
Current technologies, in particular developments in the field of photo and sensor technology, make it possible to obtain a wide range of heterogeneous data. These can be used to obtain a detailed image of spatial conditions. Due to the multitude of heterogeneous data sources and the increased quality of the measurements, however, the complexity of the data model also increases. Therefore, appropriate preparation, processing and representation of the data obtained are a prerequisite for efficient usage. Our method offers a multidimensional approach, which considers the current and upcoming development of sensor technologies, combines them, and prepares them as multidimensional models for visualisation. The robust design and the simple and intuitive handling of multiple sensors minimise the probability of errors during data acquisition. The resulting high-quality models form the base for further processing of the data, the representation using different techniques as well as archiving. By using appropriate procedures, the data and models are also digitally signed to prove their authenticity. We are pursuing these goals with the patent pending Vestigator method (A50147/2020).
The ever-improving methods for recording and evaluating spatial data with a high degree of detail are applicable to many applications – for example, the digital reconstruction of crime scenes, historical buildings and excavation sites, or for the documentation of the progress at construction sites. Accordingly, there is a multitude of individual products, procedures, and approaches that work independently from each other.
In most cases it is common to use 360-degree cameras to depict an environment. This creates individual high-resolution photographs that allow the preservation of details that are particularly relevant to an application. Depending on the application, further data is obtained from additional sources and analyses. In the context of forensics, these would include fingerprints or biochemical laboratory reports of previously unidentified substances. This results in a collection of facts about the site, but the potential to combine the data into a holistic model is very restricted.
A number of problems arise from the current procedures. On the one hand, the components used cannot be adapted to current developments, which means that they cannot keep pace with the state of the art. Furthermore, they are expensive to purchase and operate, and therefore in many cases are not available in sufficient quantities. In addition, some widely used methods to map a 360-degree environment are very time consuming: the creation of the photographs and the subsequent image processing (for example with stitching algorithms) can take a long time, as they are sensitive to errors. Incorrectly recorded images (i.e. insufficient quality, poor lighting conditions, etc.) must usually be reacquired completely. Additionally, collected information such as detailed images or analysis reports are loosely combined into a collection of facts. However, the complete processing of the different data cannot be completely guaranteed due to their diversity and quantity. In many cases, it is this comprehensive mapping that is crucial.
In addition, it must be ensured that the authenticity of the collected data is verified to prevent any subsequent manipulation of the data. This is relevant, for instance, if the collected data must be updated or reassessed after a longer period of time, for example in connection with a new commencement of legal proceedings.
Solution approach
Our solution approach is based on a modular, adaptable system that can be equipped with current and future sensors. The data is efficiently recorded by the simultaneous use of several heterogeneous sensors, stored in a structured manner and supplemented with additional information such as place and time. The data of a single measurement or several measurements can then be transferred into a holistic model. This model can subsequently be extended by further information, whereby a continuous signature procedure is used at any time to prove which additions have been made to the model.
Apart from the developments in the field of photo and sensor technology, new advances in the areas of data analysis and artificial intelligence are continuously emerging. To account for progress in these areas, the data is prepared accordingly. This makes it possible, among other things, to interpret individual measurements or entire models, to identify common features or noticeable divergences. Current techniques such as augmented reality (AR) or virtual reality (VR) also allow individual models to be presented in such high quality that both qualified experts and members of a jury can work with them in a meaningful way.
Vestigator model
The creation of a model begins with the definition of the task. Depending on the definition, the type of measurement and the corresponding sensors are selected. In addition to measurements with a tripod, the use of self-propelled systems or drones is also possible, for example in environments that are difficult to access, dangerous for humans or contaminated.
The measurement is performed using a carrier device on which the individual sensors are mounted. The sensors are mounted in such a way that the measuring beam would pass in a backward extension through the centre of rotation of the carrier device. The carrier device is then brought to a measuring point. In order to carry out a measurement, the carrier device performs a full, stepwise 360 degrees rotation depending on the sensors used. During this rotation process, the sensors acquire measurement data. Afterwards, further measurements can be carried out at different points of origin, or at the same measuring point in a different perspective by changing the height or by tilting the carrier device. The data obtained from the measurement is digitally signed immediately while it is saved.
From the data obtained, a model is created, which includes the different sensor measurements. This model consists of several layers (at least one per sensor), whereby the layers are evaluated individually or combined. Figure 1 shows different layers with data from four different sensors clockwise from the left: thermo scan, picture, LiDAR and UV scan. This model can be extended at any time to include additional external data independent of the individual measurements such as the time at which the measurements commenced, people in charge, climate conditions etc. After that, the model is signed like individual measurements.
Figure 1: Vestigator-different layers of spatial data from four different sensors.
Additional information relevant to the model can be obtained even after the original measurement took place, for example in the form of laboratory reports, legal assessments or research reports. These can be incorporated into the model by adding an additional layer.
This layer can be based on the optical image and the additional information can be integrated into the model using anchor points. For example, data about a person found in a database containing fingerprints can be added to the fingerprints found at a site so the information about the person concerned can be subsequently incorporated into the model. After each addition to the model, a further digital signature is added.
The representation of the data can be carried out in different ways. The layer model enables the use of VR technology to virtually move through the environment. The use as AR model or a representation as 2D model, especially to display individual details, is possible. The model itself can be enhanced with further elements for representation. For example, additional objects such as mannequins can be inserted, the target state can be compared to the measured actual state or reconstructions of damaged items can be made. The models can be sent to experts for assessment, and their reports can be included in the model. This database can be supplemented by new insights over the years. Apart from the conservation of the scene, this approach is open to current and future methods from the field of machine learning, for example in pattern recognition or the reconstruction of sequences.
Please contact:
Christian Kollmitzer, Center for Digital Safety & Security,
AIT Austrian Institute of Technology GmbH, Austria
