AI-Enabled Services for Reconnaissance

by Refiz Duro, Rainer Simon (AIT Austrian Institute of Technology GmbH) and, Christoph Singewald (Syncpoint GmbH)

Climate change, pandemics and unstable geopolitical and economic circumstances on the global level are complex challenges necessitating approaches leveraging technological advances and human cross-domain expertise and experience. One piece of the puzzle addressing these challenges is to provide efficient services assisting decision and policy makers with insights extracted from the available data. Combining AI-based natural language processing and computer vision services with human-centred annotation and information enrichment service to build knowledge graphs for reconnaissance has a prospect to make a difference in the safety and security domain. But what does such an implementation look like, and shouldn’t such services already be integrated into most decision-making processes?

We are directly witnessing the impacts that current world-changing events have on our daily lives. Pandemics have and still are affecting the movement of people and goods globally, climate change has brought more frequent and severe events and destruction, while the geopolitical situation has forced the EU and state governments to re-evaluate their current state of defence capabilities.

The commonality of the events mentioned is the necessity of acquiring insights for accurate and timely decision and policy making. Considering defence, the acquisition, processing, and analysis of data for creating intelligence are military operations’ backbone. The dynamic nature of reconnaissance operations, however, require that the information be delivered as efficiently as possible, meaning with no or minimal delay and in a form that is easily consumable. To achieve this requirement and to leverage the increasing amount of available data and data sources, it is necessary to embrace technological advancements. They offer the potential to process, analyse, filter, and deliver information much more efficiently to ultimately accelerate the decision-making process.

The PIONEER project funded by the Austrian Defence Research Programme, has taken on the challenge of improving automation and digitalisation processes for information flow for reconnaissance and intelligence of Austrian Armed Forces (AAF). The initial overview of capabilities and practices has shown that the available ICT support for automated execution of the intelligence gathering and distribution can be significantly improved. For example, it can be challenging to quickly merge, structure and analyse collected sensor data (UAV footage, written reports) due to heterogeneity of data formats and multimodalities, limiting data fusion necessary for advanced analytics. Furthermore, although the experienced personnel involved in reconnaissance activities can quickly and accurately establish intelligence products using traditional methods (e.g., populating operational maps on walls with geotagged information), the ever-increasing amount of available data and diversification of data sources are forcing the practices to be enhanced through integrating state-of-the-art information extraction technologies.

Due to the selected type of data (PDF reports and photographs), two technological fields heavily integrating Artificial Intelligence (AI) have seemed to be the most obvious for the experimental try-outs: computer vision and natural language processing (NLP). In the first one, the focus is on the object detection, leveraging an open-source machine learning model, which uses the YOLOv3 architecture for multi-class object recognition, and weights trained on the Microsoft Common Objects in Context dataset. The photographs provided by the project partner AAF are typical for reconnaissance operations. The applied model on these images shows promising results, i.e., vehicles (e.g., trucks) and persons, which are two categories of high intelligence value, are detected with high accuracies. The focus of NLP in the context of the project is on the extraction of entities through named entity recognition (NER). The basic idea is to automatically tag and classify words in a document according to predefined categories such as person or place. Similar to object detection, there are available open-source libraries for building suitable NER pipelines. In our case, the documents are reconnaissance reports integrating information collected by the field scouts and applying the model on these reports again shows promising results (Figure 1).

Figure 1: Annotation results are highlighted through boxes in photographs (left) or are highlighted with colours in text-based data (right; from [1]). Annotations can be set in relations (“person drives car”) and enriched with additional information through taxonomy feature.  

Further processing implies services exploiting the extracted information moving in the direction of building knowledge graphs (KG) through enhancing NER and entity and event linking. A KG enables semantic search and serves as a basis for different forms of representation – spatial, temporal and relational, supports analysts in recognising patterns and drawing conclusions – and for automated reasoning. Within the project, the automatically generated annotations are mapped to an ontology (schema.org) via classifiers, converted to RDF 1.1 N-Quads and integrated into the existing KG, thus allowing for automated spatial and temporal patterns and anomaly detection, detection of otherwise difficult-to-detect networks of groups of people and organisations, and in general knowledge engineering.

To conclude, integrating AI capabilities into the intelligence-gathering process looks promising. Recent advances have made it possible to achieve respectable gains with little effort, so that open-source models can be used relatively easily for tasks intended for reconnaissance systems. The suitability of these models is, however, not optimal, as misclassification can occur due to inaccuracies or training datasets not including, e.g., "tank" as a class, but "truck" class. This requires updating the existing model through "transfer learning" so that it is more appropriate for the context of reconnaissance and daily environments relevant for the AAF.  It is critical that the objects and entities found in images and documents are linked and set in a context through additional information enrichment. This is not achieved by the object detection and NER components, but by open-source annotation libraries RecogitoJS and Annotorius [L1]. The tool necessitates a human user to, e.g., link the NER-detected entities through relations (“entity-1 is seen at the location entity-2”), and thus brings in the concept of human-centred in the process, making the system less prone to errors due to uncertainties and erroneous decisions. Implementing a KG in an RDF Quad Store is limiting, as the methods for network analyses resembles a property graph. In order not to lose the advantages of an RDF store, we consider using a hybrid model in future. We conclude that AI-based natural language processing and computer vision services with human-centred annotation and information enrichment service for reconnaissance has a prospect to make a difference in the safety and security domain.

Link:
[L1] https://recogito.github.io/annotorious/

Reference:
[1] G. Chroust, P. Doucek and, V. Oškrdal (editors), "IDIMT-2022 Digitalization of society, business and management in a pandemic : 30th Interdisciplinary Information Management Talk", in 30th Interdisciplinary Information Management Talks, Prague, 2022. https://doi.org/10.35011/IDIMT-2022

Please contact:
Refiz Duro, AIT Austrian Institute of Technology GmbH, Austria
This email address is being protected from spambots. You need JavaScript enabled to view it.