ERCIM News 134
July 2023
Special theme Explainable AI
Guest editors Manjunatha Veerappa (Fraunhofer IOSB) and Salvo Rinzivillo (CNR-ISTI)
Explainable AI - Introduction to the Special Theme
- Details
- Category: Special Theme
- Hits: 4990
by Manjunatha Veerappa (Fraunhofer IOSB) and Salvo Rinzivillo (CNR-ISTI)
Artificial Intelligence (AI) has witnessed remarkable advancements in recent years, transforming various domains and enabling groundbreaking capabilities. However, the increasing complexity of AI models, such as convolutional neural networks (CNNs) and deep learning architectures, has raised concerns regarding their interpretability and explainability. As AI systems become integral to critical decision-making processes, it becomes essential to understand and trust the reasoning behind their outcomes. This need has given rise to the field of explainable AI (XAI), which focuses on developing methods and frameworks to enhance the interpretability and transparency of AI models, bridging the gap between accuracy and explainability.
Explainable AI: A Brief History of the Concept
- Details
- Category: Special Theme
- Hits: 13042
by Mihály Héder (SZTAKI)
Understandability of computers has been a research topic from the very early days, but more systematically from the 1980s, when human-computer interaction started to take shape. In their book published in 1986, Winograd and Flores [1] extensively dealt with the issues of explanations and transparency. They set out to replace vague terms like “user-friendly”, “easy-to-learn” and “self-explaining” with scientifically grounded design principles. They did this by relying on phenomenology and, especially, cognitive science. Their key message was that a system needs to reflect how the user's mental representation of the domain of use is structured. From our current vantage point, almost four decades later, we can see that this was the user-facing variation of a similar idea, but for developers – object-oriented programming, a method on the rise at the time.
Diagnostic Explainability by BrightBox
- Details
- Category: Special Theme
- Hits: 3813
by Andżelika Zalewska-Küpçü (QED Software), Andrzej Janusz (University of Warsaw & QED Software) and Dominik Ślęzak (University of Warsaw & QED Software)
BrightBox technology presents a novel approach to investigating mistakes in machine learning model operations.
An Explanation that LASTS: Understanding Any Time Series Classifier
- Details
- Category: Special Theme
- Hits: 4272
by Francesco Spinnato (Scuola Normale Superiore and ISTI-CNR), Riccardo Guidotti (University of Pisa) and Anna Monreale (University of Pisa )
We present LASTS, an XAI framework that addresses the lack of explainability in black-box time series classifiers. LASTS utilises saliency maps, instance-based explanations and rule-based explanations to provide interpretable insights into the predictions made by these classifiers. LASTS aims to bridge the gap between accuracy and explainability, specifically in critical domains.
Explaining Ensemble Models for Lung Ultrasound Classification
- Details
- Category: Special Theme
- Hits: 3709
by Antonio Bruno, Giacomo Ignesti and Massimo Martinelli (CNR-ISTI).
Correct classification is the main aspect in evaluating the quality of an artificial intelligence system, but what happens when you reach top accuracy and no method explains how it works? In our study, we aim at addressing the black-box problem using an ad-hoc built classifier for lung ultrasound images.A Governance and Assessment Model for Ethical Artificial Intelligence in Healthcare
- Details
- Category: Special Theme
- Hits: 3911
by Luigi Briguglio, Francesca Morpurgo and Carmela Occhipinti (CyberEthics Lab.)
How can clinicians be deemed responsible for basing their decisions on diagnoses generated by artificial intelligence and derived in a way that cannot be fully understood? How can patients rely and accept decisions if they are based on “black boxes” of data and algorithms? In the context of the MES-CoBraD project, CyberEthics Lab. defines a model for governing and assessing “Ethical Artificial Intelligence” (ETHAI).
Current Challenges and Future Research Directions in Multimodal Explainable Artificial Intelligence
- Details
- Category: Special Theme
- Hits: 3862
by Nikolaos Rodis (Harokopio University of Athens), Christos Sardianos (Harokopio University of Athens) and Georgios Th. Papadopoulos (Harokopio University of Athens)
Despite the outstanding advances in Artificial Intelligence (AI) and its widespread adoption in several application domains, there are still significant challenges that need to be addressed regarding the explanation of how decisions are reached to the end-user. The latter need becomes more complex and demanding when multiple types of data are involved in the AI-based generated decisions; hence, leading to the emergence of the so-called multimodal explainable AI (MXAI) field. The above challenges become even more imperative for some critical domains, for example, medical applications (where human lives are involved).
Predictive Model for Functional Outcome after Orthopaedic Surgery Using Machine Learning Methods
- Details
- Category: Special Theme
- Hits: 3647
by Alexandre Lädermann (Hôpital de La Tour, Meyrin, Switzerland), Philippe Collin (American Hospital of Paris, France) and Patrick J. Denard (Oregon Shoulder Institute, Medford, Oregon, USA)
Surgery is said to be indicated when conservative treatment fails. Previous studies reported that around 20% of patients do not improve sufficiently after surgery, inducing frustration, high societal costs, and an overload of healthcare systems. The consortium members investigated the efficacy of machine learning methods in detecting outcomes. They achieved a promising model with a recall of 32% of the cases that were inappropriate candidates for an operation.
Unleashing the Power of Artificial Intelligence for Personalised Drug Design
- Details
- Category: Special Theme
- Hits: 3261
by Michaela Areti Zervou, Effrosyni Doutsi, Panagiotis Tsakalides (University of Crete and FORTH-ICS)
Precision medicine holds the promise of personalised treatment based on an individual's genetic makeup. However, the lengthy and costly process of drug discovery hinders progress. Can artificial intelligence (AI), specifically generative models offer a solution? Is it possible to efficiently select the most promising drug candidates for validation? Our research focuses on developing robust tools that streamline the validation process of drug design – saving time and resources.
An Explainable Deep Ensemble Framework for Intelligent Ticket Management
- Details
- Category: Special Theme
- Hits: 4012
by Gianluigi Folino, Massimo Guarascio, Luigi Pontieri and Paolo Zicari (CNR-ICAR)
Pushing intelligence and integrating explainable tools in the new generation of ticket-management systems is crucial for supporting customer-support activities. To this aim, we defined a comprehensive ticket-classification framework, which integrates deep ensemble methods and AI-based interpretation techniques to help both the operator identify misclassification errors and the analyst improve the model. Tests on real data demonstrate the quality of the predictions returned by the framework and the practical value of their associated explanations.
Merging Explainable AI into Automotive Software Development
- Details
- Category: Special Theme
- Hits: 4472
by Danilo Brajovic and Marco F. Huber (Fraunhofer Institute for Manufacturing Engineering and Automation IPA)
The debate about reliable, transparent and thus, explainable AI applications is in full swing. Despite that, there is a lack of experience in how to integrate AI-specific safety aspects into standard software development. In the veoPipe research project, Fraunhofer IPA, Huber Automotive, and ROI-EFESO Management Consulting work on a joint approach to integrate these AI-specific aspects into an automotive-development process. In this article, we share details on one component of this framework – reporting the AI development.
Decoding the Unknown: Unveiling Industrial Time Series Classification with Counterfactuals
- Details
- Category: Special Theme
- Hits: 3343
by Anahid Jalali (AIT), Andreas Rauber (TUWien), Jasmin Lampert (AIT)
Deep learning models for time series prediction have become popular with the rise of IoT and sensor data availability. However, their lack of explainability hampers their use in critical industrial applications. While existing model-agnostic approaches like LIME and SHAP have been used in time series classification applications, it is worth mentioning that they may have limitations in their suitability. For example, the random sampling process used by LIME leads to unstable explanations. We propose a counterfactual explanation approach for interpretable insights into time series predictions to address this issue. We choose an industrial use case, determining machine health, and employ k-means clustering and Dynamic Time Warping (DTW) to handle the temporal dimension. DTW compares and aligns two time series by discovering the optimal path of alignment that minimizes disparities in their temporal patterns. We explain the model's decisions using local surrogate decision trees, analysing feature importance and decision cuts.
ChatGPT Responses Validation through Knowledge Graphs
- Details
- Category: Special Theme
- Hits: 5737
by Michalis Mountantonakis and Yannis Tzitzikas (FORTH-ICS and University of Crete)
The novel artificial intelligence ChatGPT chatbot offers detailed responses across many domains of knowledge; however, quite often it returns erroneous facts even for popular persons, events and places. To tackle this problem, we present GPT•LODS, a novel prototype that annotates and validates ChatGPT responses by leveraging one or more RDF knowledge graphs.
An Intuitive Architecture for a Chatbot that Exploits High-Level Reasoning for Human–Robot Interaction
- Details
- Category: Special Theme
- Hits: 3616
by Christoforos Prasatzakis, Theodore Patkos and Dimitris Plexousakis (ICS-FORTH)
In this article, we present a new, easy-to-understand and flexible chatbot architecture, which banks on ease of use and modularity. It relies on the Event Calculus, in order to perform high-level reasoning on world events and agents’ knowledge, and can answer questions regarding the other agents’ belief state, in addition to what is happening in the chatbot’s world in general.
A Multilayer Network-Based Approach for Interpreting and Compressing Convolutional Neural Networks
- Details
- Category: Special Theme
- Hits: 3714
by Alessia Amelio, Gianluca Bonifazi, Domenico Ursino and Luca Virgili (Polytechnic University of Marche
We propose an approach to map a convolutional neural network (CNN) into a multilayer network. It allows the interpretability of the internal structure of deep learning architectures. Then, we use this representation to compress the CNN.
A Pathway to Combat Climate Change with Human-Centred XAI
- Details
- Category: Special Theme
- Hits: 3885
by Anahid Jalali, Alexander Schindler (AIT) and Anita Zolles (BFW)
Explainable Artificial Intelligence (XAI) is gaining importance in various fields, including forestry and tree-growth modelling. However, challenges such as evaluating model interpretability, lack of transparency in some XAI methods, inconsistent terminology, and bias towards specific data types hinder its integration. This article proposes combining long short-term memories (LSTMs) with example-based explanations to enhance tree-growth models' interpretability. By generating counterfactual examples and engaging domain experts, critical features impacting outcomes can be identified. Addressing privacy protection and selecting appropriate reference models are also crucial. Overcoming these challenges will lead to more interpretable models, supporting informed decision-making in forestry and climate change mitigation.
The Eye of the Beholder Project: Transparent and Actionable AI Pipelines for Information Quality Prediction
- Details
- Category: Special Theme
- Hits: 3759
by Davide Ceolin (CWI) and Ji Qi (Netherlands eScience Center)
The spread of disinformation affects society as a whole and the recent developments in AI are likely to aggravate the problem. This calls for automated solutions that assist humans in the task of automated information quality assessment, a task that can be perceived as subjective or biased, and that thus requires a high level of transparency and customisation. At CWI, together with the Netherlands eScience Center, we investigate how to design AI pipelines that are fully transparent and tunable by an end user. These pipelines will be applied to automated information quality assessment, using reasoning, natural language processing (NLP), and crowdsourcing components, and are available in the form of open source workflows.
Explainable AI for Astronomical Images Classification
- Details
- Category: Special Theme
- Hits: 5635
by Mahmoud Jaziri (Luxembourg Institute of Science and Technology) and Olivier Parisot (Luxembourg Institute of Science and Technology)
AI is an indispensable part of the astronomer's toolbox, particularly for detecting new deep space objects like gas clouds from the immense image databases filled every day by ground and space telescopes. We applied explainable AI (XAI) techniques for computer vision to ensure that deep sky objects classification models are working as intended and are free of bias.
ERCIM Appoints Dominique Hazael-Massieux as Manager of ERCIM EEIG
- Details
- Category: Joint ERCIM Actions
- Hits: 3165
The ERCIM EEIG board of Directors is pleased to announce the appointment of Dominique Hazael-Massieux as the new manager of ERCIM EEIG, effective from June 2023. Dominique will be taking over from Philipp Hoschka, who has served as the manager since December 2016.
Diversity Matters
- Details
- Category: Joint ERCIM Actions
- Hits: 3185
by Léon Ouwerkerk, CWI, interviewed by Monica Divitini, as part of the ERCIM HR initiative to create awareness for diversity in the broadest sense.
Diversity remains an important topic. ERCIM News’ Monica Divitini interviewed Léon Ouwerkerk of CWI. Ouwerkerk: “At CWI, we think it is important that everyone feels safe to be themselves in the workplace. That is one of the reasons for us to take actions to reach more diversity, equity and inclusion. I am glad to be asked to tell you more about our policies”.
Workshop on Privacy, Transparency, Sovereignty and Security: A Summary
- Details
- Category: Joint ERCIM Actions
- Hits: 1945
by Peter Kunz (ERCIM)
A collaborative workshop on privacy, transparency, sovereignty, and security was held on 27-28 April 2023 at the Inria Mediterranean Campus in Sophia Antipolis. The workshop was co-organized by the European projects
TRAPEZE [L1] and E-CORRIDOR [L2], and co-sponsored by Inria and ERCIM. The two-day workshop aimed to explore new ideas, share insights and collaborate on solutions that can benefit society in the fields of privacy, cybersecurity, and technology. This report presents a brief overview of the presentations and discussions held during the workshop [L3].
11th International Workshop on Computational Intelligence for Multimedia Understanding
- Details
- Category: Joint ERCIM Actions
- Hits: 3081
by Behçet Uğur Töreyin (İTÜ), Maria Trocan (ISEP) and Davide Moroni (ISTI-CNR)
Around fifty researchers attended the International Workshop on Computational Intelligence for Multimedia Understanding (IWCIM), organized annually by the working group Multimedia Understanding through Semantics, Computation and Learning (MUSCLE) of the European Research Consortium for Informatics and Mathematics (ERCIM), which took place as a satellite workshop of IEEE ICASSP 2023 held in Rhodes, Greece, June 5, 2023. This year's workshop was also a huge success as previous venues with two oral sessions and one poster session, hosting eighteen papers authored by seventy-three researchers affiliated with nineteen institutions coming from twelve countries.
Fibre-Optic Sensing for Road-Traffic Monitoring in Remote Areas
- Details
- Category: Research and Innovation
- Hits: 5639
by Martin Litzenberger, Carmina Coronel and Kilian Wohlleben (AIT Austrian Institute of Technology GmbH)
Accurate real-time traffic sensing is of key importance for optimising and managing road traffic. Often the high density of traffic sensors, needed to achieve an accurate real-time monitoring of important roads, is difficult to implement due to technical constraints or because of high installation cost. Fibre-optic sensing (FOS) is a new and cost-effective alternative technology that allows a seamless, real-time monitoring of the road traffic over large distances of up to 50 km, even in remote areas such as on critical costal or mountain roads, using existing telecom fibre-optic cable infrastructure.
Security-by-Design IoT Development and Certification with IoTAC
- Details
- Category: Research and Innovation
- Hits: 3589
by Sascha Hackel, Martin Schneider and Ramon Barakat (Fraunhofer FOKUS)
The Horizon 2020 European research project IoTAC [L1] enhances IoT security through a multi-layered approach using best practices, standards and research findings. Its technology includes privacy-friendly access-control mechanisms, ML-based attacked detection, and security-by-design methodology. Additionally, IoTAC provides developers and service providers with SDKs and APIs to seamlessly integrate the framework.
SD4MSD: Using a Single Device for Multiple Security Domains
- Details
- Category: Research and Innovation
- Hits: 4345
by Florian Skopik, Arndt Bonitz, Daniel Slamanig (Austrian Institute of Technology), Markus Kirschner (MUSE Electronics GmbH) and Wolfgang Hacker (Austrian Ministry of Defence)
Military field operations place high demands on information and communication technology (ICT) devices, both in terms of reliability and security. These requirements include robustness against environmental influences such as vibrations, water and humidity, as well as protection against physical attacks and cyberattacks [1]. Attempts to compromise a device must be detected immediately and, if necessary, trigger automated countermeasures such as alarms, partial deactivation or emergency wiping of all data.
SPACE: Scalable Parallel Astrophysical Codes for Exascale
- Details
- Category: Research and Innovation
- Hits: 3201
by Marisa Zanotti (EnginSoft), Andrea Mignone (University of Torino) and Manolis Marazakis (ICS-FORTH)
In astrophysics and cosmology (A&C), high-performance computing (HPC)-based numerical simulations are invaluable instruments to support scientific discovery. The efficient and effective exploitation of exascale (and beyond) computing capabilities will be key to paving the way for scientific discovery in this scientific domain, but requires a coordinated effort towards adapting A&C codes for space applications on exascale HPC systems.
TRAPEZE: Transforming Data Management for All
- Details
- Category: Research and Innovation
- Hits: 3038
by Lauro Vanderborght (Digitaal Vlaanderen), Martin Kurze (Deutsche Telekom) and Ramon Martin de Pozuelo (CaixaBank)
The TRAPEZE project, titled “Transparency, Privacy, and Security for European Citizens,” is making remarkable progress as it moves forward in its journey. With its objective of demonstrating the TRAPEZE prototype solution in real-world scenarios, the three project use cases are advancing splendidly. These use cases will exemplify how the TRAPEZE solution overcomes present-day limitations and revolutionizes the way enterprises, public administration, and citizens interact with their sensitive information.
Denotational Engineering - Call for Participation
- Details
- Category: Research and Innovation
- Hits: 3208
by Andrzej Blikle (Institute of Computer Science, Polish Academy of Sciences
This project is devoted to the development of programming languages in a way that guarantees the existence of denotational semantics for them. Once such a language has been developed, we can derive such program-construction rules, which guarantee the correctness of programs constructed by their means.
Prestigious Gödel Prize for Ronald de Wolf
- Details
- Category: Announcements
- Hits: 3241
Ronald de Wolf from CWI and his co-authors receive the 2023 Gödel Prize for outstanding papers in theoretical computer science.
TRAPEZE Project Webinar: Pioneering Privacy, Transparency, and Security for European Citizens
- Details
- Category: Announcements
- Hits: 2078
Online, 17 July 2023
The European TRAPEZE project is delighted to invite you to a special webinar where the project will be showcasing the remarkable achievements. This groundbreaking initiative has made significant advancements in safeguarding individual privacy while ensuring data transparency and security.
Dagstuhl Seminars and Perspectives Workshops
- Details
- Category: Announcements
- Hits: 3224
Call for Proposals
Schloss Dagstuhl – Leibniz-Zentrum für Informatik is accepting proposals for scientific seminars/workshops in all areas of computer science, in particular also in connection with other fields.
Open Call for Innovative Extended Reality Tools and Applications Targeting Training and Educational Scenarios
- Details
- Category: Announcements
- Hits: 3835
A total funding of €2,100,000 will be distributed to projects for the development of novel XR-based applications primarily targeting the manufacturing domain. The total amount of funding per successful project will be in the range of €150,000 to €300,000. Each project may have up to three members including industrial partners, universities and research centers.
14th IFIP Trust Management Conference
- Details
- Category: Announcements
- Hits: 3169
The 14th IFIP International Conference on Trust Management (IFIPTM 2023) will be held in Amsterdam, The Netherlands on 18-20 October 2023.
MISDOOM - The 5th Symposium on Multidisciplinary International Symposium on Disinformation in Open Online Media - Call for Contributions
- Details
- Category: Announcements
- Hits: 3772
Amsterdam, The Netherlands, 21-22 November 2023
The Multidisciplinary International Symposium on Disinformation in Open Online Media (MISDOOM) is returning for its 5th edition on 21 and 22 November 2023. This time, the conference will be hosted by the National Research Center for Mathematics and Computer Science (CWI) at Amsterdam Science Park (Netherlands). MISDOOM values multidisciplinary research and is designed to be inclusive of different academic disciplines and practices.