by Alessandro Pollini, Tania Sabatini and Sara Traversari (BSD Design)
In manufacturing, Human-Line Interactions (HLI) are characterised by three aspects: the operator’s loss of situational awareness due to the opacity of the systems [1], the complexity and interdependency of all the automation functions, and the machine autonomy that always requires to be clearly communicated. The PROXIMA project solves for these aspects by creating an Augmented Reality (AR) Training Kit that supports and enhances human cognition in the act of learning in HLI-based environments.
With the focus on human cognition in the act of learning, the PROXIMA (AR) Training Kit will seamlessly integrate spatial guidance, machine and equipment training, and hazard identification, through virtual simulations and real-time contextual instructions [2, 3]. A key aspect of the Kit is proximity, both in terms of spatial factors (i.e. user’s position with relation to the line and workspace features) and human factors (i.e. the machine is shown differently based on the user’s knowledge and necessities).
Human Cognition at Heart
As a general scope, the Kit aims to increase the acquisition of production line management skills; to promote production processes knowledge, and to enhance the capability of breakdowns’ remediations. This will also bring the development of personal trust, accountability, and autonomy in high-automation lines.
The Kit reaches the above results by having human cognition at heart. In fact, the Kit is made of three key modules: 3D Visual Storytelling, Training Authoring Tool, and AR Training. The innovation of the Kit lies in the investigation of users’ psychological (knowledge, skills, and attitudes towards automation) and cognitive (learning, inspection, awareness, decision-making) dimensions via 3D storytelling; in the way the Authoring Training Tool enable tailoring of the training programmes on trainees’ needs; and in the on-field AR training that enables practice sessions immersed in relevant real-life scenarios.
The product is being prototyped with the objective to validate its efficacy in real-life situations and it will be considered successful if the modules effectively contribute to the operator’s learning experience.
Figure 1: Outline of a case study addressed within the PROXIMA project.
The Modules
- 3D Visual Storytelling Module: the Kit will present stories describing user control vs. machine automation challenges and proposing decision-making dilemmas. This approach aims to stimulate users’ reflection about different interaction styles and to collect data about their preferences, prior knowledge of and current attitude toward automation. This module will include three stories typical of HLI environments: automation monitoring; semi-automatic routines and human-machine collaboration; and breakdowns’ recovery. Data on operators’ trust, expectations, and attitudes will be collected and will inform educational scenarios.
- Training Authoring Tool: the project will develop the AR-specific version of the tool that will support expert users and trainers in building the educational scenarios by: (a) Scanning real physics about the actual production environment following this logical phase: material preparation (e.g. position and physical properties), program (e.g. the product recipe), transformation, fine tuning, product quality; (b) Retrieving documentation and technical specifications; (c) Defining the Scenario Graph with points of interest for critical interactions with industrial machines, and the related natural interaction strategies, including the triggers for the human action, the decision nodes, and the consequences of the action; (d) Producing visual training content to be visualised in AR layers.
- AR Training Module: Through the AR layer, trainees will be provided with simulations and contents as resources to learn automation processes and enable them to manage critical situations. Specifically, in relation to the trainee’s spatial position, consistent visual instructions for interacting with smart and automated machines will be provided to the trainees considering their decision-making style and system control preferences (e.g. according to the desired level of automation). These AR contents are enabled by the user’s spatial position detection and personalised according to the educational scenarios (i.e. Training Authoring Tool).
With PROXIMA, we aim to offer training experiences in symbiosis with the production line. Thus a stable and reliable framework is needed to structure tailored HLI for each project.
Although one of the goals of this project is to achieve commercialisation of the Kit, we intend to release open-source functional building blocks, giving priority to virtuous exchanges with EU open source community on XR – in line with the European sharing policies on digitalisation. This would allow for both efficient development of the technology and the blossoming of procedures that spread the know-how and encourage the dissemination of the principles of industrial upskilling and reskilling in virtual environments.
PROXIMA is proposed by BSD in partnership with PROTESA and the University of Bologna. It has received funding from the European Union’s Horizon Europe research and innovation action programme, via the XR2Learn project (grant agreement Nr. 101092851). The project started in January and will conclude by the end of December 2024.
Links:
[L1] https://www.linkedin.com/showcase/101584342/admin/feed/posts/
[L2] https://www.bsdesign.eu/PROXIMA-eng
References:
[1] M. R. Endsley, “Automation and situation awareness,” in Automation and Human Performance, pp. 163–181, CRC Press, 2018.
[2] J.K. Bologna, C.A. Garcia, A. Ortiz, P.X. Ayala, and M.V. Garcia, “An Augmented Reality platform for training in the industrial context,” IFAC-PapersOnLine, vol 53, pp. 197–202, 2020.
[3] C. Caballini, A. Carboni, F. Boero, F. Parodi, I. Valentini, M. Paolucci, and S. Pagano, “Augmented reality and portable devices to increase safety in container terminals: the testing of A4S project in the port of Genoa,” Transportation Research Procedia, vol. 69, pp. 344–351, 2023.
Please contact:
Alessandro Pollini, Tania Sabatini, BSD, Italy
