by John Soldatos (INNOV-ACTS Limited), Georgios Makridis (University of Piraeus Research Center) and Fotis Liarokapis (CYENS Centre of Excellence)
Extended reality (XR) is a term that covers all real and virtual environments, enabling human-machine interactions through computer-generated content. It includes virtual reality (VR), augmented reality (AR), and mixed reality (MR) technologies [1]. State-of-the-art XR applications for manufacturing are built based on a “one-size-fits-all” philosophy that does not consider the characteristics of individual workers. Hence, they fall short when it comes to supporting the emerging wave of Industry 5.0 (I5.0) applications that are destined to be human-centric and emphasise trustworthy human-machine collaboration. I5.0 requires XR visualisations that consider the characteristics, skills, and context of the manufacturing worker [2], along with the peculiarities of their interactions with machinery, automation devices, and cyber physical production systems.
In this context, the newly started XR5.0 Horizon Europe Project will build, demonstrate, and validate a novel person-centric and AI-based XR paradigm that will be tailored to the requirements and nature of I5.0 applications. The project will specify blueprints for using XR in I5.0 applications with emphasis on the development of innovative “XR-made-in-Europe” technology that blends with human-centric manufacturing technologies and adheres to European values (e.g. trustworthiness, security/privacy-by-design, transparency) as reflected in relevant EU regulations and policies.
Extended Reality Driven by Trusted Artificial Intelligence and Human-Centred Digital Twins
The XR5.0 applications will consider the characteristics and context of the worker based on the integration of human-centred digital twins (DTs) that comprise the “digital image” of the worker. At the same time, XR5.0 will design and implement a unique blending of XR technology and advanced AI paradigms, including AI technologies that foster the interplay between humans and AI such as explainable AI (XAI), Active Learning (AL) [3], Generative AI (GenAI), and Neuro-symbolic AI. The XR5.0 technologies will be coupled with a cloud-based XR training platform for Operator5.0 applications, which will enable ergonomic and personalised training of industrial workers on widely used manufacturing processes.
Figure 1: High-level overview of the architecture and enablers of the XR5.0 project applications.
The main results to be produced by the project are illustrated in Figure 1 and include:
- XR5.0-ReferenceModel: The project will build a reference model for “XR-made-in-Europe” technology that supports I5.0 applications in-line with European values.
- XR5.0-DigitalTwin: XR5.0 will provide a platform for the development of cognitive, human-centred digital twins that will be able to construct the worker’s digital image in a credible way. Such digital images will be accordingly used to personalise the content, the flow, the aesthetics, the locality, and the geometry of XR5.0 applications.
- XR5.0-AI: XR5.0 will build technology that blends XR into advanced, human-centric AI paradigms that are commonly deployed and used in I5.0 applications, such as GenAI, XAI, AL, and Neuro-symbolic AI. This blending will enable a variety of AI-enabled functionalities from the visualisation of AI-generated insights to AI-based interactions with cyber-representations of manufacturing tasks.
- XR5.0-AugmentedIntelligence: XR5.0 will enable new forms of multi-modal augmented intelligence based on the visualisation of AI outcomes (e.g. insights, recommendations, AI explanations) in various production scenarios. XR5.0-AugnentedIntelligence will be integrated within XR solutions of the partners to enable generation of innovative virtual worlds and construction of realistic, yet simulated environments (i.e. simulated reality).
- XR5.0-Training: XR5.0 will build its own cloud-based human-centric Operator5.0 training platform that will include XR content for common I5.0 UCs). The platform will be integrated with other XR platforms of the European XR ecosystem.
- XR5.0-Apps: The XR5.0 paradigm will empower the development of six novel pilot applications spanning the areas of AI-based product design, remote and intelligent maintenance of assets, workers’ training, support in product assembly, as well as guidance and instructions for troubleshooting. These applications will be deployed and demonstrated in realistic manufacturing environments.
- XR5.0-Ecosystem: The project will build its own community of XR stakeholders including researchers, XR vendors and providers/integrators of I5.0 enabled XR solutions. The development of this community will boost the sustainability and wider use of the project’s results.
XR5.0 will enable novel applications such as the following user journeys (UJ):
- UJ #1 (Personalised, Multi-Modal, AI-Enhanced Workers’ Training): A manufacturing worker needs to learn a new process for assembling a complex product. With the help of the XR5.0 training platform, they are trained in a highly customised, multi-modal environment that blends physical and virtual aspects. The environment imitates the manufacturing process while considering the worker’s context (e.g. physical characteristics, skills, and emotional status). Moreover, the environment integrates AI features, such as explanations for parts of the process and real-time feedback on the user’s activities in the applications. This enables workers to improve their learning experience and satisfaction while at the same time minimising the learning curve for the assembly process.
- UJ #2 (AI-based Sustainable-by-Design Products/Production): A manufacturing product manager wants to optimise a product for sustainability based on a Sustainable-by-Design (SbD) approach, which is suggested in Europe’s Circular Economy Action Plan [L1]. To contribute to SbD, XR5.0 enables the manager to visualise numerous product options in a Metaverse-like environment. Using XR5.0 technologies, the manager can visualise hundreds of production configurations to audit them against their sustainability features and CO2 footprint, but also evaluate their appearance. XR5.0 can also visualise explanations, to enable the manager to analyse “what-if” scenarios and improve their decision making. This results in increased productivity for the workers, while improving sustainability.
- UJ #3 (Effective Mass Customisation based on AI and XR): This scenario involves a product designer that needs to create a customised product for a customer, i.e. based on a Made-to-Order (MTO) approach. XR5.0 enables the designer to use a GenAI model for producing multiple design options. The XR environment provides real-time feedback, allowing the designer to make changes on the fly and improve the final product's quality.
- UJ #4 (Personalised Remote Instructions and Support in Maintenance Processes): A maintenance manager needs to service a complex piece of machinery, which requires the expertise of the machine builder. XR5.0 obviates the need for costly travels to the remote site, as the machining expert can provide personalised remote guidance to an on-site technician. The remote guidance is visualised in an XR environment based on instructions tailored to the skills of the technician. XR5.0's AI and personalisation features enrich the standard instructions with personalised information, alerts, assistance, and contextual explanations. These features enable the technician to complete service and repair processes quickly, safely, and efficiently.
The work presented in this article has received funding from the European Union’s Horizon Europe research and innovation programme under grant agreement No. 101135209 (XR5.0).
Links:
[L1] https://environment.ec.europa.eu/strategy/circular-economy-action-plan_en
References:
[1] S. Doolani, et al., “Review of extended reality (XR) technologies for manufacturing training,” Technologies, vol. 8, no. 4, p.77, 2020. https://doi.org/10.3390/technologies8040077
[2] S. Grabowska, S. Saniuk, B. Gajdzik, “Industry 5.0: improving humanization and sustainability of Industry 4.0,” Scientometrics, vol. 127, pp, 3117–3144, 2022.
[3] P. Zajec, et al., “Help me learn! Architecture and strategies to combine recommendations and active learning in manufacturing”, MDPI Information Journal, 2021. https://doi.org/10.3390/Info12110473
Please contact:
Georgios Makridis
University of Piraeus Research Centre, Greece
