A new augmented reality framework utilizes high-fidelity digital twins and machine vision to enable autonomous technical maintenance in industrial environments.
Researchers at the University of Oulu have developed an augmented reality framework designed to facilitate complex technical maintenance without requiring real-time expert intervention. The system utilizes high-fidelity digital twins generated through 3D scanning to provide precise, step-by-step guidance for specialized tasks, effectively decoupling expert knowledge from synchronous human presence.
The architecture relies on an initial phase where experts define procedural workflows using a 3D-scanned digital twin of the target hardware. These workflows are augmented with graphical overlays, including spatial annotations and text-based instructions, which are then stored in a centralized library.
Users access these instructions through a mobile interface that registers the digital overlay onto the physical device using machine vision algorithms. The system functions by aligning the virtual instruction set with the physical object, allowing for precise spatial tracking during the repair process.
This approach moves beyond traditional tutorial media by providing context-aware, geometry-linked guidance that updates based on the user’s interaction with the hardware. The project, titled IMMERSE, integrates years of research in image-based 3D reconstruction to improve the accuracy of these virtual representations.
Janne Mustaniemi, a researcher and project manager at the University of Oulu, notes that recent advancements in the field have significantly increased the fidelity of these digital twins. The development team is currently focusing on scaling the system to handle more complex mechanical environments beyond initial demonstrations.
The underlying computer vision pipeline requires robust feature extraction to maintain registration stability across varying industrial environments. By mapping the physical device’s geometry to the digital twin, the algorithm ensures that graphical annotations remain anchored to specific components regardless of the user’s viewing angle.
Future iterations aim to automate the generation of these instructions using artificial intelligence to reduce the manual labor currently required by human experts. Janne Heikkilä, Professor of Machine Vision and Signal Analysis at the University of Oulu, emphasizes that the current implementation is a foundational step toward more autonomous maintenance systems.
AR-assisted guidance has indeed been studied worldwide, and there are also commercial applications, but they typically require the presence of an expert and interaction with the user, whereas our proposed solution allows the task to be performed independently without personal guidance.
The research team is also investigating the transition from mobile-based displays to augmented or extended reality headsets to improve ergonomic efficiency. This shift would eliminate the need for manual device handling, allowing technicians to maintain focus on the physical task at hand.
The project, funded by Interreg Aurora and conducted in collaboration with Mid Sweden University, specifically addresses operational challenges in sectors such as mining and forestry. By digitizing expert workflows, the technology aims to mitigate the impact of regional skill shortages in industrial manufacturing.
The integration of machine vision into these workflows represents a shift toward more resilient, decentralized maintenance operations. As the team moves toward an open-source release, the focus remains on refining the user interface and optimizing the underlying computer vision pipelines.
These improvements are essential for ensuring that the system can reliably track hardware components in variable lighting and environmental conditions. The long-term objective is to enable manufacturers to provide standardized, high-precision maintenance protocols for any physical asset.
Success in this domain will depend on the continued improvement of spatial registration accuracy and the ability of AI models to interpret complex, multi-step mechanical procedures. The team plans to release an open-source implementation that includes all essential tools for creating digital twins and instruction sets.
Upcoming milestones include the integration of AI-driven instruction generation to further streamline the workflow for non-expert users. The project will continue to evaluate the system’s performance in real-world industrial settings to refine its operational reliability.
