Knowledge Driven Procedural and Spatial Reasoning for Context Aware Operator Assistance in Extended Reality

Sammanfattning

Industrial operators often perform procedural tasks such as assembly, inspection and maintenance under conditions where correct sequencing, spatial awareness and timely information are important. Extended Reality (XR) can support such work by placing guidance closer to the point of action, but many XR systems focus mainly on displaying instructions rather than capturing procedural activity in a structured and reusable form. This thesis addresses that problem by developing a knowledge driven approach for transforming procedural evidence into an assembly-oriented graph representation.The study follows a design oriented and development based methodology and is evaluated through two complementary empirical branches. The first branch uses data captured with a Meta Quest 3 headset to demonstrate the full transformation from sensorderived evidence to graph-based outputs. In this branch, colour images, depth information and camera pose data are organised into frame-wise records, processed into detections, spatial observations, object tracks and primitive events and then transformed into higher-level symbolic layers such as state facts, operation events, subtasks, workflow phases and an assembly graph. The second branch uses an external industrial like assembly dataset called IndustReal, to examine whether the downstream state-, procedure- and graph-reasoning layers can operate on a larger external industrial like dataset. In this branch, assembly-state labels provided by the dataset are treated as trusted input rather than being predicted from raw video. The results show that XR-captured procedural activity can be organised into a coherentgraph-based representation. They also show that the later symbolic reasoning layers can recover procedural structure and generate graph outputs from reliable assembly-state labels in an external industrial like dataset. The Quest branch supports the feasibility of a capture-to-graph prototype, while the IndustReal branch demonstrates transferability of the later reasoning layers under conditions where the assembly states are already known. At the same time, the results show important limits that the system does not yet provide full autonomous industrial assembly understanding, confirmed physical attachment detection or CAD-level verification. The main contribution of the thesis is therefore a layered and traceable representation method that connects procedural evidence, events, states, operations, phases and graph-based assembly knowledge in a form that can support future context-aware operator assistance and later reuse.