Flight Crew Coordination in Mixed Reality: The Effects of Using Video Passthrough for Vision

Sammanfattning

Head-mounted mixed reality devices are making their way into airline pilot training. Devices that use passthrough video hold much promise but come with caveats. Because current generation passthrough video devices occlude natural light, users rely entirely on a digital video feed that lacks the visual fidelity and field of view of the unaided human eye. The application of head-mounted extended reality devices in aviation has been predominantly examined in single-pilot contexts, with limited attention given to their implications for multi-crew flight decks. This study explored the effects of using a mixed reality device incorporating passthrough video in the context of a multi-crew airline operation, i.e., two co-located users wearing a head-mounted device, engaged in joint activity. The perspective taken was crew coordination. The methodology was inspired by an exploratory multiple case study research design. A naturalistic setting was achieved with experienced professional pilots flying a high-fidelity simulator in realistic scenarios. Multiple crews flew two similar scenarios, one with the head-mounted device and the other without it. The activity was observed and recorded for later analysis, and the crews were interviewed post-session about their experiences. Findings suggest that the technology is not value neutral and alters how the crew coordinates their work on a flight deck. The device disrupted normal work rhythm, created additional workload and affected how often the pilots looked at each other. These effects should be explored further and considered when designing training programs for pilots in multi-crew environments using mixed reality devices.