Combining pose estimation, machine learning, and LLMs for natural language descriptions of human motion in ergonomics analysis

Sammanfattning

Ergonomics evaluations in industrial settings commonly rely on manual observation, which is time-consuming and subjective, often leading to inconsistencies. Automating this process with large language models using raw video raises concerns regarding worker privacy and cost. This study, conducted in collaboration with EasyErgo, investigates whether LLMs can generate natural language descriptions of human motion from pose data alone. A pipeline was developed in which a machine learning classifier predicts human postures from pose estimation data, and the resulting information is passed to a large language model that produces descriptions. The pipeline was evaluated through two surveys in which participants matched generated descriptions to their corresponding videos, comparing the proposed pipeline against MG-MotionLLM, a state-of-the-art motion-to-text system, and a random-guessing baseline. The results indicate that the proposed pipeline is a viable proof-of-concept for privacy-preserving motion descriptions, outperforming both chance level and MG-MotionLLM. The implementation is open source on GitHub.