Assessment and Failure Recovery in Remote Vision-Language-Action Deployment - From Pipeline Measurement to Proprioceptive Retry

Sammanfattning

Vision-language-action (VLA) models integrate visual perception, language understanding, and motor control within a single network, enabling robots to performmanipulation tasks from natural-language instructions without task-specific programming. Because these models are computationally demanding, they are oftendeployed on a remote GPU server: the robot client sends observations over a network, receives action chunks asynchronously, and continues executing motion whilethe next chunk is being inferred. However, this asynchronous deployment introducesfailure modes that are not visible when the policy is evaluated in isolation, includingstale action chunks, waiting periods at chunk boundaries, and physical states thathave diverged from the observations used to generate the action chunks.In this thesis, we study these issues on both real robots and simulation benchmarks under a unified workflow. We collected an SO-101 dataset, fine-tuned π0.5and SmolVLA on it, ran both models on a remote GPU cluster through the LeRobot asynchronous-inference stack, and executed the returned action chunks on therobotic arm. The same workflow is also used as a measurement tool: it decomposesend-to-end latency across the pipeline and shows how image cropping and Real-TimeChunking choices shift the main bottleneck between the network and the GPU.To address execution failures in this workflow, we propose Failure-Aware Recovery(FAR), a lightweight runtime monitor that wraps an unchanged VLA policy withoutadding extra sensors. FAR jointly uses gripper intent from the action queue andgripper feedback to detect empty-grasp and slip events. The feedback comes frommotor load and position on the real robot, and from finger separation and MuJoCocontact information in simulation. Once a failure is detected, FAR drains stalequeued actions, executes a bounded recovery motion, refreshes the server context,and requests a new action chunk from the recovered state. FAR relies only onexisting gripper feedback and requires no additional sensors, no policy retraining,and no additional learned module for action evaluation, state prediction, or vision-language reasoning. With FAR, the success rate improves across most LIBEROsuites and on the real robot, while these gains are generally accompanied by shorterepisodes. Under scripted human interference that forces empty grasps, task successrises from 32% to 100% for π0.5 and from 40% to 82% for SmolVLA, with negligibleadditional latency from the recovery layer.