Combining foundation models and numerical solvers for physics-informed motion control

Sammanfattning

Multi-modal foundation models for robotics control is a rapidly evolving field. Models are trained on large amounts of robot trajectory data and, in some cases, additional web data. This approach to robot control allows for control policies that are capable of reasoning and that have the capability of performing previously unseen tasks in unseen environments. However, these models have no safety guarantee built into their motion planning. As a contribution towards increasing the safety of these models, we propose a novel scheme for capturing unsafe trajectories before they are executed, called Sparse Evaluation. This method integrates a numerical physics solver with the foundation model, to screen proposed actions and assert their safety before real-life execution. In this thesis paper, we provide a small proof-of-concept for this scheme, and the results indicate that it is possible to use this method to detect if an action will be carried out as expected or not, depending on some parameters in the system. We also provide a brief survey of existing models for robot control with foundation models.