Evaluation of LLMs within the FMI simulation standard

Sammanfattning

Simulation of complex systems plays a critical role in the development process and personnel training in industries such as aerospace. The Functional Mock-up Interface (FMI) standard is used for co-simulation, where different models can exchange information and be integrated through a standardized interface. The standard was created with deterministic, fast, and self-contained models in mind. With the rapid development and improvements of LLMs, there could be several benefits of integrating LLMs into the standard. However, LLMs differ in many aspects from the physics-based models that the FMI standard was built for. This thesis investigated the feasibility of using LLMs in simulations using the FMI standard. Different approaches were evaluated, how the FMI standard enables and constrains model interactions for LLMs was explored, and whether LLMs could comply with co-simulation requirements for models. The questions were evaluated using two use cases. The first consists of feeding the model multiple errors from an airplane, with the task of finding the root cause of the errors. The second task involved generating natural language by providing pilot training instructions based on a specific airplane scenario, with information and scenarios from pilot handbooks. Results showed that not all requirements and desired attributes could be fulfilled at the same time, and would depend on the chosen approach. An external model could provide relatively high accuracy without the need for expensive local hardware. However, it relies on external services that will introduce latency and external dependencies to the simulation. Local approaches could instead be independent of external services, but could result in a large model package and need significantly better local hardware to get acceptable performance. Therefore, depending on the requirements, different approaches and models should be considered. Trade-offs will be made, primarily between accuracy and latency, since larger models generally have higher accuracy and higher latency. The second use case demonstrated how accuracy can be increased with a RAG system, although latency would once again increase. Therefore, several trade-offs would need to be made when choosing an integration approach.