Simuleringsverktyg för analys av beslutsfattande hos LLM kontrollerade agenter : En konceptvalidering i virtuella SAR-miljöer

Sammanfattning

As autonomous systems increasingly operate in high-stakes environments such as Search and Rescue (SAR), the integration of Large Language Models (LLMs) offers unprecedented cognitive capabilities. However, standardizing the evaluation of these agents’ decision-making in these scenarios remains a significant challenge. This study presents a simulation tool, developed through a Design and Creation approach, designed to systematically quantify behavioral differences of LLM controlled agents in a simulated 3D environment. By utilizing a closed-loop architecture that measures cognitive resource consumption via generated tokens rather than execution time. Furthermore, the tool ensures hardware independent and reproducible data. Experimental validations were conducted by assigning distinct personality profiles based on “Big Five”-model to the agents across two different scenarios. The results demonstrate the system’s capability to capture variations in navigation, target prioritization and resource management. The report analyzes these variations and highlights statistically significant differences between profiles. While current LLM limitations in spatial reasoning result in some navigation anomalies, such as oscillation. The findings successfully validate the tool as a proof of concept. Ultimately, this research provides a methodological foundation for evaluating and developing reliable, safe and transparent LLM controlled agents for future critical missions.