High-Quality Source Code Generation from Design Concepts Using Generative AI : An Experimental Evaluation of Large Language Models' Multimodal Input Capabilities

Sammanfattning

Saab's Mission Support System is a software system for planning, executing, and evaluating Gripen fighter aircraft missions. Developers make use of design concepts in the form of images or mock-ups when developing new features for the system, but manually converting these design concepts into XAML code can be a time-consuming and repetitive process. With recent advancements in generative AI, there are many studies exploring its possibilities within software development. However, there is limited research regarding the topic of LLMs' multimodal capabilities, especially with generation from design concepts. This thesis aims to explore the possibilities of utilizing design concepts to generate XAML source code through multimodal LLMs. Experiments were executed on several proprietary and open-source LLMs with two datasets, each consisting of 100 hand-drawn images and screenshots. For evaluation, static analysis was done on the generated code to determine the quality, and a survey was given to two developers of the MSS to understand which type of image yields the most accurate results. The results showed that across all models, 27.88\% of the total attempts were successful, and out of those attempts, 48.76\% had high-quality code. In total, 16.81\% out of all attempts resulted in runnable and high-quality XAML code. This indicates that LLMs still have room for improvement in reliably generating high-quality XAML code from design concepts. The hand-drawn images work better as input for generating more visually accurate and useful output compared to screenshots. The results also show that proprietary models generally perform better than open-source models, and that larger models generally outperform smaller models, with some exceptions. The top-performing model for all experiments is Gemini 2.0 Flash.