BENCHMARKING LARGE LANGUAGE MODELS IN UML DIAGRAM GENERATION FROM INFORMAL NOTATIONS

Sammanfattning

Translating initial design ideas into formal representations is key to producing structured software documentation. Large Language Models (LLMs), especially those with multimodal capabilities, offer new opportunities to automate this translation by interpreting visual inputs such as whiteboard sketches, flowcharts, and architectural overviews. While previous research has explored using LLMs to generate UML diagrams from structured text inputs, their effectiveness in handling image-based, informal notations remains underexplored. This thesis investigates whether state-of-the-art multimodal LLMs can generate valid UML diagrams from informal visual inputs. To address this, we compiled a dataset of informal diagrams extracted from scientific publications and prompted four multimodal LLMs to generate five types of UML diagrams for each image. The outputs were evaluated using automated syntax validation and manual semantic analysis. Our results show that LLMs can generally create structurally valid UML diagrams with good syntactic accuracy from informal input. However, capturing the original meaning remains challenging, and performance varies between models and diagram types. While LLMs offer the potential for supporting early-stage design documentation, human oversight is still needed. This work highlights the promise and the current limitations of using LLMs to automate software documentation.