Evaluating large language model adaptation strategies for geospatial code generation

Sammanfattning

Recent advances in Large Language Models (LLMs) offer a promising alternative by enabling code generation from natural language. However, despite progress, LLMs still struggle with spatial reasoning, structural fidelity, and robustness across diverse GIS datasets. This thesis systematically compares three LLM adaptation strategies, i.e., prompt engineering, retrieval-augmented generation (RAG), and QLoRA fine-tuning, for their effectiveness in geospatial code generation. A custom multi-agent evaluation framework is developed, testing six configurations on real-world datasets and validated QGIS scripts. The evaluation combines semantic metrics (CodeBERTScore, embedding similarity) with structural measures (CodeBLEU), and introduces AST-derived indicators of structural complexity and behavioral richness on generated code. Results show that while all strategies affect code structure, their impact on semantic fidelity is limited. Prompting and RAG enhance structural conformity by imposing external scaffolds, whereas QLoRA improves fluency and intent alignment but struggles with structural generalization. These findings highlight the need for structurally diverse supervision and varied training corpora to improve the reliability of fine-tuned LLMs in GIS programming contexts.