Designing Swedish-Only Prompting Pipeline for Internal Documentation Tool with Local LLM : A multi-step prompt design with XHTML-safe editing and automated evaluation with LLM judge scoring

Sammanfattning

This thesis investigates how prompt engineering and lightweight guardrails can adapt a general-purpose LLM (Llama 4 Scout) to assist with Swedish documentation inside Arbetsförmedlingen’s Confluence workflow. The work focuses on three requirements: Swedish-only text improvement, lossless stripping and re-tagging of Confluence Storage Format XHTML and automatic evaluation using an LLM judge. A prototype system was implemented with an end-to-end pipeline consisting of plain text improvement, XHTML re-tagging, and automatic scoring using an LLM judge. Five system prompt configurations were evaluated on a small set of curated examples, with automatic scoring used to compare performance. The results show that structured prompts with explicit numbered instructions perform best for this editing task and that XHTML preservation is feasible when editing and formatting are separated. While the evaluation is limited and offline, the prototype demonstrates a practical pattern for integrating LLMs into a constrained public-sector documentation workflow and identifies a clear path for future deployment and human evaluation.