Evaluating large language models for natural language queries in manufacturing execution systems

Sammanfattning

This thesis examines whether large language models (LLMs) can translate natural language questions into SQL queries for a Manufacturing Execution System (MES). MES databases store important production data, but retrieving that data can require knowledge of the interface, database structure, or SQL. The study therefore investigates how model size and model specialisation influence effectiveness and efficiency in natural language-to-SQL generation. A controlled quasi-experiment was carried out in collaboration with Schneider Electric using four Qwen2.5 models, the same question bank, a sanitised industrial MES database, manually created retrieval-augmented generation (RAG) context, SQL validation and read-only execution. The evaluation considered both effectiveness and efficiency. Effectiveness was measured through exact match, execution success, and F1-score, while efficiency was measured through latency and token usage. The results showed that no model performed best in every area. Among the larger models, specialisation led to different strengths: one model produced more correct results regarding exact match and F1-score, while the other more often generated executable SQL. The smaller models were faster overall. The experiment results were also discussed with a Schneider Electric representative through a limited semi-structured validation interview to support the practical interpretation of the findings.