Assisted Configuration Generation Based On Device Manuals

Sammanfattning

Configuring industrial Modbus RTU gateways from device manuals currently takes a non-expert one to several hours per device, with the time scaling roughly linearly in the number of slave devices and the manual ambiguity. This thesis investigated whether a Large Language Model can extract the configuration parameters automatically, reducing the manual step to seconds per manual while remaining accurate enough to deploy under engineer review. The proposed method is a two-pass modular RAG pipeline that splits extraction into separate communication-settings and transaction-settings passes, each with its own retrieval queries and system prompt. It was evaluated on a curated evaluation dataset constructed for this thesis, comprising 51 publicly available Modbus RTU manuals with manually validated golden configurations produced using the official HMS configuration generator and reviewed by a company expert, against four baselines: three prompt-only methods of increasing structure, and a single-pass naive RAG variant. All methods extracted the same 13 scored configuration fields per manual, using gpt-5.1 with temperature 0. Statistical significance was assessed with McNemar’s exact test and cluster-bootstrap confidence intervals. Modular RAG reached 85.38% standard accuracy, with an error rate when answering of 3.79%, the lowest of any method evaluated, including the most expensive prompt-only baseline at 6.27%. It also reduced total processing cost by 26% compared to the engineered prompt and produced a substantially flatter per-manual cost profile, since retrieval caps the context length regardless of manual size. A three-tier decomposition of the prompt-engineering benefit showed that simply rewriting questions to mirror the configurator’s option set, without adding any Modbus knowledge, accounted for +16.25%, while the explicit protocol rules added a further +7.02%. The thesis makes three contributions: a deployable two-pass RAG pipeline that produces fewer wrong answers than any other method evaluated; a three-tier decomposition of the prompt-engineering benefit separating question framing from domain knowledge; and a curated evaluation dataset of 51 Modbus RTU manuals with manually validated golden configurations, released publicly on GitHub to support future research. Together, these results demonstrate that retrieval-based extraction reaches accuracy and cost levels that support deployment as a configuration assistant under engineer review.