Automated Requirement Extraction from Tender Documents : Evaluation of Classical Machine Learning and Large Language Model Classification Approaches

Sammanfattning

Manual analysis of natural language requirements documents is time-consuming and error-prone. Real-world documents contain substantial amounts of non-requirement content alongside actual specifications. This work presents a tool that automates the extraction of requirement sentences from raw documents as a preprocessing step for further analysis, developed as part of a larger requirement engineering project to reduce manual effort. The project addresses two research questions: how requirements can be automatically identified from textual specifications using natural language processing techniques, and to what extent semantic analysis can improve the identification and interpretation of requirements. Several classification models were developed and evaluated on a dataset of approximately 40,000 manually labeled sentences collected from real tender documents, primarily within the software and electronics engineering domains. The classical models evaluated were Support Vector Machine (SVM), Naive Bayes, and Decision Tree, alongside a locally hosted Large Language Model, Mistral 7B Instruct. Results demonstrate that automated requirement extraction from natural language documents is feasible. Threshold selection significantly impacts classification performance, particularly in balancing recall and precision. Because missed requirements carry severe consequences in tender documents, the system prioritizes high recall. The SVM achieved the most robust and balanced performance among classical models, with an F1-Score of 83.1% and a recall of 84.9%. The LLM demonstrated higher recall on real-world documents (87.3% across 78 files) but at a substantially lower F1-Score (53.4%), making it better suited for contexts where missing a requirement is costlier than reviewing false positives.