Automatic Discrimination of Factoid and Inferential Questions in Reading Comprehension Tests : A Comparative Study of BERT-based Transformers

Sammanfattning

Reading comprehension tests are a common way of assessing a language learner's ability. The difficulty of a reading comprehension test is often determined by the balance of factoid and inferential questions. While factoid questions test retrieval of explicitly stated facts, inferential questions demand commonsense reasoning and "reading between the lines", making inferential questions generally more difficult. Manually classifying questions can be subjective and time-consuming for test designers. This study implements four automatic binary classifiers of varying sizes for reading comprehension questions using BERT-based Pre-trained Language Models, along with four lexical baselines. The SQuAD 1.1 dataset is used for the factoid class, and the Cosmos QA dataset for the inferential class. This data was processed in a filtering step and supplemented with synthetic data to prevent the models from relying on surface-level stylistic biases between the datasets. The fine-tuned classifiers (TinyBERT, DistilBERT, DeBERTa-v3-base, and DeBERTa-v3-large) were quantitatively evaluated based on F1-performance and parameter count. Due to hardware limitations, the largest model, DeBERTa-v3-large was trained using Low-Rank Adaptation (LoRA). The comparison demonstrates a strong positive correlation between model size and classification performance. The best-performing model, DeBERTa-v3-large, achieved a macro average F1-score of 0.926, outperforming the lexical baselines by ~24%. These findings suggest that it is possible to effectively automate the classification of reading comprehension questions using modern Transformer architectures, serving as a valuable tool for automatic difficulty assessment of reading comprehension tests.