Large Language Models and the Swedish Language : A Comparison of Accuracy in Reading Comprehension, Quantitative Reasoning, and Mathematical Problem Solving

Sammanfattning

Many large language models have been evaluated on datasets assessing different capabilities in the English language, enabling models to be compared. However, there exist fewer datasets in the Swedish language, and also fewer comparisons. Reading comprehension, quantitative reasoning, and mathematical problem solving, are some capabilities that are commonly evaluated in English. In contrast, how these models perform in these capabilities in the Swedish language has not been as widely explored. The purpose was to compare how large language models measure against one another in reading comprehension, quantitative reasoning, and mathematical problem solving, respectively, in the Swedish language. The goal was to contribute to research in the applicabilities and capabilities of large language models in the Swedish language. The research was conducted using a qualitative methodology with a comparative method, supported by quantitative data. We collected tests that were originally designed for humans, to assess the large language models’ reading comprehension, quantitative reasoning, and mathematical problem-solving capabilities, in the Swedish language. The questions from the tests were processed into datasets. Three models were selected to be compared on the datasets with accuracy as the comparison criterion. The selected models were GPT-3.5 Turbo, Claude 3 Sonnet, and Gemini 1.0 Pro. We observed that Claude 3 Sonnet performed the best in quantitative reasoning and mathematical problem solving, respectively, in the Swedish language, while Gemini 1.0 Pro performed the best in reading comprehension in the Swedish language. Although we are unable to generalize the findings, the work can be useful as a starting point for more comprehensive research.