LLM Tokenization Effect on Swedish Noun Gender Classification

Sammanfattning

AI assistant models generally target English. While the instruction-following capability transfers to other languages, their words are split into more pieces called tokens, which increases cost. Whether this splitting also has a negative impact on how well grammatical rules are learned remains underexplored, despite how often they come into play. To study this, we look at the gender agreement between Swedish nouns and the indefinite articles en and ett, since a separate noun suffix token requires learning fewer associations, and some Swedish single-token nouns share spelling with English nouns, which previous work shows great spelling recall on. We test on a smaller model where efficiency will be more obvious. As a prerequisite, we also investigate variations to the prompt they are given. We see how well the large language model (LLM) Llama-3-8B-Instruct is able to recall the gender of 1 975 singular indefinite Swedish nouns of various frequencies. 6 different prompt questions are fed to the model, both with and without examples included. We penalize overconfident errors in making the correct binary choice, determined by unambiguous SALDO lexicon entries. To see whether frequency has a compensating effect, the words come from two 1 000-word samples, one sampling the tail-end of the frequency distribution. To see whether the association between noun gender and plural suffix category (declension) is at play, we balance the samples by those categories in addition to token length. Finally, we use the best-performing questions to find trends in accuracy as a noun gets broken up into more tokens. We mostly find minimal improvement when examples are included. We find that accuracy generally increases as token length increases for 1 to 4 tokens, with variation along individual plural suffix categories. The results also suggest poor generalization of the gender rule for compound words.