Romance-MoLE: Rich Cousins and Their Benefits An Approach to Low-Resource Language Modelling

Sammanfattning

Integrating low-resource languages into Large Language Models (LLMs) is fundamentallylimited by persistent data scarcity and the “Curse of Multilinguality,” where dominantsource languages overwrite minority language representations and cause structural“translationese”. To preserve the endangered Languedocien dialect of Occitan withoutcatastrophic forgetting, this thesis presents Romance-MoLE (Mixture of LoRA Experts),a parameter-efficient framework applying targeted cross-lingual transfer from Occitan’sphylogenetic “rich cousins”: Catalan and French. Using the pre-trained Llama-3.1-Carballo architecture, the pipeline applies Trans-Tokenisation to repair subword fragmentationand utilises curated synthetic data to enforce standardised dialectal norms.Independent Low-Rank Adaptation (LoRA) experts extracted for French and Catalanare fused via TIES-Merging to construct a stable baseline initialisation. Finally, a HierarchicalMixture of Ranked Adapters routing strategy (HMoRA) applies token-levelgating in shallow layers for localised morphological precision and sequence-level poolingin deep layers to eliminate mid-sentence code-switching. Romance-MoLE yields a+14.57 chrF++ improvement in Occitan translation quality over a standard fine-tuningbaseline, while mitigating catastrophic forgetting in Catalan and allowing the model tofollow French instructions with a significant reduction in structural code-switching.