Evaluation Robustness of Climate NLP Benchmarks: Revising SciDCC through Lexical Transformations in Multi-Label Scenarios

Sammanfattning

Large language models are increasingly applied to climate-related text analysis, yet existing climate benchmarks have been shown to contain annotation errors, inconsistencies, and design weaknesses. These issues cause them to fail to adequately capture model performance and raise questions about the reliability of current evaluation practices in the climate change domain. This study analyses the Science Daily Climate Change (SciDCC) benchmark for single-label topic classification and proposes a revised multi-label classification and ranking framework, including robustness testing through lexical transformations on a word and sentence level. We generate three different text variants, involving paraphrasing, masking, and code-mixing strategies, and evaluate four large language models -- Mistral-Large 2 (Mistral AI), Llama3.3 70B (Meta), GPT-OSS 20B (OpenAI), and Llama3.2 3B (Meta) on the extended benchmark through label and ranking overlap, consistency, robustness, inter-model analysis, and human annotation. The results show that the redesigned multi-label classification task is well-defined and that models remain highly robust to text variations, although the transformations are only partially effective at exposing differences among the top-performing models. The ranking task exhibits lower inter- and intra-annotator agreement and greater disagreement between models and annotators. Nonetheless, robustness scores remain high across most transformations, which may be attributed to certain limitations of the evaluation metrics. The study makes a first attempt at evaluating robustness and consistency in the climate domain and suggests strategies for further benchmark improvement and task refinement.