Jämförande undersökning av Gemini Ultra och GPT-4 med avseende på integrering inom en matematiskt pedagogisk verksamhet

Sammanfattning

This study investigates the potential of two state-of-the-art AI-based language models, OpenAI’s GPT-4 and Google’s Gemini Ultra, to improve math performance among Swedish students. In light of the latest PISA 2022 results, which show a decline in mathematical performance, the need for innovative and effective educational tools more evident than ever. The study focuses on the implementation of these language models within Mattecoach.se, a digital platform offering math assistance, and evaluates their ability to deliver pedagogically relevant and mathematically accurate answers. By integrating AI into the education sector, the study explores opportunities to relieve teachers and create a more adaptive and responsive learning environment. To assess the mathematical competence of the language models, responses were generated for 136 different math questions from national exams at the secondary and high school levels. With the assistance of employees at Mattecoach.se, these responses were evaluated to determine both the mathematical accuracy and the pedagogical adequacy of the language models. The results of the study indicate that GPT-4 performed better in terms of mathematical accuracy, with 79% correct answers, while Gemini Ultra achieved only 57% correct answers. The inability to consistently produce correct answers is reflected in the operational feedback, as employees do not see as much value in using AI if the answers may not be reliable.