Conversational Vocabulary Practice using Generation Techniques in Large Language Models

Sammanfattning

Chatbots leveraging Artificial Intelligence (AI) to communicate with humans are increasingly prevalent across various domains, from customer service to education. In language learning, communication is a central component and thus this specific area of education has a great opportunity to benefit from the recent advancements in generative AI, brought by Large Language Models (LLMs). This study seeks to address the problem of how to make LLMs incorporate specific words in the generated text and applies this to an advanced English language learning scenario. To determine the optimal approach for the task, two methods are compared: one employs prompting and one utilises a variant of the decoding method beam search known as constrained beam search. The methods are also compared to a version, with no enhancements, that acts as a baseline. These methods are evaluated through a within-subjects experiment involving 32 proficient English speakers. The evaluation metrics not only assess how effectively the methods incorporate the requested words but also how the responses are perceived by the users. The LLM used is Llama 3 8B Instruct by Meta. The results indicate that prompting is the more effective of the two methods, scoring significantly better on 3 out of 4 metrics. However, both methods are considered viable options. This demonstrates that relatively small LLMs can be used for complex tasks without the need for fine-tuning. This is positive because access to the largest commercial LLMs is not always possible and comes with its own challenges.