Generering av kunskapsgrafer med hjälp av stora språkmodeller : En automatiserad kartlägningsprocess av kursmaterial på lärandeplattformen Kollin

Sammanfattning

This thesis investigates how Large Language Models (LLMs) can be utilized to automatically generate knowledge graphs from mathematical theory chapters stored in Kollin’s database. Two prompting techniques were evaluated - stepwise prompting, where all subtasks are executed in one single LLM call, and prompt chaining, where a sequence of LLM calls tackle one subtask each. To create the prompts for both prompting techniques, a method called Recursive Meta Prompting was used - where an LLM iteratively creates and updates the prompt with the help of human input. Both techniques produced coherent graphs, but due to the inherent stochasticity in LLMs, the results varied from graph to graph. However, prompt chaining offered superior trans- parency and debuggability at the cost of multiple LLM calls. Furthermore, it was investigated to which degree adjusting the hyperparameter settings of the LLM had a significant effect on output stability in the keyword extraction step of the prompt chaining sequence. A temperature setting of t ≤ 0.3 ensured stable keyword extraction, while variations in top-k and top-p did not have a sig- nificant impact. Varying the level of detail in the system prompts revealed that both minimal and highly detailed instructions led to lower tolerance for temperature changes, whereas a moderate level of detail allowed for greater robustness to such variations. Qualitative analysis of selected chapters confirmed that using a higher temperature on occasion included keywords with less relevance, but also showed that some of the variation in the output could be explained from the LLM extracting different grammatical variants of the same concept. Future studies should focus on incorporating Retrieval-Augmented Generation (RAG) to surpass context window constraints (token limits), and on hybrid solutions that combine automation with expert-crafted ontologies.