Knowledge Graph-Based GraphRAG for Clinical Question Answering in Hashimoto’s thyroiditis Built from Peer-Reviewed Literature

Sammanfattning

Background. Hashimoto’s thyroiditis is the most common autoimmune thyroid disease, and its rapidly expanding literature makes evidence synthesis increasingly difficult. Large language models offer support for medical question answering but suffer from hallucination and limited reasoning over relational knowledge. Objectives. This thesis investigates whether a domain-specific GraphRAG system built from peer-reviewed Hashimoto’s thyroiditis literature can produce more accurate and clinically relevant answers than vector-based RAG and vanilla LLM approaches. Methods. A corpus of 115 peer-reviewed papers was processed through a pipeline covering metadata extraction, section-aware chunking, schema-constrained entity and relation extraction, and ingestion into a Neo4j graph database and a Qdrant vector store. The pipeline produced a knowledge graph of 10,596 nodes and 17,580 re-lationships, structured under a corpus-derived schema of 23 entity types and 32 rela-tion types, alongside a vector store of 1,765 chunk embeddings. A hybrid GraphRAG retrieval and generation layer combining graph traversal, dense vector search, and chain-of-thought prompting was evaluated against vector-based RAG and vanilla LLM baselines on multiple-choice and open-ended question answering tracks, with an ablation study isolating the contribution of each component.(code available at https://github.com/chappiii/hashimotos-graph-rag) Results. The hybrid system achieved 85.0% aggregate accuracy on the retained multiple-choice benchmarks, a 6.67 percentage point advantage over both baselines.On open-ended evaluation, the hybrid system led on Faithfulness (0.941 vs. 0.905for vector-only), confirming stronger grounding in corpus evidence, while BERTScore confirmed marginally higher semantic and biomedical lexical alignment across both encoders. The ablation study identified chain-of-thought prompting as the dominant factor for reasoning quality, vector retrieval as the foundational component forretrieval completeness, and graph retrieval as the primary contributor to relational reasoning on complex clinical questions. Conclusions. A domain-specific GraphRAG pipeline built from scientific literature is a viable and effective approach for structured biomedical question answering. The combination of knowledge graph retrieval, dense vector retrieval, and chain-of-thought reasoning produces a system that is more robust, faithful, and domain-aligned than either retrieval modality or prompting strategy operating in isolation.