Designing and Evaluating AI Optimized Datasets to Enhance Telecom Retrieval Performance in RAG Systems

Sammanfattning

Developing a domain-specific dataset to enhance retrieval relevance in the telecommunications field is a complex task that requires expertise in natural language processing. This project aimed to improve the performance of Ericsson’s CoreChat system by leveraging artificial intelligence techniques and advanced language models. Python was utilized for data collection, preprocessing, and dataset creation, with OpenAI’s GPT-3 model employed to generate related questions. Text embedding was facilitated through the Ericsson Language Intelligence API, resulting in a dataset specifically aligned with telecommunications queries. The effectiveness of the dataset was evaluated using two vectorization techniques, traditional TF-IDF and ELI embeddings via the specter2-base model. Both methods used cosine similarity to calculate the relevance scores between the user query, the reference document, and the dataset entries. The evaluation revealed that searches conducted with the enhanced dataset consistently produced better match results. Specter2-base embeddings demonstrated superior semantic depth and contextual relevance, achieving higher alignment with complex queries, as evidenced by consistently higher cosine similarity scores. This project highlights the value of domain-specific datasets in improving the retrieval relevance for industry-specific applications. By enhancing CoreChat’s ability to handle telecommunications-specific queries, the project sets the stage for further system enhancements. Leveraging Ericsson’s technical documents, the project creates a robust foundation for future advancements in AI-driven retrieval systems.