LLM-assisted trend scout

Sammanfattning

Abstract It is important for a large company like Bosch to be at the forefront of technological development, and a central part of this entails scouting for trends in the market. To achieve this, they search the Internet for news articles that could be of interest, providing them with vast amounts of information. A problem, however, is that they receive information about very many different areas in many different regions, which makes it difficult and time-consuming to sort out what may be interesting and relevant. This thesis explores how LLMs can be used to assist with finding relevant information on the internet. The thesis contains two main parts: One explores how a pre-trained LLM can be optimized through different techniques to evaluate given information according to its relevance to Bosch. The other explores how this solution could be integrated into a web-based application. To test how an LLM could be optimized, 5 different LLM techniques (zero-shot prompt, few-shot prompt, multi-stage pipeline, RAG 1, and RAG 2) were found and created by doing literature research, and each LLM technique were tested on a test-set of articles that already were labeled on a scale from 0-3 by a Bosch employee were the score 0 meant not relevant and score 3 meant highly relevant. To test how this solution could be integrated into a web-based application, a prototype was made upon requirement from a Bosch employee and some user-tests were done by a Bosch employee. The results of the different LLM techniques showed that multi-stage pipeline performed best in F1-score, Cohens kappa and Quadrant kappa in predicting the right relevant score to an article. But in another test where scores 2 and 3 were seen as positive relevance, and scores 0 and 1 were seen as negative relevance. Few-shot performed better than multi-stage in recall of articles with positive relevance, meaning it found most of the positively relevant articles. To minimize the risk of missing a relevant article in the prototype, a few-shot prompt would be the best LLM technique for the prototype. Due to the limitation of training data in the RAG 1 technique, a reliable conclusion could not be drawn from the results for RAG 1, and since RAG 1 is similar to the few-shot technique, RAG 1 was used in the prototype although few-shot prompt performed better in the LLM technique tests. But to make a better decision about which LLM technique to use in the prototype, more test data should be used to compare the different LLM techniques. The final prototype was tested by a Bosch employee according to the user-tests and the result showed that the prototype was easy to use and had a high number of relevant articles.