Understanding Ambiguity: Designing, evaluating, and improving an LLM-powered virtual assistant for ambiguous task resolution in realistic scenarios

Sammanfattning

Introduction This thesis investigates how large language model (LLM)-based virtual assistants interpret ambiguous task descriptions, and how their understanding can be improved through targeted techniques. This is crucial for the development of acceptable LLM-powered virtual assistants for retrieval-augmented generation and real-world task solving. The thesis work is motivated by and written in collaboration with the company Destiny Sweden Service Center. Research Question How effectively can an LLM-powered virtual assistant interpret and solve ambiguous instructions in a realistic usage setting? How can ambiguity resolution be improved through targeted modifications to prompt and software design? Method An LLM-based virtual assistant was built on top of an existing cloud communications software platform. The assistant was then evaluated in two iterations. In the first iteration, a baseline version of the assistant was evaluated both qualitatively, to identify areas for improvement to design, and quantitatively, to measure the assistants performance. In the second iteration, the improved version of the assistant was re-evaluated quantitatively in order to determine if the improvements identified in the first iteration had any significant impact on the assistants performance. Results The results indicate that the assistant has achieved a high degree of both measurable performance and effective handling of ambiguous tasks. The results suggest that increasing LLM size had the highest impact on performance, although targeted improvements to design were also significant. Discussion These findings suggest that LLM size is the most important predictor for increasing the ambiguity resolution of virtual assistants. However, targeted prompt design and code constraint techniques seem to elicit significant enhancements to ambiguity resolution, particularly in scenarios when LLM size is limited.