Do Large Language Models Know Their Chemistry? - An Evaluation of Open-Source LLMs for Chemical Reaction Feasibility

Sammanfattning

In drug discovery, a crucial task is determining whether two molecules can form aspecified product. Large language models (LLMs) have recently emerged as promising tools for such tasks, but their ability to reason about molecular structures beyondsurface-level pattern memorization remains uncertain. This thesis evaluates variousopen-source LLMs on binary chemical reaction feasibility classification. In this task,the models must determine whether a candidate product can be formed from twogiven reactants.For this evaluation, we tested ten open-source models ranging from 7 to 80 billionparameters. The best-performing model was further evaluated using randomizedSMILES and few-shot prompting.The results show that most models perform above the random baseline, with thebest-performing model achieving an accuracy above 0.9 while maintaining a balanced error profile. We also found that parameter count is not the only predictorof performance; the model developer and architecture appear to play a more significant role, although larger models generally performed better than smaller ones.Randomizing the SMILES resulted in only a small decrease in accuracy, indicatingthat the models possess some capability to reason beyond surface-level memorization.Surprisingly, few-shot prompting did not improve performance but instead led to asmall decrease in accuracy. Together, these findings suggest that some open-sourceLLMs can serve as useful tools for reaction feasibility assessment.