Profiling and Visualizing CPU-Based LLM Inference in llama.cpp

Sammanfattning

Large language models (LLMs) are increasingly deployed on local hardware ratherthan through cloud services, motivated by concerns around data privacy, networkdependence, and operational cost. In many local deployments, the CPU serves asthe primary or sole compute resource, since dedicated accelerators such as GPUsare not always available. Running LLMs on CPUs is technically demanding, as inference performance is shaped by complex interactions between the workload andthe underlying hardware. Identifying performance bottlenecks is therefore essentialfor improving local LLM inference, but is difficult without structured measurementand analysis.This bachelor’s thesis presents a profiling and visualization tool for characterizingLLM inference on CPUs using an inference serving framework called llama.cpp.The tool collects hardware performance counters, which are low-level CPU metricssuch as cache accesses and floating-point operations, at four levels of granularity: theentire inference run, the prefill phase in which the user’s input prompt is processedand the decode phase in which the response is generated token by token, individualdecoder blocks, and individual tensor operations. Measurements are stored in astructured database and can be explored through a graphical dashboard, allowingusers to inspect runtime, memory traffic, cache behavior, and other metrics acrossdifferent quantization formats and model sizes. By applying roofline analysis, thetool distinguishes between compute-bound and memory-bound behavior and supports the identification of performance bottlenecks during local LLM inference.