Performance Bottleneck Evaluation of llama.cpp on Jetson and H100

Sammanfattning

Efficient large language model inference depends strongly on the interaction betweenworkload shape, numerical precision, serving configuration, and hardware platform.This thesis evaluates the performance bottlenecks of llama.cpp on two contrastingNVIDIA platforms: the datacenter-class H100 GPU and the edge-oriented JetsonAGX Orin. The study uses Llama 3.1 8B models in BF16, Q8_0, and Q4_K_Mformats, and separates inference into prefill and decode phases using controlledsingle-sequence workloads and concurrent serving experiments.The evaluation first establishes baseline performance across balanced, prefill-heavy,and decode-heavy workloads. It then applies targeted profiling with Nsight Systems,server-side timing logs, and power measurements to explain the observed behavior.On H100, the results show a stable phase-dependent precision trade-off: BF16 ismost effective for long-prefill workloads because execution is dominated by optimized BF16 GEMM and attention kernels, while Q4_K_M is more favorable fordecode-heavy workloads where execution shifts to repeated matrix-vector kernels.Flash Attention further improves long-prefill throughput, but has a smaller effecton decode.On Jetson Orin, the dominant tuning problem is different. Performance and energyefficiency depend strongly on the selected power mode. The results show that 50Wprovides a strong energy-oriented operating point, while MAX mode gives the highest throughput and lowest latency. Orin also shows power-mode-dependent precisionbehavior: Q8_0 remains competitive at lower power, while Q4_K_M becomes morefavorable for decode at higher power modes. Concurrent serving experiments furtherreveal a trade-off between throughput, energy per token, and time to first token.Overall, this thesis shows that inference optimization cannot rely on a single globalconfiguration. Instead, effective deployment requires phase-aware, platform-aware,and power-aware tuning. The final guidelines recommend precision, Flash Attention, power mode, and concurrency settings based on the dominant workload anddeployment objective.