How Far Is Too Far? Temporal Memory Limits in Large Language Models

Sammanfattning

Modern Large Language Models (LLMs) perform strongly across many tasks but struggle with information that changes over time. When asked time-dependent factual questions, a model may default to the most recent fact, the most prominent fact, or a nearby historical change point. This thesis investigates how LLMs internally process temporal information and how temporal reasoning gradually degrades with increasing temporal distance. We construct a controlled, Wikipedia-grounded dataset of 1140 time-sensitive questions covering five relations, two time modes (absolute, namely as of YYYY, and relative, namely X years ago), two reference years, and two entity-prominence levels. We analyze four open instruction-tuned models using behavioral evaluation paired with attention and MLP analyses implemented in TransformerLens: Gemma-2-2B-it, Gemma-3-4B-it, Llama-3.2-1B-Instruct, and Qwen2.5-7B-Instruct-1M. We also include control experiments on combined temporal formulations and ambiguous transition years, as well as exploratory robustness analyses.Our central observation is that absolute and relative temporal queries are associated with different attention heads in the Gemma family (Gemma-2: L15H1 versus L13H6; Gemma-3: L21H1 versus L22H5), partially in Llama (L11H26 for absolute, L8H2 for relative, with cross-talk), and not at all in Qwen, where a single head (L22H20) dominates both modes. Central processing appears distributed and not localized to a single component. Attention and MLP analyses suggest partially distinct routing and answer-construction behavior across the tested architectures and temporal formulations. Accuracy degrades gradually with temporal distance and shows no sharp cutoff within our observed range, while entity prominence shifts the failure boundary more strongly than time mode in three of the four models. Incorrect answers frequently reflect unstable competition between internally represented candidates rather than a complete absence of temporal knowledge.The contributions of this thesis are: (i) evidence of mode-specific temporal routing as an architecturally diverse phenomenon, (ii) a mechanistic interpretation of temporal reasoning as distributed temporal processing rather than a single localized mechanism, (iii) evidence that temporal failures may reflect biased candidate resolution rather than complete temporal knowledge absence, (iv) characterization of temporal degradation as gradual and prominence-sensitive rather than boundary-like, and (v) a Wikipedia grounded controlled dataset usable for follow-up mechanistic work on time-sensitive question answering.