Benchmarking LLM performance on financial data mapping : Using LLMs to automate financial data restruction

Sammanfattning

This thesis investigates whether large language models (LLMs) can automate the generation of SQL-based mappings for heterogeneous financial data sources while preserving both structural correctness and financial consistency. The work was conducted in collaboration with Swimbird, a wealth management software company that aggregates portfolio and transaction data from multiple banks and custodians into a unified canonical schema. Because incoming financial data differs substantially between institutions in structure, semantics, and business conventions, onboarding new data sources currently requires extensive manual mapping work. To address this, an LLM-driven pipeline was developed that generates DuckDB-compatible SQL views and refines them through two correction loops driven by structural validation and financial reconciliation. A central contribution of the thesis is the use of reconciliation-based evaluation as a reference-free correctness signal: instead of comparing generated SQL against gold-standard queries, the system evaluates whether mapped transaction movements reconcile against the corresponding changes in holdings. Experiments were conducted on four anonymised institutional sources covering transactions, holdings, and cash balances, comparing pipeline configurations that varied the use of grounding and the number of correction iterations. On standardised single-currency sources, several runs achieved near-complete reconciliation. Grounding improved compile success and gave reconciliation a better starting point on more complex multi-currency sources, albeit at a higher token cost. The dominant remaining failures were not related to SQL syntax generation but to higher-level financial semantics. Across 24 runs, transaction classification errors alone accounted for 50.7% of all reconciliation diagnoses, followed by cash-leg routing and foreign-exchange settlement handling. The findings suggest that LLMs reliably produce structurally valid transformation pipelines but struggle with the domain-specific accounting logic required to reconcile complex transactions autonomously. The thesis concludes that LLM-based systems are viable as assisted mapping tools that substantially reduce manual workload, while fully autonomous mapping of heterogeneous institutional data remains an open challenge.