Evaluating GenAI for Database Design-Rule Review in FinTech: Industry experiment

Sammanfattning

Background. In a large-scale FinTech environment, the database consists of hundreds of interconnected tables where a single incorrect change can spread across the schema. Database-related merge requests are reviewed by a single database administrator (DBA) to prevent this, creating a bottleneck for the development teams. LLMs have shown potential across software engineering tasks, but none has been evaluated on proactive database design-rule review against what a human DBA actually flagged. Objectives. We evaluate whether GenAI can support DBA review tasks by comparing two LLMs, a commercial model (Claude Opus 4.6) and an internal open-source model (MiniMax M2.5), against a historical human baseline on three dimensions: (1) detection capability, (2) detection precision, and (3) code correctness. Methods. We conducted an experiment on 18 micro-cases extracted from real merge requests at a FinTech company. Both models received the same prompt template with database access via tool use. Two domain experts validated the output. Analysis combined bucket-based exploratory methods with hypothesis testing in a secondary role. Results. Opus covers a broader set of findings than MiniMax (79% of MiniMax's findings are also caught by Opus, 37% in reverse). Precision is inverted. The internal model is more precise on violations (78.8% vs 69.2%), Opus on suggestions (78.4% vs 48.6%), with the difference associated with vocabulary breadth. Model and expert effects cannot be fully separated in this validation design. Both models produce working code at similar rates, but Opus produces 28 invalid violations paired with working code compared to 3 for MiniMax, a confident hallucination where the code compiles but the problem does not exist. These are point estimates from a single run per case and may vary across runs. Conclusions. GenAI can replicate a subset of DBA decision-making on routine design rules, but cannot substitute the DBA's contextual judgment. The practical deployment model is to use the LLM as a first pass, reducing the DBA's search effort while retaining human judgment for contextual decisions.