Governed Natural Language to SQL Generation : A Behavioral and Mechanistic Analysis

Sammanfattning

Enterprise analytics environments impose strict governance constraints on data access, including restrictionson personally identifiable information (PII) and approved join paths between database tables. This thesisinvestigates governed natural language to SQL (NL-to-SQL) generation from three linked experimental tracks,with the mechanistic interpretability findings constituting the primary contribution.In the mechanistic track (Track 2), TransformerLens is applied to Pythia-2.8B to identify internal componentsassociated with constraint violations. Layer 23 Head 28 (L23H28) is identified as a PII-associated attentionhead, showing a 24× higher attention signal to the email token on PII-violating prompts (mean 0.307) comparedto compliant prompts (mean 0.013). Correlation and prediction evidence are strong: L23H28 fires strongly onPII prompts, and its attention weights classify violations at 94% cross-validated accuracy. Intervention evidenceis weak but directionally consistent: ablating L23H28 alone reduces the PII violation count only marginally,from 18/18 to 17/18. This supports the claim that L23H28 is a high-signal component associated with PII-related SQL generation, not that it is the sole or dominant causal driver. A matched-pair replication on 96pairs (192 total prompts) yields 97.4% classifier accuracy (±1.7%), but lexical baselines achieve comparableperformance (Table 9); therefore the current controlled dataset does not isolate activation-based predictionfrom surface-token prediction. Two violation types show different internal signatures: PII violations route moreconsistently through heads associated with PII-related tokens (L23H28 and Layer 17 Head 13), while unapprovedjoin violations show no consistent dominant head, suggesting they may require structurally different mitigationstrategies.In the behavioral track (Track 1), a 52-question benchmark with verified ground truth evaluates three instruction-tuned models. Mistral-7B-Instruct was evaluated across all six semantic context conditions; Qwen2.5-7B-Instruct and Phi-3-mini were evaluated on five primary conditions to test whether the main schema and policyeffects replicated across model families. Schema context alone improves SQL executability by 41–50 percentagepoints across all three models, while on the evaluated benchmark, full policy context was the only conditionthat achieved complete governance compliance (6/6 violations caught on all models). A schema-complianceinteraction anomaly is identified in which models with stronger implicit governance knowledge (Qwen, Phi-3)lose compliance when explicit schema context is added. Chain-of-thought reasoning imposes a consistent 3.5–4×latency overhead across all models regardless of family.The core finding is: governed NL-to-SQL should be evaluated on compliance separately from SQL execution.Semantic schema context improves SQL executability, while explicit policy context is required for complianceon the evaluated benchmark. In a mechanistic case study using Pythia-2.8B, PII-related SQL generation isassociated with identifiable internal activation patterns, but single-head ablation and lexical baselines showthat the present evidence supports a high-signal association rather than a complete causal circuit or a surface-feature-independent monitor.