An Agentic System for Schema Aware NL2SQL Generation
arXiv:2603.18018v1 Announce Type: new Abstract: The natural language to SQL (NL2SQL) task plays a pivotal role in democratizing data access by enabling non-expert users to interact with relational databases through intuitive language. While recent frameworks have enhanced translation accuracy via task specialization, their reliance on Large Language Models (LLMs) raises significant concerns regarding computational overhead, data privacy, and real-world deployability in resource-constrained environments. To address these challenges, we propose a schema based agentic system that strategically employs Small Language Models (SLMs) as primary agents, complemented by a selective LLM fallback mechanism. The LLM is invoked only upon detection of errors in SLM-generated output, the proposed system significantly minimizes computational expenditure. Experimental results on the BIRD benchmark demonstrate that our system achieves an execution accuracy of 47.78% and a validation efficiency score of
arXiv:2603.18018v1 Announce Type: new Abstract: The natural language to SQL (NL2SQL) task plays a pivotal role in democratizing data access by enabling non-expert users to interact with relational databases through intuitive language. While recent frameworks have enhanced translation accuracy via task specialization, their reliance on Large Language Models (LLMs) raises significant concerns regarding computational overhead, data privacy, and real-world deployability in resource-constrained environments. To address these challenges, we propose a schema based agentic system that strategically employs Small Language Models (SLMs) as primary agents, complemented by a selective LLM fallback mechanism. The LLM is invoked only upon detection of errors in SLM-generated output, the proposed system significantly minimizes computational expenditure. Experimental results on the BIRD benchmark demonstrate that our system achieves an execution accuracy of 47.78% and a validation efficiency score of 51.05%, achieving over 90% cost reduction compared to LLM-centric baselines as approximately 67% of queries are resolved using local SLMs. The system achieves an average cost per query of 0.0085 compared to 0.094 for LLM-only systems, achieving near-zero operational costs for locally executed queries. [Github repository: https://github.com/mindslab25/CESMA.]
Executive Summary
This article proposes an innovative system for natural language to SQL (NL2SQL) generation that leverages Small Language Models (SLMs) as primary agents, complemented by a selective Large Language Model (LLM) fallback mechanism. The system achieves significant computational savings and improved real-world deployability in resource-constrained environments. Experimental results demonstrate a substantial reduction in execution accuracy compared to LLM-centric baselines, while maintaining a high validation efficiency score. The system's cost-effectiveness and near-zero operational costs for locally executed queries make it a promising solution for democratizing data access. However, the system's performance on more complex queries and its potential limitations in handling diverse schema types remain areas of concern.
Key Points
- ▸ The proposed system employs SLMs as primary agents for NL2SQL generation, complemented by a selective LLM fallback mechanism.
- ▸ The system significantly minimizes computational expenditure and achieves near-zero operational costs for locally executed queries.
- ▸ Experimental results demonstrate a substantial reduction in execution accuracy compared to LLM-centric baselines, while maintaining a high validation efficiency score.
Merits
Efficient Computational Resource Utilization
The system's strategic use of SLMs and selective LLM fallback mechanism enables efficient computational resource utilization, making it suitable for resource-constrained environments.
Improved Real-World Deployability
The system's cost-effectiveness and near-zero operational costs for locally executed queries enhance its real-world deployability and make it a promising solution for democratizing data access.
High Validation Efficiency Score
The system's experimental results demonstrate a high validation efficiency score, indicating its ability to accurately resolve queries and achieve near-zero operational costs.
Demerits
Performance on Complex Queries
The system's performance on more complex queries remains an area of concern, as the authors do not provide comprehensive evaluation on this aspect.
Potential Limitations in Handling Diverse Schema Types
The system's limitations in handling diverse schema types, if any, are not explicitly discussed, which may impact its applicability in real-world scenarios.
Expert Commentary
The proposed system's innovative approach to NL2SQL generation, leveraging SLMs as primary agents, demonstrates a significant step forward in addressing the challenges associated with computational overhead, data privacy, and real-world deployability. However, further evaluation on the system's performance on complex queries and its limitations in handling diverse schema types is necessary to ensure its broad applicability. Additionally, the system's potential policy implications regarding data accessibility and user experience warrant careful consideration.
Recommendations
- ✓ Future research should focus on evaluating the system's performance on complex queries and its limitations in handling diverse schema types.
- ✓ The authors should provide a comprehensive evaluation of the system's performance on various schema types and query complexities to ensure its broad applicability and reliability.
