What the studies show
In a vendor's 2026 comparison, questions answered through defined metrics were right more often than queries written straight against tables: 98.2% and 100.0% against 90.0% and 84.1% on modelled data, and 72.7% against 64.5% on the original schema, where it could not answer questions needing too many joins. A separate preprint found that a short file of written definitions lifted accuracy by 17 to 23 points. Enterprise-scale benchmarks from 2024 were much harsher, with a code agent built on a 2024 model solving about a fifth of tasks.
Read them as a direction, not a rate
The dbt post is by a vendor of the semantic layer and used one dataset, and the preprint used one retail dataset. Newer tools may do better. None of it says what a tool will score on your data.
What to put in place first
Write each metric once, get it signed, and reconcile it to a control total. Write 30 questions with answers you know, and keep 10 back from whoever builds the tool. Make the tool refuse questions outside what is defined.
- A signed definition per metric.
- Control totals per metric.
- A held-back question set.
- Refusals for anything outside the definitions.
Controls for a question tool
Use a read-only role on a copy with personal columns masked, approved views, row and time limits, and a log of every question and query. Show the query and the row count with every answer. A tool whose answers cannot be traced is hard to trust.
Sources and limits
- dbt: semantic layer vs text-to-SQL, 2026 Checked 2026-10-12.
- The benchmark used a 15-table insurance dataset with 11 questions each run 20 times across several models.
- On modelled data text-to-SQL scored 90.0% and 84.1% for two models against 98.2% and 100.0% for the semantic layer.
- On the original schema both models scored 64.5% for text-to-SQL against 72.7% for the semantic layer.
- The post says the semantic layer can only answer questions within what has been modelled, and that when it cannot answer it says so while text-to-SQL will cheerfully give you a wrong number.
- arXiv 2604.25149: Semantic Layers for Reliable LLM-Powered Data Analytics Checked 2026-10-12.
- 100 questions on one retail dataset, three frontier models: a roughly 4 KB hand-written markdown file of definitions and rules improved accuracy by 17 to 23 percentage points, from 45.5-50.5% to 67.7-68.7%.
- arXiv 2411.07763: Spider 2.0 Checked 2026-10-12.
- Spider 2.0, a benchmark of 632 enterprise text-to-SQL problems, was solved at only 21.3% by a code agent built on o1-preview, against 91.2% on Spider 1.0; this was a late-2024 result.