A 90% success rate gets treated as a win for AI-driven analytics. Anyone who has spent a weekend repairing a database after a hallucinated JOIN or a race condition in a batch update knows what the other 10% costs. Text-to-SQL accuracy in enterprise data is binary, not a sliding scale.
Fourteen years of WordPress and WooCommerce databases have shown me plenty of broken wp_options tables, and the thing that never fully recovers is trust. Hand a business owner an AI that chats with their data, watch it misread a filter or invent a table that does not exist, and the damage does not stop at that one query. The client stops trusting the entire stack.
The myth of “good enough” text-to-SQL accuracy
Spider 1.0 gave the field a false sense of security for years. Its databases were tiny, clean SQLite files, LLMs were scoring above 90% on them, and the problem looked solved. Enterprise data is not a clean SQLite file with ten tables. It is a BigQuery instance with 3,000 columns where a definition like “churn rate” lives in documentation rather than in the schema.
Older benchmarks also scored with “Exact Match” (EM). Write SELECT * FROM users when the gold query was SELECT id, name FROM users and the model was marked wrong, which tells you nothing useful. Execution Accuracy (EX) is the metric worth watching: run the generated query and the gold query against the live database and compare the result sets. If they differ, Text-to-SQL accuracy for that task is zero. Production has no partial credit.
If you want to see how AI performance gets measured inside the WordPress ecosystem, I went through it in my guide on WP-Bench AI Benchmarks.
Why naive LLMs fail at enterprise SQL
Take a request like “Find the total revenue from customers in Berlin.” A naive LLM writes standard MySQL and misses that the warehouse is Snowflake or BigQuery, with dialect quirks such as UNNEST or FLATTEN. In a WordPress context it skips post_status altogether and counts draft orders as revenue. Here is the naive version next to a grounded one:
// Naive approach: Letting the LLM guess the schema
// Result: Likely fails because it doesn't know about custom meta keys
$sql = "SELECT SUM(meta_value) FROM wp_postmeta WHERE meta_key = 'total_sales'";
// Senior approach: Grounding the LLM with a schema map and dialect-specific instructions
// We use a filter to inject context before the LLM generates the query
add_filter( 'bbioon_llm_schema_context', function( $context ) {
$context['wp_postmeta'] = 'Contains order totals. Use CAST(meta_value AS DECIMAL) for sums.';
return $context;
});Spider 2.0 and what it exposed
The Spider 2.0 benchmark landed recently and current LLMs did badly on it. Success rates fell from 90% to somewhere between 10% and 20%, because it tests the things that actually break production:
- Schema linking that has to find 10 relevant columns among 3,000.
- Dialect diversity, since the same question has to work in T-SQL, Snowflake and BigQuery.
- External knowledge, where the logic sits in a Markdown or YAML file rather than in the database.
All of that is a grounding problem, which is why RAG pipelines need a refactor before they go anywhere near production. Throwing a prompt at an API and hoping is not a pipeline.
BigQuery and native AI integration
I have been looking at how BigQuery integrates Gemini natively, and the appeal is that you stop hopping between platforms. Generating embeddings, storing them in a vector database and running semantic searches all happen through the same SQL interface. The catch is lock-in and cost. One unoptimized query will drain your credits faster than a memory leak in a badly written loop.
Transients take some of that pressure off. Cache the analytical results people ask for repeatedly, or cache the gold SQL for the common questions, so the LLM is not regenerating the same logic, and getting a fresh chance to hallucinate it, on every request.
If this Text-to-SQL accuracy work is eating your dev hours, I can take it on. I have been wrestling with WordPress since the 4.x days.
Stop betting on 90%
High Text-to-SQL accuracy in a lab setting is a vanity metric. In an enterprise the bar is binary: the query returns the right rows or it does not. If you are building a data-driven application, you need evaluation that reflects that, which means Execution Accuracy and Soft-F1 rather than a string comparison against a gold query.
