For years, the advice for scaling WooCommerce or enterprise WordPress was to hand your data to the “Big Data” giants. That was the Modern Data Stack pitch: pipe your SQL rows into Snowflake, run transformations in dbt, then pay a BI tool to draw the charts. Lately I keep hitting the thing I have suspected for a decade, which is that we over-engineer ourselves into a corner.
The Modern Data Stack is hitting a ceiling. That shows up as a valuation problem for Databricks and Snowflake, but for developers it is a practical one. We build fragile pipelines to shuttle data that a well-indexed table and a custom WP-CLI command would have handled. Hugo Lu calls this “The Great Data Closure”: incumbents now have enough gravity to make it harder, and more expensive, to actually use your own information.
Where the modern data stack runs out of road
Lu’s argument is that Databricks and its peers are branching out because the core business is saturating. Storing and processing raw data has become a commodity, so they push into AI and applications instead. For a WordPress developer, that closure shows up as the “Salesforce Tax”: rising fees just to connect your own data sources to an outside platform.
I have had clients spend $2,000 a month on Fivetran connectors to sync WooCommerce order data into a warehouse, then admit the insights were not worth the overhead. Raw metrics without context do not tell you much. I have argued before that human-centered data analytics beats raw metrics, and that goes double in e-commerce.
Why simpler architecture is winning
The move now is toward what people call “Lean Data.” Rather than carry the whole Modern Data Stack, a lot of high-traffic stores process locally instead. DuckDB will run analytical queries straight against CSV or Parquet files with no server cluster behind it. So proprietary warehousing keeps shrinking as a requirement for everyone except the largest enterprises.
A refactor story: bypassing the middleware
Last year I worked with a store doing 50k orders a month. Their data sync had a race condition: orders hit the warehouse before the tax metadata was finalized. A better sync tool would not have fixed that. What did was a simplified SQL view inside WordPress that formatted the data for the CFO’s reporting tool, generated on-site by a custom WP-CLI command. That saved them thousands in SaaS fees.
A rough version of that looks like this, pulling aggregated numbers straight out of the database instead of routing basic reporting through a pipeline:
<?php
/**
* Generate a lean revenue report without external warehouses.
* Prefixing with bbioon_ for safety.
*/
function bbioon_get_monthly_revenue_report() {
global $wpdb;
$results = $wpdb->get_results( "
SELECT
DATE_FORMAT(post_date, '%Y-%m') as order_month,
SUM(meta_value) as total_revenue
FROM {$wpdb->posts} p
JOIN {$wpdb->postmeta} pm ON p.ID = pm.post_id
WHERE p.post_type = 'shop_order'
AND p.post_status = 'wc-completed'
AND pm.meta_key = '_order_total'
GROUP BY order_month
ORDER BY order_month DESC
LIMIT 12
" );
return $results;
}
?>It is not as pretty as a Databricks dashboard. It also does not need a $50k annual contract, and the data never leaves your own database. Permissions get harder once more people need access, and you may need to master robust specialized data roles for that, but the query logic does not change.
Brace for the Great Data Closure
As the big platforms raise prices and lock down their APIs, the open web starts to look like an advantage rather than a compromise. WordPress keeps your data in tables you control. The Modern Data Stack maximalists still talk about a borderless future, but the ceiling the incumbents built is real. The teams that do well here will be the ones getting more out of what they already run.
If this Modern Data Stack work is eating your dev hours, I can take it off your plate. I have been wrestling with WordPress since the 4.x days.
What to do instead
A tool will not fix a logic problem. A messy data architecture moved into Snowflake is just messy data that costs more. Index properly, keep the SQL lean, and process locally where you can. Your P&L and your server response times both benefit.
