Ibrahim Salami published a 12-month self-study plan on Towards Data Science for going from data analyst to data engineer: performance SQL, Python built for pipelines instead of notebooks, Git, Spark, Airflow, Databricks, three to four hours a day. My first reaction was that none of this touches my week. That lasted about a minute, because WooCommerce data pipelines already run on the stores I look after, under other names.
What the roadmap covers
His argument for the move is simple: analysts work with data after it arrives, engineers build the systems that move, store and refresh it, and he wants to be upstream instead of downstream. He is open about the money and about AI, which now handles a lot of the analytical layer while the plumbing underneath still needs people. His line is that he’d rather build the pipes than use them, and it’s a fair summary of the post.
The stack order is the useful part. SQL first, but engineering SQL: query optimization, indexing, large datasets rather than pulls and filters. Then production Python. Then Git, which he admits is currently copy the command and hope it works. Then Spark for distributed processing, Airflow for orchestration, and one platform, Databricks, picked over Snowflake and BigQuery because depth in one beats surface knowledge of three. He also expects 12 months to stretch to 15 and says so on purpose, because he’d rather come out solid than fast.
I liked that he said it out loud.
For store owners the framing matters at hiring time. Someone who only reads finished tables builds you a dashboard. Someone who thinks about how data moves asks why the nightly export takes an hour, and those are different conversations.
WooCommerce data pipelines already run on your store
The store you maintain has a job queue. Action Scheduler ships inside WooCommerce and runs subscription renewals, webhook deliveries, emails and stock imports in the background. The WooCommerce docs describe batches of 20 actions with up to five queues processing at once, and the project’s own site talks about millions of payments a month. The tooling is smaller than Airflow, but the work is the same kind. If you have ever debugged a stuck subscription renewal, you have already debugged a pipeline by hand.
Storage has been through the migration he’ll study in month six. Action Scheduler 3.0 moved its queue out of a post type into dedicated tables, and High-Performance Order Storage has been the default for new stores since WooCommerce 8.2, which puts orders into wc_orders and its siblings with their own indexes instead of rows of postmeta. It’s the same instinct he assigns to data engineering: pick storage based on how the data gets queried. The WordPress-specific version of that argument is choosing a custom table instead of post meta, and the trade is the same.
Who notices when a job fails
Here the comparison breaks. Airflow’s value is mostly that retries, logging and alerting are defaults rather than projects. On a WordPress site, WP-Cron kicks the queue off, and WP-Cron depends on site traffic. A quiet store overnight means jobs drift late, and a failed action doesn’t page anyone. It sits in the failed queue until a renewal doesn’t happen and a ticket arrives. Some hosts run cron properly already and some don’t, and the store owner usually has no way to know which, so checking costs a minute.
The fix on a normal store is two settings and a habit.
# wp-config.php
define( 'DISABLE_WP_CRON', true );
# crontab, every five minutes
*/5 * * * * cd /var/www/example.com && wp cron event run --due-now >/dev/null 2>&1The habit is reading the queue. WooCommerce, Status, Scheduled Actions has Failed and Pending filters worth checking on any maintenance pass, and WP Crontrol shows the core cron events, so you can spot the plugin that quietly registered five of them. I don’t deal with multisite much, so treat all of this as single-site advice.
What I’d steal from his ordering
SQL first, aimed at performance, is the right call and the part we skip most. On our side it means knowing what’s indexed on wc_orders before writing a report query, and knowing when wc_get_orders is the correct call and when an admin report over a few hundred thousand orders needs to read the tables directly. Store analytics is usually written last, by whoever had time that month, and that’s why his point about engineering SQL lands here too.
The other thing worth stealing is the notebook to production step. Moving from a throwaway script that fixes 400 products to something scheduled, logged and reversible is the same jump as Pandas to pipeline code. A WordPress ETL pipeline can start in a notebook, but it shouldn’t end there.
One disagreement: Git at month three is too late. If you bill clients, branching and pull requests come before Spark, because the cost of skipping them lands in someone’s production store.
What I’d do on a client store this week
None of this needs a 12-month plan. A weekly pass over three screens covers most of it:
- WooCommerce, Status, Scheduled Actions: check the Failed and Pending queues. If something has been sitting in Failed for more than a day, find out why before you delete it.
- wp-config and the crontab: confirm cron is real and not traffic-driven. On low-traffic stores this is the one change that matters most.
- WooCommerce, Status, Tools: run the HPOS data sync check so the two order storages agree before a report or export depends on them.
That’s a morning on one store. I’d start with the failed actions queue, since it’s the only one of the three that tells you something already went wrong. If you’d rather hand it over, I regularly set up real cron, clear stuck queues and add alerts for failed WooCommerce jobs.
