Ibrahim Salami’s writeup of his first ETL pipeline on Towards Data Science does what a first pipeline should: it pulls the thirty most starred Python repos from the GitHub Search API, cleans them with pandas, and writes a CSV. No Airflow, no Spark, just extract, transform, load in that order. I like that he kept it small. His own next steps, scheduling it daily and swapping the CSV for SQLite, happen to be the same two steps that turn a WordPress ETL pipeline from a script into infrastructure.
Moving that notebook into a plugin changes the job more than you’d think, though. The three steps stay the same, but the constraints around them change. A PHP request has a timeout, the schedule depends on traffic unless you fix that, and the database you’re loading into belongs to a live store. I’d handle each of the three steps differently than the notebook does.
The extract changes shape in PHP
His Python calls requests.get() with a params dictionary, then prints the status code. That manual status check is a good instinct, because requests will happily call .json() on a 403 body and hand you an error document instead of data. PHP has the same problem, but the HTTP API gives you three outcomes to handle explicitly: a transport error (DNS, timeout, connection refused), a non-200 response (GitHub returns 403 when you’re rate limited), and a body that decodes to something you didn’t expect.
Two details matter in practice. The default timeout for wp_remote_get() is 5 seconds, which is short for an API having a bad day, so I raise it. And GitHub allows 60 unauthenticated requests per hour, with a separate, tighter bucket for search endpoints, so a daily job is fine while a loop over pages every hour will get you shut out for a while. The whole extract looks like this:
function bbioon_fetch_trending_repos() {
$url = add_query_arg(
array(
'q' => 'language:python created:>' . gmdate( 'Y-m-d', strtotime( '-30 days' ) ),
'sort' => 'stars',
'order' => 'desc',
'per_page' => 30,
),
'https://api.github.com/search/repositories'
);
$response = wp_remote_get( $url, array( 'timeout' => 15 ) );
if ( is_wp_error( $response ) ) {
return $response; // transport failed, retry on the next run
}
$code = wp_remote_retrieve_response_code( $response );
if ( 200 !== $code ) {
return new WP_Error( 'bbioon_github', 'GitHub returned ' . $code ); // 403 is usually the rate limit
}
$body = json_decode( wp_remote_retrieve_body( $response ), true );
return $body['items'] ?? array();
}(One annoyance worth knowing: WordPress sends its own user-agent, WordPress/x.x plus the site URL, and some APIs block that, so I usually pass a custom user-agent in the args.)
The transform loses data you may want back
One line in his transform is dropna(subset=[‘description’]), which deleted one of the thirty repos before anything was saved. For a CSV you read once, fine. Inside a site it’s a trap, because once you’ve loaded the cleaned rows and thrown away the raw response, a filter you got wrong can’t be undone. You can’t re-run and compare either, because the source has moved on: repos gain stars, and the created:> date filter means yesterday’s result set doesn’t exist anymore.
So I load raw first and transform on read. Each item becomes one row with the columns I actually query, repo id, stars, owner, name, plus the untouched JSON in a text column. The shaping, dropping empty descriptions, sorting, happens in a query or a function rather than as a destructive edit at import time. His viral column is the other thing I wouldn’t store. A Yes/No flag frozen at import is wrong twice: the threshold will change, and the stars change daily. Compute stars > 50000 where it’s displayed. At WordPress scale, thousands of rows rather than millions, reading raw and shaping on the fly costs almost nothing, and you skip needing anything like pandas.
Where a WordPress ETL pipeline should load
The notebook writes a CSV and downloads it. Inside WordPress you have four realistic options. A custom table is my default: you index what you query, you can truncate and re-import without touching content, and you leave the posts table alone. wp_options only fits a single small snapshot, and never with autoload on, because autoloaded options load on every request and that’s a common way to slow a site down. A custom post type makes sense only when the rows are editorial, when editors search them and templates render them; a price feed doesn’t need permalinks. And sometimes the CSV in uploads is genuinely right, when the consumer is a person opening a spreadsheet.
Scheduling is the other comparison. WP-Cron runs when the site gets traffic, which on a busy store is fine and on a quiet brochure site means your 3am import fires at 9:15 when the first visitor arrives. If timing matters, replace WP-Cron with a real cron entry on the server. If the job is big, hundreds of API calls or an update per product, use Action Scheduler instead: the queue lives in the database, jobs run in batches, failures are logged with the error message and visible in admin, and you can retry a single action by hand. I’ve written about splitting big background jobs before, and it’s the same idea.
If you have a feed that should land in WordPress or WooCommerce on a schedule, product syncs, distributor pricing, stock levels, that’s work I do regularly. Send me the API docs and the shape you need on the other end, and I can tell you quickly whether it’s a cron job or a queue job.
The thing I still haven’t settled is where the cutoff sits for pulling a pipeline like this out of WordPress entirely and running it as a small worker outside the site. Below some size, a plugin plus one cron entry is fewer moving parts than a deployed script with its own logging and secrets. Past it, you’re asking a PHP process that also serves shoppers to do data engineering on the side. I don’t know exactly where that line is, and it matters, because it decides whether your sync fails with the store or independently of it.
