Missing information is the part nobody plans for. In 14 years of wrestling with WordPress and high-traffic WooCommerce sites, I have watched plenty of developers stall because they had no real data to test against. A grad student trying to pass Calc 1 and a senior engineer refactoring a legacy checkout flow are stuck on the same thing, and in both cases the way out is synthetic training data.
The standard advice has settled on “just wing it” or “test on staging with 10 rows of dummy data.” That is how production incidents happen. When the real training data does not exist, like the non-existent past exams for a niche grad course, you have to manufacture your own. Doing that well is what makes the system hold up later.
The human training data problem
Researchers have a name for this: human underfitting. It is the feeling you get when you have read every page of the documentation, or the lecture notes, and still cannot solve the actual problem on the exam. The mental model does not have enough training data behind it. On a dev team it shows up as a junior who knows the syntax of WP_Query cold but cannot architect a faceted search that performs.
This is where LLMs earn their keep. Feed a model like Claude or Gemini your notes, your Q&A sessions and the constraints you are actually working under, and it will produce mock exams or test cases with something close to real-world complexity. Jonathan Yahav wrote about this at length in his piece on solving the human training data problem.
Generating synthetic training data for WordPress testing
In WordPress work, synthetic data exists so you can hit edge cases without touching real customer records. Build a custom reporting engine for WooCommerce and you need thousands of orders that differ from each other: different payment gateways, shipping zones and tax rates, because that is where the race conditions hide.
Here is the naive version I keep running into, usually a loop around a single product. It is far too uniform to surface a real bug.
// The Naive Approach - Don't do this
for ($i = 0; $i < 100; $i++) {
$order = wc_create_order();
$order->add_product( get_product(123) );
$order->calculate_totals();
$order->save();
}The better move is to have an LLM write a JSON schema of varied scenarios, then ingest that to build the orders. Your local environment ends up resembling the messy version of production rather than the tidy one. Here is how I rework that loop:
<?php
/**
* Generate varied synthetic WooCommerce orders for testing.
* Prefixing with bbioon_ to keep the namespace clean.
*/
function bbioon_generate_synthetic_orders( $scenarios ) {
foreach ( $scenarios as $scenario ) {
$order = wc_create_order();
// Add random products based on the LLM-generated scenario
foreach ( $scenario['products'] as $item ) {
$order->add_product( wc_get_product( $item['id'] ), $item['qty'] );
}
$order->set_billing_country( $scenario['country'] );
$order->set_payment_method( $scenario['gateway'] );
// Force specific statuses to test reporting hooks
$order->set_status( $scenario['status'] );
$order->calculate_totals();
$order->save();
error_log( "Synthetic Order Created: " . $order->get_id() );
}
}Context rot and bias
You still cannot take the output on trust. Two things go wrong with synthetic data. The first is context rot, which Anthropic defines as a model’s declining ability to recall information as the context window fills up. Keep the same chat open for weeks and the quality of what it generates quietly drops, so I start a fresh session for each testing phase.
The second is bias, and it is harder to spot. Prompt only for standard orders and you will never see what your plugin does with a botched POST request from a failing payment gateway. Your mental model overfits to a perfect world that does not exist. I got into nearby territory in my post on Technical Debt in AI Development.
If this kind of test-data work is eating your dev hours, hand it to me. I have been wrestling with WordPress since the 4.x days.
What I take from it
LLMs are what you make of them. They can be a shortcut that leaves you worse off, or a coach that takes on the heavy lifting while you learn. Used deliberately, with separate chats, an open mind and real-world snippets mixed into the synthetic set, they get you on top of a complex system far faster than anyone could a decade ago.
If the overlap between AI and development interests you, I have also written about AI in the real world, and the IBM Research write-up on synthetic data is worth the time.
