WordPress 6.x and the Core AI team have shipped something that was missing for a long time: the WP-Bench AI Benchmark. General-purpose models like GPT-4o or Claude 3.5 Sonnet handle Python or plain JS well enough, but they still treat WordPress like a legacy hobbyist project instead of the application framework it has become.
Anyone who has asked an AI assistant to refactor a WooCommerce checkout or write a custom REST API endpoint knows the pattern. It reaches for deprecated hooks, forgets wp_unslash(), or invents a Transient API function that never existed. That is a large part of why technical debt in AI development keeps piling up, and why we need a way to measure whether a model really knows WordPress.
What the WP-Bench AI benchmark actually is
WP-Bench is a two-part evaluation framework rather than a set of multiple-choice questions. It measures two things:
- Knowledge: whether the model understands WordPress concepts, security patterns, and newer additions like the Interactivity API.
- Execution: the benchmark hands the model code generation tasks, then runs the result in a real WordPress runtime with Docker and
wp-envto see whether it works.
WordPress itself is the grader, through static analysis plus runtime assertions. A function that triggers a PHP notice or skips a capability check fails the test, so the score comes from what the code does rather than from how confident the answer sounds.
Why a WordPress-specific benchmark was needed
A model can score 90% on HumanEval, which is Python, and still fail to register a block in the Gutenberg editor because it does not understand the block.json schema. When that happens, debugging the output costs more time than writing the thing by hand would have.
A standardized WP-Bench AI Benchmark puts real pressure on the AI labs. The goal is for providers like OpenAI and Anthropic to run these tests during their pre-release cycles, so WordPress performance gets checked alongside everything else they measure.
Setting up the WP-Bench AI benchmark
If you are comfortable with Python and the command line, the setup takes a few minutes. You need a local WordPress runtime to act as the grader. The quick start sequence:
# Create a virtual environment and install the harness
python3 -m venv .venv && source .venv/bin/activate
pip install -e ./python
# Boot up the WordPress runtime (requires Docker)
cd runtime && npm install && npm start
# Execute the benchmark against your chosen model
cd .. && wp-bench run --config wp-bench.example.yamlThe current dataset leans toward newer features like the Abilities API. That is deliberate, since the newer APIs are where models struggle most, but it does run straight into the training cutoff problem. A model trained six months ago has never seen the 2025 Core updates.
Where the benchmark still falls short
In early testing, models scored well on older WordPress patterns, the add_action('init', ...) sort of thing, then fell apart on transient race conditions or on the newer wp_interactivity_state(). So the job for the community now is writing harder test cases. Things like:
- Complex SQL queries that use
$wpdb->prepareproperly. - Sanitizing data with the function that matches the context.
- Using the object cache well enough to keep repeat queries off the database.
If the WP-Bench AI Benchmark work is eating your dev hours, or you are tired of cleaning up AI-generated spaghetti code, I can take it on. I have been wrestling with WordPress since the 4.x days and I know where the models fail and where the real logic has to sit.
How to contribute to WP-Bench
The WP-Bench GitHub repository is open. If you have a WordPress pattern that reliably trips up your coding assistant, turn it into a test case. The benchmark is only as good as the collective gotchas people feed it. You can also help improve the grading logic or submit results from new models to the public leaderboard.
The loop should feed itself: tougher tests, better models. The gap left to close is between an assistant that can write a hello world plugin and one that can refactor an enterprise WooCommerce site.
