Fourteen years of WordPress work has left me suspicious of code that merely looks right. Coding assistants are useful, and they also turn out code that passes for WordPress at a glance before it falls over on a Friday afternoon. WP-Bench is the first project I have seen that measures that gap instead of arguing about it.
Why a WordPress-specific benchmark
Most models are benchmarked on general programming. Sorting a list in Python tells you nothing about whether a model can wire up the WordPress Abilities API without opening a security hole. So far, the answer has usually been no.
WP-Bench is the official benchmark for WordPress-specific work. It scores two things separately: knowledge, through multiple choice questions, and execution, where the model has to write code that works. The questions cover the Interactivity API, current security practice, and the rest of the architecture that only exists in WordPress.
WordPress grades the answers
This is my favorite design decision in the project. Nobody reads the output and decides it looks fine. WP-Bench runs the generated code in a sandbox with static analysis and runtime assertions, and a model that breaks the site fails. A confident answer scores nothing on its own; only code that runs and follows the coding standards does.
For where this sits in the wider plan, I wrote earlier about the WordPress Core AI Evolution, which covers why benchmarks like this are appearing now.
Running it yourself
The harness takes a few minutes to stand up, whether you are testing a hosted model or something you fine-tuned locally:
# Install the benchmark harness
python3 -m venv .venv && source .venv/bin/activate
pip install -e ./python
# Fire up the WordPress runtime environment
cd runtime && npm install && npm start
# Run the benchmark against your config
cd .. && wp-bench run --config wp-bench.example.yamlYou also need a .env file with your API keys. Results land as JSON in the output/ directory, which makes it easy to line up OpenAI, Anthropic and Google against each other.
Where it falls short today
This is an early release and it shows. The dataset leans hard on newer features like the Interactivity API, and those APIs landed after the training cutoff for a lot of the models being tested. A bad score there can mean the model never saw the API, not that it cannot use one.
Then there is benchmark saturation. Models handle the older WordPress patterns well enough that those tests no longer separate one model from another. The suite needs harder material: transient logic, WP-CLI commands, WooCommerce integration that goes past the surface.
If WP-Bench work is eating your dev hours, I can take it over. I have been wrestling with WordPress since the 4.x days.
Adding your own test cases
The project lives in the official WP-Bench GitHub repository, and the test suite is still thin. If you have a pattern that reliably catches out junior devs, or AI, send it in as a test case. The dataset only reflects the work people put into it.
I plan to keep running this against whatever model I use for plugin work, if only to know where it is weak. The official announcement on Make WordPress has the full technical breakdown.
