WordPress performance work usually comes down to database indexing and object caching. Deep Q-learning in a multi-player environment, say a game of Connect Four, puts “state management” in a different category entirely. Instead of a handful of transients, you have a state space that grows combinatorically.
Plenty of developers throw compute at reinforcement learning (RL) problems without looking at the stability issues underneath. It is the same instinct as fixing a slow WooCommerce checkout by renting a bigger server when the real problem is a race condition in the cart. Get the move from tabular methods to function approximation wrong and your agent spins its wheels no matter how much hardware you feed it.
Why tabular methods fail at scale
An earlier post in this series covered tabular approaches. They hold up in a simple environment like GridWorld. Connect Four has too many possible board configurations for a literal lookup table to survive. That is where Deep Q-learning (DQN) takes over: a neural network approximates the action-value function instead of a table storing it.
The catch is that neural networks go unstable when the training data is highly correlated. Update the model after every single move (on-policy) and it oscillates: it forgets old lessons about as fast as it picks up new ones. Two pieces fix most of that, a replay buffer and batched updates.
I wrote about how some of this maps back onto the usual WordPress stack in 3 machine learning lessons for WordPress development.
Replay buffers and batches
A serious Deep Q-learning setup does not learn from experience as it arrives. Transitions (state, action, reward, next state) go into a buffer, and training pulls a random batch back out. That breaks the correlation between consecutive steps and smooths the learning curve. Batching also suits the hardware, since a GPU is built for batch work and single updates leave it idle.
Handling illegal moves in Connect Four
A recurring gotcha in RL implementations is how the agent handles illegal moves, like dropping a disc into a column that is already full. The obvious approach is a negative reward. Masking the action during training is cheaper: set the Q-value of every illegal move to negative infinity and the agent never picks one in the first place.
# Constructing the target and masking illegal actions in PyTorch
q_next = self.q(batch.next_states, ...)
# Apply a mask to ensure only valid moves are considered
q_next_masked = q_next.masked_fill(~legal_moves_mask, float("-inf"))
max_next = q_next_masked.max(dim=1).values
# The core bootstrapped target calculation
target = batch.rewards + gamma * (~batch.dones).float() * max_next
loss = F.smooth_l1_loss(q_sa, target)Throughput and the Python parallelism myth
Training speed comes down to how many games per second you can simulate, which is why vectorized environments run several games at once. Working with the PettingZoo environment, we were aiming for 50 to 100 games per second.
Then the Global Interpreter Lock (GIL) showed up. Multi-threading does nothing for CPU-bound environment stepping, and multi-processing works but often hands the gains straight back in inter-process communication overhead. It plays out the same way as high-frequency work in WordPress: sooner or later the heavy lifting moves to a dedicated background service or an external API.
I covered more of the failure modes in machine learning pitfalls that hide behind high accuracy.
If this kind of work is eating your dev hours, I can take it on. I have been wrestling with WordPress since the 4.x days.
Refining the offensive and defensive policy
The Deep Q-learning agent picked up offense fast and went hunting for four in a row. Defense was another matter. It kept missing simple blocks because the function approximation limited how far ahead it could see. That plateau is normal in competitive RL: as the agent improves, so does the opponent pool, even when the opponents are just older copies of itself, so the targets never sit still.
Getting past that needs specialization. Later posts in this series go beyond the general framework in Sutton’s RL book and into methods tuned for this specific problem. If you want the code side, the official PyTorch DQN tutorial pairs well with this.
