Most of the trouble I see with Reinforcement Learning Agents starts with developers treating AI as a black box they can drop into a project. Debug one reward function that is spiraling out of control and that illusion goes away fast. It is state logic, not magic. Fourteen years on complex systems has convinced me the discipline is the same one I use on a heavy WooCommerce checkout flow: keep the state honest and know what the next move will be.
Reinforcement learning from a systems point of view
Reinforcement learning, meaning learning from observations and rewards, is the closest machine learning gets to the way people actually learn. It is also the most complicated and vexing corner of the field. Andrej Karpathy put it best: “Reinforcement Learning is terrible. It just so happens that everything we had before was much worse.”
The loop itself is short. The agent picks an action, the environment reacts, and the agent reads the new state along with whatever reward or punishment came with it. That stays simple until you notice you are building a very large state machine that has to converge without eating all your memory and CPU. For where this sits in a wider corporate stack, there is a separate guide on AI implementation strategy.
The Bellman equation, one pass at a time
Finding an optimal policy is an iterative job, and the Bellman Equation is how you get there. The long-term reward of an action is the immediate reward plus the expected reward from every action that follows it. In practice you are propagating values outward from the goal tile, with a discounting factor (gamma) shrinking the payoff the further out you go.
// The Bellman equation implementation in C#
private double GetNewValue(VTile tile)
{
return Agent.Actions
.Select(a => tileGrid.GetTargetTile(tile, a))
.Select(t => t.Reward + gamma * t.Value)
.Max();
}
private void CalculateValues()
{
for (var y = 0; y < TileGrid.BOARD_HEIGHT; y++)
{
for (var x = 0; x < TileGrid.BOARD_WIDTH; x++)
{
var tile = tileGrid.GetTileByCoords<VTile>(x, y);
if (tile.TileType == TileEnum.Grass)
{
tile.NextValue = GetNewValue(tile);
}
}
}
}Q-learning: action quality instead of state values
State values are useful for pathfinding, but Reinforcement Learning Agents usually care about action quality instead, the Q-values. Q-learning assigns a quality value to every (state, action) pair, which lets the agent keep working when its observations are incomplete or the environment is moving under it. That describes most game worlds, and plenty of web apps too.
A learning rate (alpha) controls how fast new information overwrites the old. The step logic for a Q-learning agent in Unity looks like this:
private void Step()
{
if (_agent.State.TileType != TileEnum.Grass)
{
ResetAgentPos();
}
else
{
QTile s = _agent.State;
foreach (var a in Agent.Actions)
{
double q = s.GetQValue(a);
QTile sPrime = tileGrid.GetTargetTile(s, a);
double r = sPrime.Reward;
double qMax = Agent.Actions.Select(sPrime.GetQValue).Max();
double td = r + gamma * qMax - q;
s.SetQValue(a, q + alpha * td);
}
ActionEnum chosen = PickAction(s);
_agent.State = tileGrid.GetTargetTile(s, chosen);
}
}Epsilon: exploration against exploitation
The usual failure mode is an agent stuck in a local optimum. It finds a “good enough” path and quits looking for a better one. An ε-greedy policy fixes that: pick a random action now and then to explore, take the best known action the rest of the time, and decay epsilon as training goes on so the agent settles down.
If this reinforcement learning agents work is eating your dev hours, hand it over to me. I have been wrestling with complex logic since the WordPress 4.x days and I know how to get these systems to converge.
From Q-tables to neural networks
The example above stores everything in a Q-table, which is fine for a 40-state grid. Chess, or an ecommerce recommendation engine, has a state space far too large to keep in a table. That is where Unity ML-Agents comes in, swapping the table for a deep neural network that generalizes to states it has never seen. The vocabulary does not change whether you are training a car in a simulator or an intelligent backend: state, action, reward.
