If mistakes during learning are expensive, on-policy is the safer default; if you have to learn from data you already collected, off-policy is the only thing that works. That’s the practical version of the on-policy vs off-policy question. Ananya Bhattacharyya’s walkthrough on Towards Data Science covers the theory behind it properly, from SARSA and Q-learning through Expected SARSA and Double Q-learning, and I won’t repeat all of that here. I want the parts that matter when you build for the web.
Two definitions carry the whole distinction. The target policy is the strategy you’re estimating. The behavior policy is the one generating the experience. On-policy learning means they’re the same. Off-policy learning means they differ, and that difference decides whether yesterday’s data is still usable.
On-policy vs off-policy lives in one line of each update
SARSA and Q-learning are both temporal-difference methods and they differ in exactly one place: what they bootstrap from. Here are both updates, with the differing line first in each.
# SARSA: bootstraps from the action the agent actually takes next
next_action = choose_action(Q, next_state, eps)
target = reward + gamma * Q[next_state][next_action]
# Q-learning: bootstraps from the best next action
target = reward + gamma * max(Q[next_state].values())
# both then apply the same correction
Q[state][action] += alpha * (target - Q[state][action])SARSA uses the action the agent will take next, so under an ε-greedy policy it learns the value of its own imperfect behavior. Q-learning uses the max, so every update assumes perfect play from the next step onward. In the cliff walking grid, each step costs -1, and stepping onto the cliff costs -100 and puts you back at the start. The setup comes from Sutton and Barto’s textbook, which is free online. With ε held at 0.1, SARSA learns the inland route because it prices in its own random stumbles, while Q-learning hugs the cliff because it imagines flawless execution, falls a few times during training, and then walks the shortest path once you switch exploration off. You get the better final policy at the cost of worse behavior while it learns.
How this maps to store and site work
Your order log is a replay buffer
Off-policy updates don’t assume the data came from the current policy, so every transition can be stored and resampled many times. That’s the whole reason DQN and SAC can train on millions of stored experiences, and why on-policy methods like PPO have to collect fresh data after every update. A store’s orders and analytics events are the same kind of object: behavior generated under policies you no longer run. Train a model on that history and you’re doing off-policy learning whether you planned to or not. It also means the model goes stale as the store’s behavior changes, which is model drift.
Live traffic is the cliff
The SARSA argument is a safety argument. When the agent making occasional random mistakes is a real storefront, an exploratory action costs real orders or a broken checkout instead of a -100. If learning has to happen where customers can see it, you want behavior that prices in its own errors. Usually the cheaper option is to keep exploration out of the live path entirely, learn from logs, and only promote a policy once it holds up. If you want to see what happens when that restraint is missing, the real cost of over-engineering a recommendation engine is a whole post on its own.
The max of noisy estimates runs high
Q-learning has a known flaw called maximization bias: the estimates are noisy, and the max of noisy estimates is usually an overestimate, so the agent becomes overconfident. Double Q-learning fixes it by letting one independent estimate pick the action and a second one score it. I see the same shape outside RL. Compare several variants and ship the one with the best measured number, and you’ve taken a max over noisy estimates, so the winner’s number is biased up and the lift tends to shrink after launch. Score the winner on data you didn’t use to pick it. I’m not sure the effect is worth worrying about when you have few variants and lots of traffic, but the check itself is one query.
If you’re weighing logged data against live experiments, I do this kind of work and I’m glad to look at what your setup supports before you commit either way.
On a client site this week I’d do the boring version of both. Retrain from the order log instead of the live feed, hold out the last few weeks, and re-score the winner on that held-out slice. If the lift survives, ship it. If it shrinks, the estimate was optimistic and we found out before the customers did.
