The actor-critic method and the bugs that stall it
REINFORCE was too slow for a warehouse path-finding bot, so I switched to the actor-critic method and hit three problems on the way: an undetached TD target, snapshot rewards the agent gamed, and a gamma that erased the payoff.