Q-learning: a standalone construct

Christopher Watkins named and defined Q-learning in his 1989 Cambridge PhD thesis Learning from delayed rewards, and Watkins and Peter Dayan (1992) gave the algorithm its first convergence proof. Q-learning is a method by which an agent, a decision maker interacting with an environment step by step, learns which action is best in each situation purely from the rewards it receives, without ever being told or having to model how the environment works. It does this by repeatedly updating an estimate called Q, the expected long-run reward of taking a given action in a given state and then acting optimally afterward, nudging each estimate toward the reward just observed plus the best estimated value of the next state. Because it is model-free, it needs no map of the world's dynamics, and because its update always looks toward the best available next action rather than the action actually taken, it is called off-policy: it can learn the optimal behaviour while following an exploratory one. What makes it a landmark is that it was the first reinforcement-learning rule shown to converge to the optimal action-value function under general conditions, and its update remains a building block of later methods, including the deep Q-networks that learned to play Atari games from raw pixels.[1][2][3]

Unit

A learned action-value function Q(state, action) over an agent's state and action space, expressed in the same units as the cumulative reward it estimates

Recovery means

Not applicable

Measured by

Operationalised as an iterative update rule applied to observed state, action, reward, and next-state transitions; performance is judged by convergence of the Q estimates and by the cumulative reward the resulting policy earns