Bellman equation: a standalone construct

Richard Bellman introduced the equation in his 1957 book Dynamic Programming, building on the principle of optimality he had set out in 1954, which states that an optimal plan has the property that whatever the first decision is, the remaining decisions must themselves form an optimal plan from the situation that results. The Bellman equation turns that principle into a recursive relation: the value of being in a given state equals the best immediate reward obtainable there plus the discounted value of the state that follows, so a problem spread over many steps is expressed in terms of itself one step ahead. Martin Puterman's 1994 treatment placed the equation at the center of Markov decision processes, the standard model of sequential decision-making under uncertainty, and Richard Sutton and Andrew Barto's Reinforcement learning made it the foundation of methods by which an agent learns good behavior from trial and error. What makes the equation a landmark is that it converts a search over whole sequences of actions into a fixed-point condition on a value function, one equation per state, which is what lets dynamic programming, optimal control, and reinforcement learning compute optimal policies rather than enumerate every path.[1][2][3][4]

Unit

A fixed-point equation on a value function, one relation per state of a decision process

Recovery means

Not applicable

Measured by

Written as the value of a state equal to the best immediate reward plus a discount factor times the expected value of the next state; solved by value iteration or policy iteration, or approximated by temporal-difference learning