强化学习与控制
On this page
Terms and formulas quick reference
When reading a formula, first ask what it calculates, then identify each symbol, and finally try small numbers. This page explains common concepts in RL foundations. The MDP and temporal-difference chapters also have expandable explanations beside key formulas. Follow a term link to its entry, then use your browser's Back button to resume reading.
Reading the symbols
| Symbol | Read it as | Small example |
|---|---|---|
| Current state, current action, reward received afterward | A robot reaches a junction, turns right, then receives reward | |
| Add the terms in the specified range | ||
| The discount weight after | ||
| Average the outcomes using their probabilities | Outcomes | |
| Probability of action | The bar means “given,” not division | |
| Replace the stored value on the left with the result on the right | ||
| Highest score / option with the highest score | Left scores | |
| An estimate, model parameters | The hat marks an estimate; |
The same letter can mean different things in different chapters. In an MDP,
States, observations, and MDPs
A state contains the information needed for prediction; an observation contains what sensors actually measure. A photo can show a ball's position without revealing where it is rolling. Velocity may also belong in the state.
An MDP (Markov decision process) organizes a task into states
The Markov property requires that, given the current state and action, past history provides no additional information needed to predict the next state. A POMDP describes partial observability, such as occluded information. Stacking past frames may improve observations but does not guarantee recovery of the full state.
Return to the chapter: MDP definition, Markov property.
Rewards, returns, and discounting
A reward is one step's feedback; a return is the total of future rewards from the current time, possibly discounted. A robot might earn
The discount factor
The next-time return is
Return to the chapter: returns and their recursive form.
Policies, expectations, and values
A policy
An expectation is a probability-weighted average, not a hoped-for outcome or a result guaranteed on one run.
- State value
: starting here and following , what return do we get on average? - Action value
: starting here, taking first, and then following , what return do we get on average?
If left and right have
Return to the chapter: policies and values.
Bellman equation
The Bellman equation splits the average future total into “one immediate step + the average remaining future”:
For a fixed policy, average over both possible actions and possible next states using their probabilities. The equation relates expectations; a particular trajectory that falls into a pit may still return much less than average.
Return to the chapter: Bellman equations, from Bellman to the TD target.
Bootstrapping, TD targets, and learning rate
In this context, bootstrapping means updating one estimate using another. Before finishing a journey, use your estimate for the next stop to improve your estimate for the current stop. This is different from bootstrap resampling in statistics.
The TD target is the observed reward plus the discounted next-state estimate. The TD error is that target minus the current estimate. The learning rate
With an old estimate of
Return to the chapter: calculating a TD update.
On-policy and off-policy
The behavior policy collects data. The target policy is evaluated or improved. When they agree, learning is on-policy; when they differ, it is off-policy.
Sarsa uses the value of the action actually selected next. Q-learning uses the highest estimated next-action value, even if exploration selected a different action. The distinction concerns whose behavior is learned, not whether a neural network is used.
Return to the chapter: Q-learning and Sarsa, on-policy and off-policy.
Actor, Critic, and advantage
The Actor selects actions; the Critic estimates value. Think of a decision-making part and a scoring part, but remember that the Critic's scores are learned estimates, not ground truth.
The advantage function asks how much better an action is than the current policy's average at that state:
Continue reading: Actor-Critic, PPO.
PPO probability ratios and clipping
The probability ratio
Clipping limits incentives in the surrogate objective. With advantage
For negative advantage, the incentive being limited is the gain from excessively reducing the action's probability. Do not omit the
Continue reading: PPO's Clip objective.
Further references
- Sutton & Barto: Reinforcement Learning, An Introduction: Chapters 3, 4, and 6 cover MDPs, Bellman equations, and TD learning.
- OpenAI Spinning Up: Key Concepts in RL: formal definitions of states, policies, returns, values, and advantages.
- OpenAI Spinning Up: PPO: probability ratios, the clipped objective, and its limitations.