Skip to main content

View as Markdown

Page content converted to Markdown. Use the original page link at the end to explore interactive graphics.

Terms and formulas quick reference

When reading a formula, first ask what it calculates, then identify each symbol, and finally try small numbers. This page explains common concepts in RL foundations. The MDP and temporal-difference chapters also have expandable explanations beside key formulas. Follow a term link to its entry, then use your browser's Back button to resume reading.

Reading the symbols​

SymbolRead it asSmall example
Current state, current action, reward received afterwardA robot reaches a junction, turns right, then receives reward
Add the terms in the specified range
The discount weight after steps
Average the outcomes using their probabilitiesOutcomes with equal probability have expectation
Probability of action , given state The bar means “given,” not division
Replace the stored value on the left with the result on the right increases by
/ Highest score / option with the highest scoreLeft scores , right scores : maximum , maximizing option right
An estimate, model parametersThe hat marks an estimate; can be policy-network weights

The same letter can mean different things in different chapters. In an MDP, is a reward; in PPO, is a new-to-old policy probability ratio. Read the definition next to the current formula first.

States, observations, and MDPs​

A state contains the information needed for prediction; an observation contains what sensors actually measure. A photo can show a ball's position without revealing where it is rolling. Velocity may also belong in the state.

An MDP (Markov decision process) organizes a task into states , actions , transition rules , reward rules , and discount factor . It is a problem model, not a particular neural network: what information is available, what can be done, what happens next, and what counts as good?

The Markov property requires that, given the current state and action, past history provides no additional information needed to predict the next state. A POMDP describes partial observability, such as occluded information. Stacking past frames may improve observations but does not guarantee recovery of the full state.

Return to the chapter: MDP definition, Markov property.

Rewards, returns, and discounting​

A reward is one step's feedback; a return is the total of future rewards from the current time, possibly discounted. A robot might earn per forward step and at its destination. One step's reward and the whole journey's return are different quantities.

The discount factor determines how much future rewards count now. Suppose the next two rewards are , followed by termination, and :

The next-time return is , so we can also write . No rewards follow termination, and we set . Infinite-horizon tasks commonly use with bounded rewards to ensure a finite return; finite episodes can also use .

Return to the chapter: returns and their recursive form.

Policies, expectations, and values​

A policy is an action-selection rule. For discrete actions, means a chance of selecting in state . Continuous actions are usually represented by probability densities: the density at a point is not the probability of selecting that exact action.

An expectation is a probability-weighted average, not a hoped-for outcome or a result guaranteed on one run.

  • State value : starting here and following , what return do we get on average?
  • Action value : starting here, taking first, and then following , what return do we get on average?

If left and right have values and the policy selects each with probability one half, then . This averages according to the policy; it does not take the maximum .

Return to the chapter: policies and values.

Bellman equation​

The Bellman equation splits the average future total into “one immediate step + the average remaining future”:

For a fixed policy, average over both possible actions and possible next states using their probabilities. The equation relates expectations; a particular trajectory that falls into a pit may still return much less than average.

Return to the chapter: Bellman equations, from Bellman to the TD target.

Bootstrapping, TD targets, and learning rate​

In this context, bootstrapping means updating one estimate using another. Before finishing a journey, use your estimate for the next stop to improve your estimate for the current stop. This is different from bootstrap resampling in statistics.

The TD target is the observed reward plus the discounted next-state estimate. The TD error is that target minus the current estimate. The learning rate determines how far to move toward it:

With an old estimate of , a target of , and , the new estimate is . Learning rate and discount factor differ: controls updating, while weights future rewards. Do not bootstrap after true termination; whether a time-limit truncation permits bootstrapping depends on the task's termination definition.

Return to the chapter: calculating a TD update.

On-policy and off-policy​

The behavior policy collects data. The target policy is evaluated or improved. When they agree, learning is on-policy; when they differ, it is off-policy.

Sarsa uses the value of the action actually selected next. Q-learning uses the highest estimated next-action value, even if exploration selected a different action. The distinction concerns whose behavior is learned, not whether a neural network is used.

Return to the chapter: Q-learning and Sarsa, on-policy and off-policy.

Actor, Critic, and advantage​

The Actor selects actions; the Critic estimates value. Think of a decision-making part and a scoring part, but remember that the Critic's scores are learned estimates, not ground truth.

The advantage function asks how much better an action is than the current policy's average at that state: . An action value of and state value of give advantage ; a negative value means below the baseline. PPO's is an advantage estimate, not an immediate reward.

Continue reading: Actor-Critic, PPO.

PPO probability ratios and clipping​

The probability ratio compares the new and old policies for the same action. If a discrete action's probability rises from to , the ratio is ; for continuous actions, it compares densities.

Clipping limits incentives in the surrogate objective. With advantage , , and ratio , this sample's PPO-Clip objective is . Increasing this ratio further does not increase that sample's contribution to the objective.

For negative advantage, the incentive being limited is the gain from excessively reducing the action's probability. Do not omit the or interpret clipping as locking all probabilities into a fixed interval. Clipping does not guarantee that the new and old policies remain close.

Continue reading: PPO's Clip objective.

Further references​