Skip to main content

View as Markdown

Page content converted to Markdown. Use the original page link at the end to explore interactive graphics.

6. DQN

DQN extends Q-learning by using a deep neural network to approximate the action-value function , allowing it to handle high-dimensional state spaces. Experience replay and a target network improve training stability and performance.

Embodied AI perspective: Approximating value functions with neural networks is a foundation of many robot-control algorithms. Understanding DQN helps prepare for DDPG, SAC, and other advanced methods.

This chapter follows the MDP reward-index convention: executing produces immediate reward and next state . A transition is consistently written as .

6.1 The Q network​

Neural networks can approximate state values and learn through gradient descent on TD errors. The same idea applies to action values . Recall the Q-learning update:

Here is the target Q value, the current estimate, and the learning rate. With a parameterized approximation:

The network parameters are optimized by gradient descent to reduce the difference between target and predicted Q values:

Because this is a TD update, check whether the next state is terminal. A terminal transition retains its immediate reward and removes only the continuation value:

6.2 Experience replay​

In basic Q-learning, each newly observed transition immediately updates the Q function:

Using this approach with neural networks presents several difficulties. Single-sample updates can be unstable, consecutive transitions are temporally correlated rather than independent samples, and discarding each transition after one update wastes data. Interaction can be expensive, so poor data reuse reduces training efficiency.

DQN addresses these problems through experience replay. Each transition is stored in a replay buffer:

For each network update, randomly sample a minibatch:

Here is a minibatch sample index, not a time index. The reward and next state belong to the same transition as ; does not mean the state of sample . If that transition came from time step , then . The minibatch contains samples. Compute their targets and update by gradient descent:

Before introducing a target network, use , or if is terminal. With a target network, replace here with . Both versions use the reward and next state from the same sample .

Instead of feeding only the newest transition directly into the model, DQN stores experience and trains on randomly selected batches from the buffer:

Direct sample-to-model updates compared with storing samples in a replay buffer before training

Experience replay (labels in Chinese)

6.3 The target network​

So far, the target and current prediction use the same parameters . Updating those parameters also changes the target, forcing the model to chase a moving value and potentially destabilizing learning.

DQN maintains a target network with the same architecture but separate parameters. It computes the target as:

Periodically copy the current network's parameters into the target network:

The hyperparameter controls the copy interval. Keeping target parameters fixed between copies makes targets more stable and helps learning converge:

The current network learns from replay samples; its parameters are copied to the target network every C steps

The target network (labels in Chinese)

This periodic copy is called a hard update. An alternative is a soft update, which gradually moves target parameters toward current parameters:

The coefficient is usually small, for example . Hard updates are standard in DQN, whereas soft updates are widely used in later methods such as DDPG and TD3.

6.4 Epsilon-greedy exploration​

DQN balances exploration and exploitation with an -greedy policy:

  • With probability , choose a random action.
  • With probability , choose an action with the highest Q value.

As training progresses, usually decreases, shifting from extensive exploration toward exploiting learned values.

6.5 Algorithm​

The full DQN procedure is:

  1. Initialize the current network , target network with , and replay buffer .
  2. Reset the environment to for each episode. Starting at , select and execute using an -greedy policy. Observe , , and the terminal flag; store the transition in .
  3. Once enough samples are available, draw a minibatch and the associated terminal flags from . Set for terminal transitions, otherwise .
  4. Update by minimizing . Every environment steps, copy .
  5. End the episode if the environment transition was terminal. Otherwise set and repeat the action selection and updates from step 2 onward.

6.6 Interactive demonstration​

In this 5×5 grid world, the agent (🤖) learns to avoid traps (🔥) and reach the goal (🚩):

  • Left: a heatmap shows how estimated state values change during training.
  • Center: the neural network's inputs, hidden layers, and action-value outputs.
  • Right: replay-buffer contents and training loss.

The shared demonstration currently uses Chinese interface labels. Select 训练 (Train) to watch the agent progress from random exploration toward a good path.

DQN 算法原理可视化

Deep Q-Network: 结合深度学习(感知)与强化学习(决策)

高价值区域
陷阱
目标
🤖
🔥
🔥
🚩
Episode
0
Total Reward
0.0
Epsilon (探索率)
50%
Step Count
0
速度:

神经网络 (Q-Network)

DQN (Deep Q-Network)
X
0
Y
0
Hidden Layers
0.00
UP
0.00
DOWN
0.00
LEFT
0.00
RIGHT
输出: 各动作的Q值 (Q-Values)

当前决策解析

等待开始... 点击训练按钮。

经验回放 (Replay Buffer)

存储最近的经验元组 (s, a, r, s')
Buffer Empty

训练损失 (Loss)

暂无 Loss 数据
Loss (MSE)

原理说明: DQN 使用神经网络(此处简化为 Q-Table 可视化)来近似 Q 函数。 它通过 Experience Replay(经验回放) 打破数据相关性,并利用 Target Network(目标网络)思想稳定训练。 观察上面的"热力图":随着训练进行,到达终点的路径颜色会变深(Q值变高),陷阱周围会变暗(Q值变低)。

6.7 Limitations and improvements​

LimitationImprovement
Overestimated Q valuesDouble DQN selects actions with the online network and evaluates them with the target network
Inefficient uniform samplingPrioritized Experience Replay samples using TD-error priorities
Discrete actions onlyDDPG and SAC extend related ideas to continuous actions
Expected return onlyDistributional DQN models the distribution of returns

Later chapters introduce further developments.

6.8 Questions​

How does DQN improve on Q-learning?

Neural networks replace the Q table and handle high-dimensional, continuous state inputs. Experience replay stores state–action–reward–next-state transitions, reduces temporal correlation in training batches, and reuses data. A separate target network stabilizes the values used for TD updates.

Why use an -greedy policy?

The agent needs to balance exploiting actions with high estimated Q values and exploring actions whose values are still uncertain. Always following current estimates can miss better strategies; always acting randomly prevents effective exploitation. An -greedy policy provides both behaviors.

Why add a target network?

When one network produces both predictions and targets, every update changes both. Holding target parameters fixed for a period reduces this feedback and stabilizes learning. The target network has the same structure as the current network, but is updated less frequently.

What does experience replay do?

It stores past transitions and randomly resamples them for training. This reduces correlations within batches, increases data reuse, and makes updates less dependent on the newest trajectory. Together with a target network, it helps stabilize neural value-function learning.