强化学习与控制
On this page
6. DQN
DQN extends Q-learning by using a deep neural network to approximate the action-value function
Embodied AI perspective: Approximating value functions with neural networks is a foundation of many robot-control algorithms. Understanding DQN helps prepare for DDPG, SAC, and other advanced methods.
This chapter follows the MDP reward-index convention: executing
6.1 The Q network
Neural networks can approximate state values
Here
The network parameters
Because this is a TD update, check whether the next state
6.2 Experience replay
In basic Q-learning, each newly observed transition immediately updates the Q function:
Using this approach with neural networks presents several difficulties. Single-sample updates can be unstable, consecutive transitions are temporally correlated rather than independent samples, and discarding each transition after one update wastes data. Interaction can be expensive, so poor data reuse reduces training efficiency.
DQN addresses these problems through experience replay. Each transition
For each network update, randomly sample a minibatch:
Here
Before introducing a target network, use
Instead of feeding only the newest transition directly into the model, DQN stores experience and trains on randomly selected batches from the buffer:

6.3 The target network
So far, the target
DQN maintains a target network
Periodically copy the current network's parameters into the target network:
The hyperparameter

This periodic copy is called a hard update. An alternative is a soft update, which gradually moves target parameters toward current parameters:
The coefficient
6.4 Epsilon-greedy exploration
DQN balances exploration and exploitation with an
- With probability
, choose a random action. - With probability
, choose an action with the highest Q value.
As training progresses,
6.5 Algorithm
The full DQN procedure is:
- Initialize the current network
, target network with , and replay buffer . - Reset the environment to
for each episode. Starting at , select and execute using an -greedy policy. Observe , , and the terminal flag; store the transition in . - Once enough samples are available, draw a minibatch
and the associated terminal flags from . Set for terminal transitions, otherwise . - Update
by minimizing . Every environment steps, copy . - End the episode if the environment transition was terminal. Otherwise set
and repeat the action selection and updates from step 2 onward.
6.6 Interactive demonstration
In this 5×5 grid world, the agent (🤖) learns to avoid traps (🔥) and reach the goal (🚩):
- Left: a heatmap shows how estimated state values change during training.
- Center: the neural network's inputs, hidden layers, and action-value outputs.
- Right: replay-buffer contents and training loss.
The shared demonstration currently uses Chinese interface labels. Select 训练 (Train) to watch the agent progress from random exploration toward a good path.
DQN 算法原理可视化
Deep Q-Network: 结合深度学习(感知)与强化学习(决策)
神经网络 (Q-Network)
当前决策解析
经验回放 (Replay Buffer)
训练损失 (Loss)
6.7 Limitations and improvements
| Limitation | Improvement |
|---|---|
| Overestimated Q values | Double DQN selects actions with the online network and evaluates them with the target network |
| Inefficient uniform sampling | Prioritized Experience Replay samples using TD-error priorities |
| Discrete actions only | DDPG and SAC extend related ideas to continuous actions |
| Expected return only | Distributional DQN models the distribution of returns |
Later chapters introduce further developments.
6.8 Questions
How does DQN improve on Q-learning?
Neural networks replace the Q table and handle high-dimensional, continuous state inputs. Experience replay stores state–action–reward–next-state transitions, reduces temporal correlation in training batches, and reuses data. A separate target network stabilizes the values used for TD updates.
Why use an
The agent needs to balance exploiting actions with high estimated Q values and exploring actions whose values are still uncertain. Always following current estimates can miss better strategies; always acting randomly prevents effective exploitation. An
Why add a target network?
When one network produces both predictions and targets, every update changes both. Holding target parameters fixed for a period reduces this feedback and stabilizes learning. The target network has the same structure as the current network, but is updated less frequently.
What does experience replay do?
It stores past transitions and randomly resamples them for training. This reduces correlations within batches, increases data reuse, and makes updates less dependent on the newest trajectory. Together with a target network, it helps stabilize neural value-function learning.