Skip to main content

View as Markdown

Page content converted to Markdown. Use the original page link at the end to explore interactive graphics.

DreamerV3 experiment: reach a target and hold

Watch the arm on the right approach the red target, then remain nearby. This is a physics replay of an actually trained policy for DMControl reacher_easy, using two-joint state observations.

Watch: what did it learn?​

Left: random actions. Right: trained DreamerV3. The panels share the same initial condition. These are the first 5 simulation seconds, played at half speed in roughly 10 seconds. Follow the orange endpoint of the yellow arm toward the red target. The labels embedded in the shared video are Chinese; panel order is the same in every language.

  • Reach: the endpoint starts about 28.7 cm from the target center. It first earns reward at simulation time 0.66 s, approximately 1.32 s in the slowed video.
  • Hold: little movement later is expected: remaining in the target region keeps earning reward. Constant motion is not the task.
  • Compare: random actions can cross the target accidentally; purposeful approach and sustained occupancy are different.

This 1280×800 comparison replays the recorded actions with higher-resolution physics rendering. Joint positions, velocities and target-relative positions were checked at every step for the first 250 steps; maximum error was below 4e-7. Actions were not edited. Initial condition 200 was selected in advance for recording, rather than chosen as the best of the final ten.

An exercise without a GPU

Pause near playback time 1.3 s: has the right arm entered the target region? Continue to distinguish passing through from holding. Then compare the clip with the full-episode metrics below.

Observations, actions and reward​

ItemConfiguration
ObservationTwo joint positions, two velocities and a 2D endpoint-to-target vector: 6 values
ActionTwo continuous joint control values
Reward1 inside the target region; 0 outside
Episode1000 control steps, 20 simulated seconds; maximum return 1000
ObjectiveReach early and remain in the target region

The agent does not see the video. This experiment uses state inputs; MuJoCo provides physics and the video illustrates behavior. It is neither a visual-control experiment nor a hardware validation.

Two networks, or one world model?​

DreamerV3 is a model-based reinforcement learning algorithm. It has three functional parts, implemented by several neural modules:

PartQuestionModules in the pinned implementation
World modelWhat is the current state, and what happens after an action?Encoder enc, recurrent state-space model dyn (RSSM), observation decoder dec, reward head rew, continuation head con
ActorWhich action should be taken now?pol
CriticHow much future return is expected?val, with a slowly updated value target slowval

The encoder extracts observation features. The RSSM combines them with history and the previous action, using both deterministic recurrent state and stochastic latent variables. Training also predicts observations, rewards and continuation. One checkpoint can store all these modules; file count is not network count.

Training versus inference​

Training alternates interaction and learning​

The world model learns from actual interaction sequences; Actor and Critic learn from imagined trajectories. Inaccurate predictions can mislead the policy, so independent physics evaluation remains necessary. This experiment does not fully pretrain and freeze the world model before training the policy: interaction and learning repeat throughout training.

Inference fixes parameters but still uses part of the world model​

The actor does not simply receive raw observations, and the world model does not directly output actions. Critic, reward prediction and imagined trajectories mainly support training. The pinned implementation may compute auxiliary outputs during policy calls, but the decision path is encoding/state update followed by the Actor. Unlike MPC, it need not search many action sequences at each control step.

Actual results​

One RTX 4090, one training seed (0), the author's small dmc_proprio model, and approximately 638,000 trainable parameters. These are experiment measurements, not a hardware performance guarantee.

ConfigurationEnvironment stepsTraining ratioMean return on the same 5 development initial states
Round 1: pipeline check100,0081281.4
Round 2: restore author DMC update ratio100,0081024734.6
Round 2: continued training300,0001024969.0
Random baselineNo training—32.6

train_ratio measures training samples relative to new environment steps. 1024 does not mean 1024 optimizer updates per step or 1024 imagined future steps. Round 2 took about five hours including the intermediate evaluation; final evaluation took additional time.

A further 10 new initial states, evaluated for 1000 steps each:

MetricRandomFinal policy
Mean return11.7954.8
Mean target occupancy1.17%95.48%
Lowest episode return0851

Occupancy over the last 100 steps was 91%–100% for each case. This supports reaching and holding on these ten conditions, not arbitrary conditions, multiple training seeds, image inputs or hardware. Binary rewards make occupancy and return directly related; they are not independent evidence.

Why retain the failed first round?​

Checkpoint saving and finite losses did not imply success. Round 1 was worse than random on new initial states; four initial resets from training environments also returned zero. More updates helped at the same roughly 100,000-step budget in this single-seed comparison. The 300,000-step stage also changed the interaction budget, so gains cannot all be attributed to one factor.

SymptomCheck first
Low return with finite lossesReward-bearing experience and update budget
Good training logs, poor independent replayCheckpoint, action wrappers, observations, reset and recurrent state
Video appears staticResolution, speed, first arrival time and whether holding is the objective
Good reconstruction, poor controlReward prediction, value estimates and policy learning

Reproduce and experiment​

Companion code and per-condition results include the patch and independent evaluation script. Follow its README for setup and commands; watching the video requires no training GPU.

Pin the author source to e01491fad6434b2245a3b8ca201dd7faedcc458c. The patch adds environment seeding, experiment configuration and final checkpoint saving; it does not change algorithm losses. The tested combination was Python 3.11, JAX 0.4.33, DMControl 1.0.34 and MuJoCo 3.3.6. Training dependencies and GPU setup are separate from the website.

Change only train_ratio from 1024 to 128 while holding task, seed, model and environment budget fixed. Compare whole-episode return and final occupancy. Stronger conclusions require multiple training seeds, not a selected best replay.

References and next steps​

Preserve this state-input baseline before moving to pixels or harder tasks.