Skip to main content

View as Markdown

Page content converted to Markdown. Use the original page link at the end to explore interactive graphics.

CarDreamer intersection experiment: turning with cross traffic

An empty intersection is manageable. What changes when cross traffic is added to the same right turn? Compare two public checkpoints, then inspect predicted future BEV against the actual rollout.

Start with the DreamerV3 reaching experiment. Here you will evaluate public checkpoints, distinguish actual interaction from imagined training, and inspect prediction limits visually. Watch driving outcomes, read the code, inspect forecasts, then change one variable. Training a driving policy from scratch is not required.

Switch scenarios and inspect the behavior​

This run: simple checkpoint / empty road: 3/3 reached; simple checkpoint / traffic: 1/3 reached and two collisions; medium checkpoint / traffic: 3/3 reached. These are actual closed-loop simulations with no new training.

ScenarioPublic checkpointThree-episode runInterpretation
Empty right turnsimple3 reached, no collisionBasic turn under these conditions
Cross traffic, spawn spacing 8–16 msimple1 reached, 2 collisionsEmpty-road success does not establish traffic handling
Same traffic configurationmedium3 reached, no collisionTask-matched weights perform better in this small test

Each video is the first episode of its group, selected in advance. Chase camera left, actual BEV input right.

The simple checkpoint approaches, turns and follows the route out.

Each group has three consecutive episodes, fixed seed 2024, a separately started CARLA instance and the same road/configuration. This is not multi-seed robustness testing; traffic trajectories are not guaranteed identical across policies. Time below 0.2 m/s is 0.3–0.8 seconds including startup, so reliable stopping and yielding has not been established.

We use CARLA 0.9.15 Town03_Opt, while the author specifies Town03. The OpenDRIVE hashes match, but full scene equivalence is unproven. No real-car validation has been performed.

What does the car observe?​

ViewSourcePurpose
HD chase cameraCARLA renderingShows vehicle motion; not a policy input
birdeye_wptSimulator ground-truth BEV with roads, vehicles, route and visible background-vehicle waypointsActual visual input of the public policy
World-model predictionsLearned latent dynamics and decoderExported from saved latent states and actions; see prediction videos above

This is not front-camera-only driving. Ground-truth BEV and route information reduce perception complexity and make the control pipeline easier to study. Real-camera input requires additional perception, training and validation.

Where is the world model used?​

CarDreamer supplies driving tasks and a Dreamer-style model/training implementation. We reproduce its right_turn_simple.ckpt and right_turn_medium.ckpt; the reaching checkpoint cannot directly control a car.

During training the world model learns dynamics, rewards and continuation from environment interactions. Actor and Critic learn using imagined trajectories. At evaluation time observations and history update the latent state, and the Actor produces actions. This evaluation does not perform online MPC trajectory search at every step.

Code comparison: where do PPO and Dreamer learn?​

Separate upstream training from this evaluation. These excerpts explain how the public implementation trains. This run only loaded checkpoints, executed policies and exported forecasts; it performed no training updates. Links point to the pinned source.

StageTypical PPO trainingDreamer training here
Environment interactionCollect observations, actions and rewards in the physical simulatorAlso collect interactions, stored in replay
Learn dynamicsNo separately trained transition modelTrain a world model on actual sequences
Policy update targetsRewards, values and advantages from actual trajectoriesPredicted rewards and values along imagined trajectories
ChallengesInteraction cost, exploration and optimizationAlso exploiting errors in the learned model

1. Actual interaction trains the model. The training loop stores tran in replay, samples sequences and calls agent.train(). The update entry point trains the world model before the behavior:

# Agent.train(): excerpt from the pinned upstream source
state, wm_outs, mets = self.wm.train(data, state)
context = {**data, **wm_outs["post"]}
start = tree_map(lambda x: x.reshape([-1] + list(x.shape[2:])), context)
_, mets = self.task_behavior.train(self.wm.imagine, start, context)

wm.train() updates the encoder, RSSM, image decoder, reward and continuation heads. post is an internal state corrected using actual observations; it supplies imagination starting points. The model loss combines observation/reward/continuation prediction losses and latent-dynamics constraints. The environment defines the reward target; the model learns to predict it.

2. The policy tries actions inside the model. The imagination loop alternates Actor actions and predicted RSSM states:

# WorldModel.imagine(): excerpt
start["action"] = policy(start)

def step(prev, _):
prev = prev.copy()
state = self.rssm.img_step(prev, prev.pop("action"))
return {**state, "action": policy(state)}

There is no CARLA step or next real BEV here. Compare PPO data collection, next_obs, reward, ... = env.step(action): the next-state source changes from the physical simulator to learned img_step(). PPO may reuse a collected batch for several updates; using actual trajectories does not mean running a new simulation for every gradient update.

3. Predicted rewards train Actor and Critic. The task behavior scores imagined states using the world-model reward head:

# Greedy.__init__(): predicted rewards for imagined trajectories
rewfn = lambda s: wm.heads["reward"](s).mean()[1:]

ImagActorCritic.train() generates imagined trajectories, updates the Actor, then trains the Critic. Continuous actions here default to Actor gradients through the differentiable model. This step optimizes Actor parameters; the world model has its own optimizer. Training alternates actual interaction, model learning and imagined behavior learning, rather than freezing the model forever after one training phase.

4. Execution still uses actual observations. The policy entry point corrects the internal state with the current image before selecting an action:

# Agent.policy(): observation correction, then action selection
embed = self.wm.encoder(obs)
latent, _ = self.wm.rssm.obs_step(
prev_latent, prev_action, embed, obs["is_first"]
)
task_outs, task_state = self.task_behavior.policy(latent, task_state)

obs_step() receives actual image features; img_step() does not. The distinction is observation correction versus prediction without the next observation. The forecast video below separately supplies recorded future actions to RSSM and decodes predicted images. It is an offline diagnostic, not Actor training or online MPC.

Code-reading question: what if actual interaction is removed and imagination runs forever?

The model receives no fresh evidence to correct errors. A policy may discover actions whose rewards the model overestimates. Higher imagined returns do not guarantee better CARLA driving. Model-based training still needs physical-simulator evaluation; driving outcomes and forecast images are complementary checks.

Inspect what the world model predicts​

BEV in the driving videos is simulator truth. The following videos show model predictions: actual future left, prediction center, absolute image error amplified 4× right. Switch between empty road and traffic to see where predictions drift.

Three windows were fixed in advance: at simulation 2 / 6 / 10 seconds, use the latent state filtered from observations so far and predict 20 steps / 2 seconds. Playback is 0.5×, roughly four seconds per window. Native BEV is 128×128, enlarged for viewing.

  • The model receives the recorded actions actually executed during the window, with no future images. This is offline action-conditioned forecasting.
  • Driving actions still come from the Actor; these videos are not online search over candidate routes.
  • RSSM prediction samples are stochastic. We export one sample per window, without estimating uncertainty.
Mean normalized RGB pixel MAE over fixed windows (lower is better)ModelHold current frame
Empty / simple, 9 windows0.02510.0896
Traffic / simple, 9 windows0.05120.0922
Traffic / medium, 9 windows0.03370.0953

Errors use RGB values in 0–1 and include all preselected windows, including collision episodes. Road pixels dominate: low mean pixel error does not establish accurate vehicle positions or collision prediction. Each checkpoint contains both world model and policy, so driving differences cannot be attributed solely to forecasting accuracy.

Make the experiment auditable​

The runner checks for an empty scene, cleans up and audits after every episode, logs collision partners and saves partial records. All nine episodes completed driving and forecasting, with empty post-cleanup scenes. Camera and BEV use matching simulator frame IDs; 3–10 missing frames per video are omitted while simulation timestamps remain visible, so playback is slightly shorter.

The original Town03 still crashes in the Vulkan pipeline. Town03_Opt runs after rebuilding its problematic traffic-manager cache; see the code README. Client timeout was increased from 20 to 60 seconds; none of these nine episodes timed out. This does not establish the root cause of the earlier timeout.

Earlier four-success/one-system-timeout results and reused-scene collisions remain debugging records, excluded from this batch. The two failures in this batch are retained.

What counts as success?​

CheckCriterion
DestinationEnvironment destination_reached: at most three route waypoints remain
No collisionis_collision, observed collision and independent sensor all stay clear
Stay on routeout_of_lane does not trigger; this is a waypoint-distance threshold, not lane-marking detection
No timeoutNeither time_exceeded nor evaluator truncation occurs

Count success only when destination is reached and all failure flags are false. Keep model/task parameters fixed and save consecutive episodes instead of selecting attractive clips. Report termination reasons, steps and video per episode; reward is supplementary.

The upstream collision check uses the latest observation and can lag by one frame. Passing a finite evaluation only supports those simulation conditions. Additional traffic seeds and collision checks near termination are needed.

Reproduce the experiment​

Pinned versions: CarDreamer 160132436aeda1de54956f9910e56f3970a565aa, CARLA 0.9.15, Python 3.10, Town03_Opt, tasks carla_right_turn_simple / carla_right_turn_medium (the author uses Town03). Follow the author setup instructions for dependencies and offscreen rendering, and obtain the public right-turn checkpoint.

The new experiment.py completed three groups of three episodes, forecasts and cleanup audits. Follow the code README for dependencies, eval-compat.patch and a dedicated Town03_Opt server. Each group starts a separate server; the runner requires no existing vehicles or sensors.

# Start CARLA in a separate terminal on an unused port.
./CarlaUE4.sh -RenderOffScreen -nosound -carla-port=22000 -benchmark -fps=10

python experiment.py --source /path/to/CarDreamer \
--checkpoint /path/to/right_turn_simple.ckpt \
--output results/traffic --port 22000 --task carla_right_turn_medium --episodes 3 --seed 2024

Outputs include each episode’s replay.mp4, terminal flags, collisions.json, scene-after-cleanup.json, trajectory.npz, three forecast-*.npz files and forecasts.json. Render the three-panel forecast below (system FFmpeg required). See the code README for all three groups and checkpoint hashes. The display camera is separate from policy input; actions are not manually altered. Load trusted checkpoints only.

python render_forecasts.py results/traffic/episode-00 --output prediction-traffic.mp4

Three exercises​

  1. Predict whether empty-road weights can handle traffic before watching, then inspect collision records.
  2. Pause a forecast window at +1 second. Compare prediction, truth and the frozen-frame baseline; which regions dominate error?
  3. Change only the traffic seed and report every episode. Three same-seed episodes are not three independent seeds.

This chapter provides an inspectable public-checkpoint demo and prediction export. Before training a new policy, fix the task, acceptance criteria and evaluation seeds.