拓展阅读 / NanoJev：小型决策模型

知识库收录材料 · 原始文件 ↗ · 来源、版本与许可

Unified MC versus multi-step TD: completed results

Eight-step TD mixed with actual terminal outcomes is the strongest follow-up candidate in this pilot: mean test game success reaches 57.22%, compared with 45.97% for MC. The predeclared three-step mixture reaches 46.25% on test and falls from MC's 17.22% to 12.08% on OOD. The TD bootstrap horizon and retaining real terminal supervision materially affect the result.

Method and commands · Frozen protocol · Complete tables · Machine-readable results

All eight arms completed 200 updates. Each selected checkpoint played the same 228 cases, giving 1,824 new episodes across Maze, Snake, ViZDoom Basic, and Predict Position. Seeds 17 and 29 share the frozen dataset and rollout seed 17. Each arm is one shared Qwen3-0.6B model across all tasks.

Game success

Values are the mean of two training seeds. Within each seed, task-macro success equally weights Maze, Snake, and shooting. Test and OOD each contain 10 Maze, 8 Snake, and 12 shooting cases. Every controller uses the same Q-based action selection with 15% uniform exploration.

Eight-step mixing gains 11.25 percentage points on test and 3.06 points on OOD relative to MC. Both seeds improve over their matched MC run. The condition was a registered secondary comparison; the result motivates a fresh replication. Its OOD mean remains below the initial controller's 25.28%.

Predeclared primary comparison

Success counts by game and seed

Each cell is Maze / Snake / shooting, with denominators 10 / 8 / 12 respectively.

The 50×50 cases remain unsolved by every arm across all splits. Predict Position has isolated successes: mix_n3_seed17 reaches 1/6 on test and mc_seed17 reaches 1/6 on OOD. The eight-step mixed arms score 0/6 on Predict Position in both held-out splits; their shooting gains come from Basic. All scenario counts, including zero-success cells, appear in the complete tables.

Fixed-policy probability prediction

These scores evaluate real terminal outcomes under the frozen continuation policy pi_1. They measure a different task from playing with the newly improved controller. NLL uses natural logarithms; vector Brier is 2 * (p_true - Y)^2. Both are averaged within each task and then equally across tasks. Lower is better.

Three-step mixing improves OOD Brier from 0.249597 to 0.214354, while OOD game success falls. Pure eight-step TD has the lowest mean NLL/Brier among the trained conditions, but a lower mean game score than MC. Good prediction scores alone do not select the strongest game controller.

The constant task-prior reference has the lowest test NLL and Brier in this table. It uses only training outcome frequencies: Maze 0.120221, Snake 0.788945, and shooting 0.067639. Outcome proportions differ across tasks and splits; the frozen development shooting set contains 0 successful episodes out of 12. These distributions are preserved in the prior reference.

Test contains 2,716 outcome questions and OOD 3,107. Questions from one episode are correlated, and longer episodes contribute more questions. These pi_1 scores use a new dataset and must be kept separate from the first iteration's pi_0 outcome scores.

Training and cost

The predictor learns deadline-success probability with gamma=1. Multi-step targets use the recorded frozen behavior probabilities and a detached target network, refreshed every 25 updates. Outcome loss mixes observed MC Brier with soft-target TD Brier; 25% policy CE is retained. The TD mixture weight is 0.5 in both mixed conditions and 1 in the pure-TD condition. This experiment tests TD targets using direct Brier optimization; the earlier paired proper-reward objective is documented separately.

Every arm uses 4,800 online question instances, including 3,600 outcome and 1,200 retained policy questions. For a given seed, all four arms use exactly the same 200 sampled batches. Checkpoints minimize task-balanced development CE with 25% policy and 75% observed-outcome weight at updates 0, 100, and 200.

All arms completed 200 updates regardless of the selected checkpoint. Training time includes the training loop and its development evaluations; initialization, cache preparation, and final held-out evaluation are outside that timer. Target inference is additional computation. The successful runs used seven A100 80GB GPUs with the eighth arm queued. An earlier storage-blocked attempt was restarted in full from identical initialization; none of its incomplete results enter these tables. This round made zero API calls.

Validation and next iteration

All 152 unit tests passed on the remote environment. A real-model smoke test verified backbone/head updates, detached target predictions, and exact hard target copies. The independent training audit verified every same-seed batch, the target refresh schedule, development selection, and held-out metrics against observed outcomes. The summarizer rehashed all selected weights and bound each rollout to the exact checkpoint and configuration.

Independent simulator replay verified all 1,824 episodes and 142,944 executed decisions with zero mismatches. The replay summary retains each report hash, episode hash, and matching collection/replay implementation hashes.

A local reproduction from downloaded predictions and episodes matches all eight arms' metrics and every game result under the unchanged protocol.

Keep the current default model while testing the eight-step mixture on fresh environment groups and fresh outcomes from its own frozen controller. Retain MC and the initial controller as controls, and expand successful Predict Position and large-maze experience. The next comparison should be registered before inspecting its new test outcomes. The two seeds here provide a pilot comparison, not a precision estimate across datasets or rollout randomness.

快照未包含部分工程附件或配图；请通过上游链接或原始材料查看，收录范围见来源清单。