[Algorithm] Add a reproducible DreamerV3 DMC Walker benchmark

[vmoens]

Stack from ghstack (oldest at bottom):

• [Doc] Explain DreamerV3 architecture and nomenclature #4095
• -> [Algorithm] Add a reproducible DreamerV3 DMC Walker benchmark #4075
• [Feature] Complete DreamerV3 continuation training semantics #4074
• [BugFix] Match DreamerV3 categorical and KL semantics #4073
• [Feature] Add the DreamerV3 block RSSM core #4072
• [Feature] Add DreamerV3 slow-critic regularization #4068
• [Feature] Normalize DreamerV3 REINFORCE return scale #4067
• [BugFix] Decode DreamerV3 rewards and values end to end #4066
• [BugFix] Correct DreamerV3 symexp two-hot semantics #4065

Summary:

• add a proprioceptive DMC Walker Walk preset for 1.1 million environment steps
• support Gym and dm_control environments through one maintained training script
• record evaluation return by environment step and aggregate multi-seed median/IQR curves
• add optional-dependency and short end-to-end Walker smoke coverage

Rationale:
The existing Pendulum example validates wiring but cannot establish DreamerV3
training parity on a representative continuous-control task. A fixed Walker
protocol, machine-readable curves, and a multi-seed acceptance check make the
reference comparison reproducible while keeping full learning runs outside
pull-request CI.

Test plan:

• pytest test/objectives/test_dreamer_v3.py -k 'dmc_benchmark or sota_shares' -q
• pytest test/smoke_test_deps.py -k dreamer_v3_dmc_walker_env -q
• compose config_dmc_walker with Hydra and inspect the resolved job config
• run 20-step Pendulum and DMC Walker training smokes with JSON metrics
• run benchmark.py for one shortened Walker seed and validate summary.json
• run the full three-seed 1.1M-step benchmark as scheduled/manual validation

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/4075

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 5 Cancelled Jobs, 2 Unrelated Failures

As of commit a42b53a with merge base d5a62cb ():

CANCELLED JOBS - The following jobs were cancelled. Please retry:

• Build Linux Wheels / generate-matrix / generate (gh)
• Build M1 Wheels / generate-matrix / generate (gh)
• Build Windows Wheels / generate-matrix / generate (gh)
• SOTA Tests on Linux / tests (3.10, 13.0, 1) / linux-job (gh)
• SOTA Tests on Linux / tests (3.10, 13.0, 2) / linux-job (gh)

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• Unit-tests on Linux / tests-cpu (3.10, bulk) / linux-job (gh) (trunk failure)
  test/rb/test_storages.py::TestStorages::test__rand_given_ndim_recompile
• Unit-tests on Linux / tests-cpu (3.13, bulk) / linux-job (gh) (trunk failure)
  test/rb/test_storages.py::TestStorages::test__rand_given_ndim_recompile

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[vmoens]

Update after the stack-level ghstack rebase: The full eight-PR Dreamer stack was checked out from #4075, rebased onto current origin/main, linted with pre-commit run --all-files, and submitted successfully as vmoens. This PR is now at 7bb418c6ae6f. The core Dreamer suites pass (267 tests) and the two doctest modules pass (15 tests); the DMC-only smoke still requires dm_control, which is not installed locally. Any earlier conflict/head-status note below is superseded; the substantive code-review observations remain applicable.

Reviewed current PR head e61ee26b60c0a8b0622cdafd2d45afcd057a6fbf and the latest CI/bot state. The requested rebase onto freshly fetched origin/main could not complete: replaying lower-stack commit d48b17c13 conflicts in sota-implementations/dreamer_v3/dreamer_v3.py. I left the rebase in progress and did not force-push.

I checked the Gym/dm_control environment construction and transforms, replay-ratio-to-update conversion, step-aligned JSON metrics, median/IQR aggregation, Hydra preset composition, SOTA smoke overrides, and docs. pytest -q test/objectives/test_dreamer_v3.py -k dmc_benchmark_aggregation passes. The dm_control smoke reached the intended environment construction but could not run locally because dm_control is not installed; that integration needs the optional-dependency job. I did not find a separate functional issue in the PR-specific diff at this head.

Please resolve the stack conflict first, then run the optional-dependency suite for the new dm_control path. The three current bulk failures are the LBFGSConfig parity failure already reported against trunk/merge-base.

[vmoens]

Addressed the optional-dependency collection failure and resubmitted the rebased stack. Hydra and OmegaConf are now detected at module scope, configuration-dependent tests skip when unavailable, and OmegaConf imports stay lazy. The targeted config tests pass, both modules import with Hydra/OmegaConf unavailable, and the full all-files lint suite passes at this PR boundary.