RSSM on WorldSense Tech Blog

What Does a World Model Actually Do in a Robot? From Perception to Action

Wed, 26 Aug 2026 00:00:00 +0000

Dreamer Series · Part 2

Part 1 broke down Dreamer’s overall architecture. This article zooms out one step further: where exactly does a world model sit in a robotic system? What happens between sensor data and the final action? Rather than discussing how to train a general-purpose robot world model, this article focuses on its functional position within robotic systems.

1. A Fundamental Difference: Robot World Models Are Nothing Like Language Models

Over the past few years, the word “model” has been used repeatedly across AI. Language models predict the next token, vision models predict the next frame, autonomous driving models predict the behavior of traffic participants. They all do some form of “prediction,” but the objects and constraints differ enormously.

Understanding Dreamer: How World Models Learn to Imagine

Tue, 25 Aug 2026 00:00:00 +0000

Dreamer Series · Part 1

This article covers Dreamer’s overall architecture at a conceptual level. If you’ve already read the RSSM Code Walkthrough Series, this article will help you connect the scattered code details into a coherent architectural understanding.

Dreamer’s most important idea is not “training a model that generates future frames,” but rather training a latent world model sufficient to support decision-making, then letting policies learn inside that internal world.

Understanding RSSM Through Code (6): Default Config, Four Formulas, and the Code↔Math↔Semantics Map

Mon, 24 Aug 2026 00:00:00 +0000

Understanding RSSM Through Code · 第 6 篇 / 共 6 篇

Series contents (you are on part 6, bolded; prev/next at the bottom):

(1) Where RSSM Sits & the Stochastic State

(2) Prior/Posterior, Straight-Through & unimix

(3) _core(), deter=8192 & Block GRU

(4) KL Balancing, Free Nats & Final KL

(5) Imagine, Observe vs. Imagine & Reset 6. (6) Default Config, Four Formulas & the Map

25. Default Config: Split into “RSSM Architecture” and “World-Model Training”

To avoid mixing “RSSM architecture parameters” with “world-model / agent training config,” the configuration is split into two tables below.

Understanding RSSM Through Code (5): Imagine, Observe vs. Imagine, Sequence Training, and Reset

Sun, 23 Aug 2026 00:00:00 +0000

Understanding RSSM Through Code · 第 5 篇 / 共 6 篇

Series contents (you are on part 5, bolded; prev/next at the bottom):

(1) Where RSSM Sits & the Stochastic State

(2) Prior/Posterior, Straight-Through & unimix

(3) _core(), deter=8192 & Block GRU

(4) KL Balancing, Free Nats & Final KL 5. (5) Imagine, Observe vs. Imagine & Reset

(6) Default Config, Four Formulas & the Map

20. Imagine: How Does RSSM Run Without Observations?

This is the most elegant part of RSSM.

Understanding RSSM Through Code (4): KL Balancing, Free Nats, and the Final KL Combination

Sat, 22 Aug 2026 00:00:00 +0000

Understanding RSSM Through Code · 第 4 篇 / 共 6 篇

Series contents (you are on part 4, bolded; prev/next at the bottom):

(1) Where RSSM Sits & the Stochastic State

(2) Prior/Posterior, Straight-Through & unimix

(3) _core(), deter=8192 & Block GRU 4. (4) KL Balancing, Free Nats & Final KL

(5) Imagine, Observe vs. Imagine & Reset

(6) Default Config, Four Formulas & the Map

14. Complete Observe Phase Data Flow (Observe / Imagine in One Framework)

We can now draw the RSSM state-transition framework and put Observe and Imagine inside the same _core() transition framework:

Understanding RSSM Through Code (3): Deterministic Transition _core(), deter=8192, and Block GRU

Fri, 21 Aug 2026 00:00:00 +0000

Understanding RSSM Through Code · 第 3 篇 / 共 6 篇

Series contents (you are on part 3, bolded; prev/next at the bottom):

(1) Where RSSM Sits & the Stochastic State

(2) Prior/Posterior, Straight-Through & unimix 3. (3) _core(), deter=8192 & Block GRU

(4) KL Balancing, Free Nats & Final KL

(5) Imagine, Observe vs. Imagine & Reset

(6) Default Config, Four Formulas & the Map

9. The Real Deterministic Transition: `_core()`

This is the most worthwhile part of rssm.py to read.

Understanding RSSM Through Code (2): Prior/Posterior, Straight-Through Sampling, and unimix

Thu, 20 Aug 2026 00:00:00 +0000

Understanding RSSM Through Code · 第 2 篇 / 共 6 篇

Series contents (you are on part 2, bolded; prev/next at the bottom):

(1) Where RSSM Sits & the Stochastic State 2. (2) Prior/Posterior, Straight-Through & unimix

(3) _core(), deter=8192 & Block GRU

(4) KL Balancing, Free Nats & Final KL

(5) Imagine, Observe vs. Imagine & Reset

(6) Default Config, Four Formulas & the Map

4. Translating the Source Code into Mathematical Formulas (Watch the Time Index)

DreamerV3’s recurrence relation:

Understanding RSSM Through Code (1): Where RSSM Sits in DreamerV3 and the Stochastic State

Wed, 19 Aug 2026 00:00:00 +0000

Understanding RSSM Through Code · 第 1 篇 / 共 6 篇

Series contents (you are on part 1, bolded; prev/next at the bottom): 1. (1) Where RSSM Sits & the Stochastic State 2. (2) Prior/Posterior, Straight-Through & unimix 3. (3) _core(), deter=8192 & Block GRU 4. (4) KL Balancing, Free Nats & Final KL 5. (5) Imagine, Observe vs. Imagine & Reset 6. (6) Default Config, Four Formulas & the Map

Source Code Reading Guide

When World Models Meet Transformers: From RSSM to Large-Scale Sequence Modeling

Wed, 12 Aug 2026 00:00:00 +0000

In previous articles, we covered the RSSM architecture and training techniques in DreamerV3 in depth. RSSM is a classic design in reinforcement learning world models, but if you follow recent research, you’ll notice a clear trend: world models are becoming Transformer-based.

From Google’s UniSim to Wayve’s GAIA-1, from NVIDIA’s Cosmos to solutions from domestic embodied AI teams, the Transformer is emerging as a key technical approach for large-scale world models.

DreamerV3 Training Tips: Lessons from Real-World Debugging

Tue, 11 Aug 2026 00:00:00 +0000

In the previous article, we walked through four representation approaches for world models. Today, we shift back to the practical side of DreamerV3 and talk about the pitfalls and tricks you encounter during training. This article is based on local experiments using DreamerV3 commit e3f02248, JAX + Haiku, and MuJoCo + DM Control. Parameter names and configurations may differ across versions.

DreamerV3 is currently one of the most open-source and mature world model implementations available. But if you’ve actually trained it, you know the process is far from easy — environment setup, hyperparameter tuning, training instability, slow convergence… the list of gotchas goes on.

Deep Dive into RSSM: The Core Engine of World Models

Sun, 02 Aug 2026 00:00:00 +0000

In the previous article, we covered the basic concepts of world models and the overall architecture of DreamerV3. Some readers asked for a deeper explanation of how RSSM actually works. This article dissects the core component of the Dreamer family of world models.

I’ll try to make the math clear without being overly formal. After all, our goal is to understand the principles, not prove theorems.

Why We Need State-Space Models

Before discussing RSSM, let’s step back and ask: why do we need state-space models at all? Can’t we just use an RNN or Transformer directly?