World Models on WorldSense Tech Blog

What Does a World Model Actually Do in a Robot? From Perception to Action

Wed, 26 Aug 2026 00:00:00 +0000

Dreamer Series · Part 2

Part 1 broke down Dreamer’s overall architecture. This article zooms out one step further: where exactly does a world model sit in a robotic system? What happens between sensor data and the final action? Rather than discussing how to train a general-purpose robot world model, this article focuses on its functional position within robotic systems.

1. A Fundamental Difference: Robot World Models Are Nothing Like Language Models

Over the past few years, the word “model” has been used repeatedly across AI. Language models predict the next token, vision models predict the next frame, autonomous driving models predict the behavior of traffic participants. They all do some form of “prediction,” but the objects and constraints differ enormously.

Understanding Dreamer: How World Models Learn to Imagine

Tue, 25 Aug 2026 00:00:00 +0000

Dreamer Series · Part 1

This article covers Dreamer’s overall architecture at a conceptual level. If you’ve already read the RSSM Code Walkthrough Series, this article will help you connect the scattered code details into a coherent architectural understanding.

Dreamer’s most important idea is not “training a model that generates future frames,” but rather training a latent world model sufficient to support decision-making, then letting policies learn inside that internal world.

Understanding RSSM Through Code (6): Default Config, Four Formulas, and the Code↔Math↔Semantics Map

Mon, 24 Aug 2026 00:00:00 +0000

Understanding RSSM Through Code · 第 6 篇 / 共 6 篇

Series contents (you are on part 6, bolded; prev/next at the bottom):

(1) Where RSSM Sits & the Stochastic State

(2) Prior/Posterior, Straight-Through & unimix

(3) _core(), deter=8192 & Block GRU

(4) KL Balancing, Free Nats & Final KL

(5) Imagine, Observe vs. Imagine & Reset 6. (6) Default Config, Four Formulas & the Map

25. Default Config: Split into “RSSM Architecture” and “World-Model Training”

To avoid mixing “RSSM architecture parameters” with “world-model / agent training config,” the configuration is split into two tables below.

Understanding RSSM Through Code (5): Imagine, Observe vs. Imagine, Sequence Training, and Reset

Sun, 23 Aug 2026 00:00:00 +0000

Understanding RSSM Through Code · 第 5 篇 / 共 6 篇

Series contents (you are on part 5, bolded; prev/next at the bottom):

(1) Where RSSM Sits & the Stochastic State

(2) Prior/Posterior, Straight-Through & unimix

(3) _core(), deter=8192 & Block GRU

(4) KL Balancing, Free Nats & Final KL 5. (5) Imagine, Observe vs. Imagine & Reset

(6) Default Config, Four Formulas & the Map

20. Imagine: How Does RSSM Run Without Observations?

This is the most elegant part of RSSM.

Understanding RSSM Through Code (4): KL Balancing, Free Nats, and the Final KL Combination

Sat, 22 Aug 2026 00:00:00 +0000

Understanding RSSM Through Code · 第 4 篇 / 共 6 篇

Series contents (you are on part 4, bolded; prev/next at the bottom):

(1) Where RSSM Sits & the Stochastic State

(2) Prior/Posterior, Straight-Through & unimix

(3) _core(), deter=8192 & Block GRU 4. (4) KL Balancing, Free Nats & Final KL

(5) Imagine, Observe vs. Imagine & Reset

(6) Default Config, Four Formulas & the Map

14. Complete Observe Phase Data Flow (Observe / Imagine in One Framework)

We can now draw the RSSM state-transition framework and put Observe and Imagine inside the same _core() transition framework:

Understanding RSSM Through Code (3): Deterministic Transition _core(), deter=8192, and Block GRU

Fri, 21 Aug 2026 00:00:00 +0000

Understanding RSSM Through Code · 第 3 篇 / 共 6 篇

Series contents (you are on part 3, bolded; prev/next at the bottom):

(1) Where RSSM Sits & the Stochastic State

(2) Prior/Posterior, Straight-Through & unimix 3. (3) _core(), deter=8192 & Block GRU

(4) KL Balancing, Free Nats & Final KL

(5) Imagine, Observe vs. Imagine & Reset

(6) Default Config, Four Formulas & the Map

9. The Real Deterministic Transition: `_core()`

This is the most worthwhile part of rssm.py to read.

Understanding RSSM Through Code (2): Prior/Posterior, Straight-Through Sampling, and unimix

Thu, 20 Aug 2026 00:00:00 +0000

Understanding RSSM Through Code · 第 2 篇 / 共 6 篇

Series contents (you are on part 2, bolded; prev/next at the bottom):

(1) Where RSSM Sits & the Stochastic State 2. (2) Prior/Posterior, Straight-Through & unimix

(3) _core(), deter=8192 & Block GRU

(4) KL Balancing, Free Nats & Final KL

(5) Imagine, Observe vs. Imagine & Reset

(6) Default Config, Four Formulas & the Map

4. Translating the Source Code into Mathematical Formulas (Watch the Time Index)

DreamerV3’s recurrence relation:

Understanding RSSM Through Code (1): Where RSSM Sits in DreamerV3 and the Stochastic State

Wed, 19 Aug 2026 00:00:00 +0000

Understanding RSSM Through Code · 第 1 篇 / 共 6 篇

Series contents (you are on part 1, bolded; prev/next at the bottom): 1. (1) Where RSSM Sits & the Stochastic State 2. (2) Prior/Posterior, Straight-Through & unimix 3. (3) _core(), deter=8192 & Block GRU 4. (4) KL Balancing, Free Nats & Final KL 5. (5) Imagine, Observe vs. Imagine & Reset 6. (6) Default Config, Four Formulas & the Map

Source Code Reading Guide

The World Model Hype Has Been Exaggerated: A Technical Analysis

Tue, 18 Aug 2026 00:00:00 +0000

Here is my take: world models are not just hype — they represent a genuinely real technical direction. However, much of the current discourse focuses on generative capabilities rather than the problems world models actually need to solve: environment prediction, causal modeling, and planning.

Let’s First Clarify: What People Call “World Models” Today Are Not the Same Thing

The world models proposed by LeCun (the JEPA architecture) and the World Space concept discussed by Fei-Fei Li overlap in the problems they address — both involve representation learning and world understanding — but their technical approaches and emphases differ. LeCun aims to make predictions in an abstract representation space, sidestepping the computational cost of pixel-level generation; Fei-Fei Li leans more toward 3D scene understanding and generation. Meanwhile, what industry is building today — such as Genie and Genie 2 in the gaming domain — is closer to an action-conditioned interactive world model: you give it an action, and it predicts the next environment state. Yet its core capabilities remain concentrated in visual environment generation and short-horizon dynamics prediction.

When World Models Meet Transformers: From RSSM to Large-Scale Sequence Modeling

Wed, 12 Aug 2026 00:00:00 +0000

In previous articles, we covered the RSSM architecture and training techniques in DreamerV3 in depth. RSSM is a classic design in reinforcement learning world models, but if you follow recent research, you’ll notice a clear trend: world models are becoming Transformer-based.

From Google’s UniSim to Wayve’s GAIA-1, from NVIDIA’s Cosmos to solutions from domestic embodied AI teams, the Transformer is emerging as a key technical approach for large-scale world models.

DreamerV3 Training Tips: Lessons from Real-World Debugging

Tue, 11 Aug 2026 00:00:00 +0000

In the previous article, we walked through four representation approaches for world models. Today, we shift back to the practical side of DreamerV3 and talk about the pitfalls and tricks you encounter during training. This article is based on local experiments using DreamerV3 commit e3f02248, JAX + Haiku, and MuJoCo + DM Control. Parameter names and configurations may differ across versions.

DreamerV3 is currently one of the most open-source and mature world model implementations available. But if you’ve actually trained it, you know the process is far from easy — environment setup, hyperparameter tuning, training instability, slow convergence… the list of gotchas goes on.

Four Paradigms of World Model Representations: A Comparative Analysis

Mon, 10 Aug 2026 00:00:00 +0000

In a previous post, we compared MuJoCo and Isaac Sim to clarify simulator selection. But regardless of which simulator you use, world models face a more fundamental question: what exactly should be used to represent the “world”?

This may sound abstract, but it directly determines what a world model can and cannot do. It is like choosing the wrong data structure — no matter how clever your algorithms are downstream, you cannot recover.

TD-MPC: How World Models Enable Robot Control

Fri, 07 Aug 2026 00:00:00 +0000

In the previous article, we broke down RSSM’s dual-track state design and understood how the Dreamer series “imagines” the future in latent space. But RSSM is not the only approach to using world models for robot control. Today we discuss another important route: TD-MPC (Temporal Difference Model Predictive Control).

If Dreamer’s core idea is to leverage a learned world model to perform imagined rollouts in latent space and optimize the policy via actor-critic methods, then TD-MPC takes a more direct approach: learn a world model, then plan within that model to select the optimal action sequence for execution. Rather than relying on a policy network as the sole decision-making mechanism, it uses the world model for online planning, combined with a learned policy prior to improve search efficiency.

World Models as Synthetic Data Engines for VLA Training

Thu, 06 Aug 2026 00:00:00 +0000

In the previous post, we set up and ran DreamerV3 from scratch. A reader asked: what can a trained world model actually do?

Today we discuss a more cutting-edge topic: how to use world models to generate synthetic data that enhances the training of VLA (Vision-Language-Action) models. This has become one of the widely researched directions in robot foundation models in recent years.

Why VLA Needs Synthetic Data

The core capability of a VLA is enabling a robot to “understand human language” — you point at a cup on the table and say “hand me the red one,” and it can understand the language instruction and execute the corresponding action.

ABot-World-0: 24-Hour Stable Inference from an Interactive World Model

Wed, 05 Aug 2026 00:00:00 +0000

Having worked in the world models space for over half a year now, I think the latest ABot-World-0 release from Amap is worth paying attention to, because it tackles a problem this field has never been able to sidestep: world consistency over long time horizons.

What Has Been the Biggest Bottleneck for Interactive World Models

It’s not that they “can’t generate visuals” – it’s that they can’t maintain long-term consistency.

Building a World Model Lab from Scratch: A MuJoCo + DreamerV3 Practical Guide

Wed, 05 Aug 2026 00:00:00 +0000

I’ve written several theoretical articles on world models so far — from the mathematics of RSSM to Sim-to-Real transfer, to the comparison between VLAs and world models. A reader asked: “I get the theory, but how do I actually run something?”

Today’s article answers that question. I’ll walk you step by step through setting up a complete world model experimentation environment — from installation to training to visualization. Once you’ve gotten through it, you can build your own experiments on this foundation.

VLA vs World Models: Which Will Prevail?

Tue, 04 Aug 2026 00:00:00 +0000

Between 2025 and 2026, two distinctly different technical directions have emerged in robot AI. One is the VLA (Vision-Language-Action) approach, represented by RT-2, OpenVLA, and pi-0. The other is the World Models approach, represented by DreamerV3, Genie, and DIAMOND.

Many colleagues have asked me: between these two directions, which one should I bet on? My answer is: the question itself is wrong.

In today’s article, I want to break down and compare these two approaches, explain the logic, advantages, and bottlenecks of each, and then discuss why I believe they will ultimately converge.

Is World Model a Good Research Direction? An Engineer's Honest Assessment

Mon, 03 Aug 2026 00:00:00 +0000

As an engineer who has been working in this field for over half a year, here are my thoughts.

Let me start with the conclusion: it is a good direction, but not everyone should jump in right now.

Why It’s a Good Direction

World models address a very fundamental problem: enabling AI not just to “see” the world, but to “understand” it.

Large language models have already demonstrated that when a model is large enough and the data is sufficient, strong capabilities can emerge. But language models understand the world of text, not the physical world. For robots to truly operate in real-world environments, they need to understand physical laws — gravity, friction, collisions, causality. These things cannot be learned from text data alone.

World Models in 2026: Where Are the Real Opportunities?

Mon, 03 Aug 2026 00:00:00 +0000

Let me start with the conclusion: it is a boom for world models, but not a boom for everyone.

In 2025-2026, world models have undeniably heated up. Video generation models like Sora, Kling, and Vidu are all essentially learning “how the world changes”; DreamerV3 has demonstrated the sample-efficiency advantage of world models in robotic control; and a 2026 survey from the Chinese Academy of Systems Science lays out the four major technical paradigms clearly, marking the field’s transition from “scattered efforts” to a “systematized” stage.

World Models: 8 Years and the Same Bottleneck

Mon, 03 Aug 2026 00:00:00 +0000

After working in the world models direction for over half a year, my biggest takeaway is this: there are plenty of papers, but very few that truly land in practice. Recently I read a 2026 survey from the Chinese Academy of Sciences and several universities that does an excellent job of mapping out eight years of progress. Drawing on my own hands-on experience, I want to discuss a few bottlenecks that still haven’t been fundamentally broken.

Deep Dive into RSSM: The Core Engine of World Models

Sun, 02 Aug 2026 00:00:00 +0000

In the previous article, we covered the basic concepts of world models and the overall architecture of DreamerV3. Some readers asked for a deeper explanation of how RSSM actually works. This article dissects the core component of the Dreamer family of world models.

I’ll try to make the math clear without being overly formal. After all, our goal is to understand the principles, not prove theorems.

Why We Need State-Space Models

Before discussing RSSM, let’s step back and ask: why do we need state-space models at all? Can’t we just use an RNN or Transformer directly?

What Is a Robot World Model? An Engineer's Deep Dive

Sat, 01 Aug 2026 00:00:00 +0000

If you’ve been following the latest developments in AI, you may have noticed a trend: from ChatGPT to Sora, from AlphaFold to robotic manipulation, AI is moving from “understanding language” to “understanding the world.” At the heart of this transition lies an increasingly central concept — the World Model.

In today’s post, I want to discuss, from an engineer’s perspective, what a world model is, why it matters so much for robotics, and what DreamerV3 — currently one of the most representative approaches — actually does.

World Models on WorldSense Tech Blog

What Does a World Model Actually Do in a Robot? From Perception to Action

1. A Fundamental Difference: Robot World Models Are Nothing Like Language Models

Understanding Dreamer: How World Models Learn to Imagine

Understanding RSSM Through Code (6): Default Config, Four Formulas, and the Code↔Math↔Semantics Map

25. Default Config: Split into “RSSM Architecture” and “World-Model Training”

Understanding RSSM Through Code (5): Imagine, Observe vs. Imagine, Sequence Training, and Reset

20. Imagine: How Does RSSM Run Without Observations?

Understanding RSSM Through Code (4): KL Balancing, Free Nats, and the Final KL Combination

14. Complete Observe Phase Data Flow (Observe / Imagine in One Framework)

Understanding RSSM Through Code (3): Deterministic Transition _core(), deter=8192, and Block GRU

9. The Real Deterministic Transition: _core()

Understanding RSSM Through Code (2): Prior/Posterior, Straight-Through Sampling, and unimix

4. Translating the Source Code into Mathematical Formulas (Watch the Time Index)

Understanding RSSM Through Code (1): Where RSSM Sits in DreamerV3 and the Stochastic State

The World Model Hype Has Been Exaggerated: A Technical Analysis

Let’s First Clarify: What People Call “World Models” Today Are Not the Same Thing

When World Models Meet Transformers: From RSSM to Large-Scale Sequence Modeling

DreamerV3 Training Tips: Lessons from Real-World Debugging

Four Paradigms of World Model Representations: A Comparative Analysis

TD-MPC: How World Models Enable Robot Control

World Models as Synthetic Data Engines for VLA Training

Why VLA Needs Synthetic Data

ABot-World-0: 24-Hour Stable Inference from an Interactive World Model

What Has Been the Biggest Bottleneck for Interactive World Models

Building a World Model Lab from Scratch: A MuJoCo + DreamerV3 Practical Guide

VLA vs World Models: Which Will Prevail?

Is World Model a Good Research Direction? An Engineer's Honest Assessment

Why It’s a Good Direction

World Models in 2026: Where Are the Real Opportunities?

World Models: 8 Years and the Same Bottleneck

Deep Dive into RSSM: The Core Engine of World Models

Why We Need State-Space Models

What Is a Robot World Model? An Engineer's Deep Dive

9. The Real Deterministic Transition: `_core()`