Here is my take: world models are not just hype — they represent a genuinely real technical direction. However, much of the current discourse focuses on generative capabilities rather than the problems world models actually need to solve: environment prediction, causal modeling, and planning.
Let’s First Clarify: What People Call “World Models” Today Are Not the Same Thing
The world models proposed by LeCun (the JEPA architecture) and the World Space concept discussed by Fei-Fei Li overlap in the problems they address — both involve representation learning and world understanding — but their technical approaches and emphases differ. LeCun aims to make predictions in an abstract representation space, sidestepping the computational cost of pixel-level generation; Fei-Fei Li leans more toward 3D scene understanding and generation. Meanwhile, what industry is building today — such as Genie and Genie 2 in the gaming domain — is closer to an action-conditioned interactive world model: you give it an action, and it predicts the next environment state. Yet its core capabilities remain concentrated in visual environment generation and short-horizon dynamics prediction.
These efforts have value, but they address different problems. If equating “can generate the next frame” with “has formed a stable, transferable, planning-usable representation of environment state” — where “understanding” here does not mean human-level semantic understanding, but rather this specific capability — then yes, the hype has been exaggerated.
The Real Challenge Is Not Prediction, It’s Stable Environment Representations
Running DreamerV3 myself gave me a deep insight: the RSSM (Recurrent State-Space Model) can indeed learn certain dynamics patterns — for instance, on the MuJoCo cartpole task, it can predict the pole’s swing trajectory in imagined space, and the trained policy actually works. But if you examine the imagined states closely, you’ll find that for even slightly more complex tasks, predictions start to blur and diverge.
The reason is not complicated: the RSSM learns task-specific latent state dynamics, not an explicit 3D physical world model. It performs well within the training distribution, but it does not necessarily form explicit representations akin to physical laws, and therefore may not reliably extrapolate when faced with out-of-distribution scenarios the way a physics model would.
This gap is the most fundamental problem in current world model research.
Injecting Physical Priors Is One Important Direction
If the goal is to improve the stability and generalization of world models, simply scaling up data volume and stacking more Transformer layers may not be enough. Injecting physical constraints — such as energy conservation, collision detection, and rigid body dynamics — as priors into the model architecture is a path worth exploring, so that the model does not start from scratch when learning dynamics but instead follows physically plausible learning trajectories.
In my experiments, adding specific physical constraints improved training stability and long-horizon prediction performance on certain tasks — for example, reducing common issues in purely data-driven models like “pole passing through objects” or “objects suddenly disappearing.” Of course, physical priors are not the only path; scaling, multimodal pretraining, and object-level learning each have their own potential.
Object-Centric World Models: Another Direction That Cannot Be Avoided
The RSSM and JEPA approaches discussed earlier mostly make predictions in an overall latent space or pixel space. But “understanding the physical world” has another path: shifting the model from pixels or latent representations toward modeling entities, relationships, and state changes — in other words, object-centric representations. Through methods like slot attention and scene graphs, the scene is decomposed into individual objects and the interaction relationships between them, with dynamics modeled separately for each object.
The advantage of this approach is stronger generalization — if you’ve seen a cup being pushed on a table, and then encounter a bowl being pushed on a table, the object-centric model transfers more easily. An overall latent model, by contrast, may need to relearn from scratch.
The Relationship Between World Models and VLAs Needs a More Nuanced View
The VLA (Vision-Language-Action) approach is very popular in the industry right now — examples include RT-2 and π0. Current mainstream VLAs primarily optimize the mapping from vision and language to actions, while explicit environment prediction and long-horizon planning capabilities remain their weak points. It should be noted that the internal Transformers in these models do have latent prediction and action chunking mechanisms, so they are not entirely devoid of planning ability — it’s just that this planning is implicit and short-sighted.
World models address a different layer of the problem: first predicting “if I take this action, what will the environment state become,” then planning within this imagined space to select the optimal path for execution. Perception is just one input to a world model, not the whole picture.
In tasks requiring long-horizon planning and environment prediction, world models could become an important foundational module alongside VLAs. Future architectures might take the form of a hierarchical VLA + world model + planner structure, or end-to-end VLAs might handle everything directly — there is no consensus yet.
Let’s Also Consider: World Models May Not Be the Final Answer
After discussing the advantages, it’s important to address the risks. The world model approach faces several fundamental challenges: the cost of learning a complete physical world may be too high, long-horizon prediction is inherently difficult (errors accumulate), and a planner may not need a precise simulator — a “good enough” rough model might suffice. It’s even possible that direct policy learning could be more efficient than “first learn a world model, then plan” on certain tasks.
These issues don’t mean the world model direction is wrong, but they do suggest it may not be the only endpoint — it is more likely a key module within an agent architecture.
So, Hype or Critical Milestone?
My assessment: world models are not important because they “can generate future frames,” but because they attempt to solve the prediction and planning problems agents face in unknown environments. However, most current systems remain at the stage of task-specific latent state prediction, still a clear distance away from general physical world understanding.
Between latent state prediction and general physical understanding, there are many unsolved problems: causal reasoning, 3D representation, and the reliability of Sim-to-Real transfer, among others. But this precisely shows the direction is worth investing in. If all the problems were already solved, that would mean there was no room left to explore.
Comments