Summary: A push to move AI from predicting the next text token toward models that understand the physical world, for robotics, autonomous driving, and long-term planning. “World model” is overloaded: it covers video generators that predict frames (Decart, Genie), 3D scene builders (Fei-Fei Li’s Gaussian splats), and Yann LeCun’s JEPA family, which predicts abstract latent representations instead of pixels. Critics of frame prediction argue it’s wasteful and weakly tied to intelligence. Latent approaches can represent uncertainty properly but are not yet proven at scale.
Key ideas
- Plato’s cave: LLMs learn from a text projection of the world, like prisoners seeing shadows. They don’t know what a person is, or how beer tastes (hidden-layers—manifold-gilad-levi, gilad-levi).
- Next-frame prediction over all of YouTube is infeasible and no human does it; it has a weak link to intelligence (hidden-layers—manifold-gilad-levi, gilad-levi).
- JEPA (Joint Embedding Predictive Architecture, from LeCun’s AMI lab): encode two frames and minimize the distance between their latents. A car approaching a three-way split is predicted as a one-third chance for each branch, where a video model must pick one path and is wrong two-thirds of the time (hidden-layers—manifold-gilad-levi, gilad-levi).
- Collapse problem: both encoders could output the zero vector. Fix: regularize the distribution of embeddings (VICReg: variance, invariance, covariance; then SIGReg: just an isotropic Gaussian) (hidden-layers—manifold-gilad-levi, gilad-levi).
- JEPA limits: it’s non-generative, so it can’t produce text, and it’s unproven at scale; one paper shows it needs data with particular distributional properties (hidden-layers—manifold-gilad-levi, gilad-levi).
- Gaussian splats represent images as sums of Gaussians, which have no order, so transformers drop positional encoding to become set-invariant (hidden-layers—manifold-gilad-levi, gilad-levi).
- New paradigms start as underdogs (LeCun and Bengio struggled to publish on neural nets), but with AI money even a $1B+ lab is an underdog bet (hidden-layers—manifold-gilad-levi, gilad-levi).
- Evidence from Nexar: fine-tuned apples-to-apples on ~2M driving clips (crashes and near misses vs normal driving), V-JEPA 2 beat autoregressive video models (Cosmos, VideoMAE) by about 7 points on collision prediction. It generalized to other customers’ cameras and resolutions, and even to generated videos of colliding dinosaurs, horses, and spaceships it never saw (hidden-layers—nexar-roni-goldshmidt, roni-goldshmidt).
- Latent prediction skips unpredictable detail: an autoregressive model wastes capacity tracing a falling leaf’s exact path, while a latent model learns “the leaf swirled and fell” (hidden-layers—nexar-roni-goldshmidt, roni-goldshmidt).
- Vision models need far fewer parameters than language: V-JEPA tops out around 1B (with 300M and 700M variants), perhaps because physics is lawful while language is ambiguous (hidden-layers—nexar-roni-goldshmidt, roni-goldshmidt).
- Attention maps show the model looking ahead along each moving object’s path, anticipating where it will be (hidden-layers—nexar-roni-goldshmidt, roni-goldshmidt).
Disagreements & open questions
- More compute vs new architectures: Levi grants that “add compute and data” keeps working for many cases (ViT, AlphaFold 3) but argues diminishing returns (10× compute for 99% → 99.9%) open the door to cheaper approaches (hidden-layers—manifold-gilad-levi, gilad-levi).
Takeaways
Related
continual-learning · llm-pretraining · multimodal-llms · image-generation · small-language-models