After 600 experiments, here's what I discovered about JEPA world models

There’s a repeated theme here that I don’t understand. For example:

> (3) That's where I decided to build a harder test, where each task takes its start from one demonstration and its goal from another

> (4) I also built an interactive demo where I could place the block anywhere and pick any goal. [but] The goals in the harder test are frames from expert demonstrations

Why lean so hard on the demonstration examples? Surely the actual task here is to move between any two arbitrary poses.

So if the demonstration set is statistically strange in any way, you keep banging into that. And even if you get great results, you still haven’t shown that it generalizes to the generic system. And the one thing you know 100% about a physical robot system is that you will never encounter exactly the same situation twice.

Is there actually a reason to pick eval examples from the demonstrations instead of independent samples from the underlying task?

I’m pretty sure that the T-shoving problem doesn’t have goal reachability issues. And if it did, test 4 could run into them since the start pose is arbitrary.

JEPA is very similar to Model Predictive Control. Here, you swap the model of the system with black-box simulator (a neural net), apply some version of dynamic programming / Bellman equation to find the optimal action over planning horizon.

In the planning stage, by factoring out the optimization step from the rollout step, you can free the model to focus on learning the dynamics in latent space. (The other case, where the rollout+optimization happen in the learning loop, just becomes model-free RL.)

My background straddles classical & data-driven controls. This was the first similarity that popped out to me when I first started learning about world models.

Is JEPA just MPC but where the "M" is a latent-space model?

Hands-on learning is an awesome thing - especially in the age of AI!

But maybe worth reframing away from “discovering” - hierarchical planning, sub-goals, action chunking etc. are well understood - this is more of a… transcript of your personal learning journey?

Which is still cool to see. But I read through expecting there to be some kind of novel results. We probably need to categorize these kinds of posts a bit better.

Let me know if you have any questions!

The theme is really hard to read. Dont mean to be a dick, I love the article. I tried a few browsers and my eyes really strained.

MCTS also solves push-T problems without training: https://github.com/olivkoch/gumbel-mcts

Whenever I read stuff like this, I wonder how could I get into this field?

Awesome read!

as someone with an oled monitor that background effect gave me a small heart attack

In a world where I'm increasingly being triggered by the fonts default Claude uses, I really liked the font used for the title here - it's Faculty Glyphic FYI.

AI; DR

If you’re going to use AI to generate text, keep it short and concise and meaty. I don’t hate reading AI, but I can’t read large walls of text for presumably some useful nugget of information in it.

https://www.pangram.com/history/2dd2b019-221a-4bee-bb6a-6b7e...

There's weird subtle moving rainbow color hues on this webpage on mobile

We are doomed if most of the comments here are just talking about the website layout and not the contents of the article.

Anyway:

> The physics of pushing a block doesn't change if you shift, rotate or mirror the whole table. So when the graph has no states near a goal, it can take demonstration segments from elsewhere, transform them to fit the gap, and plan through those. With these transformed segments, success came back to 85%

This is an interesting technique to recover incomplete states or lost memories after demonstration. I'm guessing that the original LeWM does not have technique of transformative memories if I am correct?

Cool

[flagged]

I'm calling it: JEPA is what unlocks general robotics. In a autonomy stack, LLMs are only useful at the planning level and above.

Website is borderline unusable on my phone. I suspect because of the animations? Either way not a great state of affairs