This is the same reasoning behind why Yann Lecun thought test-time scaling would not work for LLMs: compounding error.
Instead, the more tokens LLMs use, the better their performance on many tasks. LLMs can self-correct, evidenced by the power of getting models to question themselves by emitting "Wait," in S1. https://arxiv.org/abs/2501.19393
ssivark 42 minutes ago [-]
Ha, interesting. I wasn't aware of Sutton's blog post, but if I might make a shameless plug, we demonstrated [1] exactly this problem (see section 4.4.3), and how multi-step world models (using diffusion models as the substrate) could be one potential answer.
Since then, I have come to like temporally-abstract models more and more. Rolling out in time -- either step-by-step or many steps at once -- suffers from the tyranny of the specific. For long horizon planning with agents, I care (often only approximately) about where I can end up, and seldom about exactly when I end up there. Successor features, GVFs, Forward-Backward representations, and the like seem like they have an elegant approach for structuring thinking at a "high level", instead of generating exponentially large search trees by rolling out microscopic world models.
Instead, the more tokens LLMs use, the better their performance on many tasks. LLMs can self-correct, evidenced by the power of getting models to question themselves by emitting "Wait," in S1. https://arxiv.org/abs/2501.19393
Since then, I have come to like temporally-abstract models more and more. Rolling out in time -- either step-by-step or many steps at once -- suffers from the tyranny of the specific. For long horizon planning with agents, I care (often only approximately) about where I can end up, and seldom about exactly when I end up there. Successor features, GVFs, Forward-Backward representations, and the like seem like they have an elegant approach for structuring thinking at a "high level", instead of generating exponentially large search trees by rolling out microscopic world models.
[1] https://arxiv.org/abs/2410.05364 (funnily, from around the same time / few months after Sutton's blog post)