They ended up using a well-known conjecture to show that the computational power of even multilayer transformers is limited when it comes to solving complicated compositional problems. Then, in December 2024, Peng and colleagues at the University of California, Berkeley posted a proof(opens a new tab) — without relying on computational complexity conjectures — showing that multilayer transformers indeed cannot solve certain complicated compositional tasks. Basically, some compositional problems will always be beyond the ability of transformer-based LLMs.
You are viewing a single comment's thread from: