The researchers found that even state-of-the-art LLMs struggle to solve simple math problems when presented with irrelevant or extraneous information. For example, in a problem involving Oliver picking kiwis, the model was able to solve the problem correctly when the information was straightforward. However, when the problem was modified to include a random detail, such as kiwis being smaller than average, the model's performance dropped significantly. This suggests that LLMs do not truly understand the problem, but rather are able to respond with the correct answer through pattern recognition and replication of training data.
You are viewing a single comment's thread from: