Apple Study Reveals Advanced AI Faces Serious Limitations in Complex Tasks

A study by Apple researchers reveals that large reasoning models (LRMs) experience ‘complete accuracy collapse’ when tackling complex problems. The paper highlights that standard AI models perform better in simpler tasks but both types struggle significantly as complexity increases. This raises serious questions about the industry’s rapid drive towards artificial general intelligence, with critics highlighting fundamental limitations in current approaches.

In a striking new study, researchers from Apple are shining a light on serious shortcomings in cutting-edge artificial intelligence systems. They reveal that large reasoning models (LRMs), which are touted as the future of AI, can suffer a “complete accuracy collapse” when faced with complex challenges. The paper was released just over the weekend and has already stirred up quite the discussion in tech circles.

The core finding indicates that these advanced AI models struggle with complex queries, revealing what Apple describes as “fundamental limitations” in their reasoning abilities. Interestingly, traditional AI models actually outperformed these LRMs during low-complexity tasks, but both kinds faltered with more difficult problems. It’s almost as if the LRMs, despite their sophisticated design to deconstruct issues into manageable steps, simply could not keep up.

Apple’s testing included various puzzles, like the Tower of Hanoi and the River Crossing conundrum, and it showcased an alarming trend: as the LRMs approached what researchers termed a “performance collapse,” these models began cutting back on their reasoning efforts. This revelation raised a big red flag for the researchers, highlighting a troubling tendency as the complexity of problems increased.

Gary Marcus, a noted academic and voice of prudence in the AI conversation, described the implications of the Apple findings as “pretty devastating.” According to him, these insights could undercut the rush toward artificial general intelligence (AGI), a point at which machines could potentially replicate human-like reasoning across tasks. He said, “Anybody who thinks LLMs are a direct route to the sort [of] AGI that could fundamentally transform society for the good is kidding themselves.”

The study found that not only were LRMs wasteful in their computational processes on simpler problems, they really went astray when the going got tough. Models would wander down incorrect pathways before finally stumbling onto correct solutions, and in extreme cases, they couldn’t even generate a single correct answer, even with an algorithm in hand. The researchers noted that as difficulty ramped up, the models began to cut back on their analytical efforts, which is rather counterintuitive.

The Apple paper bluntly stated: “Upon approaching a critical threshold – which closely corresponds to their accuracy collapse point – models counterintuitively begin to reduce their reasoning effort despite increasing problem difficulty.” This alludes to what’s perceived as a scaling limitation in the cognitive capabilities of these kinds of reasoning models.

While the study certainly offers valuable insights, the focus on puzzles was acknowledged as a limitation. The research looked at several notable models including OpenAI’s o3, Google’s Gemini Thinking, Anthropic’s Claude 3.7 Sonnet-Thinking, and DeepSeek-R1. Some of these companies have yet to respond to inquiries for comment, while OpenAI chose not to weigh in at all.

In the paper, the notion of “generalisable reasoning” comes up too, which speaks to an AI’s ability to stretch narrow conclusions into broader applications. The Apple team stated, “These insights challenge prevailing assumptions about LRM capabilities and suggest that current approaches may be encountering fundamental barriers to generalisable reasoning.”

Andrew Rogoyski from the University of Surrey’s Institute for People-Centred AI said this research signals that the industry is still figuring out its approach to AGI and that it might be facing a “cul-de-sac” in its current methodology. He concluded that the findings suggesting LRMs falter when problems get complex implies that the path forward might not be as clear as once hoped.

The Apple study raises serious concerns about the reliability of large reasoning models in handling complex tasks, highlighting a potential barrier to achieving advanced AI capabilities. With traditional models outperforming LRMs in simpler scenarios and both types collapsing under complexity, the findings might signal a critical juncture in AI development. Observers, like Gary Marcus, suggest this could derail the pursuit of practical AGI, leaving many in the industry to reconsider their strategies and expectations.

Original Source: www.theguardian.com