A study from Carnegie Mellon University revealed that AI agents struggle to execute complex job tasks. The simulation, TheAgentCompany, involved various AI models trying their hand at real-world software company duties, with dismal performance results. The best AI only completed 24% of its tasks, exposing serious limitations in their capabilities and understanding. Workers can relax; the AI takeover isn’t coming soon.
In a twist that might soothe workers’ frayed nerves about AI replacing them, a recent study from Carnegie Mellon University took a deep dive into AI capabilities. They set up a mock software firm entirely staffed by AI agents. These bots, crafted by industry giants like Google and OpenAI, were tasked with real-world job duties. And the findings? Well, they weren’t exactly impressive.
Dubbed TheAgentCompany, this experiment saw various AI agents assume roles—like financial analysts and project managers—amidst some simulated coworkers. Tasks mirrored the daily grind of a software company, from navigating file directories to crafting performance reviews for their peers. But as reported by Business Insider, the outcomes were a spectacle of confusion.
The standout performer was Anthropic’s Claude 3.5 Sonnet, managing to complete a sad 24 percent of its assignments. That’s right—just a quarter of the tasks! Meanwhile, Google’s Gemini 2.0 Flash came in second worst, wrapping up only 11.4 percent of its tasks after racking up a hefty average of 40 steps per job. Ouch. Meanwhile, the clear loser was Amazon’s Nova Pro v1, with a staggering completion rate of merely 1.7 percent.
Researchers speculated on these eye-opening results, suggesting that these agents suffer from a lack of common sense and poor social skills. They’re like kids trying to play grown-up in the workplace! The bots even had a tendency to trip themselves up by creating shortcuts that led to miscommunication: an agent might rename a user to contact the right person, effectively missing the mark.
So while some smaller tasks can be handled by AI, this study, alongside others, shows that when it comes to more complex jobs—those messy, human-centered gigs—we’re not about to be bowled over by machines anytime soon. Let’s just say our beloved bots are, for now, still living in the realm of sophisticated predictive text, rather than viable workplace teammates.
In a nutshell, the experiment from Carnegie Mellon University illustrates that AI agents still have a long way to go before they can take on jobs meant for humans. Misguided decision-making and a basic lack of understanding for the tasks at hand leave these bots struggling. So, workers can rest easy—for now—while the AI revolution remains firmly in the realm of possibility rather than reality.
Original Source: futurism.com
