Anthropic Wants Its AI Agent to Control Your Computer
Claude is the first major AI model to be able to take control of a computer to do useful work.
Demos of AI agents can seem stunning but getting the technology to perform reliably and without annoying, or costly, errors in real life can be a challenge. Current models can answer questions and converse with almost human-like skill and are the backbone of chatbots such as OpenAI’s ChatGPT and Google’s Gemini. They can also perform tasks on computers when given a simple command by accessing the computer screen as well as input devices like a keyboard and trackpad or through low-level software interfaces.
Anthropic says that Claude outperforms other AI agents on several key benchmarks including SWE-bench, which measures an agent's software development skills and OSWorld, which gauges an agent's capacity to use a computer operating system. The claims have yet to be independently verified. Anthropic says Claude performs tasks in OSWorld correctly 14.9 percent of the time. This is well below humans, who generally score around 75 percent, but considerably higher than the current best agents, including OpenAI’s GPT-4, which succeed roughly 7.7 percent of the time.
Anthropic claims that several companies are already testing the agentic version of Claude. This includes Canva, which is using it to automate design and editing tasks and Replit, which uses the model for coding chores. Other early users include The Browser Company, Asana and Notion.
Ofir Press, a postdoctoral researcher at Princeton University who helped develop SWE-bench, says that agentic AI tends to lack the ability to plan far ahead and often struggle to recover from errors. “In order to show them to be useful we must obtain strong performance on tough and realistic benchmarks,” he says, like reliably planning a wide range of trips for a user and booking all the necessary tickets.
Article