Introduction to Atlas Agent
The Atlas agent is a new feature being tested by OpenAI that allows users to automate certain tasks on the web. In this article, we will explore the capabilities and limitations of the Atlas agent through a series of tests.
Test Results
The Atlas agent was tasked with finding and downloading game demos on Steam. Rather than navigating to the “Free Demos” category, the agent started by searching for “demo.” After eventually finding the macOS filter, it wasted minutes looking for a “has demo” filter, even though the search for the word “demo” already narrowed it down.
This search results page was about as far as the Atlas agent was able to get when asked for game demos.
Credit:
Kyle Orland
After a long while, the agent finally clicked the top result on the page, which happened to be visual novel Project II: Silent Valley. But even though there was a prominent “Download Demo” link on that page, the agent became concerned that it was on the Steam page for the full game and not a demo. It backed up to the search results page and tried again.
Evaluation
The Atlas agent scored a 1/10 on this task, as it technically found some macOS game demos but utterly failed to even attempt to download them.
Final Results
Across six varied web-based tasks, the Atlas agent scored a median of 7.5 points (and a mean of 6.83 points) on a somewhat subjective 10-point scale. The agent was generally able to correctly interpret what was being asked of it and was able to navigate and process information on webpages carefully (if slowly).
Limitations
The major limiting factor in many of the tests was the “technical constraints on session length” that seem to limit most tasks to a few minutes. Given how long it takes the Atlas agent to figure out where to click next—and the repetitive nature of the kind of tasks one would want a web-agent to automate—this severely limits its utility.
Conclusion
The Atlas agent isn’t yet reliable enough to use as a kind of “set it and forget it” background automation tool. However, for simple, repetitive tasks that a human can spot-check afterward, it already seems like the kind of tool one might use to avoid some of the drudgery in their online life.
FAQs
Q: What is the Atlas agent?
A: The Atlas agent is a feature being tested by OpenAI that allows users to automate certain tasks on the web.
Q: How did the Atlas agent perform in the tests?
A: The Atlas agent scored a median of 7.5 points (and a mean of 6.83 points) on a 10-point scale across six varied web-based tasks.
Q: What are the limitations of the Atlas agent?
A: The major limiting factor is the “technical constraints on session length” that seem to limit most tasks to a few minutes.
Q: Is the Atlas agent ready for use?
A: The Atlas agent isn’t yet reliable enough to use as a background automation tool, but it can be useful for simple, repetitive tasks that can be spot-checked by a human.
