Honestly I think we will get something in the middle. A lot of stuff will become API calls, but we will also have a lot of people-facing interfaces, and the need to use them automatically (for QA at the very least least). And then there are games
I get the point, but you can't always use the API.
1. Presumably you want to trust your agent to do no shady sheningans behind your back when you give it a simple task
2. Sometimes it just doesn't work.
Does this look like an efficient way to solve the task to you? 500 steps of fiddling with a JS injection, followed by hacking the task.
The funniest part is that it needs to achieve a score of 100, but puts 150 "just in case". I don't think I want it to take the same approach when e.g. fixing a customer's balance.
Well, there’s a bigger issue of AIs not doing what we tell them to do, but I think that’s orthogonal to this UI issue. If you want to fault the model for not obeying the prompt or for a poor solution, then fine. My point is, it solved the problem. If the API isn’t there, then that eliminates the API as a solution, but it says nothing about how well the model can solve the UI problem. Perhaps the model can’t solve it. If we really want to benchmark the ability of models to use human UIs to solve problems, then perhaps we need to choose benchmarks that don’t have APIs available such that the model cannot get “creative” in any way and must use the UI as part of the task. More simply, maybe the model isn’t the problem; maybe the benchmark designer is.
> If we really want to benchmark the ability of models to use human UIs to solve problems, then perhaps we need to choose benchmarks that don’t have APIs available such that the model cannot get “creative” in any way and must use the UI as part of the task. More simply, maybe the model isn’t the problem; maybe the benchmark designer is.
That's a valid point, yet it's hard to blame authors of OSWorld and ALE. They created an env for benchmarking long horizon task completion to be as close to real computer as possible. And for this goal CLI/API access is generally useful, yet when the model not defaults to it for the majority of subtasks.
There're benchmarks that would measure UI literacy (Webgames Benchmark is one). But they are far from the task we want to benchmark in the end.
Sure, “blame” is a strong word. My point isn’t that anyone deserves actual blame, but just that the models are proving creative. We already know that they will comment out unit tests to make them “pass,” for instance. If we want to focus on a particular skill or behavior, we need the tests to highly constrain the model. If we don’t create our benchmarks like that, shame on us.
OSWorld (even the old ones, v1 and verified) runs the model in a full linux VM. And using the terminal is allowed. The models are given tools that provide a subset of possible UI actions. They then use them to open the terminal and script their way to solutions.
Even when this is prohibited, models hack their way around restrictions. Like in OSWorld v2 Claude Opus 4.8 inserts some JS in the browser search bar to execute it instead of playing the Chrome dinosaur.
WebArena is better in that regard because it doesn't allow to use the terminal. But it's worse in another regard: only allows limited actions and the tasks are only such that an agent can win by using a few pre-defined actions (click, scroll, new tab, etc). Which is a tiny subset of what real browser use is, in my opinion.
Thanks, this makes things clearer. Sounds like the idea is that LLMs are competent enough at computer use to bootstrap their way into a terminal or DevTools environment (and from there complete the task using their legitimately impressive scripting skills), but not so good at figuring out how to navigate a UI they've never seen before the way a human does.
Agreed and a lot of companies seem to have their head in the sand, just hoping AI agents will magic their way through the legacy
But I don't think RPA is the way. Or at least, not in the current state. I think AI agents with proper computer use are what RPA should be. Usual RPA is too brittle and maintaining it often requires more work than just doing the work
I was most excited about the "frontier tuning." Like, it will actually watch you do stuff and learn to do it for you? That would be actually interesting.
But no, it's just a data labelling interface: https://learn.microsoft.com/en-us/microsoft-365/copilot/copi.... You have to provide the instruction and give feedback and there is a whole UI with hour-lonf wait between steps. So basically they want you to do the labelling to train a model, or at least that's how it looks from the outside
Also the mission statement of Humanist AI is the most boring, but tries to sound way too grand. Like "all the cool labs have a mission statement, so we should also have one" vibes
I wouldn't call it a benchmark since it's just one sample. They do highlight a real problem, though. Computer use is immature right now and far behind language agents
Try playing fruit ninja via text and llm toolcalls though
Appreciate the effort, but this is very basic and nothing you need the source code to understand. I was expecting a deep dive into what specific decisions they made, but not how an loop of tool calls works
I found it a useful overview. My primary question about the client source was - is there any secret sauce in it? Based on this site, the answer is no, the client is quite simple/dumb, and all the secret sauce resides on the server/in the model.
I particularly valued the tool list. People in these comments are complaining about how bad the code is, but I found the client-side tools that the model actually uses to be pretty clean/general.
My takeaway was more that at a very basic level they know what they are doing - keep the client general, so that you can innovate on the server side without revving the client as much.
