r/automation
i tested 7 browser setups on the same 100 live web tasks
- upvotes
- 5
- comments
- 7
Post
Highlighted: the lines this signal was extracted from
100 Online-Mind2Web tasks on live websites, scored with the benchmark's official judge. six local setups on the same datacenter server/IP, claude haiku 5.5, headed browsers, 10-minute cap, one attempt. plus a guest appearance by grok bot. results (WebJudge on GPT-6 Luna, actions and screenshots only): browseros + claude code: 40/100 claude in chrome: 37 5dive + claude code: 34 openclaw: 29 5dive + pi: 28 hermes: 27 grok bot: 18 (own cloud/model, different judge) why browseros came first it had more ways to recover when normal clicks failed: 94 screen-coordinate clicks and 216 page javascript calls. our browser doesn't support those fallbacks yet. why pi was cheaper than claude code pi made 20% fewer tool calls and started each call with far less context: ~5.6k vs 17.7k tokens. cost per completed task: 2.8¢ vs 6.1¢. but pi also scored lower. it often extracts page text using grep and python, leaving less visual evidence. the judge sees actions and screenshots, not the final answer. some apparently correct answers weren't credited. what didn't help a residential IP barely improved results in an earlier test. sonnet didn't outperform haiku in a separate test despite costing 10x more. separate everyday-task test 4 simpler tasks, 3 runs each. all six local setups passed all 12 runs. browseros was fastest at 19.1s; 5dive + pi was 0.7s behind at roughly 1/8 the...
Keep reading with a free account
The rest of this post, and every signal for xAI, is in your free account.
Also quoted as evidence
[comment u/Thomas_yang1] useful write-up, especially the note that grok bot used a different model and judge, so that 18 isn't a straight comparison.