i ran openclaw's browser on 100 live-web tasks. it finished the most and cost the most
openclaw finished the most tasks of the five setups. it was also the slowest and cost the most.
setup
tasks come from Online-Mind2Web, a deliberately hard benchmark.
every setup used claude haiku 5.5, headed chrome, the same prompts, server and IP.
10-minute limit, one attempt per task.
preview.redd.it
finished / median time / total cost:
- openclaw: 35, 141s, $4.80
- 5dive (claude code): 32, 105s, $1.39
- claude in chrome: 29, 67s, $0.91 (an estimate, it runs on a subscription)
- 5dive (pi): 29, 105s, $0.65
- hermes: 18, 86s, $0.72
the differences among the top weren't statistically significant in my paired tests.
raw results: gist.github.com
why the gaps
- hermes stops sooner: 11.6 steps a task, against 23.5 for openclaw.
- leaving out tasks it got blocked on, hermes still finished 22%, against 45% for openclaw.
- openclaw's batching was on by default, and its agent used it in 5 of the 100 tasks.
- in smaller tests, each openclaw call re-read 72K tokens, against 43K for claude code and 4.5K for pi.
- claude in chrome was fastest (67s a task): its steps took 4s, against 8 to 10s for the others. my guess: they wait for each page to settle and double-check every click.
- agents on a server get a datacenter IP, and some sites block it: each setup hit a bot wall on 20 to 26 of the 100 tasks.
how the browsers differ
claude in chrome drives your own chrome through anthropic's extension.
5dive runs chrome on the agent's own server and stays logged in. when a site throws a robot check, the agent sends you a one-time link to that live window. you solve it, it carries on.
limitations
cheap model, one attempt per live task.
my agents built the test rig. the judge used haiku 5.5, so these scores don't compare to the benchmark's leaderboard.
claude in chrome and hermes ran a day or two after the others, and live sites change.
i built the browser, so i'm not neutral.
i'd rerun openclaw with any setting its users think cuts the calls.
5dive, open source (MIT): 5dive-ai/5dive
browser plugin (Apache-2.0): 5dive-ai/5dive-browser
12 points · 6 comments
Mentions
i ran openclaw's browser on 100 live-web tasks. it finished the most and cost the most openclaw finished the most tasks of the five setups. it was also the slowest and cost the most. setup tasks come from Online-Mind2Web, a deliberately hard benchmark. every setup used claude haiku 5.5, headed chrome, the same prompts, server and IP. 10-minute limit, one attempt per task. preview.redd.it finished / median time / total cost: - openclaw: 35, 141s, $4.80 - 5dive (claude code): 32, 105s, $1.39 - claude in chrome: 29, 67s, $0.91 (an estimate, it runs on a subscription) - 5dive (pi): 29, 105s, $0.65 - hermes: 18, 86s, $0.72 the differences among the top weren't statistically significant in my paired tests. raw results: gist.github.com why the gaps - hermes stops sooner: 11.6 steps a task, against 23.5 for openclaw. - leaving out tasks it got blocked on, hermes still finished 22%, against 45% for openclaw. - openclaw's batching was on by default, and its agent used it in 5 of the 100 tasks. - in smaller tests, each openclaw call re-read 72K tokens, against 43K for claude code and 4.5K for pi. - claude in chrome was fastest (67s a task): its steps took 4s, against 8 to 10s for the others. my guess: they wait for each page to settle and double-check every click. - agents on a server get a datacenter IP, and some sites block it: each setup hit a bot wall on 20 to 26 of the 100 tasks. how the browsers differ claude in chrome drives your own chrome through anthropic's extension. 5dive runs chrome on the agent's own server and stays logged in. when a site throws a robot check, the agent sends you a one-time link to that live window. you solve it, it carries on. limitations cheap model, one attempt per live task. my agents built the test rig. the judge used haiku 5.5, so these scores don't compare to the benchmark's leaderboard. claude in chrome and hermes ran a day or two after the others, and live sites change. i built the browser, so i'm not neutral. i'd rerun openclaw with any setting its users think cuts the calls. 5dive, open source (MIT): 5dive-ai/5dive browser plugin (Apache-2.0): 5dive-ai/5dive-browser