The real question is can AI maintain the workflows I depend on?
Web agent benchmarks are broken. Frameworks like Browserbase’s Stagehand evals and agent-evals have fundamental gaps that make them poor predictors of real-world automation success.
Here’s the uncomfortable truth: current benchmarks tell us AI isn’t ready to automate the web. But that’s not the whole story.
The Compound Action Problem
Stagehand’s atomic action evals report ~85% accuracy per action. That sounds solid until you multiply across a real workflow.
Booking a restaurant table might take 10 steps—navigate, select date and time, enter details, submit. At 85% accuracy per action, end-to-end success is 0.85¹⁰ = 19.7%.
One in five attempts succeeds.
Stagehand’s end-to-end tasks show ~65% success, which is closer to reality for multi-step flows—but still low. Benchmarks suggest we can’t reliably automate web workflows today.
Except people already are. Every day. So what’s missing?
The Reliability Question
A 65% success rate means different things depending on distribution.
If 65% of tasks always succeed, that’s transformative. You automate the reliable ones and handle the rest manually.
If every task succeeds only 65% of the time, it’s useless—you can’t trust any single run.
Benchmarks don’t tell us which case we’re in. They give aggregate numbers without showing the distribution of successes across tasks. Without that, we can’t answer the most important question: Which processes can actually be automated today?
Also Read: CIO Influence Interview with Duncan Greatwood, CEO at Xage Security
The Agent Harness Gap
Nobody runs models raw in production. Every real system includes an agent harness—retry logic, validation checks, caching, fallbacks, confidence thresholds.
A model that scores 60% raw might hit 90% with solid harness engineering. Another at 65% might plateau at 70% because its errors are harder to recover from. Benchmarks don’t reveal that.
Different models fail differently. Some make locator errors that are easy to detect and retry; others misread intent in ways that defy recovery. Raw accuracy hides these differences.
Real automation layers recovery: when Playwright code fails mid-run, AI can step in to complete it; when a selector breaks, it tries alternates; when validation fails, it retries. This isn’t magic—it’s engineering. But it changes success rates dramatically.
Benchmarks measure model capability in isolation. Production success is model capability + engineering resilience. Without testing the full stack, we’re optimizing for the wrong thing.
The Auto-Healing Problem
Writing a browser automation script is easy. Maintaining it is not.
A Selenium or Playwright script can automate nearly any workflow in an afternoon. But next week, it breaks when the UI changes—new buttons, banners, validation rules, spinners. Maintenance is the killer.
This is where LLMs matter—not for generating one-off scripts, but for keeping them alive.
Benchmarks miss this entirely. They test one-shot performance, not how systems recover. The best real-world setup isn’t an AI that rediscovers everything from scratch—it’s an AI that maintains deterministic code.
Generate Playwright code as the source of truth. When it breaks, let AI update the code and submit it as a PR. You get version control, human review, and traceable change history. Failures become clear, actionable reports: this selector broke, here’s the fix.
That’s the real automation problem: not “Can AI complete a random task?” but “Can AI maintain the workflows I depend on?”
What We Really Need
Enterprise automation isn’t about zero-shot generalization. It’s about reliable repetition with graceful recovery.
A realistic automation loop looks like this:
- Define a repeatable workflow (AI can generate Playwright code).
- Add structure, guardrails, and validation.
- Run it daily or continuously.
- If code breaks mid-run, AI recovery completes the task.
- After runs, the system reviews failures and submits fixes.
- The automation adapts as sites change.
- Reports are specific—“selector broke, fixed,” not “AI failed.”
This assumes human oversight at the start and an evolving code base, not a black-box agent guessing every time.
Benchmarks that test one-shot tasks measure something, but not the thing that matters most: resilience over time with human-reviewable artifacts.
The Path Forward
We need benchmarks that:
- Report per-task success, not just averages. Show which workflows are automatable now.
- Include agent harnesses. Measure real-world systems with recovery logic, not raw models.
- Track adaptation over time. Run workflows, change sites, see what survives.
- Value reliability over breadth. Ten workflows at 95% reliability beat 100 at 50%.
- Test code generation and healing. Can it produce reviewable code and fix its own errors?
Today’s benchmarks tell us AI isn’t ready. Yet AI already automates real web work—meaning we’re measuring the wrong things.
We’re building a benchmark that reflects production realities—not to replace Stagehand or others, but to measure what they don’t: resilience, healing, and practical automation value.
The web is already being automated by AI. The question isn’t “Can it work?” It’s “Which workflows work now—and how do we make them work better?”
Better benchmarks are how we find out.
Why It Matters Now
Engineering teams spend hours writing and maintaining tests. Building one new test can take five hours or more, slowing feature delivery. And when bugs slip to production, fixing them costs 5–10x more.
At the same time, developers are expected to move faster, leaning on AI assistants like Cursor or Copilot. Modern teams need modern quality assurance.
Better, more realistic benchmarks are a critical step toward automating end-to-end software quality testing.
At Checksum, we generate and maintain Playwright tests with AI, and we’ve learned firsthand what makes browser automation succeed in production. That’s why we’re launching a new benchmark—one that measures what actually matters for reliable web automation.
Catch more CIO Insights: The CIO’s Role In Data Democracy: Empowering Teams Without Losing Control
[To share your insights with us, please write to psen@itechseries.com ]
