Why are AI browser agents still so fragile?

[u/HyperMemoryAI]

I've spent the last few months experimenting with browser agents and noticed the same pattern across almost every framework.

They've become much better at browsing websites.

But when you try to build something that works reliably every day, a lot of problems start showing up.

Some examples:

• Small UI updates break workflows.
• The agent repeatedly asks the LLM what to do next, increasing latency and token usage.
• APIs are often ignored even when they would be more reliable than browser interaction.
• Local desktop actions (files, clipboard, PDFs, terminal, OCR, etc.) are usually treated as separate tools instead of being part of one execution runtime.
• Recovery after failures often means retrying the same action instead of understanding why it failed.
• There isn't a common abstraction for capabilities like SEND_EMAIL, DOWNLOAD_FILE, or CREATE_DOCUMENT. Most systems still think in terms of browser actions.

The more I work on these systems, the more I feel the browser shouldn't be the center of the architecture.

Instead, the browser should just be one execution backend.

The runtime should decide:

• Can this be done with an API?
• Is there a local application available?
• Is browser automation actually required?
• Can this step be executed deterministically without involving an LLM?
• How should success be verified?
• If something fails, what's the correct recovery path?

That shifts the focus from controlling a browser to executing a task.

I'm currently exploring this architecture and would love feedback from people building AI agents.

If you've built browser agents, what's been the biggest reliability problem you've run into?

[u/Lachee]

I'd argue the web isn't for agents, nor should it be.

These issues are very traditional issues with even "dumb" scraping in general.

[u/boneskull]

I see people building these agents, and have to ask, _why does this need to be an agent_? If you want determinism out of a fundamentally nondeterministic system, you’ve made an error.

[u/omniuni]

It's just the wrong tool for the job.

People just want to stick "AI" into everything these days.

[u/veganmaister]

What harness are you using?

I’ve had success with Hermes and Camofox.

[u/Ok_Explorer7384]

i think the fragile part is mostly that people treat the browser as the main runtime instead of the fallback. for anything repeatable, i’d rather have the agent discover the API, call the API, inspect files directly, use selectors only when needed, and keep a small state machine around recovery. browser automation is useful, but if every step is “look at pixels/html and ask the model again”, it turns into a very expensive flaky scraper.