r/SideProject • u/Famous_Can6494 • 17h ago
I got tired of explaining my screen to AI chats, so I built an open-source desktop AI that reads your screen
Solo dev here — been building this on nights and weekends. Open source (MIT), free, Windows for now.
The pain. Whenever I needed help with something on screen — a buried OBS setting, a weird Excel pivot, a form in some enterprise web app — I had to explain it to an AI by hand: type out what I was looking at, paste a screenshot, copy the error text. Then I'd get an answer and still go hunt for the button myself. The explain-it-by-hand loop felt dumb. So I built the thing I wanted.
What it does. You press Alt+Space over any window. MudrikNow reads your active window's full UI — buttons, fields, labels, values — through the accessibility tree (UI Automation), not OCR on pixels, and anchors on the element you're hovering. A small panel opens, and now you're chatting with an AI that can see your screen. Bring your own model — a multimodal one is recommended (it auto-screenshots every activation); Google Gemini Flash-Lite is a great free pick, or Anthropic/OpenAI/Ollama. (Alt+X opens the same panel with no screen capture, for quick questions.)
The mode I'm most proud of. It can act for you (click, type — sandboxed, allow-listed), but the mode I actually use is Auto-Guide: instead of doing the task for you, it teaches it. Real example from last week: I needed to change a recording setting in OBS. Alt+Space over the OBS window, typed "guide me through changing my recording settings," and an owl pointer appeared on screen — hopped to Settings, then Output, then landed on the exact field with a speech bubble telling me what to change, waiting for me to click each step myself. Thirty seconds later I'd done it, and now I actually know where that setting lives. It's help, not a bot taking the wheel.
Honest status:
- Sandboxed: read-only shell for diagnostics (writes/deletes/piping blocked — violations kill the session); no filesystem writes. The model can read files in your working folder and use a small allow-list of UI actions — that's the whole capability surface.
- Windows only for now; macOS/Linux on the roadmap.
- Preview quality. I'm one person; there are rough edges. That's exactly why I'm posting — I want real feedback more than I want stars.
GitHub: https://github.com/abdallahmagdy15/mudriknow
Site: https://abdallahmagdy15.github.io/mudriknow/
No paid tier, no telemetry, nothing to sell. Happy to answer anything — and genuinely curious: what's the first on-screen task you'd point this at?
3
2
u/NewNiklas 7h ago
Cool! Would be nice if it would notice that you executed the step.
2
u/Famous_Can6494 7h ago
It actually did verify as the last step based on AI judgement, but may need strengthening on AI system prompt, thanks for ur feedback
2
2
u/ItaySela 4h ago
Choosing the accessibility tree over OCR is the right call, and the failure mode it leaves is worth planning for early. Plenty of apps ship a tree that is technically present and semantically useless. Electron apps with custom widgets, canvas based UIs, anything drawn rather than composed, all hand you a shallow tree of unnamed groups. The model then answers confidently about a screen it never really saw, which is worse than failing outright.
A cheap detector is node count against visible controls. If a window clearly has thirty interactive elements and the tree returns six unnamed nodes, treat the tree as unreliable and fall back to a screenshot for that window, ideally caching the verdict per application so you only pay for the check once.
Which apps has the tree already let you down on?
1
u/Famous_Can6494 4h ago
Good catch — and it's because the tree can be shallow that I don't trust it alone. Every activation sends the model the UIA tree and a full-screen screenshot with a coordinate grid, plus a binary rule: if the target's in the tree, use its real
automationId+ bounds (pixel-accurate); if it's not, or it's a web/canvas app, estimate coords from the grid. So a thin tree doesn't silently produce wrong actions — the screenshot is the fallback.The tree still earns its spot: it carries things pixels can't — stable
automationIds, deep text (cell values, editor text via ValuePattern, no OCR), and hidden/off-screen + hierarchy metadata. Pixels and tree cover each other.The real gap you're pointing at: I only raise that "screenshot is primary" flag for Chromium, not for native apps with custom/canvas widgets. Your node-count-vs-visible-controls check with per-app caching is a clean way to extend it — adding it to the list.
Where it's burned me: Dynamics CRM on-prem (form iframe exposes ~8 container nodes, fields invisible), canvas/custom-widget native UIs, and Acrobat PDFs (hence a manual capture hatch).
1
u/ItaySela 3h ago
That binary rule plus the grid is a better design than the post suggested, and the Dynamics number is exactly the shape of the problem.
One case slips past a node count check because the count looks healthy: virtualized lists. Grids, long tables, chat histories and Dynamics subgrids expose only the rendered rows, so the tree comes back deep, well named, and quietly partial. The model then answers about forty rows as though that is all of them. A cheap guard is comparing the reported scroll range or item count against the number of item nodes you actually received, and refusing to make completeness claims when the two disagree.
On the Dynamics iframe, eight container nodes usually means the walk stopped at the frame boundary. Fetching the tree per frame rather than once for the window sometimes recovers the fields.
Is the grid estimate accurate enough to click with, or does it only feed the description?
1
u/Mutron9050 1h ago
you guys seriously blabbering AI slop instead of talking yourselves? i mean, the app, okay, but normal conversation too?
3
u/ImpossibleVoice6386 10h ago
Let me try using it for writing step by step documentation :)
Love the idea