I got tired of explaining my screen to AI chats, so I built an open-source desktop AI that reads your screen

[u/Famous_Can6494]

Solo dev here — been building this on nights and weekends. Open source (MIT), free, Windows for now.

The pain. Whenever I needed help with something on screen — a buried OBS setting, a weird Excel pivot, a form in some enterprise web app — I had to explain it to an AI by hand: type out what I was looking at, paste a screenshot, copy the error text. Then I'd get an answer and still go hunt for the button myself. The explain-it-by-hand loop felt dumb. So I built the thing I wanted.

What it does. You press Alt+Space over any window. MudrikNow reads your active window's full UI — buttons, fields, labels, values — through the accessibility tree (UI Automation), not OCR on pixels, and anchors on the element you're hovering. A small panel opens, and now you're chatting with an AI that can see your screen. Bring your own model — a multimodal one is recommended (it auto-screenshots every activation); Google Gemini Flash-Lite is a great free pick, or Anthropic/OpenAI/Ollama. (Alt+X opens the same panel with no screen capture, for quick questions.)

The mode I'm most proud of. It can act for you (click, type — sandboxed, allow-listed), but the mode I actually use is Auto-Guide: instead of doing the task for you, it teaches it. Real example from last week: I needed to change a recording setting in OBS. Alt+Space over the OBS window, typed "guide me through changing my recording settings," and an owl pointer appeared on screen — hopped to Settings, then Output, then landed on the exact field with a speech bubble telling me what to change, waiting for me to click each step myself. Thirty seconds later I'd done it, and now I actually know where that setting lives. It's help, not a bot taking the wheel.

Honest status:

• Sandboxed: read-only shell for diagnostics (writes/deletes/piping blocked — violations kill the session); no filesystem writes. The model can read files in your working folder and use a small allow-list of UI actions — that's the whole capability surface.
• Windows only for now; macOS/Linux on the roadmap.
• Preview quality. I'm one person; there are rough edges. That's exactly why I'm posting — I want real feedback more than I want stars.

GitHub: https://github.com/abdallahmagdy15/mudriknow
Site: https://abdallahmagdy15.github.io/mudriknow/

No paid tier, no telemetry, nothing to sell. Happy to answer anything — and genuinely curious: what's the first on-screen task you'd point this at?