GitHub Repository

AI that acts on your screen, not just reads it. Snip → execute. MCP plugins, sandboxed, local LLM.

6 starsRust

Omni-Glass – Rust app that turns screen pixels into MCP tool calls

Name: Omni-Glass – Rust app that turns screen pixels into MCP tool calls
Availability: InStock
Author: goshtasb

by goshtasb·Feb 22, 2026·3 points·1 comment

Visit Project View on HN

AI Analysis

●●●BangerWizardryZero to OneShip It

Screen-to-action execution via MCP plugins, fully sandboxed, offline-capable.

Strengths

•MCP plugin architecture is genuinely extensible—anyone writes Node/Python server, actions appear automatically with no app code changes
•Kernel-level sandbox (sandbox-exec) walling off home directory is real security, not theater—differentiates from browser-based LLM tools
•Offline execution with local Qwen/llama.cpp + native OCR means nothing leaves your machine; solves privacy failure mode of cloud AI tools

Weaknesses

•macOS-only limits audience significantly; Windows version promised but not shipped
•Requires LLM setup (cloud key or local model); not plug-and-play for non-technical users

Post Description

Omni-Glass is an open-source macOS app (Rust/Tauri) that sits in your menu bar. You draw a box around anything on your screen — a terminal error, a data table, a foreign-language doc — and it runs local OCR, sends the text to an LLM, and gives you a menu of executable actions in under a second. Not explanations. Actions. It fixes the error, exports the CSV, creates the GitHub issue, runs the command. The LLM layer supports Claude Haiku, Gemini Flash, or Qwen-2.5 running locally via llama.cpp (fully offline, nothing leaves your machine). The part I'm most excited about: it's built on MCP (Model Context Protocol). Anyone can write a plugin — a standard MCP server in Node.js or Python — and their actions show up in the menu automatically. The app translates raw OCR text into structured JSON arguments matching your tool's schema. You just write the API call. Every plugin runs inside a kernel-level macOS sandbox (sandbox-exec). Your entire home directory is walled off unless you explicitly approve access. Environment variables are filtered. Commands require confirmation.

Looking for help with:

Build a plugin. Jira, Slack, Notion, Linear, Datadog — if it has an API, it can be an Omni-Glass action. Most plugins are under 100 lines. Break the sandbox. If you can read ~/.ssh/id_rsa from a plugin process, I want to know. Windows and Linux. The code compiles on Windows but hasn't been tested on real hardware. Linux needs Tesseract OCR and Bubblewrap sandbox work.

GitHub: https://github.com/goshtasb/omni-glass