HN Showcase
Back to browse
GitHub Repository

If it's on the screen, it's an API. Control any desktop app via REST + MCP. Rust.

123 starsRust

OculOS – Give AI agents control of your desktop via MCP

by stif1337·Mar 6, 2026·1 point·0 comments
Visit ProjectView on HN

AI Analysis

●●●BangerWizardrySolve My ProblemSlick

Accessibility tree API beats OCR—Claude controls Spotify without screenshots.

Strengths
  • •Reads native OS accessibility APIs (UIA/AXUIElement/AT-SPI2) instead of fragile vision, enabling exact semantic control
  • •Single Rust binary, zero dependencies, runs locally with MCP integration for Claude/GPT agents
  • •Built-in dashboard + automation recorder provides immediate inspection and debugging without leaving the tool
Weaknesses
  • •Maturity unclear—8 stars and 9 commits suggest early-stage; no published benchmarks vs UIAutomator or similar tooling
  • •macOS accessibility permission requirement is a friction point for adoption; unclear how many apps fully expose their tree
Target Audience

AI agent builders, automation engineers, Claude/Cursor power users

Similar To

UIAutomator (Android) · Accessibility Inspector (Apple) · PyAutoGUI with accessibility APIs

Post Description

Hi HN,

I built OculOS because giving AI agents (like Claude Code or Cursor) control over desktop apps is still surprisingly difficult. Most current solutions rely on slow OCR/Vision or fragile pixel coordinates.

OculOS is a lightweight daemon written in Rust that reads the OS accessibility tree and exposes every button, text field, and menu item as a structured JSON API and MCP server.

Why this is different:

Semantic Control: No screenshots or coordinates. The agent interacts with actual UI elements (e.g., "Click the 'Play' button").

Rust-powered: Single binary, zero dependencies, and extremely low latency.

Universal: Supports Windows (UIA), macOS (AXUIElement), and Linux (AT-SPI2).

Local & Private: Everything runs on your machine; no UI data is sent to the cloud.

It also includes a built-in dashboard for element inspection and an automation recorder. I’m looking forward to your feedback and technical questions!

Similar Projects