SoMatic – Vision-based OS automation framework for AI agents
Brings Set-of-Marks prompting to native OS apps where DOM trees don't exist.
A vision-first browser agent with self-healing deterministic replay.
Vision-only coordinates beat DOM selectors where Stagehand and browser-use still stumble on UI changes.
Automation engineers, QA teams, and backend developers building brittle web scrapers
browser-use · Stagehand · Playwright
Problem
Browser automation is fragile. Scripts break constantly and agents waste tokens getting stuck in loops.
Today there are two options:
Selector-based scripting like Playwright and Puppeteer: these require you to target specific DOM elements.
First-generation browser agents (Stagehand, browser-use): use natural language interfaces but still resolve instructions into selectors under the hood.
Selector-based scripts can break every time the UI changes. You end up maintaining selectors instead of building features. First-gen agents inherit the same brittleness, especially when they misidentify the right element.
Solution:
Lumen is vision-first. It sees the screen and acts like a human. Every natural language instruction resolves into an x,y coordinate on the screen. Three layers of stuck detection keep it on track and a dual-history system with context compaction lets it handle 20+ step workflows without blowing up the context window.
We ran a WebVoyager eval (25 tasks across 15 sites, scored by LLM judge, 3 trials per task, all frameworks on Claude Sonnet 4.6):
Lumen: 100% success rate, 77.8s avg time, ~104K tokens.
browser-use: 100% success rate, 109.8s avg time.
Stagehand: 76% success rate, 207.8s avg time, ~200K tokens.
Lumen matches browser-use on accuracy while completing tasks ~30% faster, and beats Stagehand on every metric.
Get Started
Start using Lumen today:
Docs: https://lumen.omlabs.xyz/
Support us: GitHub star: https://github.com/omxyz/lumen
Brings Set-of-Marks prompting to native OS apps where DOM trees don't exist.
Retina-aware screenshot + deterministic coordinate mapping for agent desktop control.
Lets agents actually see the screen and act on it by returning OCR text with pixel coordinates and offering commands like click_at, type_text, and press_key. You can run it instantly with npx (it auto-creates a Python venv and hooks into Apple Vision/Quartz), and there are ready-made integration snippets for Claude, VS Code, and Cursor — a pragmatic, technically neat tool for closed-loop agent UI work. It’s limited to macOS 13+ and Apple APIs, but within that niche it removes a lot of friction.
Git for agent reasoning state solves the multi-agent coordination collision problem.
SkillForge turns the old 'show, don't tell' trick into code: record a task, and their AI teases clicks, keystrokes and navigation out of pixels into a stepwise skill file you can edit and export. The ability to trim video, rewrite steps via AI, and output a SKILL.md for agent frameworks is a practical, opinionated workflow that could shortcut lots of brittle RPA scripting — my main questions are reliability across dynamic UIs and privacy/recording controls, but the product direction is smart and tangible.
Git for agent reasoning state beats orchestrators like LangGraph on parallel work.