AI · GitHub repo
OmniParser
Parses UI screenshots into structured interactive elements for vision LLMs
OmniParser is an open-source screen parsing system from Microsoft that converts graphical interface screenshots into structured, labeled elements with bounding boxes and functional descriptions. It allows vision LLMs such as GPT-4o, DeepSeek-R1, and Qwen 2.5VL to accurately ground actions on desktop or web interfaces. It also includes OmniTool for testing and running computer-use automation directly inside a Windows virtual machine.
What it's for
- Ground vision LLM actions to precise UI coordinates for computer-use agents
- Detect clickable icons and interactive bounding boxes across desktop and mobile screens
- Automate desktop tasks inside a Windows VM using custom multi-modal models
- Generate structured UI trajectories and interaction data to train custom GUI models
Features
- Interactive region detection using YOLOv9 models to isolate small icons and UI controls
- Element interactability prediction to distinguish clickable elements from static graphics
- Icon captioning model using Florence to produce functional element descriptions
- OmniTool integration to operate and evaluate actions inside a Windows 11 virtual machine
- Multi-LLM compatibility supporting OpenAI, DeepSeek, Qwen, and Anthropic backends
- Local trajectory logging to collect data for agent training pipelines
- Gradio-based interactive visual demo for testing screenshot parsing locally
Pricing
Open source
Details
- Type
- GitHub repo
- Category
- AI
- Topic
- Computer Use
- Open source
- Yes
- Platforms
- Linux, Windows, macOS, Code library
- For
- AI researchers and developers building vision-based computer-use agents and RPA workflows
- Alternative to
- Anthropic Computer Use, UI-TARS, SeeClick
- Tags
- gui agent, computer use, vision llm, screen parsing, object detection, automation
Why it was saved
- @Sumanth_077: A drop-in component for building screenshot-driven browser/desktop automation agents with a model of your choice.
Compare
Related
Curated collection of papers, tools, and frameworks for computer-controlling AI agents
Open-source computer use agent operating GUI interfaces via mouse and keyboard
Realtime desktop autopilot that controls mouse and keyboard for native apps.
No-code visual canvas for building AI mini-apps from plain English prompts
Open-source, self-hosted AI agent with persistent memory and messaging integrations.
A practical three-step approach to adopting AI tools with minimal risk.
Updated