# OmniParser

> Parses UI screenshots into structured interactive elements for vision LLMs

OmniParser is an open-source screen parsing system from Microsoft that converts graphical interface screenshots into structured, labeled elements with bounding boxes and functional descriptions. It allows vision LLMs such as GPT-4o, DeepSeek-R1, and Qwen 2.5VL to accurately ground actions on desktop or web interfaces. It also includes OmniTool for testing and running computer-use automation directly inside a Windows virtual machine.

Website: https://github.com/microsoft/OmniParser

## What it is for

- Ground vision LLM actions to precise UI coordinates for computer-use agents
- Detect clickable icons and interactive bounding boxes across desktop and mobile screens
- Automate desktop tasks inside a Windows VM using custom multi-modal models
- Generate structured UI trajectories and interaction data to train custom GUI models

## Features

- Interactive region detection using YOLOv9 models to isolate small icons and UI controls
- Element interactability prediction to distinguish clickable elements from static graphics
- Icon captioning model using Florence to produce functional element descriptions
- OmniTool integration to operate and evaluate actions inside a Windows 11 virtual machine
- Multi-LLM compatibility supporting OpenAI, DeepSeek, Qwen, and Anthropic backends
- Local trajectory logging to collect data for agent training pipelines
- Gradio-based interactive visual demo for testing screenshot parsing locally

## Pricing

Open source

## Details

- Type: GitHub repo
- Category: [AI](https://xiry.app/topics/ai.md)
- Open source: yes
- Platforms: Linux, Windows, macOS, Code library
- For: AI researchers and developers building vision-based computer-use agents and RPA workflows
- Alternative to: Anthropic Computer Use, UI-TARS, SeeClick
- Tags: gui agent, computer use, vision llm, screen parsing, object detection, automation

## Why it was saved

- [@Sumanth_077, 23 February 2025](https://x.com/Sumanth_077/status/1893665895672627406): A drop-in component for building screenshot-driven browser/desktop automation agents with a model of your choice.

## Compare

- [Alternatives to Anthropic Computer Use](https://xiry.app/alternatives/anthropic-computer-use.md)

## Related

- [ACU - Awesome Agents for Computer Use](https://xiry.app/resources/acu-awesome-agents-for-computer-use): Curated collection of papers, tools, and frameworks for computer-controlling AI agents (Website, Free)
- [Agent S](https://xiry.app/resources/agent-s): Open-source computer use agent operating GUI interfaces via mouse and keyboard (Desktop app, Open source)
- [Ace](https://xiry.app/resources/ace): Realtime desktop autopilot that controls mouse and keyboard for native apps. (Desktop app)
- [Google Opal](https://xiry.app/resources/google-opal): No-code visual canvas for building AI mini-apps from plain English prompts (Web app, Free)
- [Hermes Agent](https://xiry.app/resources/hermes-agent): Open-source, self-hosted AI agent with persistent memory and messaging integrations. (Open source)
- [3 Steps to Start Using AI](https://xiry.app/resources/3-steps-to-start-using-ai): A practical three-step approach to adopting AI tools with minimal risk. (Tip, Free)

Updated 10 October 2026. Page: https://xiry.app/resources/omniparser

Make your own library: Xiry takes your saved reels, tweets, Reddit bookmarks and Watch Later videos, makes them searchable and pulls the useful stuff out, like this library. It's invite-only for now; join the waitlist at https://xiry.app/#waitlist.