xiry.

AI · Desktop app

Agent S

Open-source computer use agent operating GUI interfaces via mouse and keyboard

From the Xiry library · Updated 3 October 2026

Visit github.com ↗

Agent S is an open-source agentic framework that allows multimodal AI models to operate desktop and web software like a human user. By interpreting screen visuals and generating mouse, keyboard, and scroll actions, it completes multi-step tasks across apps without custom APIs. Its third-generation version (Agent S3) and related hosted agent Sai lead benchmarks like OSWorld.

What it's for

  • Automate complex multi-step workflows across desktop and browser apps
  • Benchmark and evaluate computer-use agents on OSWorld and WindowsAgentArena
  • Conduct autonomous web research, trip planning, and price comparisons
  • Execute repetitive cross-application data entry without dedicated APIs

Features

  • GUI interaction via mouse movements, clicks, keystrokes, and scrolling
  • Direct visual perception of desktop and browser screens
  • Multi-OS support across macOS, Windows, and Linux
  • Compatibility with multiple model providers including OpenAI, Anthropic, and open-weight models
  • Distribution via the gui-agents Python package
  • Autonomous planning and step-by-step reasoning sidebar

Pricing

Open source. Free and open-source under Apache-2.0; users pay for their own LLM API tokens. Hosted version Sai available separately.

Details

Type
Desktop app
Category
AI
Topic
Computer Use
Open source
Yes
Platforms
macOS, Windows, Linux, Code library
For
Developers and AI researchers building autonomous computer-use agents or desktop automation workflows.
Alternative to
Anthropic Computer Use, OpenAI Operator
Tags
computer use, ai agent, automation, osworld, gui agent, python

Why it was saved

  • @tectalks.io, 26 April 2025: It automates repetitive web-based and computer tasks 24/7 without manual intervention, such as price comparison, trip planning, online ordering, and research.

Compare

Related

  • ACU - Awesome Agents for Computer Use

    Curated collection of papers, tools, and frameworks for computer-controlling AI agents

  • OmniParser

    Parses UI screenshots into structured interactive elements for vision LLMs

  • Ace

    Realtime desktop autopilot that controls mouse and keyboard for native apps.

  • AutoGPT

    Open-source platform and framework for building autonomous AI agents

  • Claude Opus Multi-Agent Orchestration Demo (opus.py)

    A Python multi-agent pattern pairing Opus orchestration with Haiku worker instances.

  • Gobi

    Cloud browser AI agent API to scrape structured data via natural language

Updated