CLI-Anything Guide: Turning Desktop GUIs into Stateful, Agent-Native CLIs

Every few months, the AI ecosystem gets captivated by a demo where an autonomous agent takes screenshots of a desktop, calculates pixel coordinates, and clicks through a graphic user interface like a tired human intern. It looks impressive on Twitter, but as an engineer building production systems, you quickly discover the reality: screenshot-based computer use is slow, brittle, non-deterministic, and burns through your context window like dry brush in a wildfire.

If a UI button moves five pixels to the left, an OS display scaling factor changes, or a tooltip renders half a second late, the visual agent derails. Worse, you are trading hundreds of thousands of vision tokens just to execute something that should take a single line of structured text.

That is why HKUDS/CLI-Anything (created by the Data Intelligence Lab at the University of Hong Kong) is one of the most important architectural developments of 2026. The thesis behind their research paper ("CLI-Anything: Towards Agent-Native Computer Use", arXiv:2606.03854) is straightforward: stop making agents play video games with user interfaces. Instead, give them structured, deterministic command-line harnesses with JSON output, session persistence, and instant execution.

In this comprehensive guide, I will break down the origin and verified identity of CLI-Anything, warn you about the unofficial clones polluting search results, show you how to install pre-built harnesses for tools like GIMP and Blender, demonstrate how to generate new harnesses from source inside Claude Code, and provide an architectural decision matrix so you know when to use CLI-Anything and when to stick with OpenAPI or MCP.

Origin, Identity & Canonical Verification

Before running a single installation command, let us establish exact provenance. In high-velocity open-source ecosystems, popular repositories inevitably attract copycats, unofficial forks, and scam downloads. CLI-Anything is no exception.

The canonical upstream repository is HKUDS/CLI-Anything. GitHub treats hkuds/cli-anything as the exact same repository under case-insensitive resolution. The project is an original creation from the Data Intelligence Lab at the University of Hong Kong—not a fork or side branch of another tool.

What CLI-Anything Actually Is (And What It Isn't)

To use CLI-Anything effectively, you need to understand its boundaries. It is not a generic runtime bus like Zapier, nor is it a drop-in replacement for an enterprise REST API. Instead, it is a disciplined four-part system:

• A Standardized Methodology (HARNESS.md): A rigorous blueprint for translating complex software into an agent-native CLI surface with persistent state.
• A 7-Phase Agent Generation Pipeline: A multi-step workflow (/cli-anything) that guides coding models through analysis, design, implementation, test planning, test writing, documentation, and packaging.
• A Curated Package Manager (cli-hub): A centralized catalog containing 40+ pre-built, tested harnesses for major software suites (including Blender, GIMP, LibreOffice, FreeCAD, and OBS).
• Agent-Native Runtime Primitives: Built-in --json flags for machine consumption, interactive REPLs for stateful multi-turn sessions, undo/redo state stacks, and auto-generated SKILL.md files for automated agent discovery.

How does it talk to software? It avoids GUI vision models completely. Instead, typical CLI-Anything backends hook directly into the software's native scripting interfaces (such as Blender's bpy Python engine or GIMP's Script-Fu), parse structured interchange formats (SVG, XML, JSON), or drive headless process adapters.

However, there are three hard engineering constraints you must respect before planning a build:

• Requires a Capable Coding Model: Generating a new harness requires an advanced reasoning model like Claude 3.7 Sonnet or Claude Opus. Smaller local models struggle with the 7-phase harness discipline.
• Source Code is Paramount: While you can attempt to wrap closed binaries via headless wrappers, the quality of generated harnesses is dramatically higher when the agent has access to full open-source code.
• The Upstream Software Must Still Be Installed: CLI-Anything generates the CLI bridge and harness; it does not replace the underlying rendering engine, CAD kernel, or graphic runtime.

Installation & Hands-On Workflow Guide

Prerequisites: Python 3.10 or higher installed on your system, plus the native upstream software (e.g., Blender, GIMP) if you intend to execute real commands. On Windows environments running Claude Code, ensure Git for Windows or WSL is installed because cygpath is required for Unix-style path translation.

There are two primary ways to work with CLI-Anything: consuming existing harnesses from the hub, or generating custom harnesses for unadapted codebases.

Workflow A: Consuming Existing Harnesses with cli-hub

Most developers start by using pre-built harnesses. You can search, inspect, and install verified harnesses directly from PyPI:

# 1. Install the official HKUDS hub package from PyPI
pip install cli-anything-hub

# 2. Search and discover available harnesses
cli-hub list
cli-hub search "3d modeling"
cli-hub info gimp

# 3. Install the GIMP harness
cli-hub install gimp

# 4. Inspect available commands
cli-hub launch gimp --help

# Optional: Opt out of hub telemetry
export CLI_HUB_NO_ANALYTICS=1

Once installed, your coding agent can invoke the CLI in one of two modes: direct one-shot execution with machine-readable JSON flags, or an interactive stateful REPL session that preserves project state across multiple agent steps:

# Structured JSON output for agent parsing
cli-anything-gimp --json project list
cli-anything-gimp --json canvas resize --width 1920 --height 1080

# Interactive REPL mode for stateful agent sessions (with undo/redo)
cli-anything-gimp

# Alternatively, run directly from a local harness checkout in editable mode
cd gimp/agent-harness
pip install -e .
cli-anything-gimp --help

Workflow B: Generating a Brand-New CLI Harness with Claude Code

When you have an internal codebase or an open-source tool with no CLI, you point Claude Code at the source repository or local checkout and run the CLI-Anything plugin pipeline:

If you are working in Cursor or other IDE environments, HKUDS ships installation scripts that set up the exact same command playbook:

# Cursor Plugin Setup
git clone https://github.com/HKUDS/CLI-Anything.git
bash CLI-Anything/cursor-plugin/scripts/install.sh

# Pi Extension Setup
bash CLI-Anything/.pi-extension/cli-anything/install.sh

Community adapters are also available for OpenCode, Codex, Hermes, Reasonix, OpenClaw, Goose, Qodercli, and GitHub Copilot CLI. They all enforce the exact same HARNESS.md methodology, ensuring that whichever agent you deploy produces consistent, verifiable CLI tooling.

Architectural Decision Matrix: Is CLI-Anything the Right Choice?

One of the biggest mistakes I see teams make when modernizing their developer tooling is reaching for the shiny new framework without evaluating interface maturity. CLI-Anything is an exceptional tool when you need to bridge desktop software into agent workflows, but it is not a hammer for every nail.

Here is how I evaluate when to use CLI-Anything versus alternative patterns:

The Sherpa Takeaway: Determinism Over Desktop Screenshots

In systems architecture, the most reliable interface is always the one with the smallest blast radius and the highest signal-to-noise ratio.

Visual computer use—asking a multimodal LLM to parse a 4K desktop display, predict coordinates, and issue simulated mouse clicks—is an impressive parlor trick, but it is an operational liability. It fails silently, burns token budgets on redundant pixels, and cannot easily be tested or versioned.

CLI-Anything provides the pragmatic alternative. By converting GUI engines into command-line harnesses with structured JSON responses, undo/redo state tracking, and isolated REPLs, it gives AI agents the exact abstraction layer they were designed to excel at: text-based, deterministic tools.

If you have legacy desktop software, specialized creative tools, or internal utilities that your team wants Claude Code or Cursor to drive autonomously, do not build a visual bot. Run CLI-Anything, generate a clean harness, verify it with tests, and let your agents work with the speed and reliability of the terminal.