Web Observer

Tag: open source

  • Vercel agent-desktop: Accessibility-Tree Computer Use for Coding Agents (2026)

    Vercel agent-desktop: Accessibility-Tree Computer Use for Coding Agents (2026)

    Takeaway: Vercel agent-desktop is an open-source Rust CLI and C library from Vercel Labs that gives coding agents desktop computer-use by reading OS accessibility trees — not by guessing from screenshots. Chris Tate (@ctatedev) framed the Oct 31, 2026 launch as “agent-browser, for your whole computer.” Built by Mohamed Lahfir (@mdlahfir), who joined Vercel Labs, it ships under Apache 2.0 and works with any agent that can run shell commands (Claude Code, Codex, and similar).

    Primary sources for this guide: the vercel-labs/agent-desktop repository and the docs at agent-browser.dev/desktop. This post is a practical explainer for developers wiring agents to real desktop apps — not a pixel computer-use model tutorial.

    What Vercel agent-desktop actually is

    agent-desktop is a tool agents call, not an agent itself. There is no LLM inside, no GUI, and no interactive prompt. Every command prints one JSON envelope and exits. The observe → act → verify loop stays in your harness or script.

    It reads a desktop app’s accessibility tree and returns structured JSON. Every element the agent can act on gets a ref qualified with its snapshot id (for example @s8f3k2p9:e8). The agent acts on that ref, snapshots again to check the result, and repeats. Refs are tied to the snapshot that produced them; bare refs like @e8 are rejected with INVALID_ARGS.

    • Platforms: macOS 13+ (AX API + Accessibility permission), Windows 10 1809+ / Server 2019+ (UI Automation), Linux Ubuntu 24.04+ with AT-SPI (glibc 2.35+). npm ships x64 and ARM64 binaries.
    • Install: npm install -g agent-desktop, then agent-desktop version and agent-desktop status.
    • Surface: ~60 command names / ~56 operational commands covering observe, click/type, windows, notifications, clipboard, sessions, and more. Four held-input commands (key-down/key-up/mouse-down/mouse-up) fail closed until a daemon can release a held key or button.
    • Not for web pages: use agent-browser for the browser. agent-desktop cannot see apps that publish no accessibility tree.

    Why accessibility trees beat screenshot computer-use for agents

    Screenshot-based computer-use asks a vision model to locate buttons from pixels. That path is flexible, but it burns tokens, drifts when themes or DPI change, and makes safe retries hard — you never know whether a click already landed.

    An agent-desktop accessibility tree snapshot returns roles, names, values, and actionable refs. Ref actions go through the OS accessibility API by default (headless): no focus steal and no physical pointer move unless you pass --headed. Multi-agent sessions can show colored cursors so parallel workers stay visible.

    DimensionAccessibility-tree (agent-desktop)Screenshot / vision computer-use
    What the agent seesStructured UI tree + refs from OS a11y APIsPixels / screenshots interpreted by a model
    TargetingStable snapshot-qualified refs; re-resolve against live UICoordinate or visual match guesses
    Retry safetyErrors carry delivery/retry disposition; STALE_REF / AMBIGUOUS_TARGET force a fresh snapshotHard to know if input already delivered
    Default interactionHeadless semantic API (no focus steal); optional –headed physical inputUsually moves cursor / focuses UI
    Blind spotsCannot see UIs with no a11y tree; not for raw web pagesCan “see” anything on screen, including unlabeled chrome
    Best fitIDEs, native apps, Electron shells with a11y; coding-agent loopsOpen-ended GUIs, custom-drawn canvases, a11y-poor apps

    The observe → act → verify loop

    Docs show the same four-step pattern on every OS. A trimmed Finder / Calculator style flow looks like this:

    1. agent-desktop snapshot --app Finder -i — map interactive elements; each gets a ref.
    2. agent-desktop click @s8f3k2p9:e8 — act only with a ref from that snapshot.
    3. agent-desktop snapshot --app Finder -i (or wait / get) — verify the UI actually changed.
    4. On dense apps: start with snapshot --skeleton -i --compact, then snapshot --root @…:eN -i --compact to drill one region.

    When the UI changes, an old ref returns STALE_REF or AMBIGUOUS_TARGET. Take a new snapshot and use the new refs. Successful delivery does not prove the app effect happened — always read back.

    Install, skills, and wiring Claude Code or Codex

    Install globally, grant permissions, then load the embedded skill that matches the binary version:

    1. npm install -g agent-desktop
    2. macOS: agent-desktop permissions --guide for Accessibility. Windows needs no grant for same-integrity apps. Linux needs a graphical session with an AT-SPI bus.
    3. agent-desktop skills get desktop and agent-desktop skills get platform before the first action.
    4. Or install into a harness: npx skills add https://github.com/vercel-labs/agent-desktop --skill agent-desktop (see skills.sh). Prefer the CLI skill command so docs stay version-matched.
    5. Optional AGENTS.md line: tell the agent to run those skills before first use.

    Because there is no model inside, usage limits on Claude Code, Codex, or other hosts stay whatever your provider already bills — agent-desktop only adds shell/FFI calls. For how OpenAI is reshaping ChatGPT + Codex shared usage, see our December 2026 shared usage limits explainer.

    Timeline: Linux 1.0, 1.0.2, and the Oct 31 announce

    Project changelog notes: 1.0.0 (2026-10-27) added Linux (CLI, C library, npm) via AT-SPI; 1.0.1 the same day cleaned skills/version publishing; 1.0.2 (~2026-10-30) tightened npm launcher notices and macOS permission guidance. Chris Tate’s public announce on Oct 31 put the “whole computer” framing in front of coding-agent users. Docs at agent-browser.dev/desktop were updated through late October.

    Privacy, Electron, and JEV

    Snapshots, screenshots, clipboard reads, and debug artifacts can contain private desktop content — keep them out of public logs and uploads. For Chromium/Electron web contents, agent-desktop can open a DevTools port that agent-browser connects to; the desktop a11y tree still owns native chrome.

    An optional JEV helper (jev-desktop skill) can send screen descriptions to a configured model provider. It is separate from the native CLI and needs its own setup/API key — mention it only when you intentionally want vision-style narration on top of structured refs.

    How this fits the broader agent stack

    Desktop computer-use sits beside open coding agents and platform connectors. Compare the tool-not-agent shape with DeepSeek’s open-source super-agent prototype, Google’s Connected Apps platform for AI Mode & Gemini, and evaluation pressure around Gemini 4 Carbon benchmarks. If you publish agent tooling docs, keep FAQ and how-to blocks machine-readable — our schema markup for AI search guide covers FAQPage patterns.

    Developer checklist: ship a safe desktop loop

    1. Confirm platform readiness with agent-desktop status (permissions + supported_surfaces).
    2. Load skills (desktop + platform) before the first click.
    3. Prefer headless refs; enable --headed only when physical input is required.
    4. Always qualify refs as @<snapshot_id>:eN; never reuse after STALE_REF / AMBIGUOUS_TARGET.
    5. Verify after act with snapshot, wait, get, or is — do not trust delivery alone.
    6. Use sessions when multiple agents share a machine; assign agent ids for multi-agent cursors.
    7. Keep private data out of public logs (snapshots, clipboard, screenshots).
    8. Pair with agent-browser for web pages; do not force agent-desktop onto sites.
    9. Treat a11y-poor apps as out of scope — pixel computer-use or human handoff may still be needed.

    FAQ: Vercel agent-desktop and accessibility-tree computer use

    Is Vercel agent-desktop an AI agent?

    No. It is a CLI and C library that agents call. There is no embedded LLM, GUI, or interactive prompt. Your coding agent (Claude Code, Codex, etc.) owns the planning loop.

    How does agent-desktop accessibility tree targeting work?

    A snapshot reads the OS accessibility tree (AX API, UI Automation, or AT-SPI) and assigns snapshot-qualified refs to actionable elements. Commands re-resolve those refs against the live UI. If the tree changed, you get STALE_REF or AMBIGUOUS_TARGET and must snapshot again.

    Does agent-desktop automate websites?

    No. Use agent-browser for web pages. For Chromium or Electron apps you launch, agent-desktop can open a DevTools port that agent-browser connects to for web contents, while native chrome stays on the a11y path.

    What platforms and install command should I use?

    macOS 13+, Windows 10 1809+ / Server 2019+, and Linux Ubuntu 24.04+ (glibc 2.35+) with AT-SPI. Install with npm install -g agent-desktop (x64/ARM64 binaries). Grant Accessibility on macOS via agent-desktop permissions --guide.

    Headless vs –headed — which should agents default to?

    Default is headless: ref actions go through the accessibility API without moving the pointer or stealing focus. Pass --headed only when you need physical input. Multi-agent sessions can use colored cursors so parallel workers stay visible.

    What about privacy and the optional JEV helper?

    Snapshots, screenshots, clipboard, and debug output can contain private data — do not put them in public logs. The optional JEV helper sends screen descriptions to a configured model provider and is separate from the native CLI.

    Bottom line: Vercel agent-desktop turns desktop computer use into a structured, retry-aware CLI contract for coding agents. Prefer accessibility-tree refs over screenshot guessing when apps expose a11y; keep web automation on agent-browser; verify every act; and treat private desktop payloads as sensitive. Star the GitHub repo, read the desktop docs, and load agent-desktop skills get desktop before your first production loop.

  • DeepSeek Super-Agent Open Source Coding Prototype (2026)

    DeepSeek Super-Agent Open Source Coding Prototype (2026)

    Bottom line: DeepSeek has published dsh-superagent-prototype (also called Super-Agent in secondary reporting)—an MIT-licensed research prototype of a proactive coding agent that watches one GitHub repository through a DeepSeek Harness bundle. The repo’s own banner is blunt: it is not an official DeepSeek product, may still contain bugs, and was adapted for publication so it is not identical to the internal system or every design in the accompanying Super-agent article. Treat it as a reference architecture you can study and harden—not a drop-in rival you turn on tomorrow next to Claude Code.

    Primary artifacts live on github.com/deepseek-ai/dsh-superagent-prototype (see the raw README) and the numbered docs from philosophy through cookbook. Bloomberg’s November 1 wire, carried by Moneycontrol, frames the release as aiming to challenge Anthropic’s Claude Code and notes Harness-team WeChat commentary about monitoring developer group chats for coding feedback. That positioning is useful context; the GitHub docs are the source of truth for what the code actually does. The piece sits beside today’s coding-agent surface notes on ChatGPT Codex shared usage limits, Gemini 4 Carbon / Fairwind, and Google’s Connected Apps Platform preview.

    What DeepSeek Super-Agent is (and is not)

    Per the README, dsh-superagent is a proactive coding agent for one GitHub repository, packaged as a Harness bundle. It listens to GitHub and, optionally, Feishu. When an allowed person asks for work, it opens a work item (a Ticket), works in its own agent session and directory, and answers in the original conversation—commonly with a pull request. A web panel surfaces only what still needs a human.

    The design philosophy, spelled out in docs/000-philosophy, is attention economics: take as little human attention as possible, and give as much value as possible. The agent acts only when asked or when a team-written rule allows it, and speaks only when someone must act or can use a result. Setup is deliberately agent-guided: you ask your coding agent to read SETUP.md and walk the install with you one short question at a time. License is MIT; pnpm check runs typecheck, tests, and build.

    • Is: open research prototype + Harness plugin surface + GitHub/Feishu Sources + Router/Worker execution model.
    • Is not: a polished SaaS coding product, a multi-repo fleet manager, or a claim that every design in the Super-agent article is already implemented in this tree.
    • Operator burden: you own what it posts under your team’s everyday GitHub/Feishu identities—the philosophy doc says so explicitly.

    Router + Workers vs single-session coding agents

    Most popular coding agents (Claude Code-class tools, IDE copilots, many CLI agents) keep one interactive session glued to your terminal or editor: you prompt, it edits, you approve. DeepSeek’s prototype splits that loop across domain concepts documented in docs/007-architecture and the domain glossary:

    • Main Session (Router / orchestrator): evaluates Triggers, decides accept / stop / ignore, routes work, and keeps proactive assessments out of the worker’s hands.
    • Executor Sessions (Workers): one DeepSeek Harness Session and one working directory per Ticket—so parallel authorized jobs do not share a single chat transcript or cwd.
    • Engine + Sources: GitHub (and optional Feishu) plugins turn issues, PRs, comments, and CI into Triggers; the Engine is the sole writer of Tickets, Mandates, and Decisions.
    • Panel: human attention UI for blocked/failed work and “would accept” review—not a second IDE.

    That Router–Worker split is the practical differentiator versus a single-session coding agent. It buys concurrency and authorization boundaries; it also means you operate a small control plane (Policy, Sources, accept mode) instead of only a chat window. If your mental model is “open a shell and ask the agent to fix a bug,” this prototype is closer to “run a team bot that may open Tickets when Policy allows.”

    Authorization modes: off, observe, auto

    docs/002-authorization defines the Policy accept mode that gates inferred (proactive) work. Cite these effects exactly:

    ModeEffect on inferred candidates
    offNothing inferred becomes a candidate (unaddressed messages, listed-bot messages, follow-ups from people not on the allowed list, etc.).
    observe (default)Inferred candidates are assessed; a positive assessment is recorded as ignore marked “would accept,” and nothing is accepted.
    autoA positive assessment of an inferred candidate accepts it (still needs matching enabled rules).

    Explicit requests work in every mode—an allowed person who addresses the agent can still start work while you leave proactive accepts off. The docs’ operator advice matches the philosophy: start in observe, review recorded Decisions, then widen to auto only after “would accept” looks right. Proactive accept also requires at least one matching team rule; assessment must judge every enabled rule.

    Security caveats for observe mode: observe is safer than auto, not “read-only.” The system still ingests GitHub/Feishu traffic you configure, runs assessments in the main Session, and can create noise or cost if Sources are wide open. Unlisted people addressing the agent stay ignored by default unless you weaken the admission table. Bots never make explicit requests. Do not enable receive-all scopes or “anyone may mention” cells until you understand the attention cost. And remember: even ignored assessments are model work—observe reduces unauthorized code changes, not necessarily token spend.

    Clone and setup expectations

    This is not npm i && ship. Expect DeepSeek Harness, GitHub (and optionally Feishu) CLIs/accounts, Policy/prompt configuration, and a guided SETUP.md flow. Practical expectations before you clone:

    1. Bound the blast radius: one repository, short allowed list, accept mode observe, narrow GitHub scopes (prefer mentions over receive-all).
    2. Budget operator time: the README says setup has many steps; plan an agent-assisted install, then a human review of Decisions for a few days.
    3. Read docs in order: 000 philosophy → 001 domain → 002 authorization before you touch 006 deployment or 008 cookbook Sources.
    4. Run pnpm check after local changes; treat failing tests as a signal the prototype is still research-grade.
    5. Do not equate WeChat marketing with GA: Moneycontrol/Bloomberg coverage about rivaling Claude Code describes intent—validate behavior against the repo before promising a production coding bot.

    Fundraising figures appearing in the same Bloomberg/Moneycontrol article are secondary and optional for this story; they are not part of the GitHub release notes and are not required to evaluate the prototype.

    Practical takeaways for coding-agent teams

    • Architecture shoppers: Study Router + per-Ticket workers if you are designing multi-job agents that must not share one cwd/session.
    • Security / platform: Copy the off → observe → auto ladder and “explicit always works” gate even if you never run DeepSeek’s code.
    • Claude Code / Codex users: Keep your daily driver; use this repo as a reference for Policy and attention design. Pair with Codex shared-pool planning if your org already meters coding agents tightly.
    • Publishers / SEO tooling teams: Agent launches are stacking—Carbon Fairwind, Connected Apps, Codex limits, and this open Super-Agent prototype. Prefer primary docs over rival headlines.

    FAQ

    Is DeepSeek Super-Agent an official product?

    No. The repository banner states it is a research prototype for reference only, not an official DeepSeek product, may contain bugs, and was adapted for publication.

    What repository does dsh-superagent-prototype target?

    It is designed as a proactive coding agent for one GitHub repository, packaged as a DeepSeek Harness bundle, with optional Feishu listening.

    What are the authorization accept modes?

    Per docs/002: off (no inferred candidates), observe (default—assess and record “would accept” without accepting), and auto (accept inferred work after a positive assessment with matching rules). Explicit requests work in every mode.

    How does Router–Worker architecture differ from Claude Code-style agents?

    A main Session routes and evaluates; each Ticket gets its own Executor Session and directory. That differs from a typical single interactive coding session tied to one terminal.

    Where should teams start for a safe trial?

    Clone the MIT repo, follow SETUP.md with an agent, keep accept mode on observe, use a short allowed list, and review panel Decisions before enabling auto.

    Does the Moneycontrol/Bloomberg story change what the code does?

    No. Secondary wires supply positioning (Claude Code rivalry, Harness WeChat notes). Behavior and limits are defined by the GitHub README and numbered docs.

    Sources: deepseek-ai/dsh-superagent-prototype README and docs (000, 002, 007); DeepSeek Harness/ Moneycontrol / Bloomberg, Nov 1, 2026. Analysis original to Web Observer.