Web Observer

Tag: Codex

  • Vercel agent-desktop: Accessibility-Tree Computer Use for Coding Agents (2026)

    Vercel agent-desktop: Accessibility-Tree Computer Use for Coding Agents (2026)

    Takeaway: Vercel agent-desktop is an open-source Rust CLI and C library from Vercel Labs that gives coding agents desktop computer-use by reading OS accessibility trees — not by guessing from screenshots. Chris Tate (@ctatedev) framed the Oct 31, 2026 launch as “agent-browser, for your whole computer.” Built by Mohamed Lahfir (@mdlahfir), who joined Vercel Labs, it ships under Apache 2.0 and works with any agent that can run shell commands (Claude Code, Codex, and similar).

    Primary sources for this guide: the vercel-labs/agent-desktop repository and the docs at agent-browser.dev/desktop. This post is a practical explainer for developers wiring agents to real desktop apps — not a pixel computer-use model tutorial.

    What Vercel agent-desktop actually is

    agent-desktop is a tool agents call, not an agent itself. There is no LLM inside, no GUI, and no interactive prompt. Every command prints one JSON envelope and exits. The observe → act → verify loop stays in your harness or script.

    It reads a desktop app’s accessibility tree and returns structured JSON. Every element the agent can act on gets a ref qualified with its snapshot id (for example @s8f3k2p9:e8). The agent acts on that ref, snapshots again to check the result, and repeats. Refs are tied to the snapshot that produced them; bare refs like @e8 are rejected with INVALID_ARGS.

    • Platforms: macOS 13+ (AX API + Accessibility permission), Windows 10 1809+ / Server 2019+ (UI Automation), Linux Ubuntu 24.04+ with AT-SPI (glibc 2.35+). npm ships x64 and ARM64 binaries.
    • Install: npm install -g agent-desktop, then agent-desktop version and agent-desktop status.
    • Surface: ~60 command names / ~56 operational commands covering observe, click/type, windows, notifications, clipboard, sessions, and more. Four held-input commands (key-down/key-up/mouse-down/mouse-up) fail closed until a daemon can release a held key or button.
    • Not for web pages: use agent-browser for the browser. agent-desktop cannot see apps that publish no accessibility tree.

    Why accessibility trees beat screenshot computer-use for agents

    Screenshot-based computer-use asks a vision model to locate buttons from pixels. That path is flexible, but it burns tokens, drifts when themes or DPI change, and makes safe retries hard — you never know whether a click already landed.

    An agent-desktop accessibility tree snapshot returns roles, names, values, and actionable refs. Ref actions go through the OS accessibility API by default (headless): no focus steal and no physical pointer move unless you pass --headed. Multi-agent sessions can show colored cursors so parallel workers stay visible.

    DimensionAccessibility-tree (agent-desktop)Screenshot / vision computer-use
    What the agent seesStructured UI tree + refs from OS a11y APIsPixels / screenshots interpreted by a model
    TargetingStable snapshot-qualified refs; re-resolve against live UICoordinate or visual match guesses
    Retry safetyErrors carry delivery/retry disposition; STALE_REF / AMBIGUOUS_TARGET force a fresh snapshotHard to know if input already delivered
    Default interactionHeadless semantic API (no focus steal); optional –headed physical inputUsually moves cursor / focuses UI
    Blind spotsCannot see UIs with no a11y tree; not for raw web pagesCan “see” anything on screen, including unlabeled chrome
    Best fitIDEs, native apps, Electron shells with a11y; coding-agent loopsOpen-ended GUIs, custom-drawn canvases, a11y-poor apps

    The observe → act → verify loop

    Docs show the same four-step pattern on every OS. A trimmed Finder / Calculator style flow looks like this:

    1. agent-desktop snapshot --app Finder -i — map interactive elements; each gets a ref.
    2. agent-desktop click @s8f3k2p9:e8 — act only with a ref from that snapshot.
    3. agent-desktop snapshot --app Finder -i (or wait / get) — verify the UI actually changed.
    4. On dense apps: start with snapshot --skeleton -i --compact, then snapshot --root @…:eN -i --compact to drill one region.

    When the UI changes, an old ref returns STALE_REF or AMBIGUOUS_TARGET. Take a new snapshot and use the new refs. Successful delivery does not prove the app effect happened — always read back.

    Install, skills, and wiring Claude Code or Codex

    Install globally, grant permissions, then load the embedded skill that matches the binary version:

    1. npm install -g agent-desktop
    2. macOS: agent-desktop permissions --guide for Accessibility. Windows needs no grant for same-integrity apps. Linux needs a graphical session with an AT-SPI bus.
    3. agent-desktop skills get desktop and agent-desktop skills get platform before the first action.
    4. Or install into a harness: npx skills add https://github.com/vercel-labs/agent-desktop --skill agent-desktop (see skills.sh). Prefer the CLI skill command so docs stay version-matched.
    5. Optional AGENTS.md line: tell the agent to run those skills before first use.

    Because there is no model inside, usage limits on Claude Code, Codex, or other hosts stay whatever your provider already bills — agent-desktop only adds shell/FFI calls. For how OpenAI is reshaping ChatGPT + Codex shared usage, see our December 2026 shared usage limits explainer.

    Timeline: Linux 1.0, 1.0.2, and the Oct 31 announce

    Project changelog notes: 1.0.0 (2026-10-27) added Linux (CLI, C library, npm) via AT-SPI; 1.0.1 the same day cleaned skills/version publishing; 1.0.2 (~2026-10-30) tightened npm launcher notices and macOS permission guidance. Chris Tate’s public announce on Oct 31 put the “whole computer” framing in front of coding-agent users. Docs at agent-browser.dev/desktop were updated through late October.

    Privacy, Electron, and JEV

    Snapshots, screenshots, clipboard reads, and debug artifacts can contain private desktop content — keep them out of public logs and uploads. For Chromium/Electron web contents, agent-desktop can open a DevTools port that agent-browser connects to; the desktop a11y tree still owns native chrome.

    An optional JEV helper (jev-desktop skill) can send screen descriptions to a configured model provider. It is separate from the native CLI and needs its own setup/API key — mention it only when you intentionally want vision-style narration on top of structured refs.

    How this fits the broader agent stack

    Desktop computer-use sits beside open coding agents and platform connectors. Compare the tool-not-agent shape with DeepSeek’s open-source super-agent prototype, Google’s Connected Apps platform for AI Mode & Gemini, and evaluation pressure around Gemini 4 Carbon benchmarks. If you publish agent tooling docs, keep FAQ and how-to blocks machine-readable — our schema markup for AI search guide covers FAQPage patterns.

    Developer checklist: ship a safe desktop loop

    1. Confirm platform readiness with agent-desktop status (permissions + supported_surfaces).
    2. Load skills (desktop + platform) before the first click.
    3. Prefer headless refs; enable --headed only when physical input is required.
    4. Always qualify refs as @<snapshot_id>:eN; never reuse after STALE_REF / AMBIGUOUS_TARGET.
    5. Verify after act with snapshot, wait, get, or is — do not trust delivery alone.
    6. Use sessions when multiple agents share a machine; assign agent ids for multi-agent cursors.
    7. Keep private data out of public logs (snapshots, clipboard, screenshots).
    8. Pair with agent-browser for web pages; do not force agent-desktop onto sites.
    9. Treat a11y-poor apps as out of scope — pixel computer-use or human handoff may still be needed.

    FAQ: Vercel agent-desktop and accessibility-tree computer use

    Is Vercel agent-desktop an AI agent?

    No. It is a CLI and C library that agents call. There is no embedded LLM, GUI, or interactive prompt. Your coding agent (Claude Code, Codex, etc.) owns the planning loop.

    How does agent-desktop accessibility tree targeting work?

    A snapshot reads the OS accessibility tree (AX API, UI Automation, or AT-SPI) and assigns snapshot-qualified refs to actionable elements. Commands re-resolve those refs against the live UI. If the tree changed, you get STALE_REF or AMBIGUOUS_TARGET and must snapshot again.

    Does agent-desktop automate websites?

    No. Use agent-browser for web pages. For Chromium or Electron apps you launch, agent-desktop can open a DevTools port that agent-browser connects to for web contents, while native chrome stays on the a11y path.

    What platforms and install command should I use?

    macOS 13+, Windows 10 1809+ / Server 2019+, and Linux Ubuntu 24.04+ (glibc 2.35+) with AT-SPI. Install with npm install -g agent-desktop (x64/ARM64 binaries). Grant Accessibility on macOS via agent-desktop permissions --guide.

    Headless vs –headed — which should agents default to?

    Default is headless: ref actions go through the accessibility API without moving the pointer or stealing focus. Pass --headed only when you need physical input. Multi-agent sessions can use colored cursors so parallel workers stay visible.

    What about privacy and the optional JEV helper?

    Snapshots, screenshots, clipboard, and debug output can contain private data — do not put them in public logs. The optional JEV helper sends screen descriptions to a configured model provider and is separate from the native CLI.

    Bottom line: Vercel agent-desktop turns desktop computer use into a structured, retry-aware CLI contract for coding agents. Prefer accessibility-tree refs over screenshot guessing when apps expose a11y; keep web automation on agent-browser; verify every act; and treat private desktop payloads as sensitive. Star the GitHub repo, read the desktop docs, and load agent-desktop skills get desktop before your first production loop.

  • ChatGPT Codex Shared Usage Limits Coming December 2026

    ChatGPT Codex Shared Usage Limits Coming December 2026

    Bottom line for Plus, Pro, and Business power users: Starting in December 2026, OpenAI will merge ChatGPT and Codex usage into one shared token pool. If you already burn heavy coding sessions in Codex and long agentic chats in ChatGPT on the same bill, plan for a single meter—not two independent allowances. OpenAI says it is preparing carefully and expects most people will not hit resets more often than they do today, but the company has not published plan-specific token numbers or an exact December calendar day.

    The change is documented in OpenAI’s help article Understanding ChatGPT and Codex usage limits (updated about a day before this writing), echoed in an OpenAI Community notice on November 1, and covered in secondary reporting such as TokenPost’s Oct 31 summary. It lands in the same season as other stack shifts we have been tracking—from Gemini 4 Carbon’s Fairwind rollout to Google’s Connected Apps Platform preview and Alphabet’s AI Mode monetization notes—where chat, agents, and coding tools keep collapsing into one product surface.

    What changes in December

    Today, many Plus, Pro, and Business customers treat ChatGPT and Codex as related but separately metered products. That framing ends as OpenAI brings more of Codex’s agentic “Work” capabilities into the main ChatGPT experience.

    • One allowance across ChatGPT and Codex for Plus, Pro, and Business.
    • Usage is measured in tokens—the units models process and generate—not arbitrary “messages” or “runs.”
    • Advanced models and complex tasks consume more of the pool; light Instant chat does not.
    • Settings → Usage will show combined remaining usage and reset times.
    • If you hit the limit: wait for reset, upgrade your plan, or buy credits for eligible features.

    OpenAI’s own framing: “We expect most users won’t need to wait for their usage to reset more often than they do now.” That is a reassurance, not a guarantee for heavy dual-stack workflows.

    Who is affected

    The shared pool applies to Plus, Pro, and Business. If you are on those plans and use both ChatGPT and Codex—or expect Codex-style agent work inside ChatGPT—you are in scope. Indie hackers who keep a Plus seat for chat and a parallel Codex habit for repo work should treat December as a budgeting event, not a cosmetic UI tweak.

    Enterprise and other offerings outside the named plans are not described in the same paragraph of the help article. Do not invent Enterprise quotas from Plus/Pro/Business language. If your org mixes personal Plus seats with company Business seats, map which identity actually owns each workload before you assume “the company meter” will absorb personal Codex spikes.

    What Instant chat means for the meter

    OpenAI explicitly states that Instant chat will not draw from your usage. That matters if your day mixes quick lookups with deep coding or agent runs. Instant chat remains a pressure valve: triage questions, skim docs, and draft outlines without draining the shared pool that Codex and heavier ChatGPT work share.

    Everything else that pulls advanced models or complex tasks still burns tokens. Treat Instant chat as the exception that proves the rule—not as proof that “chat is free.” When Work-style agent capabilities sit inside main ChatGPT, a long agentic thread can look like ordinary chat while still drawing from the same allowance Codex used to hold alone.

    How to monitor Settings → Usage

    OpenAI says you will see combined remaining usage and reset times under Settings → Usage. Build a habit before December:

    1. Check the combined remaining figure after a heavy Codex or agent session—not only at month start.
    2. Note reset times so team handoffs do not stack three large jobs against the same cliff.
    3. Screenshot or log a week of typical dual usage now (ChatGPT vs Codex) so you have a baseline when the UI becomes a single meter.
    4. If you buy credits, confirm which features are “eligible” for credit top-ups before you depend on them for production CI-style loops.

    A shared meter without published token amounts means operational visibility beats speculation. Your Settings panel will be more trustworthy than third-party rumor tables.

    Practical workflow tips before the merge

    Power users who already run coding plus chat on one bill can reduce surprise with a few process changes:

    • Separate exploratory chat from expensive agent work. Use Instant chat (and lighter models when available) for brainstorming; reserve advanced models for diffs, multi-file refactors, and long agent runs.
    • Batch Codex-style jobs. Prefer fewer, well-scoped agent sessions over dozens of half-started threads that each pay a context tax.
    • Upgrade vs credits vs wait. OpenAI lists three outs when you hit the limit: wait for reset, upgrade the plan, or buy credits for eligible features. Decide in advance which path Finance/your indie budget prefers.
    • Team hygiene on Business. Shared culture of “who is burning the pool” matters more when ChatGPT and Codex no longer look like separate budgets in the UI.
    • Keep an eye on peer stacks. Publisher and SEO readers watching AI product economics—see also Google’s AI Mode monetization discussion and Bing’s AI training vs grounding controls—should expect more “one surface, one meter” policies across vendors.

    What OpenAI has not published yet

    For SEO and product readers who want hard numbers, the official page still leaves gaps:

    DetailPublished status
    Exact December calendar dayNot published
    Plan-specific token amounts (Plus / Pro / Business)Not published
    Per-model token multipliersDescribed qualitatively only (“advanced models… use more”)
    Which features accept purchased credits“Eligible features” — list not fully spelled out in the usage-limits article
    Enterprise treatment under the same mergeNot covered in the Plus/Pro/Business paragraph

    Until those land, any viral spreadsheet claiming “Plus gets X million tokens shared” is speculative. Stick to OpenAI’s help center and the in-product Usage panel.

    FAQ

    When do ChatGPT Codex shared usage limits start?

    OpenAI says starting in December (year of the help-article update: 2026). No exact calendar day is published in the usage-limits article.

    Which plans share ChatGPT and Codex usage?

    Plus, Pro, and Business. Usage becomes one allowance across ChatGPT and Codex as Work agentic capabilities move into main ChatGPT.

    Does Instant chat count against the shared pool?

    No. OpenAI states Instant chat will not draw from your usage. Advanced models and complex tasks still consume tokens from the shared allowance.

    How do I see remaining ChatGPT Codex shared usage?

    OpenAI says Settings → Usage will show combined remaining usage and reset times after the transition.

    What if I hit the shared usage limit?

    You can wait for the allowance to reset, upgrade your plan, or buy credits to continue using eligible features.

    Will most users hit limits more often?

    OpenAI expects most users will not need to wait for resets more often than they do now. Heavy dual ChatGPT+Codex users should still baseline current usage before December.

    Sources: OpenAI Help — Understanding ChatGPT and Codex usage limits; OpenAI Community — Usage allowance will be shared across ChatGPT and Codex; TokenPost — OpenAI shared ChatGPT/Codex usage (Oct 31). Analysis original to Web Observer.