# Chrome Relay full documentation # https://chrome-relay.kushalsm.com --- # Chrome Relay docs > Your agent drives the Chrome you're signed into. It reads pages, clicks buttons, and fills forms. Nothing leaves the machine. Chrome Relay lets a terminal agent operate your real Chrome. Claude Code, Codex, anything that runs shell commands. Not a headless automation browser with an empty profile. The actual browser you're signed into, with your cookies, SSO sessions, extensions, and localhost tabs. It works on background tabs without stealing focus, so you keep browsing while the agent works. Version 0.8 reaches multiple browser profiles and multiple supported browsers from the same CLI. With several connected, Chrome Relay makes the agent choose; it never silently guesses which browser to control. ```sh chrome-relay tabs # find or open a tab chrome-relay navigate "https://kushalsm.com" --new # opens in the background chrome-relay snapshot --tab 1234 -i # see the page: actionable elements get @refs chrome-relay click @e12 # act on a ref, no selectors, no --tab chrome-relay fill @e14 "hello" chrome-relay snapshot --tab 1234 -i # look again after the page changes ``` That's the whole loop. A snapshot of the Hacker News front page is ~14 KB of plain text. A click round-trip is ~150 ms. ## What it is, in one diagram ``` your agent (terminal) -> chrome-relay CLI -> instance registry (choose exactly one browser/profile) -> that profile's localhost host on an authenticated ephemeral port -> that profile's Chrome Relay extension -> CDP (chrome.debugger) -> your real tabs ``` Everything is local. There is no cloud relay, no account, no telemetry. The extension talks to one whitelisted local process through Chrome's own native-messaging permission system. The same Chrome you'd have to trust anyway. ## Where to go | You want to | Read | |---|---| | Install it | [Installation](/docs/installation/) | | Drive a page in 2 minutes | [Quickstart](/docs/quickstart/) | | Understand why it uses *your* Chrome | [Why your real Chrome](/docs/why-your-real-chrome/) | | Know how the pieces connect | [Architecture](/docs/architecture/) | | Read pages cheaply | [Snapshots](/docs/snapshots/) | | Click things reliably | [Refs](/docs/refs/) · [Clicking strategies](/docs/clicking/) | | Look up a command | [Command reference](/docs/commands/) | | Teach your agent | [The agent skill](/docs/skill/) | | Use several browsers or profiles | [Installation: browsers and profiles](/docs/installation/#multiple-browsers-and-profiles) | | Compare with agent-browser | [vs agent-browser](/docs/vs-agent-browser/) | ## Design positions, briefly - **CLI, not MCP.** Tool schemas cost thousands of context tokens on every agent turn; a CLI costs nothing until invoked, and every agent already knows how to run shell commands. - **Verbs, not magic.** There's no "smart click" with hidden fallbacks. Each addressing mode (`@ref`, CSS selector, coordinates) is explicit, and each failure is a structured error code the agent can branch on. - **Strict by default.** No silent partial success. If a click would land on an overlay instead of the target, you get `click_intercepted` naming the overlay. Not a success report and a broken page. - **Output is the product.** Agents pay for every byte on stdout. The snapshot format exists to be small. --- # Installation > Two pieces. The Chrome extension and the CLI. One command wires them together. Chrome Relay is two artifacts that meet in the middle: 1. **The extension** runs inside Chrome, holds the CDP session, and executes browser actions. 2. **The CLI** is what your agent invokes. It ships with the native messaging host that the extension connects to. ## 1. Install the extension From the [Chrome Web Store](https://chromewebstore.google.com/detail/chrome-relay/cpdiapbifblhlcpnmlmfpgfjlacebokb). Install it once in every browser profile you want your agent to reach. The primary supported browsers are: - **Google Chrome**, including several Chrome profiles at the same time. - **Dia**. - **Brave**. The installer also writes native-host manifests for detected Chrome Canary, Chromium, Edge, Vivaldi, Arc, and Opera installations. Those are compatibility targets; manifest detection does not by itself mean they have received the same end-to-end testing as Chrome, Dia, and Brave. Permissions it asks for, and why: | Permission | Used for | |---|---| | `nativeMessaging` | connecting to the local CLI host. This is the only channel out of the extension, and it goes to your own machine | | `debugger` | CDP: trusted clicks, snapshots, screenshots, network/console capture | | `tabs`, `tabGroups` | listing and targeting tabs, workspaces | | `storage` | ref-map persistence across extension service-worker restarts | ## 2. Install the CLI Needs [Node.js](https://nodejs.org) 18 or newer — the CLI ships as an npm package and works on macOS, Windows, and Linux. ```sh pnpm add -g chrome-relay # or npm i -g chrome-relay chrome-relay install ``` `install` detects every Chromium-family browser on the machine and registers the native messaging host for each. On Windows it also writes the HKCU registry entries Chrome reads, and drops a `run-host.cmd` wrapper under `%USERPROFILE%\.chrome-relay`. **Windows note:** if `chrome-relay` isn't recognized right after the global install, restart your terminal so `PATH` picks up the npm global bin (`%APPDATA%\npm`), then re-run `chrome-relay install`. ## 3. Verify ```sh chrome-relay doctor chrome-relay --version chrome-relay profile list ``` Doctor checks the whole chain: wrapper script, per-browser manifests, connected extension instances, local hosts, versions, and upload file access. Multi-browser/profile routing requires CLI and extension 0.8.0 or newer. If `chrome-relay --version` stays below 0.8 after installing the latest package, find the stale executable first (`which -a chrome-relay` on macOS/Linux; `where chrome-relay` on Windows). `profile list` is the authoritative answer to “which running browsers and profiles can my agent reach right now?” Then: ```sh chrome-relay tabs ``` If you get your window and tab list as JSON, you're done. ## Multiple browsers and profiles One CLI serves every supported browser/profile where the extension is installed. Each extension instance has a stable generated ID; Chrome does not expose profile names, so you can attach your own labels: ```sh chrome-relay profile list chrome-relay profile label work # one instance connected chrome-relay --profile 3f2a profile label personal # several: select by ID prefix chrome-relay --profile work tabs ``` With one connected instance, every command routes there and no flag is needed. With several, an unscoped command returns `profile_ambiguous` as a picker containing each label, browser, ID prefix, and exact `--profile` retry. Chrome Relay never guesses. Snapshot refs are profile-qualified (`@3f2a:e12`), so actions on a ref route themselves after the first snapshot. ## 4. Teach your agent (optional but recommended) If the CLI is installed, print the version-matched playbook: ```sh chrome-relay skills get core ``` For agents with a skills directory: ```sh npx -y skills add kiluazen/kstack@chrome-relay ``` The `-y` auto-confirms npx's "install the `skills` package?" prompt, so a headless agent doesn't hang on a TTY question. This installs the [agent skill](/docs/skill/). It teaches the snapshot/ref loop, which text tool to pick for which editor, the fallback ladder, and the failure modes. Also available raw at [/skill.md](/skill.md). ## Troubleshooting | Symptom | Fix | |---|---| | `extension_not_connected` | Install the extension, re-run `chrome-relay install`, then restart Chrome | | `doctor` says no wrapper | Run `chrome-relay install` | | Worked yesterday, dead today | Chrome may have restarted the native host. Retry once, then toggle the extension off/on | | `profile_ambiguous` | Pick one of the exact `--profile` choices in the error and rerun the command | | Browser missing from `profile list` | Install the extension in that exact browser/profile, then rerun `chrome-relay install` | | `profile` is an unknown command | Your PATH is resolving a pre-0.8 CLI; locate and remove the stale executable | | `unsupported_tool` on `snapshot` | Extension is older than the CLI. Update the extension from `chrome://extensions` | ## Updating ```sh chrome-relay update # updates the CLI and prints what changed, as JSON chrome-relay release-notes --since 0.5.22 # just read the changelog ``` The changelog is agent-readable on purpose. Your agent can check what changed before relying on a new flag. --- # Quickstart > The snapshot → ref → act loop, end to end, with real output. Everything an agent does with Chrome Relay is one loop: look at the page, act on what you saw, look again. This page runs it once, for real. ## Open a page in the background ```sh chrome-relay navigate "https://news.ycombinator.com" --new ``` ```json { "tabId": 460154464, "windowId": 460153891 } ``` `--new` opens a **background** tab — the user's focus is never stolen. Keep the `tabId`. ## Look at the page ```sh chrome-relay snapshot --tab 460154464 -i ``` ``` Page: Hacker News URL: https://news.ycombinator.com/ Tab: 460154464 - link "Hacker News" [ref=e4] - link "new" [ref=e5] - link "past" [ref=e7] - cell "1." [ref=e16] - cell "MiMo Code Is Now Released and Open-Source (xiaomi.com)" [ref=e18] - link "MiMo Code Is Now Released and Open-Source" [ref=e19] - link "xiaomi.com" [ref=e20] ... ``` `-i` prints only actionable elements. The whole front page is ~14 KB of this. Every `[ref=eN]` is a handle you can act on. ## Act on a ref ```sh chrome-relay click @e5 ``` ```json { "clicked": true, "x": 250, "y": 20, "ref": "e5", "tabId": 460154464 } ``` No `--tab`. No CSS selector. The ref knows which tab it came from — it acts there even if you (the human) are reading a different tab right now. ## Fill, type, press ```sh chrome-relay fill @e41 "search term" # inputs, textareas, selects — atomic write chrome-relay type "hello" -s @e23 # contenteditable / Lexical / ProseMirror chrome-relay keys Enter --tab 460154464 # chords: Cmd+K, Shift+ArrowDown, Escape ``` ## Look again After anything that changes the page, re-snapshot: ```sh chrome-relay snapshot --tab 460154464 -i ``` Refs survive same-page DOM churn (a toast appearing won't kill them — there's a heal step that re-finds moved nodes). They **die on navigation**, on purpose. If you act on a dead one: ```json { "relayError": { "code": "stale_ref", "message": "chrome_click_element: @e19 is not a known ref (...). Re-run `chrome-relay snapshot` and use a fresh ref." } } ``` The fix is always the same: snapshot again. See [Refs](/docs/refs/) for the full lifetime rules. ## Capture proof ```sh chrome-relay screenshot --tab 460154464 -o /tmp/evidence.png ``` Works on background tabs without activating them. ## The loop, named ``` snapshot -i → click/fill @ref → snapshot -i → ... ``` That's the whole interface. Three more things worth knowing before you go deep: - `js` runs arbitrary JavaScript in the page when the DOM doesn't expose what you need: `chrome-relay js --tab "return document.title"` - Errors are structured — branch on `relayError.code`, never regex the message. [Error reference](/docs/errors/). - For everything the CLI can do: [Command reference](/docs/commands/). --- # Why your real Chrome > Automation browsers give your agent a browser. Chrome Relay gives it yours — and that difference is the whole product. Every browser-automation tool for agents makes one of two choices: 1. **Launch a browser for the agent.** Playwright, Puppeteer, agent-browser — they download or spawn a separate Chrome (usually Chrome for Testing), drive it over CDP, and give the agent a clean, disposable sandbox. 2. **Attach to the browser you already use.** That's Chrome Relay. Neither is wrong. They're different products. The sandbox is right for e2e tests, scraping, CI. But the moment the task involves *your* accounts, the sandbox has nothing to offer: ## What only exists in your real browser **Your logins.** Here's a live test from our own benchmarking: we opened `news.ycombinator.com` in an automation browser and through Chrome Relay, same machine, same minute. The automation browser saw the logged-out page — a `login` link. Chrome Relay saw a logged-in session: username in the header, no login link anywhere. Same URL, two different realities. Every dashboard, admin panel, inbox, and internal tool your agent might operate lives exclusively in the second one. **Cloudflare and friends.** Chrome for Testing is fingerprinted by anti-bot systems and routinely blocked or challenged. Your real Chrome is indistinguishable from you, because it *is* you. There's no stealth plugin arms race when there's nothing to disguise. **Your extensions.** Password manager, ad blocker, that internal company extension — present and working. **localhost.** The dev server you're running, with whatever auth state your browser already negotiated with it. **No credential handoff.** The alternative way to get an agent "logged in" is exporting cookies or storing credentials in a vault it can replay. With Chrome Relay there's nothing to export — the session never leaves the browser it lives in. ## The objections, honestly **"Isn't attaching to my real browser dangerous?"** It's a real trade and you should make it consciously. Two things bound the risk: - *Scope.* Everything is local: CLI → `127.0.0.1` → native messaging → extension. The bridge binds to localhost only and rejects browser-origin requests, so a web page can't call it. There's no cloud relay that could be compromised, no account that could leak. - *Visibility.* The agent acts in tabs you can see (or background tabs you can switch to). Actions are auditable in your own browser history and the agent's transcript. Contrast with a sandbox browser doing things in a window you'll never look at. What it doesn't bound: an agent with your sessions can do what you can do. That's the point, and it's why Chrome Relay favors strict, loud failure over silent cleverness — structured error codes, no auto-fallbacks, refusal to click through overlays (`click_intercepted`). You should know what your agent did and why. **"Why not just launch Chrome with `--remote-debugging-port`?"** Because that requires *relaunching* Chrome (killing your session), and it exposes the raw debugging protocol — every tab, every cookie — to any process on the machine that can reach the port. The extension + native-messaging route attaches to the already-running browser, with Chrome's own permission model in front of it, scoped to one whitelisted local binary. [Architecture](/docs/architecture/) covers this in detail. **"Won't the agent steal my focus while I work?"** No — that constraint shaped the whole design. Commands run on background tabs via CDP; the extension even patches `document.visibilityState` so pages keep behaving normally while backgrounded. You browse in one tab while the agent works in another. Multiple agents can drive separate named windows ([workspaces](/docs/workspaces/)). ## When you should NOT use Chrome Relay Honesty section. Use a sandbox browser instead when: - You're running **CI or e2e tests** — you want determinism and a clean profile, not your cookies. - You're **scraping at volume** — you want parallel disposable sessions. - The task needs **no auth at all** — the sandbox is simpler and can't touch anything of yours. - You can't supervise it and the blast radius of "the agent can do what I can do" is unacceptable. The two-line summary we'd want an agent to remember: > agent-browser (and Playwright) are what your agent uses to browse *the web*. > Chrome Relay is what your agent uses to operate *your* web — logged-in, real-profile, without taking the browser away from you. --- # Architecture > Five hops, each with one job. Why the extension route is the only safe way into a running Chrome. ``` agent (any shell) → chrome-relay CLI parses args, renders output → instance registry discovers and verifies connected profiles → localhost HTTP authenticated ephemeral port for one profile → native messaging host one process spawned by that extension instance → extension service worker that browser/profile's tools, refs, buffers → chrome.debugger (CDP) trusted input, snapshots, capture → your tabs ``` Every hop is local. The wire format is one JSON envelope per call: `{ name, args }` in, `{ ok, data } | { ok: false, errorDetails }` out. Each browser profile running the extension spawns its own native host. The host registers an authenticated ephemeral localhost port under `~/.chrome-relay/instances/`; every CLI call verifies those descriptors, resolves exactly one profile, and sends the command there. With one connected profile this is invisible. With several, `--profile ` chooses one and profile-qualified refs such as `@3f2a:e12` route themselves. Port `12122` exists only as a legacy compatibility path for pre-0.8 installations. ## Why not `--remote-debugging-port`? The standard way tools attach to a real Chrome is launching it with a CDP debug port. Two problems: 1. **You have to relaunch Chrome.** Your running session — every tab, every WebSocket, every half-written form — dies so the flag can take effect. 2. **The port is a skeleton key.** Anything on your machine that can reach `localhost:9222` gets the full protocol: read any cookie, open any URL, in every profile window. No permission prompt, no scoping. The extension + native-messaging route inverts both. It attaches to the *already running* browser, and the only process that can talk to the extension is the one binary whitelisted in the native-messaging manifest — a file Chrome itself verifies. The browser's own permission model is the gate. The cost is more moving parts (install registers the host; the extension must be present). `chrome-relay doctor` exists because of that cost. ## What lives where | Piece | Owns | |---|---| | **CLI** (`chrome-relay`, npm) | argument parsing, profile discovery/routing, output rendering (snapshot text, JSON), exit codes. Stateless — every invocation resolves one host and makes one HTTP call. | | **Protocol** (bundled into the CLI) | tool names, argument schemas, error codes, the snapshot renderer. One source of truth both ends import. | | **Native host** | spawned once per extension instance on `connectNative()`; registers its authenticated localhost endpoint and relays frames. | | **Extension** | everything that touches the browser: the [ref map](/docs/refs/), the [snapshot builder](/docs/snapshots/), per-tab console/network ring buffers, CDP sessions, trusted input dispatch. | ## Security model - **Localhost only.** The bridge binds `127.0.0.1`. It also rejects requests carrying a browser `Origin` header, so a web page you visit can't script your own bridge. - **No cloud, no account, no telemetry.** There is nothing to sign into and nowhere for your data to go. The extension's only outbound channel is native messaging to your own machine. - **Chrome-verified pairing.** The native-messaging manifest whitelists exact extension IDs; Chrome enforces it in both directions. - **One trust decision.** Installing Chrome Relay means: processes on my machine that can run `chrome-relay` may drive my browser. That's the honest statement of the model — the same trust you extend to anything you `npm i -g`. If your threat model can't accept it, use a sandboxed browser instead ([when not to use this](/docs/why-your-real-chrome/)). ## Trusted input, specifically Clicks and keys go through CDP `Input.dispatchMouseEvent` / `dispatchKeyEvent`, so the page receives `isTrusted: true` events with `pointerType: "mouse"` — indistinguishable from a human. This matters on real apps: React-Aria/Radix widgets listen for pointer events and ignore synthetic `.click()`; login and payment flows gate on trusted events. JS-side `.click()` is available via `chrome-relay js` when you want it, but it's the fallback, not the path. ## Background tabs that act foregrounded Chrome throttles and hides backgrounded tabs (`document.hidden`, paused rAF, visibility events). The extension patches visibility APIs on relay-driven tabs so pages behave normally while the agent works and you keep your focus elsewhere. The one hard limit: Chrome doesn't *paint* background tabs, so `screencast` (video capture) needs the tab active — screenshots don't. --- # Snapshots > One command turns a page into ~1–15 KB of text where every actionable element has a handle. ```sh chrome-relay snapshot --tab -i ``` ``` Page: Hacker News URL: https://news.ycombinator.com/ Tab: 460154464 - link "Hacker News" [ref=e4] - textbox "Search" [ref=e41]: current value - checkbox "Remember me" [checked, ref=e42] - cell "1." [ref=e16] - link "MiMo Code Is Now Released and Open-Source" [ref=e19] - clickable "Open card" [ref=e88] ``` Agents pay context tokens for every byte a tool prints, so the snapshot format exists to be small. Measured: the HN front page is **14 KB** with `-i` (the DOM is 599 KB; the old JSON reader was 61 KB). A typical app view is 1–5 KB. ## The grammar One line per node, indented to tree depth: ``` - {role} "{name}" [{attrs}]: {value} ``` - `role` — from the accessibility tree: `link`, `button`, `textbox`, `heading`, `cell`… - `name` — what a screen reader would announce. Quoted, escaped. - attrs, in order: `level=N`, `checked`/`checked=false`/`checked=mixed`, `expanded`, `selected`, `disabled`, `required`, `readonly`, `pressed`, `ref=eN`, `url=…` (with `-u`) - `: value` — current input value, only when it differs from the name. A form readback is free with the snapshot; no second round-trip to ask "what does the field say now?" ## What gets a ref Refs are handles you can [act on](/docs/refs/). Three kinds of node earn one: 1. **Interactive roles** — button, link, textbox, checkbox, combobox, menuitem, tab, … always, even unnamed. 2. **Named content roles** — heading, cell, listitem, img, dialog … only when they have a name (an anonymous heading is noise). 3. **Cursor-interactive sweep finds** — see below. Anonymous structural nodes (layout divs, unnamed containers) collapse: their children promote up a level. That's most of why the output is small. ## The sweep — catching div-soup The accessibility tree only knows what page authors told it. Modern Tailwind-style SPAs are full of clickable `
`s and ``s with no role: measured on a real app, one email-list view had **37 clickable elements the AX tree couldn't see at all** (cursor-pointer rows, dates, sender names). A well-built dashboard (Cloudflare's, measured) had zero. So every snapshot also runs a one-pass sweep for elements that *behave* clickable — `cursor: pointer`, `onclick`, `tabindex`, `contenteditable` — deduplicated to the topmost clickable (cursor inherits; you want the row, not its twelve children). They appear as: ``` - clickable "sylvain@zerolooplabs.dev" [ref=e704] ``` Same refs, same actions. Without the sweep, "interactive only" filtering would silently hide real targets — that was the dealbreaker that decided this design. ## Flags ```sh chrome-relay snapshot --tab # full tree chrome-relay snapshot --tab -i # ref-bearing elements only — for ACTING chrome-relay snapshot --tab -d 3 # cap depth chrome-relay snapshot --tab -s "#main" # scope to a CSS subtree — refs outside it are never issued chrome-relay snapshot --tab -u # include link hrefs as url= attrs chrome-relay snapshot --tab --json # structured envelope: { title, url, tabId, nodes, refs } chrome-relay snapshot --tab --diff # only what changed since this tab's last snapshot ``` **`-i` is for acting, not fact extraction.** It deliberately drops non-interactive content — measured live on a Cloudflare Pages metrics page, all the metric values vanished under `-i` while every button survived. To *read* facts, use the full `snapshot`, [`get text`](/docs/commands/#get--one-value-no-snapshot), or a `js` projection. Action loops and fact reads are different jobs; pick the mode for the job. `--diff` attacks the re-snapshot tax directly: after an action, print the handful of changed lines (~100 tokens) instead of the whole page. A full snapshot is still taken and the ref map still refreshes — refs in the diff are current and clickable. Use consistent flags between snapshots (`-i` vs full) or the diff gets noisy. `--json` includes the full refs map — each ref's `backendNodeId`, role, name — for programmatic callers. The text form is for reading; the JSON form is for building on. ## Frames and shadow DOM - **Shadow DOM: pierced.** The accessibility tree sees through shadow roots, so web-component internals get refs that `querySelector` could never reach. (Pinned by an e2e test: a `` inside a shadow root gets a ref, and `fill @ref` writes through the boundary.) - **Same-process iframes: included** — they appear in the tab's AX tree. - **Cross-origin iframes: not yet.** Snapshot is top-frame scoped for OOPIFs. Coordinate clicks still land in them (input is dispatched at page level); ref/selector addressing doesn't. Stated limitation, on the roadmap. ## Deprecated: `read` and `ax` Earlier versions had two separate readers (`read` = DOM walk with CSS selectors, `ax` = accessibility tree with its own id space). Both are now aliases for `snapshot` and print a deprecation notice; they'll be removed. If you have scripts parsing the old shapes, switch to `snapshot --json`. --- # Refs > @eN handles that carry their own tab, heal across DOM churn, and refuse to click through overlays. Every actionable element in a [snapshot](/docs/snapshots/) gets a ref — `@e12` — and every element-addressed command accepts one: ```sh chrome-relay click @e12 chrome-relay fill @e14 "value" chrome-relay hover @e3 chrome-relay type "text" -s @e7 ``` A ref is not a CSS selector and not a coordinate. Under the hood it's the element's Chrome-internal node id (`backendDOMNodeId`) plus enough semantics (`role`, `name`, position-among-twins) to find the element again if the id goes stale. ## Refs carry their own tab This is the rule that makes refs safe in a browser a *human* is also using. `click @e12` acts on the tab that produced `e12` — never "the active tab", which is whatever you happen to be reading right now. Concretely: - No `--tab` needed on ref actions. - Ref numbers are unique across the whole browser (the counter never resets), so `@e12` can't mean two things. - A contradicting `--tab` is an error, not a guess: ```json { "relayError": { "code": "target_conflict", "message": "@e16 belongs to tab 460154464 but --tab 99999 was passed. Refs carry their own tab — drop --tab or use a ref from that tab's snapshot." } } ``` ## Lifetime: churn heals, navigation kills **Same-page DOM churn: refs survive.** Pages re-render constantly — a toast appears, a list reorders, React swaps a subtree. The resolver tries the cached node id first (one CDP call verifies it's live), and if the node was replaced, re-finds it by role + name + position and *heals* the ref in place. Healed actions report it: ```json { "clicked": true, "ref": "e16", "healed": true } ``` **Real navigation: refs die, on purpose.** Chrome reuses node-id integers in a new document — a ref from the old page could silently resolve to an unrelated element on the new one. (We caught this live: a stale ref "successfully" clicked a random link on the page that replaced it.) So navigation invalidates the tab's refs, and acting on one fails loud: ```json { "relayError": { "code": "stale_ref", "message": "@e19 is not a known ref (no snapshot produced it, or its tab was re-snapshotted). Re-run `chrome-relay snapshot` and use a fresh ref." } } ``` SPA route changes (pushState — the document persists) keep refs alive; the heal step covers the re-render. The agent rule is one line: **`stale_ref` → snapshot again.** Never retry the same ref; it isn't coming back. ## Interception: no clicking through overlays Before dispatching, a ref click hit-tests its target point. If an unrelated element owns it — a modal, cookie banner, sticky header — the click is refused, naming the interceptor: ```json { "relayError": { "code": "click_intercepted", "message": "@e8 resolved, but an unrelated
owns the click point (412, 96) — an overlay, sticky header, or modal is covering it. Dismiss it or scroll, then retry.", "details": { "interceptor": { "nodeName": "DIV", "backendNodeId": 1042 } } } } ``` Without this check, the click "succeeds", the overlay eats it, and the agent's next snapshot makes no sense. The check allows same-lineage hits (a button's inner text, a label wrapping its input) and is best-effort — if the hit-test itself can't run, the click proceeds rather than false-positive. `fill` and `type` skip the check deliberately: they write via the node, not the pointer, and writing into a visually covered input is legitimate. ## Service-worker restarts Chrome kills idle extension service workers whenever it likes. The ref map persists across restarts (and is re-verified through CDP before any action — persistence restores ids, not trust), so an agent's refs don't vanish mid-task because Chrome got tidy. ## When refs aren't the tool Refs cover the DOM. Two things aren't in the DOM in any useful way: - **Canvas internals** (Figma-style UIs) — nothing to reference. - **SVG chart segments** — anonymous `` elements; no role, no name, no sane selector either. For those, get coordinates from a `js` probe or a screenshot and use `click --x N --y N`. The full decision tree is in [Clicking strategies](/docs/clicking/). --- # Clicking strategies > Three addressing modes, each with a distinct failure mode. Pick by what the page gives you. There is no single "click" because there is no single way to identify an element across the web. Chrome Relay exposes the strategies as explicit modes instead of hiding them behind a smart wrapper — when something fails, your transcript already contains the diagnosis. ## The ladder | You have | Use | |---|---| | A `@ref` from `snapshot -i` | `chrome-relay click @e12` — default, start here | | A CSS selector you know statically | `chrome-relay click 'button[aria-label="Save"]'` | | Pixel coordinates | `chrome-relay click --x 540 --y 320` | | Only visible text | `js` probe → coordinate click (recipe below) | All three dispatch the same trusted CDP mouse sequence — hover, press, release, `pointerType: "mouse"`, `isTrusted: true`. Anti-bot heuristics and React-Aria/Radix widgets can't tell them from a human. ## 1. Refs — `click @e12` The default. Covers buttons, links, inputs, named content, cursor-pointer div-soup (via the [sweep](/docs/snapshots/)), and shadow DOM. Carries its own tab, heals across re-renders, refuses to click through overlays. Full semantics: [Refs](/docs/refs/). Fails on: real navigation (`stale_ref` → re-snapshot), covered targets (`click_intercepted` → dismiss the overlay), and anything not in the DOM. ## 2. CSS selectors — `click ''` ```sh chrome-relay click 'button[aria-label="Save"]' --tab 42 ``` For when you already know a stable selector and don't need to look at the page first. Resolution is in-page `querySelector` → scroll into view → center → trusted click. Fails on: hash-rotated class names (`._3a8K9d`), shadow DOM (querySelector can't pierce; refs can), and selectors you'd have to guess. Failure is `element_not_found` with the selector echoed; malformed CSS is `invalid_arguments`. ## 3. Coordinates — `click --x N --y N` ```sh chrome-relay js --tab 42 "const r = document.querySelector('svg path').getBoundingClientRect(); return { x: r.x + r.width/2, y: r.y + r.height/2 }" chrome-relay click --tab 42 --x 312 --y 218 ``` The escape hatch for things with no DOM handle at all: canvas UIs and SVG chart internals (anonymous `` segments have no role, no name, and no sane selector — measured 70–100 of them per analytics dashboard). Get fresh coordinates from a `js` rect probe or a screenshot, click immediately — coords go stale the moment the page scrolls. The contract is deliberately thin: "fire a click at (x, y)". It always reports `clicked: true`; whether it hit something useful is yours to verify. ## Recipe: click by visible text First try `snapshot -i` — visible text is almost always a node name with a ref next to it. When it genuinely isn't: ```sh COORDS=$(chrome-relay js --tab 42 " const target = 'the text you see'; const tw = document.createTreeWalker(document.body, NodeFilter.SHOW_TEXT); let n; while ((n = tw.nextNode())) { if ((n.textContent || '').includes(target)) { const r = n.parentElement.getBoundingClientRect(); return { x: r.x + r.width / 2, y: r.y + r.height / 2 }; } } return null; ") # parse x/y, then: chrome-relay click --tab 42 --x $X --y $Y ``` There's intentionally no `click-text` verb — it would be a smart wrapper hiding which strategy ran. ## Anti-patterns - **Retrying a stale ref.** `stale_ref` means the page changed; the ref is never coming back. One snapshot fixes it. - **`js` + `.click()` on clickable elements.** Synthetic event, `isTrusted: false` — silently ignored by anti-bot pages and pointer-event widgets. Use `js` to *find*, trusted verbs to *act*. - **Coordinate clicks from old data.** Re-read the rect right before clicking, every time. - **Strategy roulette.** Looping selector → ref → coords until something returns success burns round-trips and eventually clicks the wrong thing. Look at the page (`snapshot -i`), pick once, fail loudly. ## Typing, while you're here | Target | Tool | |---|---| | `` / `