Window state tools
Screenshots, accessibility trees, verification and visual regions of a window.
Screenshots, accessibility trees, verification and visual regions of a window.
| Tool | Description | Platforms |
|---|---|---|
get_window_state | Snapshot a window: the accessibility tree plus a screenshot. | macOS, Linux, Windows |
get_accessibility_tree | Return a lightweight snapshot of the desktop: running regular apps and on-screen visible windows with their bounds, z-order, and owner pid. | macOS, Linux, Windows |
verify_state | Deterministically verify bounded predicates against one exact window. | macOS, Linux, Windows |
parse_visual_regions | Parse one immutable registered capture into model-neutral text and icon regions. | macOS, Linux, Windows |
Served by cua-driver mcp; see MCP tools for every tool.
get_window_state#Effect: read-only. Platforms: macOS, Linux, Windows.
Snapshot a window: the accessibility tree plus a screenshot. By default the tree is ONE compact Markdown rendering (tree_format:"markdown"): a row [N] is addressed with element_token <snapshot_id>:N, where snapshot_id is in the response header and in structuredContent. tree_format:"elements" returns the structured elements array instead (element_index, element_token, role, label, value, actions, frame, parent_index, depth); "both" returns both (about twice the size). Pass the token to click, type_text, press_key, etc.
START NARROW: a read walks at most 250 nodes by default and says Tree truncated at max_elements=… (truncated:true) when it stops; raise max_elements, or use query (case-insensitive substring; matching rows plus ancestors) and max_depth. Indices and tokens stay valid for the whole snapshot. verbose:true adds _note and the full background_input report; full_output:true restores the previous full response (both representations, all metadata, ≤2 000 nodes).
DIFF READS (use them for every re-read after acting): since:"latest", or since:<snapshot_id> from an earlier read of the same window, returns only what changed: + added, ~ changed, - removed rows (removed ids are the old snapshot's), a reindexed: line if indices shifted, or no change since …; focused element is …. The response carries a NEW snapshot_id: use it in tokens. An unknown, expired, other-window or differently-scoped (query/max_elements/max_depth) since falls back to a full read; since_status says why.
A new snapshot of the same (pid, window_id) replaces the previous one and stales its tokens (invalidated_snapshot_ids). include_screenshot:false returns the tree only and keeps the last screenshot's pixel frame while the window keeps its size; include_accessibility_tree:false returns only the screenshot and keeps the current snapshot's rows and tokens valid (use it, not max_elements:1, when you just need to look); adding display_only:true (live previews) also leaves the pixel frame alone. The tree can lie on some surfaces (Electron, Catalyst, virtualized rows with h:1 frames): cross-check the screenshot, and use pixel x,y at action time only for elements missing from the tree.
Refusals: window_id_not_found; window_owner_pid_mismatch names the real owner_pid (sandboxed Open/Save panels belong to a panel service). degraded_reason ax_window_unresolved or ax_app_launching (the latter with truncation_reason: app_lookup_timeout) means an empty tree: re-snapshot, or act with delivery_mode:"foreground". px_frame_mismatch / px_capture_unavailable omit the unprovable screenshot; the AX payload stays valid.
macOS parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
include_accessibility_tree | boolean | Default true. False skips the AX walk and returns the screenshot plus window metadata. Both this and include_screenshot false is an error. | |
include_screenshot | boolean | Default true. False returns the tree only (saves image tokens and latency); screenshot_out_file still forces a capture. | |
max_depth | integer | Cap on walk depth. Default 25. Minimum: 1. | |
max_dimension | integer | Optional cap on the returned screenshot's long edge, in pixels (aspect ratio preserved): the cheap path for a small preview / thumbnail. Applied on top of the session/global max_image_dimension ceiling; the tighter of the two wins. Omit for the configured default. Minimum: 1. | |
max_elements | integer | Cap on AX nodes walked (depth-first; markdown and elements truncate together). Default 250 (2 000 with full_output:true); the response states when the tree was cut. Minimum: 1. | |
max_image_dimension | integer | Per-call override for the returned screenshot's long edge in pixels. An explicit value wins over the session/global setting; 0 returns native resolution. Omit to preserve configured behavior. Minimum: 0. | |
pid | integer | required | Target process ID. |
query | string | Case-insensitive substring filter: returns matching rows plus ancestors; element_index values are not renumbered. Try this before a full read. | |
screenshot_out_file | string | When set, write the PNG to this file path (~ expanded) instead of embedding base64 in the response. The structured output will contain screenshot_file_path instead. | |
session | string | For multi-call work, prefer a short public session label and repeat it on every call that accepts it. | |
window_id | integer | required | Target window ID from list_windows. |
Parameters on every platform
| Parameter | Type | Default | Description |
|---|---|---|---|
capture_mode | "ax" | "vision" | DEPRECATED and ignored. get_window_state always returns BOTH the element tree and a screenshot: ground on both. The modality is chosen at action time by how you address the target: an element ax action (element_token) or an element px action (x,y). Any value (including the old "som"/"screenshot" aliases) is accepted but has no effect. | |
display_only | boolean | Default false. With include_accessibility_tree:false, returns pixels for display without registering an action capture or replacing the agent snapshot. The image cannot ground input actions. | |
full_output | boolean | Default false. Set true to restore the previous full response: both elements and tree_markdown, all metadata, and the platform's own walk limits unless max_elements / max_depth are passed. Does not combine with since. | |
since | string | A snapshot_id from an earlier get_window_state of the same window, or "latest" for this window's most recent read with the same query/max_elements/max_depth. Use it on every re-read after acting. Returns only what changed against that snapshot: + added rows, ~ changed rows, - removed rows (removed ids are the old snapshot's), or no change. Falls back to a full read when the snapshot is unknown, expired, belongs to another window, or was read with a different query/max_elements/max_depth; since_status says which. The response still carries a NEW snapshot_id and its element_tokens; rows not listed keep their previous [N] unless a reindexed: line says otherwise. | |
timeout_ms | integer | 1000 | Wall-clock budget in milliseconds for the accessibility-tree walk (default 1000, min 100, max 120000). Bounds the WHOLE walk. When the budget runs out the tool returns the PARTIAL tree it has, flagged with truncated: true, truncation_reason, nodes_visited, nodes_pending and elements_complete: false; retry with a larger value (e.g. 5000) or narrow with query / max_depth. Range: 100 to 120000. |
tree_format | "markdown" | "elements" | "both" | Which tree representation to return. Default "markdown": the compact tree_markdown only, where a row [N] is addressed with element_token <snapshot_id>:N. "elements": the structured elements array only (per-element element_token, frame, parent_index, value). "both": the two together, as before; roughly doubles the size. | |
verbose | boolean | Default false. Set true to include the _note string and the full background_input capability report (macOS), which are omitted by default to save context. |
Example arguments
{"pid":1,"window_id":1}get_accessibility_tree#Effect: read-only, idempotent. Platforms: macOS, Linux, Windows.
Return a lightweight snapshot of the desktop: running regular apps and on-screen visible windows with their bounds, z-order, and owner pid.
For the full AX subtree of a single window (with interactive element indices you can click by), use get_window_state instead: that's the heavy per-window tool. This one is a fast discovery read that needs no TCC grants.
Parameters on every platform
| Parameter | Type | Default | Description |
|---|---|---|---|
session | string | For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. |
verify_state#Deterministically verify bounded predicates against one exact window. The driver evaluates structured window/accessibility state and may return the final screenshot as uninterpreted visual evidence for a multimodal caller. Predicate results are satisfied, unsatisfied, or unknown; unknown never implies success. Accessibility projections are conservative: absence remains unknown unless the observed search domain is proven exhaustive.
Effect: read-only. Platforms: macOS, Linux, Windows.
| Parameter | Type | Default | Description |
|---|---|---|---|
expect | object[] | required | One to eight predicates, combined with logical AND. Items: 1 to 8. |
include_screenshot | boolean | Return the final window screenshot as image content for a multimodal caller. The driver does not interpret that image. | |
pid | integer | required | Exact process whose window may be observed. Minimum: 1. |
session | string | For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. This field never selects capture modality or authorization. | |
stable_samples | integer | 2 | Consecutive satisfied samples required before returning success. Range: 1 to 5. |
timeout_ms | integer | 5000 | Bounded wait. Zero performs one sample. Range: 0 to 10000. |
window_id | integer | required | Exact native window identifier. |
Example arguments
{"pid":1,"window_id":1,"expect":[{}]}parse_visual_regions#Parse one immutable registered capture into model-neutral text and icon regions.
Effect: read-only, idempotent. Platforms: macOS, Linux, Windows.
| Parameter | Type | Default | Description |
|---|---|---|---|
capture_id | string | required | capture_id of a screenshot returned by get_window_state or get_desktop_state. |
options | object | {} | Optional, model-neutral controls for one bounded parse. |
options.kinds | "text" | "icon"[] | Region kinds to return (text, icon). Omit for all kinds. Items: 1 to 2. | |
options.max_regions | integer | Return at most this many regions. Minimum: 1. | |
options.min_confidence | number | Drop regions below this confidence (0 to 1). Range: 0 to 1. | |
session | string | For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. |
Example arguments
{"capture_id":"<capture_id>"}