Computer use
Drive local desktop apps from an agent via accessibility trees, screenshots, and safe UI actions.
The nethur computer CLI lets an agent inspect and control native desktop apps — list running apps, read accessibility trees, click controls, set values, type text, scroll, and take screenshots. Use it when a task needs to operate the OS or a third-party app rather than a terminal or the built-in browser.
First-time setup
Check the runtime and permissions:
nethur status --json
nethur computer permissions --json
nethur computer capabilities --jsonIf permissions reports anything missing, grant Accessibility (and Screen Recording on macOS) to Nethür Computer Use in System Settings, then re-run permissions --json to confirm.
Snapshot → act → snapshot
Every interaction follows the same loop: read the app's current state, act on a specific element, then re-read state to verify the result.
nethur computer list-apps --json
nethur computer get-app-state --app com.spotify.client --json
nethur computer click --app com.spotify.client --element-index 42 --jsonElement indexes are scoped to the latest get-app-state result and may be sparse. In --json output, read the tree from result.snapshot.treeText. Do not invent indexes from elementCount. Refresh state after navigation, focus changes, scrolling, or any app re-render before reusing an index.
Multi-window apps
nethur computer list-windows --app com.microsoft.edgemac --json
nethur computer get-app-state --app com.microsoft.edgemac --window-id <id> --json
nethur computer click --app com.microsoft.edgemac --window-id <id> --element-index 12 --jsonPrefer stable --window-id when the listed id is not none; otherwise use --window-index.
Selecting an app
Prefer bundle IDs returned by list-apps:
nethur computer get-app-state --app com.microsoft.edgemac --jsonApp names work when unambiguous (--app Spotify). Use --app pid:<number> only when bundle ID and name both collide.
Available actions
nethur computer click --app <app> --element-index <i> --json
nethur computer set-value --app <app> --element-index <i> --value "text" --json
nethur computer type-text --app <app> --text "text" --json
nethur computer press-key --app <app> --key Return --json
nethur computer hotkey --app <app> --key CmdOrCtrl+A --json
nethur computer paste-text --app <app> --text "text" --json
nethur computer scroll --app <app> --element-index <i> --direction down --json
nethur computer drag --app <app> --from-x 100 --from-y 100 --to-x 300 --to-y 300 --json
nethur computer perform-secondary-action --app <app> --element-index <i> --action <name> --jsonPrefer semantic actions (click, set-value, perform-secondary-action) over raw type-text or press-key — they target accessibility elements directly and survive focus changes that keyboard input doesn't.
When accessibility targeting fails, fall back carefully to coordinates:
nethur computer click --app com.apple.Safari --x 120 --y 340 --json
nethur computer drag --app <app> --from-element-index 3 --to-element-index 9 --jsonSensitive input
Pass secrets through stdin so they don't land in shell history:
printf '%s' "$TEXT" | nethur computer set-value \
--app com.apple.Safari --element-index 7 --value-stdin --json--text-stdin works the same way for type-text and paste-text.
On Linux and Windows, action payloads also pass briefly through a local operation file while the helper runs. Stdin keeps secrets out of shell history, but it does not hide them from every local observer; avoid sending secrets unless the task explicitly requires it.
Screenshots
get-app-state returns an accessibility tree and, by default, a screenshot. With --json, the image bytes are written to disk and the path is returned in screenshot.path rather than embedded in the response. Pass --no-screenshot when pixels aren't needed (faster, smaller payload). Pass --restore-window to bring a hidden or minimized window into view before capture.
Use it from an agent
The shipped computer-use skill packages the same command surface with safety guidance. Install it into the agent's skill directory:
nethur skills install --skill computer-useSee Skills registry & MCP for how skills are picked up.
Next steps
- Nethür CLI overview — the rest of the CLI surface (worktrees, terminal, browser).
- Skills registry & MCP — distributing this CLI to agents.