Peekaboo

Full macOS UI automation CLI: capture/inspect screens, target UI elements, drive input, and manage apps/windows/menus.

Peekaboo is a macOS-native CLI and MCP server by Peter Steinberger (steipete) that gives AI agents eyes and hands on your Mac. It captures high-fidelity screenshots of applications or the entire system, inspects UI element trees via the Accessibility API, drives mouse/keyboard input to interact with any application, and manages windows, apps, and menus — all from the command line. The architecture is built on macOS Accessibility APIs (the same ones VoiceOver uses), which means Peekaboo can see and interact with every UI element that's accessible — buttons, text fields, menus, toolbars, tables, and more. Combined with screenshot capture, this gives AI agents a complete picture: visual (what it looks like) and structural (what the UI elements are and how to interact with them). Key capabilities include: targeted screenshots (specific app, window, or screen region), UI element tree inspection with role/label/value attributes, mouse clicks and keyboard input simulation, window management (move, resize, focus, minimize), app lifecycle control (launch, quit, activate), and menu bar navigation. Every command supports JSON output for programmatic consumption. Peekaboo also functions as an MCP server, making it compatible with Claude Desktop, Cursor, and other MCP-enabled tools. The MCP interface exposes the same capabilities as the CLI, letting any MCP client leverage macOS UI automation. For OpenClaw users, Peekaboo is transformative. It's what lets your AI assistant 'see' your screen and 'use' your Mac. 'What's on my screen right now?', 'Click the Submit button in Safari', 'Read the text in that dialog box' — these become real capabilities. Combined with iPhone Mirroring on macOS Sequoia, you can even control your iPhone through Peekaboo. The tool requires two macOS permissions: Screen Recording (for screenshots) and Accessibility (for UI interaction). Both are granted through System Settings > Privacy & Security. Best suited for: AI agents needing visual awareness of the Mac desktop, macOS GUI automation without AppleScript complexity, testing and QA automation, anyone wanting their AI to see and interact with native Mac apps.

Tags: macos, ui-automation, screenshots, accessibility

Category: Automation

Use Cases

  • AI visual awareness: 'What's on my screen right now?'
  • GUI automation: click buttons, fill forms, navigate menus in any Mac app
  • iPhone control via iPhone Mirroring on macOS Sequoia
  • Automated testing: drive UI interactions and verify with screenshots
  • Screen monitoring: periodic capture for activity tracking or status checks
  • Accessibility auditing: inspect UI element trees for accessibility compliance
  • Window management: resize, move, and organize windows via AI commands

Tips

  • Grant both permissions proactively: Screen Recording AND Accessibility for your terminal app
  • Use `peekaboo capture --app Safari` to screenshot a specific app without capturing the full screen
  • Use `peekaboo inspect --app Finder` to see the full accessibility tree of an application
  • Target UI elements by accessibility label for the most reliable automation
  • Combine with iPhone Mirroring (macOS Sequoia) to control your iPhone through Peekaboo
  • Use JSON output for all automation workflows: `peekaboo inspect --app Notes --json`
  • For testing workflows, capture before and after screenshots to verify UI state changes
  • The MCP server mode works great with Claude Desktop and Cursor for visual AI workflows

Known Issues & Gotchas

  • macOS only — uses Accessibility APIs and CoreGraphics that are macOS-specific
  • Requires Screen Recording permission (System Settings → Privacy & Security → Screen Recording)
  • Requires Accessibility permission (System Settings → Privacy & Security → Accessibility)
  • Some apps (especially Electron-based) have poor accessibility tree support — UI inspection may be limited
  • Full-screen apps may require different capture approaches than windowed apps
  • Mouse/keyboard simulation works at the OS level — it physically moves the cursor and types
  • UI element targeting can be brittle if app layouts change — use accessibility labels when possible
  • Performance: full UI tree inspection of complex apps (like Xcode) can take a few seconds

Alternatives

  • AppleScript / osascript
  • Hammerspoon
  • cliclick
  • Playwright (browser)
  • OpenClaw Browser tool

Community Feedback

Peekaboo brings high-fidelity screen capture, AI analysis, and end-to-end GUI automation to macOS — as a native CLI and an MCP server.

— Peekaboo Website

OpenClaw can use the Peekaboo skill to see what is on screen and then use AppleScript or accessibility controls to drive interaction. It's like giving the AI eyes and hands.

— Reddit r/AI_Agents

Multiple backends: macOS native (Peekaboo), Rock 5B, PiKVM, NanoKVM. Regardless of which backend is connected, the AI agent sees the same set of capabilities.

— Reddit r/clawdbot

Peekaboo is a full macOS UI automation CLI: capture/inspect screens, target UI elements, drive input, and manage apps/windows/menus. Accessibility-first approach.

— AgentSkills.so

Configuration Examples

Install and grant permissions

# Install via Homebrew
brew install steipete/tap/peekaboo

# Grant permissions:
# 1. System Settings → Privacy & Security → Screen Recording → add Terminal
# 2. System Settings → Privacy & Security → Accessibility → add Terminal

Screenshot and inspect

# Capture full screen
peekaboo capture --output screenshot.png

# Capture a specific app
peekaboo capture --app Safari --output safari.png

# Inspect UI elements of an app
peekaboo inspect --app Finder --json | jq '.children[:5]'

UI automation

# Click a button by accessibility label
peekaboo click --app Safari --label 'Submit'

# Type text into a focused field
peekaboo type 'Hello from Peekaboo!'

# Press keyboard shortcut
peekaboo key --combo 'cmd+s'

# List running apps
peekaboo apps --json

Installation

brew install steipete/tap/peekaboo

Homepage: https://peekaboo.boo

Source: bundled