TaskFlow

Edit Skill: screen-control

Name

Category

Description

Skill Instructions saved to SKILL.md

---
name: screen-control
description: See and control the Mac screen. Use when asked to look at the screen, answer questions about what's on screen, click things, type text, press keys, or control the mouse and keyboard.
author: nick
version: 1.0.0
triggers:
  - "what's on my screen"
  - "look at my screen"
  - "click on"
  - "type this"
  - "press enter"
  - "take a screenshot"
  - "what do you see"
  - "control my screen"
  - "close this"
  - "open this"
  - "quit this"
metadata: {"clawdbot":{"emoji":"🖥️","requires":{"bins":[]}}}
---

# Screen Control

Two completely separate scripts. Use the correct one for each task.

## SCRIPT 1 - Screenshots (calls local server on port 9999)
```bash
/Users/dr.zoidberg/.pyenv/versions/3.11.9/bin/python3.11 {baseDir}/scripts/screenshot.py "Your question here"
```

## SCRIPT 2 - Mouse and Keyboard (runs directly, no server needed)

### Click
```bash
/Users/dr.zoidberg/.pyenv/versions/3.11.9/bin/python3.11 {baseDir}/scripts/control.py click 500 500
```

### Type
```bash
/Users/dr.zoidberg/.pyenv/versions/3.11.9/bin/python3.11 {baseDir}/scripts/control.py type "hello world"
```

### Press a key
```bash
/Users/dr.zoidberg/.pyenv/versions/3.11.9/bin/python3.11 {baseDir}/scripts/control.py press enter
```

### Hotkey
```bash
/Users/dr.zoidberg/.pyenv/versions/3.11.9/bin/python3.11 {baseDir}/scripts/control.py hotkey command c
```

### Move mouse
```bash
/Users/dr.zoidberg/.pyenv/versions/3.11.9/bin/python3.11 {baseDir}/scripts/control.py move 500 500
```

### Scroll
```bash
/Users/dr.zoidberg/.pyenv/versions/3.11.9/bin/python3.11 {baseDir}/scripts/control.py scroll 500 500 3
```

## MAC CONTROL PLAYBOOK

### App Management - ALWAYS use these first, never pixel clicks

#### Open an app
```bash
open -a "App Name"
```

#### Close an app gracefully
```bash
osascript -e 'quit app "App Name"'
```

#### Force quit an app
```bash
killall "App Name"
```

#### Check if app is running
```bash
pgrep -x "App Name"
```

#### List all open apps
```bash
osascript -e 'tell application "System Events" to get name of every process whose background only is false'
```

### Window Management

#### Minimize a window
```bash
osascript -e 'tell application "App Name" to set miniaturized of front window to true'
```

#### Maximize/fullscreen a window
```bash
osascript -e 'tell application "App Name" to set bounds of front window to {0, 0, 1920, 1080}'
```

#### Close front window without quitting app
```bash
osascript -e 'tell application "App Name" to close front window'
```

#### Bring app to front
```bash
osascript -e 'tell application "App Name" to activate'
```

### System Controls

#### Lock screen
```bash
pmset displaysleepnow
```

#### Take a screenshot to file (for debugging only)
```bash
screencapture -x /tmp/debug.png
```

#### Get mouse position
```bash
/Users/dr.zoidberg/.pyenv/versions/3.11.9/bin/python3.11 -c "import pyautogui; print(pyautogui.position())"
```

#### Get screen size
```bash
/Users/dr.zoidberg/.pyenv/versions/3.11.9/bin/python3.11 -c "import pyautogui; print(pyautogui.size())"
```

### Browser Control (Arc/Chrome/Safari)

#### Open a URL in default browser
```bash
open "https://example.com"
```

#### Open in specific browser
```bash
open -a "Arc" "https://example.com"
open -a "Google Chrome" "https://example.com"
open -a "Safari" "https://example.com"
```

### File and Finder

#### Open a folder in Finder
```bash
open /path/to/folder
```

#### Open a file with default app
```bash
open /path/to/file
```

### Notifications

#### Send a desktop notification
```bash
osascript -e 'display notification "Message here" with title "Title here"'
```

#### Send notification with sound
```bash
osascript -e 'display notification "Message here" with title "Title" sound name "Glass"'
```

### Clipboard

#### Copy text to clipboard
```bash
echo "text here" | pbcopy
```

#### Read clipboard contents
```bash
pbpaste
```

### Volume and Display

#### Set volume (0-100)
```bash
osascript -e 'set volume output volume 50'
```

#### Mute
```bash
osascript -e 'set volume output muted true'
```

#### Unmute
```bash
osascript -e 'set volume output muted false'
```

## DECISION HIERARCHY

When given a task, always follow this order:

1. **osascript** — use for anything app or window related
2. **shell commands** — open, killall, pbcopy, etc.
3. **Screenshot then click** — only when no native command exists
4. **Pixel coordinates** — absolute last resort, always take screenshot first to find them

## WORKFLOW FOR ANY TASK

1. Take a screenshot to understand current state
2. Choose the most native Mac method (osascript first)
3. Execute
4. Take another screenshot to verify it worked
5. If it failed, try next method in the hierarchy
6. Report result to user

## IMPORTANT RULES

- NEVER use screencapture for screenshots — use the server on port 9999
- NEVER guess pixel coordinates — always screenshot first
- ALWAYS verify actions worked with a follow-up screenshot
- ALWAYS try osascript before mouse/keyboard for app control
- Failsafe: slam mouse into any corner to stop pyautogui
- Screenshots are never saved to disk permanently

Cancel