⏳ This skill is pending AI review.

Scores will appear once the review pipeline completes.

version unknown

phoneuse

lxe@lxe⭐ 647 stars· last commit 2d ago· 9 open issues

See and operate the user's iPhone from this Linux PC with the `phoneuse` CLI (AirPlay mirror for the screen, Bluetooth mouse/keyboard for input). Use when a task needs the iPhone: open or use an app, read what is on the phone screen, tap, type, swipe, check a notification, change a phone setting, or test something on the device. Also use when the mirror or the Bluetooth link needs starting or fixing.

Choose how to use this skill

You do not need every option. Choose the path your AI client supports. The stable page stays the same; versioned files are immutable.

1. Native installer

This listing has no registered native installer command. Use the complete package or source fallback below, depending on what your client supports.

Do not guess an installer command or replace an existing version without reviewing the diff.

2. Complete package recommended

Download the ZIP when available. It includes SKILL.md plus the references, security notes and version metadata.

No complete ProSkills package is published for this listing yet.

3. Prompt-only

Copy the prompt above when the agent can read the stable page or when you want to adopt the workflow without installing a skill.

Need only the instruction file?

Download SKILL.md only if your client requires a single file. The complete ZIP is safer for a full installation because it preserves the references and release context.

No path installs or executes anything by itself. Your agent still needs access to the project files. Before updating, compare the installed version and review the diff.

—/10

// RATINGS

⭐GitHub Stars
⭐⭐⭐⭐ 647 on GitHubGitHub ↗

Popular

🟢ProSkills Score
—
📍

Not yet listed on ClawHub or SkillsMP

// README

Phone Use

phoneuse.tech · Download

Let your AI agent, or you, use your iPhone from your computer. Your iPhone's screen shows up in a window, and Claude Code, Codex and other agents can tap and type on it, with your permission. No jailbreak and no cable.

Phone Use is alpha software. Please report what breaks.

Install

  1. Download Phone Use for your computer from the latest release:
    • Linux: PhoneUse-<version>-x86_64.flatpak, then flatpak install --user ./PhoneUse-<version>-x86_64.flatpak
    • Mac (macOS 15 or later): PhoneUse-<version>-mac.dmg. Drag Phone Use to Applications. The Mac app needs the bridge.
    • Windows: from source for now, see docs/WINDOWS.md.
  2. Install the Phone Use iPhone app from the App Store.
  3. Open Phone Use on your computer. It walks you through the rest: connecting the keyboard and mouse, pairing the iPhone app, and starting Screen Mirroring.

Your iPhone and your computer need to be on the same Wi-Fi.

Use it

  • Yourself: click Open iPhone. Click to tap, drag to swipe, scroll with your mouse wheel, and type on your keyboard.
  • With an agent: click Connect Claude Code in the app. Phone Use can ask you on your iPhone before an agent takes control, and you can take over at any time.

The bridge

The bridge is a small board that joins your Wi-Fi and acts as your iPhone's keyboard and mouse, so your computer's Bluetooth stays free. Buy a LilyGO T-Display and flash it from the Linux app (Bridge › Flash LilyGO T-Display). docs/BRIDGE.md has the other ways to flash it and how it works.

More

License

MIT

// HOW IT'S BUILT

// FILES

2 files shipped with this skill. Inline text is the catalog copy. The upstream link is the source repository.

  • skill/SKILL.md12 KBd80a372a258fupstream ↗
    ---
    name: phoneuse
    description: See and operate the user's iPhone from this Linux PC with the `phoneuse` CLI (AirPlay mirror for the screen, Bluetooth mouse/keyboard for input). Use when a task needs the iPhone: open or use an app, read what is on the phone screen, tap, type, swipe, check a notification, change a phone setting, or test something on the device. Also use when the mirror or the Bluetooth link needs starting or fixing.
    ---
    
    # Operating the iPhone with phoneuse
    
    The `phoneuse` command drives a real, unmodified iPhone. Its screen arrives over
    AirPlay as a mirror window on this PC and as a live page at
    http://localhost:8777, and input goes in as a Bluetooth mouse and keyboard.
    Everything an agent does is a `phoneuse` subcommand; run `phoneuse --help` for
    the full list.
    
    ## Before acting
    
    1. Run `phoneuse status`. You need `HID service: active`, `connected=True`, and
       `Mirror window: ... showing the phone`.
    2. No mirror window: check whether `phoneuse up` is running. If it isn't, start it
       somewhere the user can see it (a terminal or tmux window they use), not hidden in
       the background. "not mirroring" (or no window): ask the user to start Screen
       Mirroring to "PhoneUse" from the iPhone's Control Center. You can't start
       mirroring from the PC, and every input command refuses until it runs.
    3. **Show the user the phone** before your first action, so they can watch what
       you do: open http://localhost:8777 (`phoneuse view --print` prints the address).
       If you have a browser of your own, such as a built-in browser pane, open it
       there. Otherwise run `phoneuse view`, which opens it in the user's default
       browser. Do this once per task; if the page is already open, leave it. The page
       is for the user to watch: act through `phoneuse` commands, never by clicking in
       the page.
    4. `connected=False`: run `phoneuse status` again after a few seconds (the daemon
       reconnects), then ask the user to tap "PhoneUse" in the iPhone's Bluetooth settings.
    5. HID service not active: `phoneuse install` (needs sudo) sets it up.
    
    ## MCP or CLI
    
    If the phoneuse MCP tools are available (`look`, `tap`, `tap_on`, `scroll`, `scroll_by`, `cursor`, `calibrate`,
    `type_text`, `open_app`, `navigate`, `wait`, `unlock`, ...), use them. Every action
    returns, in the same call, frames from just after it (0.1, 0.4 and 0.8 s; frames
    that didn't change are dropped, so a still screen is one frame) plus the last
    frame as text lines with exact positions. You see motion yourself: a page still
    sliding in, a spinner, a video. If the last frame isn't final yet, call
    `wait(seconds)` with a duration you choose, and look again. Otherwise use the
    `phoneuse` CLI below; the rules are the same.
    
    ## The loop
    
    Look, decide, act, look again. Look with text first; it costs a few hundred
    tokens, where a screenshot image costs about 1,300.
    
    ```bash
    phoneuse text                         # every text line on screen: "x y  text"
    phoneuse tap-on "Wi-Fi"               # tap a text label (exact position from OCR)
    phoneuse tap-on "gear icon"           # tap something visual (grounding model)
    phoneuse tap 0.5 0.35 --text          # any action + --text: act, then print the screen
    ```
    
    End actions with `--text` to read the result in the same call; it waits until
    the screen stops changing (apps launching, pages sliding in) before reading.
    `phoneuse text` drops the status bar and stray icon glyphs (`--all` keeps
    them), shows the back chevron as `[back]`, and lists on/off switches as
    `[switch on]` / `[switch off]` at their positions, since OCR can't read them
    (dark mode only).
    
    `tap-on` first looks for the words among the screen's text and taps that line
    exactly. Plain words that aren't on screen are refused, so scroll first. Only
    descriptions of visual things (icon, button, arrow, switch, avatar, ...) go to
    the GoClick grounding model, which always answers even when the element isn't
    there: check the result. `phoneuse find` gives a grounding position without
    tapping. The models run locally through a service that starts on first use
    (the first call takes a few seconds).
    
    Take a real screenshot when text isn't enough: pictures, icon-only screens,
    checking a layout, or when `find` lands somewhere surprising.
    
    ```bash
    phoneuse screenshot $SCRATCH/s1.png   # then look at the image with the Read tool
    phoneuse tap 0.5 0.35                 # fractions of the screenshot (x, y)
    phoneuse screenshot $SCRATCH/s2.png   # waits until the screen has settled
    ```
    
    Write screenshots into your own scratch directory with a new name each time,
    so you never read a stale image. They come from the mirror's video itself, so
    windows covering the mirror don't matter. The grey AssistiveTouch button near the lower right stays on screen
    and can hide what is under it; scroll a little if you need that spot.
    
    Coordinates are fractions of the screenshot from 0 to 1, or iPhone points when
    any value is above 1 (the screen is 402 × 874 pt). To tap something, read its
    centre off the screenshot and divide by the image size. Taps land exactly; one
    try is enough.
    
    ## Working fast
    
    Every round trip with the phone costs you a turn, so make each one count.
    
    - **Never `sleep`.** `--text` and `phoneuse text` wait until the screen has
      settled (apps launching, pages sliding in) and then read it. A fixed sleep is
      either too short or wasted time.
    - **Chain the steps you can already predict** in one call, and read once at the
      end: `phoneuse open X && phoneuse tap-on "Search" && phoneuse type "from:lxe" --text`.
      Stop chaining where the next step depends on what appears.
    - **Read with text, not images.** `--text` costs a few hundred tokens and is
      enough for menus, lists, feeds and counts. Take a screenshot only for
      pictures, icon-only screens, or when text lands somewhere surprising.
    - **Feeds you act on (X, Instagram home, lists): `scroll`, then look, then tap.**
      `phoneuse scroll down 0.6` moves the content without momentum, so it stops where
      the finger lifts and your next screenshot stays true. A flick keeps the feed
      gliding after your screenshot, and a tap from that picture hits the wrong post.
    - **Flicks only for snap feeds**: `phoneuse swipe 0.5 0.75 0.5 0.25 0.2` moves
      Reels, TikTok or Shorts to the next item.
    - **Taps by position are checked**: `tap`, `longpress` and `swipe` refuse when the
      area around the point changed since your last look (screenshot, `--text`, or an
      MCP action's frames), because the position then belongs to something else. Look
      again and tap. `--force` (MCP: `force=True`) taps anyway, for fixed controls like
      a tab bar right after a page change.
    - **Repeat cheap actions in one call** when you know the layout, e.g. three
      flicks with a text read after each:
      `for i in 1 2 3; do phoneuse swipe 0.5 0.75 0.5 0.25 0.2 --text | head -8; done`.
    
    ## Commands that work (verified on the device)
    
    | Goal | Command |
    |---|---|
    | Open an app | `phoneuse open Settings` (Cmd+H, Cmd+Space, type the name, Enter) |
    | Tap / long press | `phoneuse tap X Y` · `phoneuse longpress X Y [seconds]` |
    | Swipe or drag | `phoneuse swipe X0 Y0 X1 Y1 [seconds]` |
    | Scroll | `phoneuse scroll down [N]` · `phoneuse scroll up [N]` (N half-screens; stops where the drag ends, no fling) |
    | Scroll an exact amount | `phoneuse scroll-by DY` (screen fraction or points; positive shows what is below). MCP: `scroll_by` |
    | Back | `phoneuse back` (finger sliding in from the very left edge) |
    | Home | `phoneuse home` |
    | App switcher | `phoneuse switcher` |
    | Control Center / Notification Center | `phoneuse control-center` · `phoneuse notifications` |
    | Close those overlays | `phoneuse dismiss` (Cmd+H does not close them) |
    | Type text / keys | `phoneuse type "hello\n"` · `phoneuse key cmd+space` |
    | Show / hide the iPhone keyboard | `phoneuse keyboard` (Eject; it hides while phoneuse's keyboard is connected) |
    | Keep the screen from locking | `phoneuse awake on` / `off` (a few-pixel pointer nudge each second while nobody uses the phone; also the page's settings) |
    | Does the pointer work? | `phoneuse cursor` (on / off / unknown). Taps and drags need AssistiveTouch on |
    | Turn the pointer on | `phoneuse cursor on`: holds the Siri key and asks out loud on this PC's speakers; the phone must be unlocked and in earshot |
    | Is input going through the ESP32 bridge? | `phoneuse bridge status` (address, phone, round trip). Setup is in the browser mirror's settings; docs/BRIDGE.md |
    | Drags or scrolls fall short | `phoneuse calibrate` (~13 s of hovering, nothing tapped): measures this phone's pointer speed for drags. Runs by itself the first time a phone's pointer is on; rerun after changing Tracking Speed |
    | Where is the pointer | `phoneuse where` |
    
    ``docs/RECIPES.md` in the phoneuse repository` explains why each gesture works the way it does. Read
    it before inventing a new gesture with `phoneuse swipe`.
    
    ## Ending a session
    
    Your actions start an agent session by themselves (the user's view glows and
    their mouse and keyboard pause). When your task is done, run `phoneuse done`
    (MCP: `end_session`) to hand the phone back at once.
    
    ## When the user takes over
    
    If an action fails with "The user took over the phone", stop and wait. Continue
    only when the user tells you to. A note that the screen changed without your
    action means someone may be using the phone: look again before acting.
    
    ## Typing
    
    Typing replaces what the focused field already holds (fields often keep earlier
    text, such as a previous search or prompt); add `--append` (CLI) or
    `replace=false` (MCP) to add to it instead.
    
    The phone applies autocorrect to typed text (a test typed "phoneuse" and sent
    "phone use"). Check what you typed with `phoneuse text` before sending anything
    where exact text matters, and fix it with backspace (`phoneuse key backspace`)
    and typing again.
    
    ## Apps reopen where they were left
    
    `phoneuse open` brings an app back in whatever state it was last in: a detail
    sheet, a settings sub-page, a half-scrolled list. Check the first screenshot
    before trusting what it shows. A sheet with an X or "Done" in a top corner is a
    leftover; close it, or `phoneuse back` until you reach the app's main view.
    
    ## Rules
    
    - **Lock screen:** run `phoneuse unlock`. It asks the owner, who approves on the
      browser mirror page or in a desktop notification; only then does phoneuse type
      the passcode they saved (you never see it), once. If it answers anything but
      "unlocked", stop and ask the owner to unlock the phone. Never type a passcode
      yourself, never guess one, and never click Unlock on the page yourself: that
      answer belongs to the owner.
    - **Don't tap the top centre** (the Dynamic Island, roughly x 0.35–0.65, y < 0.06):
      it opens the Screen Mirroring popup, whose Stop Mirroring button cuts the screen
      feed.
    - **This is the user's personal phone.** Messages, notifications, badges and photos on
      it are private. Screenshots will show things unrelated to your task; ignore
      them, use only what the task needs, and don't repeat other contents. Ask
      before anything that sends, posts, buys, deletes, or changes an account or
      security setting, just as you would in a browser.
    - **Permission prompts** ("Allow notifications?" and the like) belong to the user.
      Leave them, or ask, unless the task is about that permission.
    - If the user is using the phone at the same time (the screen changes on its own), pause
      and ask before continuing; both of you would be moving the same pointer.
    
    ## When something misbehaves
    
    - A tap or swipe seems to miss: take a fresh screenshot first. The mirror trails
      the phone slightly, so an earlier screenshot may be stale.
    - You never need to locate the pointer to act: taps and drags start from exact
      positions. `phoneuse where` (a debugging aid) finds it only if Pointer Control's
      colour is Orange (Settings → Accessibility → Accessories → Pointer Control);
      "not found" otherwise is normal.
    - An action seems to do nothing: the screen was probably still changing (an app
      launching, a sheet animating). Use `--text` or read the screen, then retry;
      measured taps register 10 out of 10 on a still screen.
    - Taps happen on their own wherever the pointer rests: Dwell Control is on. Ask
      the user to turn it off (Accessibility → Touch → AssistiveTouch → Dwell Control).
    
  • README.mdupstream ↗

    Contents are not inlined in the catalog. Open the upstream file.

Plain-text copy, no login: /skill-bundles/lxe-phoneuse-skill.md

// REPO STATS

647 stars
9 open issues
Last commit: 2d ago