# agent-android

Adopt this skill: https://proskills.md/skills/other/aivanelabs-ai-rpa-skills-agent-android

Page: https://proskills.md/skills/other/aivanelabs-ai-rpa-skills-agent-android
This document is the public skill bundle for that page. It contains SKILL.md and every helper file the catalog carries. No login is required.

## File: skills/agent-android/SKILL.md
size: 9496
sha: ec3616a5bf33dbba4c791ec74324d21de0260824
upstream: https://github.com/aivanelabs/ai-rpa/blob/main/skills/agent-android/SKILL.md

```text
---
name: agent-android
description: Control Android over LAN without USB, ADB, or root.
metadata: {"openclaw":{"homepage":"https://github.com/aivanelabs/ai-rpa","requires":{"bins":["agent-android"]},"install":[{"id":"agent-android-uv","kind":"uv","package":"aivane-agent-android","bins":["agent-android"],"label":"Install agent-android CLI (uv)"}]}}
---

# Android REPL

Use this skill to drive an Android device through the public `agent-android` beta surface published in the `aivanelabs/ai-rpa` repo.

Runtime prerequisites:

- `agent-android` is available on `PATH`
- if it is missing, install the CLI with `uv tool install aivane-agent-android`; then run `uv tool update-shell` if the command is still not found
- the user has provided a trusted device URL such as `http://<device-ip>:8080`; use the phone-side configured port if it is not `8080`

The public path is local-first:

- the phone hosts the lightweight HTTP service locally
- the desktop connects directly to the phone URL, usually `http://<device-ip>:8080`
- UI reads, taps, inputs, and screenshots stay on the phone and controlling machine
- the current tradeoff is LAN-only control; an optional server-side or relay path may arrive later

If an `agent-android` command suddenly stops working, first check whether the AIVane app or its local API service has exited on the phone.

## Safety Boundaries

- Connect only to a device URL explicitly provided by the user.
- Do not scan local networks or guess device IP addresses.
- Treat UI trees and screenshots as sensitive because they may contain private app content.
- Save screenshots or raw UI dumps only when the user explicitly asks for a file output.
- Ask for confirmation before operating sensitive apps, private content, account settings, or irreversible actions.
- Do not expose the phone-side service on a public network.

## Core Workflow

Every Android control task should follow the same short loop:

1. Confirm connectivity with `/health`
2. Discover the target app with `apps` if needed
3. Launch one app
4. Inspect the current UI tree
5. Perform one action
6. Inspect again

Keep the loop short. Prefer inspect -> act -> inspect over long speculative command chains.

## Quick Start

Start the REPL with the user-provided device URL:

- `agent-android --repl --url http://<device-ip>:8080`

Built-in help:

- `agent-android --help`
- In the REPL: `h`

## Essential CLI Commands

Use the one-off CLI when you already know the exact action you want.

Connectivity:

- `agent-android --health --url http://<device-ip>:8080`

Discovery:

- `agent-android --apps --url http://<device-ip>:8080`
- `agent-android --list --url http://<device-ip>:8080`
- `agent-android --id com.example:id/search --url http://<device-ip>:8080`
- `agent-android --text Search --url http://<device-ip>:8080`
- `agent-android --inputs --url http://<device-ip>:8080`
- `agent-android --refId 7 --url http://<device-ip>:8080`
- `agent-android --xpath 7 --url http://<device-ip>:8080`
- `agent-android --get-attr 7 text --url http://<device-ip>:8080`

Actions:

- `agent-android --launch com.xingin.xhs --url http://<device-ip>:8080`
- `agent-android --tap 7 --url http://<device-ip>:8080`
- `agent-android --input 7 "hello world" --url http://<device-ip>:8080`
- `agent-android --swipe up --url http://<device-ip>:8080`
- `agent-android --swipe up --swipe-refid 7 --url http://<device-ip>:8080`
- `agent-android --swipe up --swipe-xpath "//RecyclerView[1]" --url http://<device-ip>:8080`
- `agent-android --back --url http://<device-ip>:8080`
- `agent-android --press home --url http://<device-ip>:8080`
- `agent-android --screenshot --url http://<device-ip>:8080`
- `agent-android --application-bundle app.zip --main-template-file __main__.json --url http://<device-ip>:8080`
- `agent-android --template template.json --async --url http://<device-ip>:8080`
- `agent-android --application-bundle app.zip --main-template-file __main__.json --async --url http://<device-ip>:8080`
- `agent-android --upload foo.json --remote-path Templates/foo.json --url http://<device-ip>:8080`

Waiting and output:

- `agent-android --wait-for Search --timeout 30 --url http://<device-ip>:8080`
- `agent-android --list --raw --url http://<device-ip>:8080`
- `agent-android --tasks --url http://<device-ip>:8080`
- `agent-android --task TASK_ID --url http://<device-ip>:8080`
- `agent-android --task-logs TASK_ID --url http://<device-ip>:8080`
- `agent-android --stop-task TASK_ID --url http://<device-ip>:8080`

## REPL Command Reference

Use the REPL for exploratory tasks and smoke runs. Short aliases and long names both work.

### Browse

- `health` or `hl`
  Check `/health`.
- `l [n]` or `list [n]`
  List the first `n` elements, or all cached elements when `n` is omitted.
- `ss` or `snapshot`
  Force-refresh the UI tree and print it again.
- `f <text>` or `find <text>`
  Filter by text or content description.
- `id <resourceId>`
  Filter by Android resource ID.
- `ref <refId>`
  Show the full element detail for one refId.
- `x <refId>` or `xpath <refId>`
  Generate XPath candidates and validate their runtime match counts.
- `xx <refId>`
  Tap by the best unique generated XPath candidate. Refuses ambiguous matches.
- `vx <xpath> [idx]` or `validatex <xpath> [idx]`
  Validate one XPath against the runtime. Optionally inspect one match by zero-based index.

### Interact

- `t <refId>` or `tap <refId>`
  Tap the element center point from the current tree.
- `tx <xpath>` or `tapx <xpath>`
  Tap one runtime-resolved XPath target.
- `i <refId> <text>` or `input <refId> <text>`
  Input text into a refId target.
  Use `--clear` or `""` to clear instead of typing.
- `ix <xpath> <text>` or `inputx <xpath> <text>`
  Input text into one XPath target.
  Use `ix <xpath> --` or `--clear` to clear the field.
- `sw <d|u|l|r> [refId] [--dur N] [--dist N]` or `swipe ...`
  Swipe down/up/left/right across the screen, or inside a refId target when provided.
- `swx <d|u|l|r> <xpath> [--dur N] [--dist N]` or `swipex ...`
  Swipe inside one XPath target.
- `p <back|home|recents>` or `press ...`
  Press a system key.
- `b` or `back`
  Press Back.
- `la <package>` or `launch <package>`
  Launch an app by package name.
- `s [path]` or `screenshot [path]`
  Capture a screenshot to an auto-generated or explicit path.
- `up <local> <path>` or `upload <local> <path>`
  Upload a local file to the phone. Overwrites by default; add `--no-overwrite` to reject an existing target.

### Wait And Inspect

- `wf <text> [--t N]` or `waitfor ...`
  Wait for an element to appear.
- `g <refId> <attr>` or `get <refId> <attr>`
  Read an attribute such as `text`, `class`, `bounds`, `x`, `y`, or `xpath`.
- `apps`
  List launcher apps from `/apps`.

### Session

- `raw`
  Toggle raw JSON mode.
- `vars`
  Show current URL, timeout, raw mode, and tree cache state.
- `set timeout 30`
  Set the default wait timeout in seconds.
- `h` or `help`
  Show the built-in help text.
- `q` or `quit`
  Exit the REPL.

## Common Patterns

### First smoke flow

1. Start the REPL with `agent-android --repl --url http://<device-ip>:8080`.
2. `health`
3. `apps`
4. `la <package>`
5. `l`
6. `t <refId>`
7. `i <refId> hello`
8. `b`
9. `s`

### Find the right package before launch

- `apps`
- `la com.example.app`
- `l`

### Inspect an element before using XPath

- `l`
- `ref 12`
- `x 12`
- `vx //EditText[@text='Search']`

### Clear and refill an input

- `i 7 --clear`
- `i 7 hello world`
- `ix //EditText[@text='Search'] --`
- `ix //EditText[@text='Search'] -- hello world`

### Run a local template bundle

- Zip the local folder containing the main template and child templates.
- Run `agent-android --application-bundle app.zip --main-template-file __main__.json --url http://<device-ip>:8080`.
- Use `--upload` only for standalone images, ordinary binary files, or one-off file sync.

## Troubleshooting

- If an `agent-android` command fails, first check whether the AIVane app or phone-side API service has exited.
- If `agent-android` is not found, run `uv tool update-shell`, reopen the terminal, and retry.
- Re-open the app or restart the phone-side service, then retry `curl http://<device-ip>:8080/health`.
- If `health` works but UI commands fail, run `ss` to force-refresh the tree before tapping or inputting.
- If `tx` or `ix` fails, run `vx <xpath>` and make sure the XPath resolves to exactly one runtime match.
- If screenshots fail the first time, confirm the on-device MediaProjection permission prompt was accepted.
- If everything suddenly stops responding, confirm the phone IP did not change and that the desktop is still on the same LAN.

## When To Stop

Stop and ask for user help when:

- the device is unreachable on LAN
- the app is not running and cannot be restarted from the current path
- required Android permissions are missing
- launcher discovery returns nothing useful
- the runtime UI no longer matches the expected screen after repeated refreshes

## References

- Smoke checklist: [GitHub](https://github.com/aivanelabs/ai-rpa/blob/main/skills/agent-android/references/smoke-flow.md)
- Quickstart: [GitHub](https://github.com/aivanelabs/ai-rpa/blob/main/docs/quickstart.md)
- Install guide: [GitHub](https://github.com/aivanelabs/ai-rpa/blob/main/docs/install-agent-android.md)
- Public protocol: [GitHub](https://github.com/aivanelabs/ai-rpa/blob/main/docs/protocol-v1.md)
- Known beta limits: [GitHub](https://github.com/aivanelabs/ai-rpa/blob/main/docs/known-limitations.md)
```

## File: skills/agent-android/references/smoke-flow.md
size: 801
sha: c5923c5c967013f205494221fef6d42e93240763
upstream: https://github.com/aivanelabs/ai-rpa/blob/main/skills/agent-android/references/smoke-flow.md

````text
# Android REPL Smoke Flow

Use this checklist when validating a fresh setup.

## Preconditions

- Android REPL app installed
- API service running
- controlling machine on the same LAN
- `agent-android` available on `PATH`

## Checklist

1. Save the device URL:

```text
set url http://<device-ip>:8080
```

2. Check service health:

```bash
curl http://<device-ip>:8080/health
```

3. List launcher apps:

```bash
agent-android --apps --url http://<device-ip>:8080
```

4. Launch the target app.
5. List the current UI tree.
6. Perform one tap.
7. Perform one text input if relevant.
8. Navigate back.

## Success Criteria

- the device responds
- launcher apps are visible
- launch succeeds
- the UI tree refreshes after actions
- the user can complete at least one inspect -> act -> inspect loop

````

## File: skills/agent-android/agents/openai.yaml
size: 281
sha: fb8bc0ed0b3a67c85cd9fc4278c73f63376f7156
upstream: https://github.com/aivanelabs/ai-rpa/blob/main/skills/agent-android/agents/openai.yaml

```text
interface:
  display_name: "agent-android"
  short_description: "agent-android for AI agents"
  default_prompt: "Use $agent-android from the aivanelabs/ai-rpa repo to connect to an AIVane device over LAN, inspect launcher apps and UI state, then control the phone step by step."

```

## File: README.md
upstream: https://github.com/aivanelabs/ai-rpa/blob/main/README.md

````text
See the Chinese overview at [README_CN.md](README_CN.md).

# AIVane AI RPA

Control an Android phone from your desktop or AI agent in minutes: inspect UI, tap, type, launch apps, and capture screenshots locally over LAN.

This repository is the public home for `aivanelabs/ai-rpa`. The current public surface is **AIVane Android REPL Beta**: a directly installable `agent-android` CLI plus a GitHub-installable `agent-android` skill for tools such as Codex, Claude Code, and OpenClaw.

## Demo

### Add a calendar event

https://github.com/user-attachments/assets/f4e48dd4-386c-4df5-936e-37c3fa650fd4

### Search on Xiaohongshu

https://github.com/user-attachments/assets/7024b34b-73cd-4cc5-857e-90a84c341110

## Start Here

- The beta already has a ready-to-install CLI and skill path; use the commands below instead of repo-local script paths.
- Download APK: [GitHub Releases](https://github.com/aivanelabs/ai-rpa/releases)
- Install CLI: `uv tool install aivane-agent-android`
- Install skill: `npx skills add aivanelabs/ai-rpa --skill agent-android`

## 3-Step Quickstart

1. Install the APK on your phone and enable the AIVane accessibility service.
2. Install the CLI:

```bash
uv tool install aivane-agent-android
```

If `agent-android` is not found afterwards, run:

```bash
uv tool update-shell
```

Then reopen the terminal. If you want the current shell to work immediately on Linux, run:

```bash
export PATH="$HOME/.local/bin:$PATH"
```

3. Verify the device is reachable:

```bash
agent-android --health --url http://<device-ip>:8080
```

If that succeeds, start the REPL:

```bash
agent-android --repl --url http://<device-ip>:8080
```

## Choose A Path

### For Humans

Use the CLI directly when you want to manually explore the phone from your desktop:

1. Download the APK from [GitHub Releases](https://github.com/aivanelabs/ai-rpa/releases)
2. Install the CLI with `uv tool install aivane-agent-android`
3. Run `agent-android --repl --url http://<device-ip>:8080`
4. In the REPL, use the short loop: `health` -> `apps` -> `la <package>` -> `list` -> one action -> `list`

### For AI Agents

Use the skill when you want Codex, Claude Code, or another coding agent to drive the phone:

1. Install the CLI with `uv tool install aivane-agent-android`
2. Install the skill:

```bash
npx skills add aivanelabs/ai-rpa --skill agent-android
```

3. Give the agent a concrete task, for example:

```text
Use the installed agent-android skill to:
1. check phone health
2. list launcher apps
3. launch Settings
4. inspect visible UI nodes
5. tap the Wi-Fi entry
```

## First Success Path

This is the shortest practical smoke flow after the phone is reachable:

1. `agent-android --repl --url http://<device-ip>:8080`
2. `set url http://<device-ip>:8080`
3. `health`
4. `apps`
5. `la <package>`
6. `list`
7. `tap <refId>`
8. `list`

If you want the full setup path, see [docs/install-agent-android.md](docs/install-agent-android.md) and [docs/quickstart.md](docs/quickstart.md).

## Template And File Sync

Use an application bundle when a workflow has one main template plus child templates. The phone executes the zip as one scoped package, so child template names do not collide with phone-local templates:

```bash
agent-android --application-bundle app.zip --main-template-file __main__.json --url http://<device-ip>:8080
```

Use the CLI upload command when you want to push a standalone image, ordinary binary file, or one-off local file from your computer to phone-local REPL storage:

```bash
agent-android --upload foo.json --remote-path Templates/foo.json --url http://<device-ip>:8080
```

Uploads overwrite by default. Add `--no-overwrite` to reject an existing target. In templates, use `http.download` when the phone should fetch and save an image, JSON template, zip, text file, or other binary file from a URL. Keep `http.get` for text or JSON responses that should be read into variables.

## Template Validation

Install the template linter when you want to check a template or application bundle before running it on a device:

```bash
uv tool install aivane-template-linter
aivane-template-lint my-app -r
```

The linter validates template JSON, OperationType parameters, variable references, and application-level relationships such as `app.json` main template resolution and `template.execute` child template calls.

See [docs/templates/](docs/templates/) for authoring guidance and [docs/operation-types/](docs/operation-types/) for the OperationType reference.

## What This Beta Is

- Local-first Android automation over LAN
- Human-friendly REPL plus agent-friendly skill
- Public CLI command: `agent-android`
- Designed for inspect -> act -> inspect workflows

## What This Beta Is Not

- Not a cloud phone farm
- Not remote control over arbitrary networks by default
- Not iOS support
- Not a visual recorder workflow yet

## Why The Phone Is The Web Server

- The phone runs the lightweight HTTP service locally and the desktop connects directly to `http://<device-ip>:8080`.
- UI inspection, taps, text input, and screenshots stay between the phone and the controlling machine.
- The first smoke flow works without a cloud relay.
- The tradeoff is that this beta is LAN-only.

## Install Sources

- PyPI package: `aivane-agent-android`
- Console command: `agent-android`
- Template linter package: `aivane-template-linter`
- Template linter command: `aivane-template-lint`
- Skill: [`skills/agent-android/`](skills/agent-android/)
- APK builds: [GitHub Releases](https://github.com/aivanelabs/ai-rpa/releases)

## Repo Layout

- `clients/python/`: publishable Python CLI package using a standard `src` layout
- `clients/template-linter/`: publishable Python template linting CLI package
- `docs/`: quickstart, install, protocol, permissions, release, and support docs
- `docs/templates/`: template authoring and linting guidance
- `docs/operation-types/`: public OperationType reference
- `docs/assets/`: README-linked media such as demo videos and screenshots
- `examples/`: smoke-flow examples and launch helpers
- `skills/agent-android/`: installable public skill definition

## Additional Resources

- [docs/quickstart.md](docs/quickstart.md)
- [docs/install-agent-android.md](docs/install-agent-android.md)
- [docs/agent-examples.md](docs/agent-examples.md)
- [docs/templates/README.md](docs/templates/README.md)
- [docs/operation-types/README.md](docs/operation-types/README.md)
- [docs/release-checklist.md](docs/release-checklist.md)
- [docs/known-limitations.md](docs/known-limitations.md)

## Contact

For questions and light coordination, please email `aivanelabs@gmail.com`.
````
