Phone-Use AI Agents: Mika vs ESP32 iPhone Agent
TL;DR: Phone-use AI agents operate real mobile apps by looking at the screen and injecting taps, swipes, and keystrokes. Mika does this on Android from a hidden background screen using the system's own debugging and accessibility hooks. ESP32 iPhone Agent does it on iOS from outside the phone, with a $7 ESP32-S3 board acting as an AirPlay receiver and a USB mouse and keyboard. The platform decides the architecture. Android lets an app reach inside, while iOS forces the agent to act like a person holding the phone.
Key Takeaways
- Two of the most-starred new GitHub agent projects this autumn are phone-use agents. Mika (Kotlin, Apache-2.0, created 2026-10-29) had about 6,600 stars by 2026-11-04. ESP32 iPhone Agent (C, MIT, created 2026-10-11) had about 4,400.
- Mika runs entirely on an Android 12+ phone. It enables Wireless debugging once to start a built-in helper, then operates apps on a separate background screen while you keep using the main one.
- ESP32 iPhone Agent never touches iOS internals. The board mirrors the screen over AirPlay to "see" and presents itself as a USB HID mouse and keyboard to "act", with AssistiveTouch turning mouse clicks into taps.
- Both projects are model-agnostic. Mika ships presets for more than 20 providers plus any OpenAI-compatible, Anthropic, Gemini, or OpenAI Responses endpoint, and the ESP32 agent accepts any OpenAI-compatible vision model.
- The bottleneck is the observation channel, not the model. The ESP32 agent's BLE link limits it to 480×1040 at 5 fps or less, and each step takes a few seconds. Neither agent can see screens the OS refuses to capture, such as payment and passcode screens.
- A phone agent holds your logged-in apps, so its safety model matters more than its benchmark. Mika's default "Ask" mode confirms only steps that spend money, which means messages, posts, and deletions run without asking.
What is a phone-use AI agent?
A phone-use AI agent is a loop that takes a screenshot of a mobile device, asks a model what to do next, and executes that action as a touch or keystroke on the real UI. It is the mobile equivalent of desktop "computer use". The agent works through the same interface a person does, not through app APIs.
The appeal is coverage. Most consumer apps (messaging, food delivery, ride hailing, job boards) have no public API, or one that excludes the actions users care about. A phone agent does not need one. As Mika's README puts it, it "looks at the screen and finds buttons the way a person does: no app APIs, no per-app integration."
The cost is the same as in [browser automation](/articles/Browser-Use-vs-Stagehand-vs-Playwright-MCP-AI-Agent-Browser-Automation): every step costs a model call, the UI can change under the agent, and recovering from mistakes is the agent's job. On a phone there are extra constraints. The OS sandboxes apps from each other, protected screens block capture, and the device is also the user's primary personal computer.
Why are phone-use agents trending in November 2026?
Two projects dominated this week's GitHub trending data for AI agents. Mika gained roughly 6,600 stars within a week of its 2026-10-29 creation. ESP32 iPhone Agent passed 4,300 stars in under a month. Both answer a question developers have been asking since desktop computer-use tools matured: can the same loop run on the device people actually use for most of their day?
The answer depends on the platform, and these two projects show the two possible answers clearly. On Android, the OS exposes hooks (developer options, accessibility services, secondary displays) that let one app operate others. On iOS, no such hook is available to third-party apps, so the only route that needs no jailbreak and no developer-signed automation is to act on the phone from outside, through standard peripherals.
How does Mika operate Android apps?
Mika is a Kotlin and Jetpack Compose app distributed as an APK. Chat, web browsing, and file work run on Android 11+. Operating other apps requires Android 12+, and the project recommends Android 14+.
Getting control. A one-time "Set up phone control" wizard walks the user through enabling Developer options, USB debugging, and Wireless debugging, then pairing with a 6-digit code. The setup guide explains that Mika "borrows Android's built-in 'Wireless debugging' to start a small built-in helper". After pairing, Mika turns on its own accessibility service, which avoids Android's "Restricted setting" block for sideloaded apps. No computer or cable is required.
Seeing. Mika reads the UI in layers. Where an app exposes readable UI, it uses that. For custom-drawn interfaces or unlabeled controls, it falls back to text recognition bundled in the APK. With a vision-capable model, it can also locate buttons and fields from screenshots.
Acting without stealing the screen. Most phone agents take over the display while they work. Mika instead "opens apps on a separate background screen and reads, taps, types and scrolls there while you keep using your phone." Android 14+ is recommended because the background screen gets its own input focus, so the agent's typing does not collide with yours. Several tasks can run at once. If two need the same app, the later one waits, and deadlocks between waiting tasks are detected rather than left hanging.
Getting faster with use. Mika ships with built-in usage knowledge for about two dozen apps and a similar number of websites (WeChat, WhatsApp, Gmail, Telegram, Uber, Amazon, YouTube, and others). It also turns tasks it has completed into reusable skills. This is the same "learned skill" pattern we examined in [Hermes Agent's self-improving skill memory](/articles/Inside-Hermes-Agent-Self-Improving-Skill-Memory), applied to UI navigation.
Model access. There is no Mika server. Users bring an API key (presets include DeepSeek, Qwen, Kimi, Zhipu, MiniMax, OpenRouter, Anthropic, Gemini, OpenAI, xAI, Mistral, Groq, Ollama, and LM Studio) or sign in with a ChatGPT, xAI, or OpenRouter account. A local model on the same network works through the "Custom" provider option.
How does ESP32 iPhone Agent operate an iPhone?
ESP32 iPhone Agent takes the opposite route. Its tagline is "No Mac, no developer account, no jailbreak". The agent never runs anything privileged on iOS. It uses the iPhone exactly as a person with a mouse and a screen mirror would.
The README's architecture fits in three lines:
iPhone ──Screen Mirroring (Wi-Fi)──▶ ESP32-S3: AirPlay receiver ──BLE──▶ ESP32 Agent app (same iPhone)
▲ own hotspot, H.264 passthrough decode → vision model → action
└──────── USB HID mouse + keyboard ◀──────────── BLE commands ◀────────────────────┘
Eyes: AirPlay. The board opens its own Wi-Fi hotspot and appears in the iPhone's Screen Mirroring list as "ESP32 Agent". It does not decode video. It passes the H.264 stream through to the companion app over Bluetooth Low Energy. Because the phone's Wi-Fi is now joined to the board's hotspot, model calls go out over cellular data.
Hands: USB HID. The board plugs into the phone's USB-C port (Lightning models need a camera adapter) and enumerates as a standard mouse and keyboard. With AssistiveTouch enabled, iOS shows a pointer, so a click becomes a tap and a drag becomes a swipe.
Brain: an iOS app on the same phone. The app decodes the stream, sends screenshots to a vision model, and runs a planner and an actor. The actor's action space is tap, swipe, draw, type, press keys, go home, and finish. Planner and actor can use different models.
Closing the loop. A USB mouse reports relative movement, so the app cannot simply "tap at (x, y)". It tracks the pointer, re-homes it in a screen corner, and, before precise taps, wiggles the pointer to find it on screen and correct the miss. After every action it waits for the screen to settle and tells the model whether anything changed. If the board, the hotspot, or the mirroring link drops, the run pauses and resumes from the same step.
The project's history makes the design point. Version 0.1.0 needed a Mac or Linux computer running an AirPlay receiver and a Python server. The current version moved the receiver onto the $7 board and the agent onto the phone itself. A separate esp32agent CLI exposes the board as an MCP server, so a coding agent such as Claude Code can drive the iPhone from a laptop over Bluetooth.
How do Mika and ESP32 iPhone Agent compare?
| Dimension | Mika (Android) | ESP32 iPhone Agent (iOS) |
|---|---|---|
| Where the agent runs | Android app on the phone | iOS app on the phone, plus ESP32-S3 firmware |
| Extra hardware | None | ESP32-S3 board with PSRAM and native USB (~$7) |
| How it gets control | One-time Wireless debugging pairing, then its own accessibility service | USB HID mouse and keyboard plus AssistiveTouch |
| How it sees | Readable UI, bundled OCR, optional vision-model screenshots | AirPlay mirror at 480×1040, ≤5 fps |
| Uses the phone while it works? | Yes, on a separate background screen | No. The agent drives the visible screen. |
| Parallel tasks | Yes, with per-app waiting and deadlock detection | One run at a time |
| Text input | Typing plus sharing text into apps | ASCII keyboard input only for now |
| Orientation | Not stated as a limitation | Portrait only for now |
| Model requirement | Any of 20+ presets or a compatible custom endpoint | Any OpenAI-compatible vision endpoint |
| Run review | Every step viewable in chat, wrap-up report on stop | Step-by-step replay with taps drawn on top, exportable traces |
| Drive it from a computer | CLI + Skill or MCP | esp32agent MCP server over Bluetooth |
| License | Apache-2.0 | MIT |
The table supports one practical conclusion: these are not competing products, they are the two shapes the problem takes. If you are on Android, Mika's in-device approach is more capable on almost every axis. It runs in parallel without taking over the screen and needs no extra hardware. If you are on iOS, an external-peripheral design like the ESP32 board is currently the only way to get a general agent operating arbitrary apps without a jailbreak, and its constraints follow from that.
Why is the observation channel the real bottleneck?
Model quality gets the attention, but in both projects the hard engineering sits in getting pixels in and touches out reliably.
The ESP32 agent states its budget plainly. BLE carries about 300–500 kbps, so the mirror is small (480×1040, ≤5 fps) and "each step takes a few seconds, most of it the model." Small frames help token cost and latency, but fine text and small icons are harder for a vision model to read. The pointer-correction loop exists because a relative mouse plus a low-frame-rate mirror means the agent is never quite sure where its finger is. That is a classic closed-loop control problem, solved by measuring after acting rather than trusting the command.
Mika's layered perception (structured UI first, OCR second, vision third) points the same way. A model that reads a list of labeled elements makes fewer mistakes per step than one guessing coordinates from a screenshot. This mirrors the split in browser agents between accessibility-snapshot tools and screenshot-based tools. Pick the richest observation the platform gives you, and fall back to pixels only when you must.
Two failure modes are shared and fixed by the OS, not the agent:
- Protected screens are invisible. iOS does not mirror passcode entry, Apple Pay, or DRM video. Android payment and password pages block screenshots. Both projects hand these steps to the human. That is the correct behavior, and it is not a bug to engineer around.
- The observation channel is fragile. Android turns Wireless debugging off when you change networks or restart (Mika re-enables it with permission). AirPlay mirroring stops when an iPhone locks. Any production-minded phone agent needs explicit pause-and-resume semantics, which both projects implement.
How should you think about phone-agent security?
A phone agent is the most privileged agent most people will ever run. It inherits every logged-in session on the device: messaging, email, banking, shopping, social accounts. Every screen it reads is untrusted input. A chat message, a product listing, or a web page can contain text aimed at the model rather than the user. That is the same indirect prompt-injection setup behind [GitLost](/articles/GitLost-Prompt-Injection-GitHub-AI-Agent-Repo-Leak), on a device that holds all three legs of the lethal trifecta by default: private data, untrusted content, and outbound channels.
Read the permission models carefully:
- Mika's default "Ask" mode confirms only steps that spend money. Its own docs say that "sending messages, posting, submitting forms, deleting and changing settings are not asked about; they are just done." "Autonomous" mode removes even the payment confirmation. Logins, verification codes, and payment passwords are always handed to the user.
- ESP32 iPhone Agent relies on supervision. Its responsible-use note says, "The agent acts with your hands: watch it, and keep the stop button within reach." The app also currently uses a private iOS API to send itself to the background, which the maintainers intend to replace before an App Store release.
Neither design is reckless for a personal tool, but neither is an authorization layer you could put in front of a business workflow. If you build on top of either, apply the principle from our [LLM vs policy-engine authorization analysis](/articles/AI-Agent-Authorization-LLM-vs-Policy-Engine). The model proposes, and a deterministic rule decides. On a phone, that means an allowlist of apps per task, a confirmation for any outbound message or post (not just payments), and no access to apps the task does not need.
A smaller but real point: automating third-party apps may violate their terms of service, and Mika's README says so directly. Batch actions such as greeting recruiters, posting, or watching ads for perks are exactly what platforms' anti-automation systems look for. Mika paces itself on websites and hands anti-bot block pages to the user instead of forcing through. Copy that behavior if you build your own.
When should you use a phone agent instead of an API or browser agent?
Use the most structured interface available, and drop to the phone only when nothing better exists:
1. An official API or MCP server exists. Use it. It is faster, cheaper, auditable, and does not break when the UI is redesigned.
2. The service has a full-featured website. Use a [browser agent](/articles/Browser-Use-vs-Stagehand-vs-Playwright-MCP-AI-Agent-Browser-Automation). DOM and accessibility snapshots give a richer observation than a phone screenshot. (Mika's built-in desktop-class browser follows the same logic inside the phone.)
3. The capability exists only in a mobile app. That is the phone agent's territory. Examples are app-only messaging, delivery and ride apps, and on-device settings like alarms. Also consider phone agents for testing your own mobile app's real user flows, where operating the real UI is the whole point.
For the third case, pick by platform: an in-device agent like Mika on Android, an external-peripheral agent like ESP32 iPhone Agent on iOS. Budget for a few seconds and one model call per step. A ten-step task is closer to a minute than to an instant, so phone agents suit errands you would otherwise do by hand, not latency-sensitive automation.
What should engineers building phone agents copy from these two projects?
Several design choices are worth copying, whatever platform you target:
- Verify after every action. The ESP32 agent checks whether the screen changed and corrects pointer misses. An agent that assumes its last tap worked compounds errors quickly.
- Make runs resumable. Both projects pause on a lost connection or a hand-off to the human and resume from the same step. Mika's stop-time wrap-up ("what it did, what it saw, how many times it sent something, and what is left") makes a later "continue" safe.
- Record everything. Step-by-step replay with drawn taps (ESP32) and per-step inspection in chat (Mika) are what make a non-deterministic agent debuggable. They are also the raw material for the [trace-to-regression-test workflow](/articles/AI-Agent-Regression-Testing-Production-Traces-to-CI) any serious agent needs.
- Separate the human's screen from the agent's. Mika's background screen is the biggest usability step in this category. An agent that blocks the phone for minutes is one people stop using.
- Keep prompts as editable data. The ESP32 project ships its planner and actor prompts as plain text files, so you change behavior by editing a file, not by rewriting code.
- Expose the device as a tool. Both projects let a desktop coding agent drive the phone over MCP. The phone becomes one more tool in a larger agent, not a separate product.
What does this mean for the future of mobile AI agents?
These projects show that general phone-use agents are now buildable by small open-source teams with off-the-shelf models and, on iOS, a $7 board. What they cannot solve is that the OS is not designed for an agent as a first-class user. Android's route runs through developer options. iOS's route runs through accessibility features and USB peripherals meant for people with motor impairments. Both work, but both rely on mechanisms built for other purposes.
Until mobile platforms offer a sanctioned agent interface with scoped permissions per app, the engineering priorities are clear. Invest in reliable observation and verification, make every run resumable and replayable, and put deterministic limits on what the agent may send or spend. The model is the easy part.
FAQ
What is the difference between Mika and ESP32 iPhone Agent?
Mika is an Android app that operates other apps from inside the phone. It uses Wireless debugging and its own accessibility service to drive apps on a hidden background screen. ESP32 iPhone Agent operates an iPhone from outside, with an ESP32-S3 board that receives an AirPlay mirror of the screen and acts as a USB mouse and keyboard. The difference follows from what each OS allows.
Does ESP32 iPhone Agent require a jailbreak or a Mac?
No. The project needs no jailbreak and no Mac at run time. A Mac with Xcode is needed only to flash the board and build the app yourself. The board can instead be flashed from Chrome or Edge, and the app is available through TestFlight. The iPhone needs iOS 18 or later and cellular data, because its Wi-Fi is joined to the board's hotspot while the agent runs.
Which AI models work with phone-use agents like Mika?
Mika supports presets for more than 20 providers, including Anthropic, OpenAI, Gemini, DeepSeek, Qwen, Kimi, xAI, Mistral, Groq, Ollama, and LM Studio. It also accepts any OpenAI-compatible, Anthropic, Gemini, or OpenAI Responses endpoint. ESP32 iPhone Agent works with any OpenAI-compatible chat API that accepts images, including self-hosted vLLM, SGLang, llama.cpp, or Ollama servers.
Can a phone AI agent make payments or log in for me?
Neither project automates logins, verification codes, or payment passwords. Both hand those steps to the user, partly because the OS blocks screen capture on those pages. Mika's default Ask mode also asks before placing an order, but it does not ask before sending messages, posting, or deleting. Review those defaults before giving a phone agent access to important accounts.
Are phone-use AI agents safe to use?
They are as safe as their permission model, and by default that model is broad. A phone agent can act in every app you are logged into, and every screen it reads may contain prompt-injection text. Use confirmation modes, watch runs on important accounts, limit tasks to the apps they need, and remember that automating third-party apps may breach their terms of service.
How fast are phone-use AI agents?
Expect a few seconds per step, mostly spent waiting on the model. ESP32 iPhone Agent mirrors the screen at up to 5 fps over a 300–500 kbps Bluetooth link, so a multi-step task usually takes tens of seconds to minutes. Mika avoids blocking the user by working on a background screen, which matters more for usability than raw step speed.
Sources
- Mika GitHub repository
- Mika features in detail
- Mika phone control setup guide
- Mika Ask and Autonomous modes
- ESP32 iPhone Agent GitHub repository
- Seeed Studio XIAO ESP32S3 getting started
---
Originally published at fp8.co. Subscribe for weekly AI engineering analysis at fp8.co.
The Trust Boundary Moved to Your Pocket: On-Device Android Agents and the Controller Duty
Last month a phone started tapping through apps it had never been given an API for, with no SDK, no per-app integration and no vendor server in the path. In the Philippines that handset is almost certainly Android, at 86.28% of mobile OS share against 13.71% for iOS in October 2026 (Source: StatCounter, 2026). The National Privacy Commission enforces RA 10173, and the duties in that law attach to whoever decides how personal information is processed, not to whoever owns the server it would have run on (Source: Republic Act 10173, 2012).
!Infographic
The integration layer is now hardware people already own
Enterprise mobile automation used to be a procurement exercise: device management, a per-app SDK, a middleware tier between phone and system of record. NIST SP 800-124 Rev. 2, published in May 2023, treats the problem as managing mobile device security across a fleet, which presumes an owner with a fleet (Source: NIST SP 800-124 Rev. 2, 2023).
The current generation of Android agents inverts that shape. One Apache-2.0 application, no vendor server, no per-app integration, no SDK negotiated with each app's publisher. The integration surface is the screen the user is already looking at.
What the agent actually does
Mika, an open-source Android agent, operates apps and the web on a separate invisible background display. It opens apps, taps, types and scrolls there while the owner keeps the main screen, and it finds buttons the way a person does, by looking at them, with no app APIs or per-app integration (Source: Mika project documentation, 2026).
Each conversation gets a Linux sandbox that runs Python, Node and shell. Keys and memory are stored encrypted on the device, and no vendor server sits in the path (Source: Mika project documentation, 2026).
Mobile-Agent, in arXiv 2401.16158, documents the same architecture class: multimodal models that use visual perception to locate on-screen elements, then plan and operate (Source: Wang et al., arXiv, 2024). What moved is packaging and price.
The platform floor is a maintenance bill, not a purchase order
The stated requirements are narrow. Android 11 and up for chat, web and files; Android 12 and up to operate apps, with Android 14 or newer recommended; arm64 for the sandbox (Source: Mika project documentation, 2026). Android held 86.28% of Philippine mobile share in October 2026, down from 91.96% in January, still an enormous installed base above that floor (Source: StatCounter, 2026).
The catch is the control channel. App operation needs a one-time wireless debugging pairing, and Android disables wireless debugging after a Wi-Fi change or reboot (Source: Mika project documentation, 2026). A phone that reboots stops being an agent until somebody pairs it again.
Accessibility was a disability channel and is now a control channel
Google Play restricts the Accessibility API and the isAccessibilityTool flag to services designed to help people with disabilities access their device (Source: Google Play Console Help, 2026). Read that precisely: it is a distribution policy aimed at a capability class, enforced at the store.
Sideloaded APKs do not pass through that review, which is not the same as going unscanned. The gate that constrains accessibility-based automation is a store gate, so the constraint exists on one distribution path and not the other.
The control channel was built for assistive input, and it is now the cheapest route into software that never exposed an interface.
When the untrusted input is the screen
OWASP's LLM01:2025 entry on prompt injection notes that such inputs can affect the model even if they are imperceptible to humans, and do not need to be human-visible as long as the model parses them (Source: OWASP Gen AI Security Project, 2025).
For a screen-reading agent, the screen is that input: a rendered label, a notification banner, a block of text inside a page the agent opened. The person watching the main screen sees an app behaving like an app; the model sees tokens.
The prompt boundary and the display boundary are two different surfaces, and only one of them has a human on it. Any review that treats "the user can see what is happening" as a control is securing the wrong surface.
Who is the controller when the phone does the intake
RA 10173 Section 12 permits processing only when a condition exists, including the data subject's consent, or where processing is necessary and related to the fulfillment of a contract with that subject (Source: Republic Act 10173, 2012). Section 20(a) puts the duty on the personal information controller to implement reasonable and appropriate organizational, physical and technical measures against accidental or unlawful destruction, alteration and disclosure, and against any other unlawful processing (Source: Republic Act 10173, 2012).
Encrypting keys and memory on the device is a technical measure, and a real one. It is also only one of the three categories that clause names. Organizational and physical measures do not disappear because no vendor server exists, and "the data never left the phone" is not an answer when the phone decides what to read.
A team that ships an agent for its own operations is the controller for what it reads on a staff handset, agreement or not. Yano.AI building that for field staff would inherit the duty, and so would any agency issuing the handsets.
What the architecture review should ask
- Where does screen content go, and is it described in the privacy notice the user sees?
- What does the agent read while no human is watching the main display?
- Which measure satisfies Section 20(a) on the organizational side, and who audits it?
- What happens to the pairing, the memory and the run logs when the handset is returned, lost or resold?
FAQ
Q: Does an Android agent need an app API to drive another app?
A: No. It identifies controls from the rendered screen, the approach documented for multimodal mobile agents, then taps or types at those elements (Source: Wang et al., arXiv, 2024).
Q: Which Android version does app operation require?
A: Android 12 and up operates apps, Android 14 or newer is recommended, and chat, web and files work from Android 11 (Source: Mika project documentation, 2026).
Q: If the agent runs entirely on the phone, does RA 10173 still apply?
A: Yes. Section 20(a) binds the personal information controller to safeguards regardless of where the processing runs (Source: Republic Act 10173, 2012).
Q: Does sideloading avoid the Google Play accessibility restriction?
A: Sideloaded builds skip Play's review of accessibility services, but the restriction is a distribution policy on one channel, and privacy duties attach to the deployment rather than the channel (Source: Google Play Console Help, 2026).
Key Takeaway
The capability stopped being an enterprise integration and became a configuration on a handset, so the review has to move with it. Name the controller, the technical measure and the organizational one, and write down what happens when the phone is rebooted, resold or lost. Which of those can you answer today without opening a vendor contract?
Sources
- StatCounter, Mobile OS Market Share, Philippines
- Mika, on-device AI agent for Android
- Google Play Console Help, AccessibilityService policy
- OWASP Gen AI Security Project, LLM01:2025 Prompt Injection
- Republic Act No. 10173, Data Privacy Act of 2012
- NIST SP 800-124 Rev. 2, Guidelines for Managing the Security of Mobile Devices in the Enterprise
- Wang et al., Mobile-Agent, arXiv 2401.16158
Mentions
Phone-Use AI Agents: Mika vs ESP32 iPhone Agent TL;DR: Phone-use AI agents operate real mobile apps by looking at the screen and injecting taps, swipes, and keystrokes. Mika does this on Android from a hidden background screen using the system's own debugging and accessibility hooks. ESP32 iPhone Agent does it on iOS from outside the phone, with a $7 ESP32-S3 board acting as an AirPlay receiver and a USB mouse and keyboard. The platform decides the architecture. Android lets an app reach inside, while iOS forces the agent to act like a person holding the phone. Key Takeaways - Two of the most-starred new GitHub agent projects this autumn are phone-use agents. Mika (Kotlin, Apache-2.0, created 2026-10-29) had about 6,600 stars by 2026-11-04. ESP32 iPhone Agent (C, MIT, created 2026-10-11) had about 4,400. - Mika runs entirely on an Android 12+ phone. It enables Wireless debugging once to start a built-in helper, then operates apps on a separate background screen while you keep using the main one. - ESP32 iPhone Agent never touches iOS internals. The board mirrors the screen over AirPlay to "see" and presents itself as a USB HID mouse and keyboard to "act", with AssistiveTouch turning mouse clicks into taps. - Both projects are model-agnostic. Mika ships presets for more than 20 providers plus any OpenAI-compatible, Anthropic, Gemini, or OpenAI Responses endpoint, and the ESP32 agent accepts any OpenAI-compatible vision model. - The bottleneck is the observation channel, not the model. The ESP32 agent's BLE link limits it to 480×1040 at 5 fps or less, and each step takes a few seconds. Neither agent can see screens the OS refuses to capture, such as payment and passcode screens. - A phone agent holds your logged-in apps, so its safety model matters more than its benchmark. Mika's default "Ask" mode confirms only steps that spend money, which means messages, posts, and deletions run without asking. What is a phone-use AI agent? A phone-use AI agent is a loop that takes a screenshot of a mobile device, asks a model what to do next, and executes that action as a touch or keystroke on the real UI. It is the mobile equivalent of desktop "computer use". The agent works through the same interface a person does, not through app APIs. The appeal is coverage. Most consumer apps (messaging, food delivery, ride hailing, job boards) have no public API, or one that excludes the actions users care about. A phone agent does not need one. As Mika's README puts it, it "looks at the screen and finds buttons the way a person does: no app APIs, no per-app integration." The cost is the same as in [browser automation](/articles/Browser-Use-vs-Stagehand-vs-Playwright-MCP-AI-Agent-Browser-Automation): every step costs a model call, the UI can change under the agent, and recovering from mistakes is the agent's job. On a phone there are extra constraints. The OS sandboxes apps from each other, protected screens block capture, and the device is also the user's primary personal computer. Why are phone-use agents trending in November 2026? Two projects dominated this week's GitHub trending data for AI agents. Mika gained roughly 6,600 stars within a week of its 2026-10-29 creation. ESP32 iPhone Agent passed 4,300 stars in under a month. Both answer a question developers have been asking since desktop computer-use tools matured: can the same loop run on the device people actually use for most of their day? The answer depends on the platform, and these two projects show the two possible answers clearly. On Android, the OS exposes hooks (developer options, accessibility services, secondary displays) that let one app operate others. On iOS, no such hook is available to third-party apps, so the only route that needs no jailbreak and no developer-signed automation is to act on the phone from outside, through standard peripherals. How does Mika operate Android apps? Mika is a Kotlin and Jetpack Compose app distributed as an APK. Chat, web browsing, and file work run on Android 11+. Operating other apps requires Android 12+, and the project recommends Android 14+. Getting control. A one-time "Set up phone control" wizard walks the user through enabling Developer options, USB debugging, and Wireless debugging, then pairing with a 6-digit code. The setup guide explains that Mika "borrows Android's built-in 'Wireless debugging' to start a small built-in helper". After pairing, Mika turns on its own accessibility service, which avoids Android's "Restricted setting" block for sideloaded apps. No computer or cable is required. Seeing. Mika reads the UI in layers. Where an app exposes readable UI, it uses that. For custom-drawn interfaces or unlabeled controls, it falls back to text recognition bundled in the APK. With a vision-capable model, it can also locate buttons and fields from screenshots. Acting without stealing the screen. Most phone agents take over the display while they work. Mika instead "opens apps on a separate background screen and reads, taps, types and scrolls there while you keep using your phone." Android 14+ is recommended because the background screen gets its own input focus, so the agent's typing does not collide with yours. Several tasks can run at once. If two need the same app, the later one waits, and deadlocks between waiting tasks are detected rather than left hanging. Getting faster with use. Mika ships with built-in usage knowledge for about two dozen apps and a similar number of websites (WeChat, WhatsApp, Gmail, Telegram, Uber, Amazon, YouTube, and others). It also turns tasks it has completed into reusable skills. This is the same "learned skill" pattern we examined in [Hermes Agent's self-improving skill memory](/articles/Inside-Hermes-Agent-Self-Improving-Skill-Memory), applied to UI navigation. Model access. There is no Mika server. Users bring an API key (presets include DeepSeek, Qwen, Kimi, Zhipu, MiniMax, OpenRouter, Anthropic, Gemini, OpenAI, xAI, Mistral, Groq, Ollama, and LM Studio) or sign in with a ChatGPT, xAI, or OpenRouter account. A local model on the same network works through the "Custom" provider option. How does ESP32 iPhone Agent operate an iPhone? ESP32 iPhone Agent takes the opposite route. Its tagline is "No Mac, no developer account, no jailbreak". The agent never runs anything privileged on iOS. It uses the iPhone exactly as a person with a mouse and a screen mirror would. The README's architecture fits in three lines: iPhone ──Screen Mirroring (Wi-Fi)──▶ ESP32-S3: AirPlay receiver ──BLE──▶ ESP32 Agent app (same iPhone) ▲ own hotspot, H.264 passthrough decode → vision model → action └──────── USB HID mouse + keyboard ◀──────────── BLE commands ◀────────────────────┘ Eyes: AirPlay. The board opens its own Wi-Fi hotspot and appears in the iPhone's Screen Mirroring list as "ESP32 Agent". It does not decode video. It passes the H.264 stream through to the companion app over Bluetooth Low Energy. Because the phone's Wi-Fi is now joined to the board's hotspot, model calls go out over cellular data. Hands: USB HID. The board plugs into the phone's USB-C port (Lightning models need a camera adapter) and enumerates as a standard mouse and keyboard. With AssistiveTouch enabled, iOS shows a pointer, so a click becomes a tap and a drag becomes a swipe. Brain: an iOS app on the same phone. The app decodes the stream, sends screenshots to a vision model, and runs a planner and an actor. The actor's action space is tap, swipe, draw, type, press keys, go home, and finish. Planner and actor can use different models. Closing the loop. A USB mouse reports relative movement, so the app cannot simply "tap at (x, y)". It tracks the pointer, re-homes it in a screen corner, and, before precise taps, wiggles the pointer to find it on screen and correct the miss. After every action it waits for the screen to settle and tells the model whether anything changed. If the board, the hotspot, or the mirroring link drops, the run pauses and resumes from the same step. The project's history makes the design point. Version 0.1.0 needed a Mac or Linux computer running an AirPlay receiver and a Python server. The current version moved the receiver onto the $7 board and the agent onto the phone itself. A separate esp32agent CLI exposes the board as an MCP server, so a coding agent such as Claude Code can drive the iPhone from a laptop over Bluetooth. How do Mika and ESP32 iPhone Agent compare? | Dimension | Mika (Android) | ESP32 iPhone Agent (iOS) | |---|---|---| | Where the agent runs | Android app on the phone | iOS app on the phone, plus ESP32-S3 firmware | | Extra hardware | None | ESP32-S3 board with PSRAM and native USB (~$7) | | How it gets control | One-time Wireless debugging pairing, then its own accessibility service | USB HID mouse and keyboard plus AssistiveTouch | | How it sees | Readable UI, bundled OCR, optional vision-model screenshots | AirPlay mirror at 480×1040, ≤5 fps | | Uses the phone while it works? | Yes, on a separate background screen | No. The agent drives the visible screen. | | Parallel tasks | Yes, with per-app waiting and deadlock detection | One run at a time | | Text input | Typing plus sharing text into apps | ASCII keyboard input only for now | | Orientation | Not stated as a limitation | Portrait only for now | | Model requirement | Any of 20+ presets or a compatible custom endpoint | Any OpenAI-compatible vision endpoint | | Run review | Every step viewable in chat, wrap-up report on stop | Step-by-step replay with taps drawn on top, exportable traces | | Drive it from a computer | CLI + Skill or MCP | esp32agent MCP server over Bluetooth | | License | Apache-2.0 | MIT | The table supports one practical conclusion: these are not competing products, they are the two shapes the problem takes. If you are on Android, Mika's in-device approach is more capable on almost every axis. It runs in parallel without taking over the screen and needs no extra hardware. If you are on iOS, an external-peripheral design like the ESP32 board is currently the only way to get a general agent operating arbitrary apps without a jailbreak, and its constraints follow from that. Why is the observation channel the real bottleneck? Model quality gets the attention, but in both projects the hard engineering sits in getting pixels in and touches out reliably. The ESP32 agent states its budget plainly. BLE carries about 300–500 kbps, so the mirror is small (480×1040, ≤5 fps) and "each step takes a few seconds, most of it the model." Small frames help token cost and latency, but fine text and small icons are harder for a vision model to read. The pointer-correction loop exists because a relative mouse plus a low-frame-rate mirror means the agent is never quite sure where its finger is. That is a classic closed-loop control problem, solved by measuring after acting rather than trusting the command. Mika's layered perception (structured UI first, OCR second, vision third) points the same way. A model that reads a list of labeled elements makes fewer mistakes per step than one guessing coordinates from a screenshot. This mirrors the split in browser agents between accessibility-snapshot tools and screenshot-based tools. Pick the richest observation the platform gives you, and fall back to pixels only when you must. Two failure modes are shared and fixed by the OS, not the agent: - Protected screens are invisible. iOS does not mirror passcode entry, Apple Pay, or DRM video. Android payment and password pages block screenshots. Both projects hand these steps to the human. That is the correct behavior, and it is not a bug to engineer around. - The observation channel is fragile. Android turns Wireless debugging off when you change networks or restart (Mika re-enables it with permission). AirPlay mirroring stops when an iPhone locks. Any production-minded phone agent needs explicit pause-and-resume semantics, which both projects implement. How should you think about phone-agent security? A phone agent is the most privileged agent most people will ever run. It inherits every logged-in session on the device: messaging, email, banking, shopping, social accounts. Every screen it reads is untrusted input. A chat message, a product listing, or a web page can contain text aimed at the model rather than the user. That is the same indirect prompt-injection setup behind [GitLost](/articles/GitLost-Prompt-Injection-GitHub-AI-Agent-Repo-Leak), on a device that holds all three legs of the lethal trifecta by default: private data, untrusted content, and outbound channels. Read the permission models carefully: - Mika's default "Ask" mode confirms only steps that spend money. Its own docs say that "sending messages, posting, submitting forms, deleting and changing settings are not asked about; they are just done." "Autonomous" mode removes even the payment confirmation. Logins, verification codes, and payment passwords are always handed to the user. - ESP32 iPhone Agent relies on supervision. Its responsible-use note says, "The agent acts with your hands: watch it, and keep the stop button within reach." The app also currently uses a private iOS API to send itself to the background, which the maintainers intend to replace before an App Store release. Neither design is reckless for a personal tool, but neither is an authorization layer you could put in front of a business workflow. If you build on top of either, apply the principle from our [LLM vs policy-engine authorization analysis](/articles/AI-Agent-Authorization-LLM-vs-Policy-Engine). The model proposes, and a deterministic rule decides. On a phone, that means an allowlist of apps per task, a confirmation for any outbound message or post (not just payments), and no access to apps the task does not need. A smaller but real point: automating third-party apps may violate their terms of service, and Mika's README says so directly. Batch actions such as greeting recruiters, posting, or watching ads for perks are exactly what platforms' anti-automation systems look for. Mika paces itself on websites and hands anti-bot block pages to the user instead of forcing through. Copy that behavior if you build your own. When should you use a phone agent instead of an API or browser agent? Use the most structured interface available, and drop to the phone only when nothing better exists: 1. An official API or MCP server exists. Use it. It is faster, cheaper, auditable, and does not break when the UI is redesigned. 2. The service has a full-featured website. Use a [browser agent](/articles/Browser-Use-vs-Stagehand-vs-Playwright-MCP-AI-Agent-Browser-Automation). DOM and accessibility snapshots give a richer observation than a phone screenshot. (Mika's built-in desktop-class browser follows the same logic inside the phone.) 3. The capability exists only in a mobile app. That is the phone agent's territory. Examples are app-only messaging, delivery and ride apps, and on-device settings like alarms. Also consider phone agents for testing your own mobile app's real user flows, where operating the real UI is the whole point. For the third case, pick by platform: an in-device agent like Mika on Android, an external-peripheral agent like ESP32 iPhone Agent on iOS. Budget for a few seconds and one model call per step. A ten-step task is closer to a minute than to an instant, so phone agents suit errands you would otherwise do by hand, not latency-sensitive automation. What should engineers building phone agents copy from these two projects? Several design choices are worth copying, whatever platform you target: - Verify after every action. The ESP32 agent checks whether the screen changed and corrects pointer misses. An agent that assumes its last tap worked compounds errors quickly. - Make runs resumable. Both projects pause on a lost connection or a hand-off to the human and resume from the same step. Mika's stop-time wrap-up ("what it did, what it saw, how many times it sent something, and what is left") makes a later "continue" safe. - Record everything. Step-by-step replay with drawn taps (ESP32) and per-step inspection in chat (Mika) are what make a non-deterministic agent debuggable. They are also the raw material for the [trace-to-regression-test workflow](/articles/AI-Agent-Regression-Testing-Production-Traces-to-CI) any serious agent needs. - Separate the human's screen from the agent's. Mika's background screen is the biggest usability step in this category. An agent that blocks the phone for minutes is one people stop using. - Keep prompts as editable data. The ESP32 project ships its planner and actor prompts as plain text files, so you change behavior by editing a file, not by rewriting code. - Expose the device as a tool. Both projects let a desktop coding agent drive the phone over MCP. The phone becomes one more tool in a larger agent, not a separate product. What does this mean for the future of mobile AI agents? These projects show that general phone-use agents are now buildable by small open-source teams with off-the-shelf models and, on iOS, a $7 board. What they cannot solve is that the OS is not designed for an agent as a first-class user. Android's route runs through developer options. iOS's route runs through accessibility features and USB peripherals meant for people with motor impairments. Both work, but both rely on mechanisms built for other purposes. Until mobile platforms offer a sanctioned agent interface with scoped permissions per app, the engineering priorities are clear. Invest in reliable observation and verification, make every run resumable and replayable, and put deterministic limits on what the agent may send or spend. The model is the easy part. FAQ What is the difference between Mika and ESP32 iPhone Agent? Mika is an Android app that operates other apps from inside the phone. It uses Wireless debugging and its own accessibility service to drive apps on a hidden background screen. ESP32 iPhone Agent operates an iPhone from outside, with an ESP32-S3 board that receives an AirPlay mirror of the screen and acts as a USB mouse and keyboard. The difference follows from what each OS allows. Does ESP32 iPhone Agent require a jailbreak or a Mac? No. The project needs no jailbreak and no Mac at run time. A Mac with Xcode is needed only to flash the board and build the app yourself. The board can instead be flashed from Chrome or Edge, and the app is available through TestFlight. The iPhone needs iOS 18 or later and cellular data, because its Wi-Fi is joined to the board's hotspot while the agent runs. Which AI models work with phone-use agents like Mika? Mika supports presets for more than 20 providers, including Anthropic, OpenAI, Gemini, DeepSeek, Qwen, Kimi, xAI, Mistral, Groq, Ollama, and LM Studio. It also accepts any OpenAI-compatible, Anthropic, Gemini, or OpenAI Responses endpoint. ESP32 iPhone Agent works with any OpenAI-compatible chat API that accepts images, including self-hosted vLLM, SGLang, llama.cpp, or Ollama servers. Can a phone AI agent make payments or log in for me? Neither project automates logins, verification codes, or payment passwords. Both hand those steps to the user, partly because the OS blocks screen capture on those pages. Mika's default Ask mode also asks before placing an order, but it does not ask before sending messages, posting, or deleting. Review those defaults before giving a phone agent access to important accounts. Are phone-use AI agents safe to use? They are as safe as their permission model, and by default that model is broad. A phone agent can act in every app you are logged into, and every screen it reads may contain prompt-injection text. Use confirmation modes, watch runs on important accounts, limit tasks to the apps they need, and remember that automating third-party apps may breach their terms of service. How fast are phone-use AI agents? Expect a few seconds per step, mostly spent waiting on the model. ESP32 iPhone Agent mirrors the screen at up to 5 fps over a 300–500 kbps Bluetooth link, so a multi-step task usually takes tens of seconds to minutes. Mika avoids blocking the user by working on a background screen, which matters more for usability than raw step speed. Sources - Mika GitHub repository - Mika features in detail - Mika phone control setup guide - Mika Ask and Autonomous modes - ESP32 iPhone Agent GitHub repository - Seeed Studio XIAO ESP32S3 getting started --- Originally published at fp8.co. Subscribe for weekly AI engineering analysis at fp8.co.
本来用 mika 来做安卓控制的，现在又得换回 ADB 直接控制。 是好项目 wangzan101/mika
Mika (4.6k stars) is an Apache-2.0 Android agent that runs your apps and the web on a hidden background screen — no Mika server, data stays on the phone. #Android #AIAgents wangzan101/Mika
Mika 的玩法越来越丰富了： 1、Mika 干不了的活，直接安排其他 AI 干，多个 AI 一起干，比强更强 2、找到免费的 API key，注册白嫖一波 期待大家挖掘 Mika 更多的使用场景。 Mika：wangzan101/mika t.co
Phone Agent 看来最近很火，现在有不少开源项目了👍 这个兄弟的开源项目在wangzan101/mika 听说dsh go做的Phone Agent完成度更高，但是我觉得控制移动端常见的玩法也就是干爬虫的活了😂 比如可以不用抓包逆向爬想了解的数据，不用写繁琐需要更新的appium uiautomater这类自动化脚本 能落地的商业玩法能想到的也就是做自动化测试/ugc平台截流，但是不知道效率如何？ 没玩，感觉对我来说能省的最多的事就是外卖优惠券比价了🥲
Mika 群的的老哥太强了，基于 Mika 二开，现在我已经看不懂他这个是用来干啥的了，有谁能看明白吗？🤣🤣🤣 Mika：wangzan101/mika t.co
Mika 现在已经支持语音功能，随时随地，想到，就能做到，欢迎下载使用体验。 Mika：wangzan101/mika
The Trust Boundary Moved to Your Pocket: On-Device Android Agents and the Controller Duty Last month a phone started tapping through apps it had never been given an API for, with no SDK, no per-app integration and no vendor server in the path. In the Philippines that handset is almost certainly Android, at 86.28% of mobile OS share against 13.71% for iOS in October 2026 (Source: StatCounter, 2026). The National Privacy Commission enforces RA 10173, and the duties in that law attach to whoever decides how personal information is processed, not to whoever owns the server it would have run on (Source: Republic Act 10173, 2012). !Infographic The integration layer is now hardware people already own Enterprise mobile automation used to be a procurement exercise: device management, a per-app SDK, a middleware tier between phone and system of record. NIST SP 800-124 Rev. 2, published in May 2023, treats the problem as managing mobile device security across a fleet, which presumes an owner with a fleet (Source: NIST SP 800-124 Rev. 2, 2023). The current generation of Android agents inverts that shape. One Apache-2.0 application, no vendor server, no per-app integration, no SDK negotiated with each app's publisher. The integration surface is the screen the user is already looking at. What the agent actually does Mika, an open-source Android agent, operates apps and the web on a separate invisible background display. It opens apps, taps, types and scrolls there while the owner keeps the main screen, and it finds buttons the way a person does, by looking at them, with no app APIs or per-app integration (Source: Mika project documentation, 2026). Each conversation gets a Linux sandbox that runs Python, Node and shell. Keys and memory are stored encrypted on the device, and no vendor server sits in the path (Source: Mika project documentation, 2026). Mobile-Agent, in arXiv 2401.16158, documents the same architecture class: multimodal models that use visual perception to locate on-screen elements, then plan and operate (Source: Wang et al., arXiv, 2024). What moved is packaging and price. The platform floor is a maintenance bill, not a purchase order The stated requirements are narrow. Android 11 and up for chat, web and files; Android 12 and up to operate apps, with Android 14 or newer recommended; arm64 for the sandbox (Source: Mika project documentation, 2026). Android held 86.28% of Philippine mobile share in October 2026, down from 91.96% in January, still an enormous installed base above that floor (Source: StatCounter, 2026). The catch is the control channel. App operation needs a one-time wireless debugging pairing, and Android disables wireless debugging after a Wi-Fi change or reboot (Source: Mika project documentation, 2026). A phone that reboots stops being an agent until somebody pairs it again. Accessibility was a disability channel and is now a control channel Google Play restricts the Accessibility API and the isAccessibilityTool flag to services designed to help people with disabilities access their device (Source: Google Play Console Help, 2026). Read that precisely: it is a distribution policy aimed at a capability class, enforced at the store. Sideloaded APKs do not pass through that review, which is not the same as going unscanned. The gate that constrains accessibility-based automation is a store gate, so the constraint exists on one distribution path and not the other. The control channel was built for assistive input, and it is now the cheapest route into software that never exposed an interface. When the untrusted input is the screen OWASP's LLM01:2025 entry on prompt injection notes that such inputs can affect the model even if they are imperceptible to humans, and do not need to be human-visible as long as the model parses them (Source: OWASP Gen AI Security Project, 2025). For a screen-reading agent, the screen is that input: a rendered label, a notification banner, a block of text inside a page the agent opened. The person watching the main screen sees an app behaving like an app; the model sees tokens. The prompt boundary and the display boundary are two different surfaces, and only one of them has a human on it. Any review that treats "the user can see what is happening" as a control is securing the wrong surface. Who is the controller when the phone does the intake RA 10173 Section 12 permits processing only when a condition exists, including the data subject's consent, or where processing is necessary and related to the fulfillment of a contract with that subject (Source: Republic Act 10173, 2012). Section 20(a) puts the duty on the personal information controller to implement reasonable and appropriate organizational, physical and technical measures against accidental or unlawful destruction, alteration and disclosure, and against any other unlawful processing (Source: Republic Act 10173, 2012). Encrypting keys and memory on the device is a technical measure, and a real one. It is also only one of the three categories that clause names. Organizational and physical measures do not disappear because no vendor server exists, and "the data never left the phone" is not an answer when the phone decides what to read. A team that ships an agent for its own operations is the controller for what it reads on a staff handset, agreement or not. Yano.AI building that for field staff would inherit the duty, and so would any agency issuing the handsets. What the architecture review should ask - Where does screen content go, and is it described in the privacy notice the user sees? - What does the agent read while no human is watching the main display? - Which measure satisfies Section 20(a) on the organizational side, and who audits it? - What happens to the pairing, the memory and the run logs when the handset is returned, lost or resold? FAQ Q: Does an Android agent need an app API to drive another app? A: No. It identifies controls from the rendered screen, the approach documented for multimodal mobile agents, then taps or types at those elements (Source: Wang et al., arXiv, 2024). Q: Which Android version does app operation require? A: Android 12 and up operates apps, Android 14 or newer is recommended, and chat, web and files work from Android 11 (Source: Mika project documentation, 2026). Q: If the agent runs entirely on the phone, does RA 10173 still apply? A: Yes. Section 20(a) binds the personal information controller to safeguards regardless of where the processing runs (Source: Republic Act 10173, 2012). Q: Does sideloading avoid the Google Play accessibility restriction? A: Sideloaded builds skip Play's review of accessibility services, but the restriction is a distribution policy on one channel, and privacy duties attach to the deployment rather than the channel (Source: Google Play Console Help, 2026). Key Takeaway The capability stopped being an enterprise integration and became a configuration on a handset, so the review has to move with it. Name the controller, the technical measure and the organizational one, and write down what happens when the phone is rebooted, resold or lost. Which of those can you answer today without opening a vendor contract? Sources - StatCounter, Mobile OS Market Share, Philippines - Mika, on-device AI agent for Android - Google Play Console Help, AccessibilityService policy - OWASP Gen AI Security Project, LLM01:2025 Prompt Injection - Republic Act No. 10173, Data Privacy Act of 2012 - NIST SP 800-124 Rev. 2, Guidelines for Managing the Security of Mobile Devices in the Enterprise - Wang et al., Mobile-Agent, arXiv 2401.16158