onehumanbeing

How to give an AI agent a real iPhone with a $7 ESP32 Most phone automation needs a computer on a cable, a developer account, or a jailbreak. ESP32 iPhone Agent takes a different route: a small ESP32-S3 board plugs into the iPhone and acts as the agent's eyes and hands, while an iPhone app runs the vision model loop. At run time it is just the phone and the board. This guide walks through the setup from the project's README, start to finish, so you can type a goal like "Open Settings and go to Wi-Fi" and watch a vision model do it on a real iPhone. How it works in one paragraph The board opens its own Wi-Fi hotspot and shows up in the iPhone's Screen Mirroring list as ESP32 Agent. The phone mirrors its screen to the board over AirPlay, and the board forwards that stream to the app over Bluetooth LE. The app sends screenshots to any OpenAI-compatible vision model, which plans the next action. The board is also a standard USB mouse and keyboard, so with AssistiveTouch on, a click is a tap and a drag is a swipe. Loop until done. What you need - An ESP32-S3 board with native USB and PSRAM. The README tested a Seeed Studio XIAO ESP32S3 (plain version, 8 MB PSRAM, not Sense, not Plus), listed at about $7. - An iPhone on iOS 18 or later with cellular data. USB-C iPhones plug in directly. Lightning models need a Lightning to USB Camera Adapter. The phone powers the board. - A vision model behind any OpenAI-compatible chat API that accepts images: a cloud API or your own server (the README names vLLM, SGLang, llama.cpp and Ollama). - A Mac with Xcode, once. Only to flash the board and build the app. It is not needed at run time. Step 1: Flash the board Plug the board into your Mac. Install arduino-cli, the ESP32 core 3.3.x and NimBLE-Arduino 2.5.x, then: arduino-cli lib install "NimBLE-Arduino@2.5.1" hardware/flash.sh # optional: [serial port] [--debug] Always build with flash.sh. It applies the BLE buffer sizes the firmware needs. Step 2: Build and install the iPhone app cp ios/Config/Local.xcconfig.example ios/Config/Local.xcconfig # set DEVELOPMENT_TEAM and IA_BUNDLE_ID open ios/IPhoneAgent.xcodeproj # run the IPhoneAgent scheme on your iPhone A free Apple ID can sign it, but you have to re-sign every 7 days. The one-tap hotspot join needs the Hotspot Configuration capability (a paid account). Without it, the app shows the hotspot password so you can join by hand. Step 3: Run the four setup screens Plug the board into the iPhone and open ESP32 Agent. Setup checks each step itself: 1. Plug Me In: pair the board over Bluetooth. 2. Let Me See the Screen: join the board's Wi-Fi, then Control Center, Screen Mirroring, ESP32 Agent. 3. Let Me Use the Pointer: turn on AssistiveTouch. The app calibrates the pointer on its own. 4. Give Me a Brain: enter your model endpoint and key (stored in the iOS Keychain), then pick a vision model from the endpoint's list or type a model ID. You can set separate planner and actor models. Step 4: Give it a task Type a goal such as "Open Settings and go to Wi-Fi" and press Start. Stop it any time from the Dynamic Island. Once you have tasks you repeat, Siri, Shortcuts, widgets and Control Center can run saved tasks. Step 5: Review what it did Every run is kept on the phone. Open Runs, pick a run and replay it step by step with the taps and swipes drawn on top. To debug with another agent, use Runs, open a run, ⋯, Export trace to get the screenshots, actions and annotations. Tuning it for your workflow The planner and actor prompts are plain text files in prompts/ and are bundled with the app. Edit them, rebuild the IPhoneAgent scheme, and compare the new run against the old one in Runs. If you suspect the board rather than the app, hardware/tools/board_test.py plays the app's role over BLE from a Mac (see hardware/tools/README.md and docs/ble-protocol.md). Limits to know before you start - Bluetooth LE carries about 300 to 500 kbps, so the mirror is small (480x1040, at most 5 fps) and each step takes a few seconds, mostly model time. - iOS does not mirror some screens (passcode entry, Apple Pay, DRM video), so the agent cannot see them. Mirroring stops when the phone locks. - While it runs, the phone's Wi-Fi is the board's hotspot, so the model is reached over cellular data. - Keyboard input is ASCII only, and landscape apps are not supported yet. - The README warns: only follow the install steps in the README, and never run "quick install" one-liners posted in issues or comments. They are not from the maintainers. Only automate devices and accounts you own or are allowed to use. The agent acts with real hands on the phone, so watch it and keep the stop button within reach. More ways to use it We wrote up five concrete use cases for this repo, with steps and copy-paste agent prompts: - Have an AI model complete a task on a real iPhone - Replay and export phone agent runs for analysis - Start saved phone tasks from Siri or Shortcuts - Edit the planner and actor prompts for a custom workflow - Test the ESP32 board over BLE from a Mac Project credit: ESP32 iPhone Agent by Henry Yuan (onehumanbeing/esp32-iphone-agent), MIT licensed, with the AirPlay part under GPL-3.0. --- Get one email every Monday with the week's hottest AI repos and exactly how to use them: subscribe free at dedupedai.com

Your iPhone can now have an AI agent in $7 without jailbreaking or no Mac required. It navigates, types, and draws by itself. A $7 ESP32 gives it eyes (AirPlay) and hands (USB HID). run locally Qwen model & it open-sources, iOS 18 or later, - onehumanbeing/esp32-iphone-agent t.co

AI agent runs on real iPhone using ESP32 ($7), AirPlay vision, USB HID hands, OpenAI-compatible model, no Mac, no dev account, no jailbreak #openai #artificialintelligence #ibl #news #opensource #ai #aiagent #technology #usb

5 GitHub Tools That Feel Illegal to Know #Shorts #AI #OpenSource 5 open-source tools on GitHub that feel illegal to know 🛠️ All free, and all launched in the last two weeks. 1. Anatomy: a Claude Code skill that turns any idea into an interactive isometric diagram. wheresryan22/anatomy 2. ESP32 iPhone Agent: a ~$7 ESP32-S3 board lets an AI model drive a real iPhone. No jailbreak needed. onehumanbeing/esp32-iphone-agent 3. ManimGX: your agent writes the Python, and out come 3D math animations (90.6× faster than ManimCE, per the project's own benchmark). academa-labs/manimgx 4. DigUp: search your files by describing what's in them, on-device, with Google DeepMind's EmbeddingGemma 2. ARahim3/DigUp 5. Helix Foundry: connect Stripe, PostHog and your database, then ask it questions, locally. HelixDB/helix-foundry The money ideas in the video are suggestions, not results. All credit goes to the creators. Forge: AI news, vibe-coding tips and open-source tools for builders. Subscribe for a new find every week. // build what you imagine. ship it today. #Shorts #AI #GitHub #OpenSource #VibeCoding

onehumanbeing/esp32-iphone-agent: Let an AI model use a real iPhone with a $7 ESP32: AirPlay eyes, USB HID hands, any OpenAI-compatible vision model. No Mac, no developer account, no jailbreak. onehumanbeing/esp32-iphone-agent

Top 5 open source do GitHub | Vol. 11 #shorts Os projetos #5 a #1 do Top 15 open source do GitHub, Vol. 11, 10 segundos para cada um. #5 Rustrak: rustrak/rustrak #4 Codewhale: codewhale-hq/Codewhale #3 ESP32 iPhone Agent: onehumanbeing/esp32-iphone-agent #2 Artemis: google/artemis #1 Mission Control: builderz-labs/mission-control Vídeo completo com os 15: youtu.be Oferecimento: Hallen Tech, hallentech.com #shorts #opensource #github

Mais 15 projetos open source do GitHub que valem o seu tempo | Vol. 11 Mais 15 projetos open source do GitHub que valem o seu tempo, em contagem regressiva e com 10 segundos para cada um: de uma placa de US$ 7 que deixa a IA usar um iPhone a um painel para operar agentes de IA, passando por avatar animado, casa 3D no Home Assistant e o fim das travadas de shader nos jogos. 📺 Décimo primeiro volume da série Open source do GitHub. Escolhidos pela análise da lista inteira (estrelas, frequência de commits, último commit, README, documentação, capturas e site), conferida em 21/10/2026. Qual desses você vai testar primeiro? Conta nos comentários! ⏱️ CAPÍTULOS E LINKS 0:00 Abertura 0:10 #15 tapflow: simuladores de iPhone e emuladores Android no navegador, no seu servidor (1,2 mil estrelas) jo-duchan/tapflow 0:20 #14 Mesh Avatar Studio: transforma uma ilustração num avatar 2D animado que pisca, fala e mexe a cabeça shinshin86/mesh-avatar-studio 0:30 #13 openGym: treinos guiados, séries e peso no celular, com os dados no seu servidor (14,5 mil estrelas) DuarteSantos8/openGym 0:40 #12 SCSKiller: compila os shaders antes do jogo para acabar com as travadas no PC (Windows) (3,3 mil estrelas) BlueHeisenberg/SCSKiller 0:50 #11 OpenFilm: o seu agente de código faz o vídeo como páginas web e você edita no navegador (1,9 mil estrelas) openfilm/openfilm 1:00 #10 Humanizer: skill que faz texto de IA soar escrito por gente, com base no guia da Wikipédia (59,3 mil estrelas) blader/humanizer 1:10 #9 NeonPlan 3D: planta 3D neon da sua casa dentro do Home Assistant, com luzes e câmeras ao vivo Mastershort/neonplan3d 1:20 #8 MangoDisk: limpeza de disco, análise de armazenamento e privacidade para macOS, Windows e Linux (4,5 mil estrelas) harry0703/MangoDisk 1:30 #7 OpenUsage: uso das assinaturas de IA de código na barra de menu do Mac (4,4 mil estrelas) robinebers/openusage 1:40 #6 Finder Menu: as opções que faltam no clique direito do Finder: novo arquivo, terminal aqui, copiar caminho nikgrbn/finder-menu 1:50 #5 Rustrak: rastreador de erros self-hosted compatível com os SDKs do Sentry rustrak/rustrak 2:00 #4 Codewhale: agente de código no terminal que funciona com mais de 40 provedores de modelos (41,1 mil estrelas) codewhale-hq/Codewhale 2:10 #3 ESP32 iPhone Agent: uma placa ESP32 de US$ 7 deixa um modelo de IA usar um iPhone de verdade, sem computador (3,5 mil estrelas) onehumanbeing/esp32-iphone-agent 2:20 #2 Artemis: IA e testes usando celulares Android reais, com MCP para Claude Code e outros (13,6 mil estrelas) google/artemis 2:30 Oferecimento Hallen Tech 2:44 #1 Mission Control: painel self-hosted para operar agentes de IA: tarefas, execuções e gastos (6,4 mil estrelas) builderz-labs/mission-control 2:54 Encerramento 🧡 Oferecimento: Hallen Tech, liderança de tecnologia sob demanda, software, cloud e segurança para a sua empresa. Site: hallentech.com E-mail: contato@hallentech.com Instagram: instagram.com LinkedIn da Hallen Tech: linkedin.com LinkedIn do Eddwan: linkedin.com Se curtiu, deixa o like, segue o canal e ativa as notificações para a próxima lista. #opensource #github #programacao #ferramentas

Been quiet lately because I'm trying to get onehumanbeing/esp32-iphone-agent onto the App Store. Still waiting on review. Expect some bugs, it's not perfect yet, but it's a big step up. If it's approved, I launch and push the new code tomorrow. You can actually try it by Download the APP Then set up your ESP32 With the tool on the website. If it's rejected, I'll resubmit, and I'll push the code either way. What's in the next release (not on GitHub yet): - Screen streams over Wi-Fi, Bluetooth handles control - OTA firmware updates from inside the app, with signature check and rollback - Record a flow once, it replays when the screen matches, falls back to AI when it doesn't - New CLI / MCP / Skill so Codex and Claude Code can drive a real iPhone - Free trial without an API key, with queue and quota hints (using my local llm so mostly it will be busy) - Lots of fixes: pairing, reconnects, text input, model config, small features. If you're planning PRs to main, please hold off until I push. Sorry for the wait! Meanwhile legacy got a lot better too: Windows support, better calibration, fixes for mirroring, connections and model compatibility. I merged a bunch of community PRs. Thanks to everyone sharing forks, sending code, testing and reporting bugs 🙏

An iPhone agent in Safari, with an ESP32 as its Bluetooth hands Our open-source project (MIT): cccat6/iphone-agent-harness Demo and browser installer · Setup guide We built iPhone Agent Harness to explore a simple interaction: describe a phone task in Safari, let a vision model inspect the screen, and use a nearby ESP32 to supply the taps and keyboard input. Our current demo shows the agent setting an alarm in the iPhone Clock app. The phone stays wireless. The ESP32 is separately powered and sits within Bluetooth range, while both devices use the same normal Wi-Fi network. How the pieces fit together The ESP32 is an input bridge. Safari runs the conversation, screen capture and model interaction. iPhone screen -> Safari screen capture -> selected context sent to your chosen vision model -> action plan validated by the Harness -> local WebRTC commands to the ESP32 -> Bluetooth mouse / keyboard input to the iPhone -> next screen observation AssistiveTouch lets the phone respond to pointer input. The board does not run the model or decode the video stream. Keeping those responsibilities on the phone lets the firmware concentrate on delivering input. That division also makes setup different from a native app workflow: the agent interface is a website, and firmware installation is available through desktop Chrome or Edge. There is no native iOS agent app to build and sign for this route. From a demo to repeatable tasks A useful phone agent needs more than a single prompt. The Harness includes conversation management and skills that can be customized, shared and updated. Skills describe how to handle recurring tasks and improve the agent's taps, typing and gestures. MCP tools can also be configured in Settings. You choose the model service. Conversations, screenshots and credentials are stored in the current browser. Selected messages and images go to your chosen AI provider, and configured MCP requests go to their respective servers. Browser storage does not mean that model inference happens locally on the phone. Trying it 1. Check the README for supported chips and flash requirements. Firmware is provided for C2, C3, C5, C6, S3 and classic ESP32. 2. Flash the board with the browser installer. 3. Join your Wi-Fi, pair the bridge over Bluetooth and enable AssistiveTouch. 4. Open the Harness in Safari, configure your model access, and follow the screen-sharing instructions. 5. Start with a short task you can observe, then inspect the result. The current requirement is iOS / iPadOS 27 or later. The phone and bridge need to share a Wi-Fi network. The setup guide also covers the optional clipboard Shortcut and installing skills. Scope and tradeoffs The alarm recording demonstrates one real workflow. It is not a benchmark of arbitrary apps or long-running tasks. A model can misread the screen or choose an incorrect action, so task success needs to be checked on the phone. Compatibility and connection feedback from additional devices are especially useful. This project was inspired by onehumanbeing/esp32-iphone-agent, which we credit in the README. We are exploring a Safari + Bluetooth HID approach; that project uses a native iOS app with an AirPlay/USB HID path. If you build with ESP32, WebRTC or phone automation, we would welcome feedback on pairing, Wi-Fi/BLE coexistence and the tasks you would like to automate. **Source, firmware and issue tracker: cccat6/iphone-agent-harness**

We built an ESP32 Bluetooth HID bridge that lets an AI agent in Safari operate an iPhone (open source) **Our GitHub repository (MIT): cccat6/iphone-agent-harness** **Demo and browser installer: igent.cc** We built iPhone Agent Harness, an open-source ESP32 Bluetooth HID bridge for an AI agent running in Safari. Our demo shows it setting an alarm in the iPhone Clock app. The hardware/software split is the interesting part: Safari captures the screen, sends selected context to your chosen vision model, validates the action plan, and sends commands over local WebRTC. The ESP32 supplies Bluetooth mouse and keyboard input through AssistiveTouch. The board does not run the model or decode video. The powered board sits nearby while the phone stays wireless. Both use the same normal Wi-Fi network. A Chrome/Edge browser installer flashes the firmware. The README lists chip and flash requirements for C2, C3, C5, C6, S3 and classic ESP32. Requirements: iOS/iPadOS 27+, Safari screen sharing, AssistiveTouch, and access to a vision model. Conversations and keys stay in browser storage; selected context goes to your model provider. Skills can be customized and shared. Inspired by onehumanbeing/esp32-iphone-agent, credited in our README. We explore Safari + Bluetooth HID instead of a native app + USB HID. I'd appreciate feedback from Wi-Fi + BLE HID builders: which board/iPhone combinations and connection edge cases should we prioritize?