I built an AI that only works when my phone is in my pocket
---
title: I built an AI that only works when my phone is in my pocket
published: true
tags: devchallenge, hf26challenge, gemma, opensource
cover_image: drift-three-nu.vercel.app
---
This is a submission for the Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass
What I Built
Every "go outside" app I have tried wants me to look at it once I'm outside: a map, a streak, a badge, a feed. I wanted the opposite: an app that is finished with me before I reach the front door.
Drift is a walk with no destination. You choose how long and a mood (quiet, colors, old things, sounds). Gemma 4, running in your browser, writes a short set of playful rules. An open-weight voice reads them to you, one at a time, while the phone sits in your pocket:
> "Trace the outline of the oldest brick you can find with your hand."
> "Follow the faint scent that seems to remember rain."
> "Stand still until the distant, muted sound settles into your bones."
(Those are real rules Gemma wrote during testing.)
The idea is borrowed from the dérive (French for "drift"), a practice from the 1950s Situationists in Paris. They walked cities by following rules instead of routes, to notice what a destination hides. Seventy years later we have a better excuse never to look up, so I gave the rules to an AI and took the screen away.
When you get home, you talk for a minute about what you noticed. Whisper transcribes it on the device, and Gemma writes it up as an entry in a field notebook. Each entry leaves one line of memory, so the next drift leans toward what you enjoyed.
There is one number Drift actually cares about: how many seconds your screen was on during the walk.
Meet Pip 🦔, who keeps you company before and after. Pip watches your finger, falls asleep while you walk, and gets star-eyed if you barely looked at your phone.
!Drift: choose a mood, walk with your phone locked, see your screen time
I took it outside
I opened Drift on my iPhone in Safari, and Gemma 4 wrote me a drift called The Quiet Current, right there on the phone. Then I went out and walked for about ten minutes.
I noticed things I hadn't noticed before: how the leaves were moving, and how the wind felt when it touched me. It was hot out, but there was a cool edge underneath the heat. It's October, and even in the sun I could feel a little of the cold that's coming. It was a very engaging feeling, and honestly I can't quite put it into words.
Then I came home and looked at what Drift had recorded, and it caught me out:
!My first real drift in the field notebook
| | |
|---|---|
| Drift | The Quiet Current |
| Device | iPhone, Safari, Gemma 4 running on the phone |
| Actually walked | about 10 minutes |
| Walk the app counted | 2 minutes (I ended the drift early and kept walking) |
| Screen on | 2 min 5 sec, the whole time it was counting |
| Notebook entry | "Thank you." |
Two honest lessons from one short walk:
- I didn't lock my phone. I kept Drift open in my hand, so the screen-time counter did exactly its job and recorded that the screen was on the whole time. The best part of the walk happened after I stopped looking at the app. That's the whole thesis of Drift, demonstrated on its own author.
- My notebook entry just said "Thank you." That's a well-known Whisper failure: when it hears silence or almost nothing, it outputs "Thank you." (it was trained on a lot of subtitles). Drift saved that as my notebook entry. It now checks whether a recording is too short or too quiet, drops Whisper's silence phrases, and asks you to try again. There's a test for it with real Whisper.
Next time the phone goes in my pocket.
What a drift actually looks like
Here is a real one, written by Gemma 4 E2B running on WebGPU in my browser. I picked 30 minutes and the mood old things. Gemma wrote this in 11.5 seconds:
> Echoes in the Cobblestones
> 1. Find the object with the deepest shadow beneath a large leaf.
> 2. Listen intently to the rhythm of the oldest wood.
> 3. Touch the texture of the wall that feels the longest.
> 4. Follow the faint scent that seems to remember rain.
> 5. Stand still until the distant, muted sound settles into your bones.
> 6. Allow your gaze to simply rest on the most enduring shape nearby.
None of these need a map, and none of them could be followed while looking at a phone. That's the point. You can't do "touch the texture of the wall that feels the longest" from a screen.
Every time you press Prepare the rules are new. Other drifts Gemma wrote while I was building: The Echo Chamber Wander, Chromatic Descent, Dust & Echoes.
Then Kokoro reads them in a calm voice, with a soft chime before each rule and minutes of silence after it. The rendered walk for The Echo Chamber Wander came out at 31 minutes of audio, from a single file. On the deployed site I watched the counter tick from "Starting" to "Rule 1 of 6" while the audio played, with the page doing nothing else.
| | Time on my laptop (Apple GPU, WebGPU) |
|---|---|
| Gemma 4 writes a drift | 11.5 s |
| Gemma 4 writes a notebook entry | 2.2 s |
| First-time model download (2 GB) | ~11–14 min on my connection |
| Network needed during the walk | none |
Demo
🌿 Try it: drift-three-nu.vercel.app · Code: codeswithroh/drift. I've run it on an iPhone (Safari) and a MacBook (Chrome). It needs WebGPU: Chrome or Edge, or Safari 26+. The first visit downloads about 2.5 GB of models, so use Wi-Fi.
!The Drift flow in under a minute
| Choose | Prepare | Walk | Screen time | Notebook |
|:---:|:---:|:---:|:---:|:---:|
| !Home | !Ready | !Walking | !Screen time | !Notebook entry |
Code
{% github codeswithroh/drift %}
How I Built It
Drift is a static web app: Vite and TypeScript, no framework, and no backend at all. Every model runs on your device.
| Job | Open model | How it runs |
|---|---|---|
| Write the rules + the notebook | Gemma 4 E2B (web build, 2 GB) | MediaPipe LLM Inference on WebGPU |
| Read the rules aloud | Kokoro-82M | kokoro-js, WebGPU or WASM |
| Listen to your debrief | Whisper base | transformers.js, WebGPU or WASM |
The trick: the whole walk is one audio file
My first idea was to speak each rule on a timer. That doesn't work: a locked phone suspends a web page's timers and background GPS, so the app goes to sleep with you.
Audio, though, keeps playing. So Drift does all the work before you leave. Gemma writes, say, 7 rules. Kokoro speaks each one. Then I stitch them into a single WAV file: a soft two-note chime, the spoken rule, then minutes of silence while you walk, all repeated.
for (const [i, step] of drift.steps.entries()) {
const raw = await tts.generate(step.say, { voice: 'af_heart', speed: 0.95 });
if (i > 0) push(chime());
push(resample(toMono(raw.audio), raw.sampling_rate, 16000));
push(new Float32Array(Math.round(step.walkSeconds * 16000))); // walk
}
You press play, lock the phone, and the next 30 minutes need no wake-ups, no GPS and no signal. The screen-time counter is just the Page Visibility API: every second the page is visible during the walk counts against you.
Things that broke (and what I learned)
I wrote end-to-end tests with Playwright that drive the real app with real models: real Gemma, real Kokoro, and a fake microphone playing recorded speech so Whisper has something to transcribe. They found bugs I would have shipped:
- Gemma 4 returned an empty string. MediaPipe passes your prompt to the model as-is, and Gemma 4 only answers inside its chat template (<|turn>user … <turn|><|turn>model). Without it, every drift silently fell back to my built-in rules. One line fixed it, and it was invisible until a test asserted that the title wasn't the fallback title.
- The 2 GB model didn't fit. Some browsers cap a site's storage below 2 GB. My cache.put() failed quietly, so the app re-downloaded Gemma on every visit. Now it checks navigator.storage.estimate() first and tells you honestly when the model can't be kept offline. The walk itself still never needs a network.
- Small models can't add up. Gemma's durations rarely total the 30 minutes you asked for, so Drift rescales the silences to fit exactly. Gemma also loved copying my example rules word for word, so the prompt now asks it to invent its own.
- The notebook made things up. My tests play a recorded debrief through a fake microphone: "I followed a yellow bicycle past a bakery. Then I stood under a huge old tree and listened to the wind in the leaves." Whisper transcribed it correctly. But with the smaller laptop fallback model (Gemma 3 4B via Ollama), the notebook entry came back mentioning "the coarse, warm texture of the brick wall against my palm." I never said anything about a wall. Gemma 4 had the opposite problem at first: it copied the transcript word for word. The prompt now asks for a rewrite in a naturalist's voice and forbids inventing sights. A notebook that lies about your own walk is worse than no notebook.
- "Thank you." My first real walk produced a notebook entry that was just Whisper's silence hallucination (see above). Silent recordings are now rejected before they reach the notebook.
- "Rule 5 of 4." The progress counter included the intro and outro clips. A classic.
Pip
Pip is drawn as two 3×3 sprite sheets (nine head directions, nine expressions), adapted from the MIT-licensed page-mascot by Kamran Ahmed. I ported the React component to plain TypeScript and gave Pip a mood for every screen. It's a small thing, but it makes a 2 GB download feel like waiting with a friend.
Why Does Open Innovation Matter?
Drift could not exist as a closed-API app, for three reasons.
1. It has to work with no signal. The walk happens where the network doesn't reach: parks, trails, the back streets the rules send you down. With an open-weight model the rules are written on your phone before you leave, and the walk needs nothing at all.
2. Your voice stays yours. The debrief is a recording of you, outside, saying what you noticed. Sending that to someone else's server to be transcribed and summarized would make me uneasy, so with Whisper and Gemma in the browser it never leaves the phone. There's no account, no API key, and no database. There is nothing to leak because there is nothing to collect.
3. It costs nothing to run. Drift is a static site. Every walk is computed by the walker's own device, so 10 users or 10,000 cost me the same: zero. A closed-model version would need a backend, a bill, and eventually a subscription. That's a strange thing to charge for in an app whose whole point is using your phone less.
Open models also let me make design choices a hosted API never would: swap Gemma for a smaller model on older phones, pick the voice, change exactly how the notebook is written, and read the source of every layer when something breaks (see "Gemma 4 returned an empty string").
Prize Categories
- Best Use of Gemma. Gemma 4 E2B runs entirely in the browser on WebGPU via MediaPipe. It writes every drift, writes every notebook entry, and keeps the memory that shapes the next walk.
---
Thanks to the Gemma, LiteRT, Kokoro, Whisper and transformers.js teams for making all of this run in a browser tab, and to Guy Debord for the idea, 70 years early. 🌿
25 reactions · 17 comments
Mentions
Developer Roadmap. Interactive roadmaps and guides for developers at every stage. nilbuild/developer-roadmap
Developer Roadmap Not sure what to learn next? Choose a path—frontend, backend, AI, security or another area—and explore its roadmap. It gives you a starting structure when there are too many tutorials to choose from. GitHub repo - nilbuild/developer-roadmap
2. Developer Roadmap 🔗 nilbuild/developer-roadmap ↳ Step-by-step roadmaps for frontend, backend, DevOps, and AI engineering.
developer-roadmap: The Hottest TypeScript Repo on GitHub Right Now In this video, we break down nilbuild/developer-roadmap - one of the hottest open-source projects on GitHub right now. Interactive roadmaps, guides and other educational content to help developers grow in their careers. It has already earned 369,844 stars on GitHub and is written primarily in TypeScript. We look at what it does, why developers are paying attention to it right now, and who should try it. 📂 Repo: nilbuild/developer-roadmap 🌐 Stars: 369,844 | Language: TypeScript | Forks: 45,111 🔔 Subscribe for daily picks of the hottest GitHub repos: / @reporadar #github #opensource #programming keywords: coding, developer tools, github, opensource, programming, software, tech
GitHub 스타 TOP 10 오픈소스 (2026.10.15 기준) GitHub에서 별(스타)을 가장 많이 받은 저장소 TOP 10 (2026년 10월 15일 기준) 00:00 10위 awesome-python 00:19 9위 coding-interview-university 00:35 8위 developer-roadmap 00:46 7위 system-design-primer 01:01 6위 openclaw 01:14 5위 free-programming-books 01:28 4위 freeCodeCamp 01:41 3위 public-apis 01:54 2위 awesome 02:06 1위 build-your-own-x 순위·별 개수는 GitHub API(api.github.com)로 조회했고, 별 개수는 계속 바뀝니다. 1위 codecrafters-io/build-your-own-x — ★553,348 — 저장소의 LICENSE 참고 codecrafters-io/build-your-own-x 2위 sindresorhus/awesome — ★519,278 — CC0-1.0 sindresorhus/awesome 3위 public-apis/public-apis — ★488,929 — MIT public-apis/public-apis 4위 freeCodeCamp/freeCodeCamp — ★457,094 — BSD-3-Clause freeCodeCamp/freeCodeCamp 5위 EbookFoundation/free-programming-books — ★399,091 — CC-BY-4.0 EbookFoundation/free-programming-books 6위 openclaw/openclaw — ★392,069 — MIT openclaw/openclaw 7위 donnemartin/system-design-primer — ★374,285 — 저장소의 LICENSE 참고 donnemartin/system-design-primer 8위 nilbuild/developer-roadmap — ★369,449 — 저장소의 LICENSE 참고 nilbuild/developer-roadmap 9위 jwasham/coding-interview-university — ★362,528 — CC-BY-SA-4.0 jwasham/coding-interview-university 10위 vinta/awesome-python — ★327,562 — 저장소의 LICENSE 참고 vinta/awesome-python ※ 화면의 저장소 이미지는 각 저장소가 GitHub에 올려 둔 대표 이미지(소셜 미리보기)이며 각 저작권자에게 권리가 있습니다. ※ 나레이션 음성(Microsoft Azure AI Speech), 배경음악(ACE-Step), 일부 장면 그림은 AI로 생성했습니다. #GitHub #오픈소스 #개발자
Install: npx skills add nilbuild/video-demo Then: /video-demo record a demo of yourapp.com The repo, MIT: nilbuild/video-demo
I built an AI that only works when my phone is in my pocket --- title: I built an AI that only works when my phone is in my pocket published: true tags: devchallenge, hf26challenge, gemma, opensource cover_image: drift-three-nu.vercel.app --- This is a submission for the Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass What I Built Every "go outside" app I have tried wants me to look at it once I'm outside: a map, a streak, a badge, a feed. I wanted the opposite: an app that is finished with me before I reach the front door. Drift is a walk with no destination. You choose how long and a mood (quiet, colors, old things, sounds). Gemma 4, running in your browser, writes a short set of playful rules. An open-weight voice reads them to you, one at a time, while the phone sits in your pocket: > "Trace the outline of the oldest brick you can find with your hand." > "Follow the faint scent that seems to remember rain." > "Stand still until the distant, muted sound settles into your bones." (Those are real rules Gemma wrote during testing.) The idea is borrowed from the dérive (French for "drift"), a practice from the 1950s Situationists in Paris. They walked cities by following rules instead of routes, to notice what a destination hides. Seventy years later we have a better excuse never to look up, so I gave the rules to an AI and took the screen away. When you get home, you talk for a minute about what you noticed. Whisper transcribes it on the device, and Gemma writes it up as an entry in a field notebook. Each entry leaves one line of memory, so the next drift leans toward what you enjoyed. There is one number Drift actually cares about: how many seconds your screen was on during the walk. Meet Pip 🦔, who keeps you company before and after. Pip watches your finger, falls asleep while you walk, and gets star-eyed if you barely looked at your phone. !Drift: choose a mood, walk with your phone locked, see your screen time I took it outside I opened Drift on my iPhone in Safari, and Gemma 4 wrote me a drift called The Quiet Current, right there on the phone. Then I went out and walked for about ten minutes. I noticed things I hadn't noticed before: how the leaves were moving, and how the wind felt when it touched me. It was hot out, but there was a cool edge underneath the heat. It's October, and even in the sun I could feel a little of the cold that's coming. It was a very engaging feeling, and honestly I can't quite put it into words. Then I came home and looked at what Drift had recorded, and it caught me out: !My first real drift in the field notebook | | | |---|---| | Drift | The Quiet Current | | Device | iPhone, Safari, Gemma 4 running on the phone | | Actually walked | about 10 minutes | | Walk the app counted | 2 minutes (I ended the drift early and kept walking) | | Screen on | 2 min 5 sec, the whole time it was counting | | Notebook entry | "Thank you." | Two honest lessons from one short walk: - I didn't lock my phone. I kept Drift open in my hand, so the screen-time counter did exactly its job and recorded that the screen was on the whole time. The best part of the walk happened after I stopped looking at the app. That's the whole thesis of Drift, demonstrated on its own author. - My notebook entry just said "Thank you." That's a well-known Whisper failure: when it hears silence or almost nothing, it outputs "Thank you." (it was trained on a lot of subtitles). Drift saved that as my notebook entry. It now checks whether a recording is too short or too quiet, drops Whisper's silence phrases, and asks you to try again. There's a test for it with real Whisper. Next time the phone goes in my pocket. What a drift actually looks like Here is a real one, written by Gemma 4 E2B running on WebGPU in my browser. I picked 30 minutes and the mood old things. Gemma wrote this in 11.5 seconds: > Echoes in the Cobblestones > 1. Find the object with the deepest shadow beneath a large leaf. > 2. Listen intently to the rhythm of the oldest wood. > 3. Touch the texture of the wall that feels the longest. > 4. Follow the faint scent that seems to remember rain. > 5. Stand still until the distant, muted sound settles into your bones. > 6. Allow your gaze to simply rest on the most enduring shape nearby. None of these need a map, and none of them could be followed while looking at a phone. That's the point. You can't do "touch the texture of the wall that feels the longest" from a screen. Every time you press Prepare the rules are new. Other drifts Gemma wrote while I was building: The Echo Chamber Wander, Chromatic Descent, Dust & Echoes. Then Kokoro reads them in a calm voice, with a soft chime before each rule and minutes of silence after it. The rendered walk for The Echo Chamber Wander came out at 31 minutes of audio, from a single file. On the deployed site I watched the counter tick from "Starting" to "Rule 1 of 6" while the audio played, with the page doing nothing else. | | Time on my laptop (Apple GPU, WebGPU) | |---|---| | Gemma 4 writes a drift | 11.5 s | | Gemma 4 writes a notebook entry | 2.2 s | | First-time model download (2 GB) | ~11–14 min on my connection | | Network needed during the walk | none | Demo 🌿 Try it: drift-three-nu.vercel.app · Code: codeswithroh/drift. I've run it on an iPhone (Safari) and a MacBook (Chrome). It needs WebGPU: Chrome or Edge, or Safari 26+. The first visit downloads about 2.5 GB of models, so use Wi-Fi. !The Drift flow in under a minute | Choose | Prepare | Walk | Screen time | Notebook | |:---:|:---:|:---:|:---:|:---:| | !Home | !Ready | !Walking | !Screen time | !Notebook entry | Code {% github codeswithroh/drift %} How I Built It Drift is a static web app: Vite and TypeScript, no framework, and no backend at all. Every model runs on your device. | Job | Open model | How it runs | |---|---|---| | Write the rules + the notebook | Gemma 4 E2B (web build, 2 GB) | MediaPipe LLM Inference on WebGPU | | Read the rules aloud | Kokoro-82M | kokoro-js, WebGPU or WASM | | Listen to your debrief | Whisper base | transformers.js, WebGPU or WASM | The trick: the whole walk is one audio file My first idea was to speak each rule on a timer. That doesn't work: a locked phone suspends a web page's timers and background GPS, so the app goes to sleep with you. Audio, though, keeps playing. So Drift does all the work before you leave. Gemma writes, say, 7 rules. Kokoro speaks each one. Then I stitch them into a single WAV file: a soft two-note chime, the spoken rule, then minutes of silence while you walk, all repeated. for (const [i, step] of drift.steps.entries()) { const raw = await tts.generate(step.say, { voice: 'af_heart', speed: 0.95 }); if (i > 0) push(chime()); push(resample(toMono(raw.audio), raw.sampling_rate, 16000)); push(new Float32Array(Math.round(step.walkSeconds * 16000))); // walk } You press play, lock the phone, and the next 30 minutes need no wake-ups, no GPS and no signal. The screen-time counter is just the Page Visibility API: every second the page is visible during the walk counts against you. Things that broke (and what I learned) I wrote end-to-end tests with Playwright that drive the real app with real models: real Gemma, real Kokoro, and a fake microphone playing recorded speech so Whisper has something to transcribe. They found bugs I would have shipped: - Gemma 4 returned an empty string. MediaPipe passes your prompt to the model as-is, and Gemma 4 only answers inside its chat template (<|turn>user … <turn|><|turn>model). Without it, every drift silently fell back to my built-in rules. One line fixed it, and it was invisible until a test asserted that the title wasn't the fallback title. - The 2 GB model didn't fit. Some browsers cap a site's storage below 2 GB. My cache.put() failed quietly, so the app re-downloaded Gemma on every visit. Now it checks navigator.storage.estimate() first and tells you honestly when the model can't be kept offline. The walk itself still never needs a network. - Small models can't add up. Gemma's durations rarely total the 30 minutes you asked for, so Drift rescales the silences to fit exactly. Gemma also loved copying my example rules word for word, so the prompt now asks it to invent its own. - The notebook made things up. My tests play a recorded debrief through a fake microphone: "I followed a yellow bicycle past a bakery. Then I stood under a huge old tree and listened to the wind in the leaves." Whisper transcribed it correctly. But with the smaller laptop fallback model (Gemma 3 4B via Ollama), the notebook entry came back mentioning "the coarse, warm texture of the brick wall against my palm." I never said anything about a wall. Gemma 4 had the opposite problem at first: it copied the transcript word for word. The prompt now asks for a rewrite in a naturalist's voice and forbids inventing sights. A notebook that lies about your own walk is worse than no notebook. - "Thank you." My first real walk produced a notebook entry that was just Whisper's silence hallucination (see above). Silent recordings are now rejected before they reach the notebook. - "Rule 5 of 4." The progress counter included the intro and outro clips. A classic. Pip Pip is drawn as two 3×3 sprite sheets (nine head directions, nine expressions), adapted from the MIT-licensed page-mascot by Kamran Ahmed. I ported the React component to plain TypeScript and gave Pip a mood for every screen. It's a small thing, but it makes a 2 GB download feel like waiting with a friend. Why Does Open Innovation Matter? Drift could not exist as a closed-API app, for three reasons. 1. It has to work with no signal. The walk happens where the network doesn't reach: parks, trails, the back streets the rules send you down. With an open-weight model the rules are written on your phone before you leave, and the walk needs nothing at all. 2. Your voice stays yours. The debrief is a recording of you, outside, saying what you noticed. Sending that to someone else's server to be transcribed and summarized would make me uneasy, so with Whisper and Gemma in the browser it never leaves the phone. There's no account, no API key, and no database. There is nothing to leak because there is nothing to collect. 3. It costs nothing to run. Drift is a static site. Every walk is computed by the walker's own device, so 10 users or 10,000 cost me the same: zero. A closed-model version would need a backend, a bill, and eventually a subscription. That's a strange thing to charge for in an app whose whole point is using your phone less. Open models also let me make design choices a hosted API never would: swap Gemma for a smaller model on older phones, pick the voice, change exactly how the notebook is written, and read the source of every layer when something breaks (see "Gemma 4 returned an empty string"). Prize Categories - Best Use of Gemma. Gemma 4 E2B runs entirely in the browser on WebGPU via MediaPipe. It writes every drift, writes every notebook entry, and keeps the memory that shapes the next walk. --- Thanks to the Gemma, LiteRT, Kokoro, Whisper and transformers.js teams for making all of this run in a browser tab, and to Guy Debord for the idea, 70 years early. 🌿