mlx-swift-lm

MLXのSwift向けライブラリに、LFM3のテキスト対応が提案されました！ 小型モデルで音声入力後の文章を整える、といった用途が想定されています。 現在は未マージ。画像・Omni対応は含まず、日本語性能も未確認です。 出典元：mlx-swift-lm#750

MLXのSwift向けライブラリに、gated-delta処理の学習を高速化する案が出ています！ 作者はM2とQwen 3.5の組合せで、学習stepが約1.9〜2.3倍速くなったと報告しています。 まだ未採用で、推論自体の高速化ではありません。 出典元：mlx-swift-lm#740

MLXのSwift向けライブラリに、Nemotron 3.5のテキスト対応が追加されました！ Lightning・Bolt・Superを使う選択肢が広がります。Qwen 3.5の内部形状を取得するAPIも追加されています。 現在は開発版で、Nemotronの画像・音声は対象外です。 出典元：mlx-swift-lm#712 mlx-swift-lm#737

MLXのSwift向けライブラリで、FastVLMの画像tokenの扱いとBERTの設定衝突が修正されました！ 画像tokenへの不適切なペナルティや、設定名の衝突によるクラッシュを防ぐ変更です。 現在は開発版への反映です。 出典元：mlx-swift-lm#735 mlx-swift-lm#736

Sloth Roll: an on-device vision model that turns my screenshots into calendar events, places and expenses. I only swipe. This is a submission for the Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady What I Built My camera roll is full of screenshots of things I meant to do later. A flyer for a tournament on Saturday. A restaurant someone recommended. A receipt I should have written down. I take the screenshot, and that's where it ends. Later never comes. So I built an app for that later. Sloth Roll reads my screenshots on the iPhone with an open-weight vision model and turns each one into a card: - an event goes to Calendar, - a place opens in Apple Maps, - a receipt is saved to an expense list inside the app. I don't type, copy, paste, or switch apps. I only swipe: right to do it, left to skip it. Tap a card to fix the title, date, place or amount first. It also does the cleanup I keep putting off. Screenshots that are blurry, near-duplicates, or that the model calls not useful end up in a Clean up tab, and I delete them in one tap. They go to Recently Deleted, through iOS's own confirmation, so nothing disappears by surprise. Everything runs on the phone. After a one-time model download, it works in airplane mode. The lazy payoff. In one scan of my real library, the app went through 50 new images, read 43 of them with the model, and found 19 actions, in about 3.7 minutes, while the phone sat on the table. My part was 19 swipes. If adding one event or expense by hand takes about a minute, that's roughly 19 minutes of copying turned into under a minute of swiping. That's my idea of Slow and Steady: the app is the slow one. It reads one image at a time, remembers what it already read, and waits for me. I swipe whenever I feel like it. Demo {% youtube mwsLE8IcGfI %} | First launch | Scan one album | Pick what to read | Reading and swiping | | --- | --- | --- | --- | | !Onboarding: You take screenshots. Sloth Roll does the rest. | !Onboarding: Scan one album, then clean it up. | !Scan from: all recent photos or one album | !A scan in progress with an event card ready to add | In the last screenshot, the app is reading image 5 of 27 while the first card is already waiting: a Facebook post that says "MAÑANA TORNEO LIGA SUPER". The card already says what will happen: "Add event · Sat 24 Oct, 3:30 p.m. · Plaza La Cachanilla". One swipe and it's in my calendar. Code {% embed Martin-EG/Sloth-Roll %} How I Built It flowchart LR UI["SwiftUI views"] --> VM["ViewModels"] VM --> SC["ScanCoordinator"] VM --> ACT["Actions: Calendar · Maps · Expenses"] SC --> PK["PhotoKit"] SC --> VN["Vision: OCR, blur, feature print"] SC --> EX["Extractor: prompt + parser"] EX --> VLM["MLX · Qwen3-VL-2B 4-bit"] SC --> SD[("SwiftData")] ACT --> SD It's a SwiftUI app (Swift 6, iOS 18+). The model is mlx-community/Qwen3-VL-2B-Instruct-4bit, a 4-bit conversion of Qwen3-VL 2B. It runs on the iPhone's GPU with MLX Swift and mlx-swift-lm. The download is about 1.8 GB, and it takes about 2 GB of memory while loaded. I tested on an iPhone 17 Pro. Triage first, model second. A vision model is slow and heavy, so a scan works in two passes. First, with no model loaded, every new image gets cheap checks. Camera photos are skipped (they have camera make, model and lens info in the file). Apple's Vision text recognition must find at least 20 characters, or the image is skipped. The shortest real action in my test set had 40. The same pass measures blur and takes a Vision feature print for finding duplicates, so each image is opened only once. Then the model loads once, reads only the images that passed, newest first, saves each answer right away, and unloads at the end. Dates relative to the screenshot, not to today. My first prompt resolved "hoy" ("today") against the day I ran the scan, so a cinema ticket from July came back dated October. Now the prompt includes the day the screenshot was taken, plus the next seven days spelled out with weekday names, in Spanish and English. "Mañana", "este sábado" and "hoy" land on the right day. The model's answer is untrusted. It answers with one JSON object: type, title, date, time, location, amount, currency, confidence. In practice it wraps the JSON in code fences, adds chatter, leaves trailing commas, and once answered with {"error": "..."}. The parser cuts the answer down to the JSON object, checks every field (a real date, a real 24-hour time, an amount that isn't negative), and turns unknown values into safe defaults. A bad answer gets one stricter retry, then the card says "couldn't read this one". It never crashes on bad output. One small prompt change mattered a lot. One version of the prompt made the model drop the "type" key on 23 of 24 images, so almost every image needed a retry. Listing "type" as a required first field brought that to zero retries. Order numbers aren't money. The model kept reading order and booking numbers as amounts. The prompt now says amounts must be money (a price or a total) and calls out folio, order and ticket numbers as not amounts. Clean up reuses the same model. Blur is a Laplacian variance score, with a contrast check so a plain blank screen doesn't count as blurry. Duplicates are screenshots whose Vision feature prints are very close; the newest one is kept. For everything else, the model that's already loaded gets one question: "Is this screenshot still useful to keep? Answer only yes or no." Only a clear "no" puts it in the pile. When in doubt, it keeps the screenshot. How well does it work? I built a debug eval screen that runs the whole pipeline on an album and lets me mark each row right or wrong. On 19 hand-checked images, the type (event, place, receipt or none) was right on 18 (95%), and all 6 junk images came back "none". The miss was a cinema purchase history read as an event instead of a receipt. The fields are weaker: date and time were both right on 6 of 10 events. The usual slip is the model taking a chat timestamp or the status-bar clock as the event time. That's why every card can be edited before you swipe, and why cards the model wasn't sure about say "not sure". Speed is about 4 to 8 seconds per image once the model is loaded. The app has 102 unit tests. Services sit behind protocols, so the tests swap in a fake model and a fake photo library. Why Does Open Innovation Matter? My screenshots stay on my phone. Screenshots are some of the most private things on a phone: chats, receipts, tickets, addresses, bank apps. A cloud vision API would see all of them. With an open-weight model running through MLX, no image, text or result leaves the phone. There's no account, no analytics and no server. It works offline. After the one-time download, scanning, cards, Calendar, Expenses and Clean up all work in airplane mode. (Opening a place hands it to Apple Maps, which uses Apple's place search when there's a connection.) Reading my whole library costs nothing. A scan reads dozens of images, and a camera roll has hundreds of screenshots. With a paid API, every scan would be a bill, and I would scan less. On the phone's GPU, the marginal cost is battery. I can swap the model. The model ID lives in one file. When a better small open vision model comes out on mlx-community, I change one line and run the eval screen again. The honest trade-offs: - The first launch needs Wi-Fi for a 1.8 GB download. - It needs a recent iPhone with enough memory; I've only tested on an iPhone 17 Pro. - The prompt and my test set cover Spanish and English only. - A 2B model gets the type right much more often than the exact time. Swiping is quick, but checking the card still matters. My Agent Session I used Claude Code as a coding partner for the Swift code, the tests and the docs. The model in the app does all the reading of screenshots. In the last sessions it prepared the repo for submission, checked that a fresh clone builds, and helped draft this post from the repo's dev log. Credits and licenses Model: Qwen3-VL-2B-Instruct by the Qwen team, 4-bit MLX conversion by mlx-community (Apache-2.0). MLX Swift and mlx-swift-lm (MIT). swift-transformers and swift-huggingface (Apache-2.0). App code under the MIT License. Full table in the repo README.

MLX Swift LMに、入力用の埋め込み表をファイル参照にする省メモリ案が出ています。 重みの精度を変えず、使わないページをOSが解放できる仕組みです。作者のM2測定ではメモリ使用量が約11〜15%減。 mlx-swift-lm#708

I built the running coach that actually listens to your body This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle What I Built I run with Nike Run Club. It tracks every run well, but its plans aren't personal. They don't know my heart rate zones, my actual paces, or what my last few weeks looked like. A tempo session says "comfortably hard" whether my resting heart rate is 50 or 70, and the plan doesn't notice when I miss a week or double my mileage. I wanted a coach that looks at my data and changes the plan because of it. OpenStride is an iOS running coach that reads your data from Apple Health and plans your week around it: - Heart rate zones from your resting HR and the highest HR seen in your own runs (Karvonen) - Pace zones from your best recent effort, or from pace plus heart rate when the effort wasn't all-out (Jack Daniels' VDOT) - Training load per run and your 7-day vs 28-day load ratio - HR drift between the first and second half of each run - How much of last week you actually did Every week, Gemma (Google's open-weight model) reviews those signals, adjusts the plan, and tells you why in plain words. It runs on the iPhone through MLX, so the app needs no account or server. With the on-device model (the default), your health data stays on the phone. Pricing One-time purchase on the App Store and Google Play: $10 (₹599 in India). No subscription or account needed. Adaptive training plans are already a paid category (Runna, TrainingPeaks), sold as monthly subscriptions. OpenStride can skip the subscription because the coach runs on the phone: after someone buys it, they cost nothing to serve. The app is iOS only today; the Play Store version comes with Android. Here's how it would look on the App Store (full sample listing): | | | | |---|---|---| | !A coach that reads your body | !It notices when you overdo it | !Runs on your iPhone | Demo Demo page with the full 60 s video: ansh-saini.github.io Recorded on my iPhone with a runner who doubled their mileage last week. Gemma 3 4B runs on the phone; the planning step is sped up. !OpenStride demo | The week | A session | Signals | What the coach saw | |---|---|---|---| | !plan | !session | !signals | !prompt | Code {% github ansh-saini/open-stride %} How I Built It I created several modules which work together in steps to deliver an optimial running plan: 1. CoachKit (pure Swift, unit tested) computes zones, paces, load and drift, then builds a draft week: phase (base, build, peak, taper), weekly volume, a long run, quality sessions. 2. Code turns the metrics into signals with fixed thresholds, each one OK or FLAG. For example: acute_chronic_load_ratio = 1.98 -> FLAG: above 1.5, injury risk. 3. Gemma decides. It sees the signals and the draft, and picks from a small action set: cut volume, slow easy pace, make_easy, shorten or drop a day, each with a reason. 4. Guardrails apply the decision and override anything unsafe. No flag means the draft stands. A flag with too small a response gets a floor. The long run can't be dropped. 5. Gemma explains. A second call writes the note from facts that code computed by diffing the draft against the final plan. Code writes the first sentence ("Your heart rate drifted 7.7% during recent runs."), and Gemma continues it. The note always matches what you see. If the model fails, a rule-based coach writes the week, so the week still gets planned. An eval harness I built six synthetic runners with seeded histories and an expected coach behavior for each: | Runner | Signal | A good coach should | |---|---|---| | steady | all in range | keep progressing | | overreaching | load ratio 1.98 | cut volume, max one hard day | | detrained | 21 days off | no hard sessions | | hr_drift | 7.7% drift | slow easy pace | | missed_sessions | did 20% of last week | not add volume | | beginner | 8 km a week | progress gently | coach-eval runs them through Ollama, and mlx-eval runs them through the exact MLX weights the phone uses. Each run is scored on whether the decision parsed, whether the final plan is right, whether the model got it right without a guardrail floor, and whether the note cites a real number. | Coach | Runs | Valid | Final plan right | Model right on its own | Note matches plan | Note cites the flagged number | Avg time | |---|---|---|---|---|---|---|---| | rules | 6 | 100% | 100% | 100% | 100% | 100% | 0.0s | | gemma3:1b (Ollama) | 18 | 100% | 100% | 33% | 100% | 100% | 1.0s | | gemma3:4b (Ollama) | 18 | 100% | 100% | 100% | 100% | 100% | 4.8s | | Gemma 3 4B (on-device) | 18 | 100% | 100% | 100% | 100% | 100% | 8.0s | | Gemma 3 1B (on-device) | 18 | 100% | 100% | 33% | 100% | 100% | 4.3s | Real data is messier than test data My own runs exposed two things the personas didn't. None of my runs from the last four weeks have heart rate recorded, so there's no drift signal, and the coach has to work without it. And my "best effort" wasn't all-out, so the race formula underrated my fitness. Now, when heart rate exists, VDOT is also estimated from pace and heart rate reserve, and the higher estimate wins. Stack SwiftUI, HealthKit, Swift Charts, XCUITest on simulator and device, MLX Swift LM with mlx-community/gemma-3-text-4b-it-4bit (2.6 GB, downloaded once), Ollama for the simulator and evals, XcodeGen. Why Does Open Innovation Matter? Privacy that holds up. Heart rate, resting heart rate and VO2 max are health data. With an open-weight model on the device, the coach has no server to send your data to. A closed API needs one. Zero marginal cost. Every plan runs on the phone's GPU. No per-token bill means a solo developer can price this as a one-time purchase instead of a subscription that has to cover inference. Works offline. After the one-time download it works on a trail with no signal, which is where runners are. Inspectable and swappable. The Insights screen shows exactly what the coach saw and what it answered. Because the weights are open, I could test the same model on my Mac, in the simulator and on the phone, and I can swap in a newer Gemma or a fine-tune without changing the app. Injury Prevention The most common mistake runners make is doing too much too fast. A good week feels great, so the next one gets longer and faster, and the body's tendons and bones can't keep up with the lungs. A static plan can't see that coming. OpenStride watches for it every week: - It measures the jump. The load ratio compares your last 7 days of training to your 28-day average. Above 1.3, the coach cuts next week's volume by at least 10%. Above 1.5, it cuts at least 20% and allows only one hard session. - It progresses slowly. Weekly volume builds about 8% at a time from your 4-week average, so one big week doesn't become the new normal. A guardrail caps any week at 110% of that average plus 2 km, whatever the model asks for. - It plans recovery weeks. Every fourth week drops volume by 20%. - It caps the long run. The long run can't be more than 15% plus 1 km longer than your longest run in the last four weeks. - It spaces hard days. Hard sessions never fall on back-to-back days, there are at most two a week, and in the base phase, runners under 15 km a week get easy running only. - It respects time off. After 14 or more days without running, the plan starts from half your recent 4-week average with no hard sessions. - It listens to your heart. When your heart rate drifts more than 5% during easy runs, it slows your easy pace. - It reacts to missed weeks. If you finished less than 60% of last week, the next week gets lighter instead of trying to catch up. In the demo, the runner doubled their mileage in one week (load ratio 1.98). The draft plan had 31 km for the next week; the coach cut it to 25 km, kept one hard day, and explained why in the note. My First User My first user is me. I ran the device test on my own iPhone with two months of my real runs: 22 runs, mostly 3 to 5 km, averaging about 14 km a week. What it got right: it planned four easy runs and a slightly longer Sunday run, about the same volume I've been doing, at an easy pace close to how I actually run. No signals were flagged, and the note said so instead of inventing a problem. What it taught me: the first estimate of my fitness was too low, because my "best" run wasn't an all-out effort. That's why the app now also estimates fitness from heart rate. And none of my runs since September 21 have heart rate recorded, so there's no drift to read and the coach is working with less than it could. Next Steps I had a ton of fun building this project as it solves a real problem for me, and I know for a fact that many people face this problem. I'm super excited to integrate this with Apple Watch and seeing how that transforms my running experience. It will give me access to real-time heart rate tracking and session feedback, ensuring the coach receives complete data for more accurate planning. Maybe the coach can say "Hey you're going too fast, slow down" into my earphones during my run itself. Prize Categories - Best Use of Gemma: Gemma 3 4B runs on the iPhone through MLX as the coach. Gemma 3 1B and Ollama-served Gemma are the fallbacks. <!-- Thank you for the opportunity! -->

Grow a Song: my run wrote its own soundtrack, with Gemma running on my phone This is a submission for the Hacktoberfest Open-Source AI Challenge, Week 1: Touch Grass. What I Built On Friday morning I went for a 35 minute run. When I started, my song was a single low drone. By the time I got back it had pads, a bass line, drums, a kalimba melody, and ten seconds of the actual street I ran down mixed in underneath. Grow a Song turns time outside into music. Every minute you spend outdoors in daylight adds a layer to today's track: | Outside time (20 min target) | Layer | |---|---| | 0 min | drone | | 2 min | pads | | 5 min | bass | | 8 min | drums | | 13 min | melody | | 20 min | a field recording of where you are | If you stay inside all day, your song stays thin, and it only fills in once you get out. There's no way to skip ahead or buy layers, so the only way to hear the whole song is to go outside. The composer is Gemma 3 1B, running on my iPhone 13 Pro. Before each walk or run it looks at the time of day, how high the sun is, what you're about to do, and how much you've been outside this week, then picks the key, scale, tempo, chord progression, instruments, a title and a one-line liner note. My run got this: > Ephemeral Echoes. D dorian, 160 bpm, chords 1 7 5 3, warm pads, pulse bass, driving drums, kalimba. > "The world stretches out beneath you, a slow, expectant invitation." There's no account or server behind it, and after a one-time model download it works in airplane mode. Demo {% youtube LCFFlzZQWFE %} Here is the song from my run, rendered by the same engine the app uses, plus a version that replays the run with the layers arriving one at a time: - Ephemeral Echoes (full song) - Ephemeral Echoes (how it grew, layer by layer) The numbers from that morning, pulled off the phone: - 35.1 minutes counted as outside, 3.4 km - all 6 layers playing by the end (my target was 20 minutes) - a 10 second clip recorded at 7:40 AM, now the top layer of the song - Gemma's field note afterwards: "You ran 3.2 km. 'Ephemeral Echoes' is now a part of the recording." (Accurate in spirit. The distance is off by about 200 m, which is what you get from a 1B model writing journal entries.) Code {% embed ansh-saini/grow-a-song %} Swift, SwiftUI, AVAudioEngine, CoreLocation, CoreMotion, HealthKit and MLX Swift LM. The music engine is a plain Swift package with 37 tests, so most of it runs and tests on a Mac without a phone. How I Built It Gemma composes, Swift plays I didn't want a 1B model generating audio or MIDI note by note. A model that small does fine at picking a mood and badly at keeping track of hundreds of notes, so Gemma makes about ten musical decisions and returns them as JSON: {"title": "Ephemeral Echoes", "key": "D", "scale": "dorian", "bpm": 160, "progression": [1, 7, 5, 3], "pad": "warm", "bass": "pulse", "drums": "driving", "melody": "kalimba", "energy": 1.0, "liner_note": "The world stretches out beneath you, a slow, expectant invitation."} A deterministic sequencer turns that into 16 bars of notes (melody in an A A' B A shape so it repeats enough to stick), and a small synth written from scratch plays it: detuned saw pads, an FM kalimba, a kick that's just a sine wave sweeping down from 155 Hz, and a reverb. The same spec always produces the same song, which turned out to matter later. Getting reliable JSON out of a 1B model My first eval on the Mac looked fine on the surface: 6 out of 6 songs were playable. Underneath, the schema-constrained decoding (MLX's MLXGuidedGeneration, built on XGrammar) was failing every time, and a fallback parser was quietly rescuing it. Logging the partial output showed why. Gemma was emitting newline after newline. JSON allows unlimited whitespace, greedy decoding found that whitespace was always a legal next token, and it filled all 360 tokens with it. Adding the library's whitespace and closing-token biases fixed it: | | Before | After | |---|---|---| | Schema-guided success | 0 / 6 | 6 / 6 | | Average compose time (M2 Pro) | 9.5 s | 4.4 s | The second problem was taste. Gemma picked D major pentatonic, chords 1 5 6 4, 60 bpm almost every time, including for a run. A 1B model leans hard on whichever option it reads first. So the prompt now shuffles the order of every option list, seeded by the current minute, and a small guard folds tempos into a range that fits the activity (a "75 bpm" run becomes 150). After that the same eval produced three keys, four scales and five different progressions in six songs. Knowing you're outside without a light sensor An Apple Watch measures time in daylight with its ambient light sensor, and the app reads that from HealthKit when it's there. But I wanted the app to work for anyone with just a phone, and iPhone apps can't read the phone's light sensor. So the phone infers it. A minute counts when: - the sun is above civil twilight (-6 degrees), computed locally from time and location with the NOAA solar equations, and - CoreMotion says you're walking, running or cycling, or the GPS fix is accurate to 20 m (usually means open sky) and you're away from home. Driving never counts. On first launch, CoreMotion's 7-day activity history gives you a week of estimated songs straight away. With a watch, the measured number takes over. The bug that would have ended my run I had a second AI agent review the code before the run. It found a real crash. iOS doesn't let background apps use the GPU. If I tapped Start and locked my phone while Gemma was still composing, MLX would hit that error and call fatalError, killing the app, the music and the minute counter all at once, a few seconds into the run. The fix has three parts. Gemma only runs while the app is on screen, and a request made in the background waits until you open the app again. The screen stays awake for the few seconds composing takes. And every MLX call is wrapped in withError so a GPU failure becomes a Swift error and falls back to a built-in song instead of a crash. The music starts immediately with the drone either way, and Gemma's song takes over at the next 16-bar phrase boundary so nothing cuts off mid-note. The same review caught that run progress was only saved when you tapped End, so it now saves every half minute. Getting the song off the phone After the run I wanted the song on my laptop, but I'd never exported it. Because the renderer is deterministic, I copied the day's record and the field clip off the phone over a cable and rendered the identical song on the Mac with the same engine. That's the audio linked above. Built with an AI agent, tested by running I built this in one long session with Claude Code: plan, code, test, review, fix. It set up a small harness (make test, make render, make eval, make device) so every change got checked against unit tests, rendered audio levels, and real Gemma output before it reached my phone. Then the real test was mine: put my shoes on and press Start. Why Does Open Innovation Matter? Grow a Song needs your location, your movement, sometimes your health data, and a microphone. That's a lot to hand to a server so it can pick a chord progression. With an open-weight model running locally, none of that leaves the phone. GPS, motion, daylight minutes and the street recordings stay on the device, and there's no backend that could leak them. It also works where runs actually happen. Trails and parks often have no signal, and after the 0.8 GB download the app doesn't need one. Running the model costs me nothing, so Gemma can compose before every walk instead of me rationing requests. And because the model is one line of config, I can move to a bigger Gemma once one fits comfortably on a phone. Prize Categories - Best Use of Gemma: Gemma 3 1B (QAT, 4-bit) runs fully on device through MLX Swift LM and composes every song with schema-guided JSON output. - Best Use of ElevenLabs: narration for the demo video. The app itself makes no network calls. --- Built with Claude Code. I directed the project, tested it on my own run, and reviewed and edited this post.