drumih

Lekh AI 8.2: run 30B-class models on an 8 GB iPhone, iPad, fully offline Hey r/LekhAI 👋 8.2 is live on the App Store. The headline: you can now run large mixture-of-experts models on devices with only 8 GB of memory. How it works A model like Qwen 3 30B has about 30B parameters, but only around 3B are active for each token. Lekh keeps the shared part of the model in memory and loads the experts each token needs from storage, as it needs them. Experts used often stay in a small cache, so after a few messages most of them come straight from memory. Everything still runs on your device. Models you can run - Gemma 4 26B (powered by TurboFieldfare (drumih/turbo-fieldfare)) - Qwen 3.6 35B - Qwen 3 30B and Qwen 3 Coder 30B - GLM 4.7 Flash - Ling mini 2.0 - Nemotron 3 Nano 30B They're under Large MoE Models in the model picker on iOS, and in LLM Models › Large MoE in Lekh AI Pro on the Mac. Faster chat on iOS We tracked down why MLX models ran slower in Lekh than they should. The chat screen was using a lot of GPU power just to draw itself, and that left less for the model. After we reworked it, Gemma 4 E2B went from about 10 tok/s to about 48 tok/s on the same iPhone. Scrolling while the model types is smooth now too. Also new - Apple Intelligence: use Apple's on-device model on iOS 26 and later - Bonsai MLX models in the catalogue - A thinking toggle to switch reasoning on or off, plus a new thinking box that shows the model's reasoning as it works - Replies now show headings, bullet lists and numbered lists properly - A refreshed chat and sidebar look - A new layout for iPhone Duo Fixes - The text field now clears after sending - Downloaded models, including Gemma 4 26B, now appear in the chat list - Fixed a crash when downloading Qwen 3 30B - Fixed a crash when the app moved to the background mid-reply - The model picker no longer disappears when you start a new chat - Long lines in replies now wrap instead of being cut off One caveat: answers depend on storage speed. The first few messages with a large MoE model are slower while the cache fills up, and then it speeds up. If you try one of the big models, please post your device, the model and the tok/s you get. We'd love to see how it runs across different hardware. Feedback and bug reports are welcome here as always 🙏

Gemma 4 26B-A4B running on an iPhone at 6–10 tok/s, in about 2 GB of RAM I've been adding on-device models to Vehla for iPhone (the companion to my Mac app), and wanted to share this one because I didn't think it would work. Gemma 4 26B-A4B runs fully offline on my iPhone Duo (12 GB) at 6–7 tokens/s. The 4-bit model is about 14 GB on disk, more than the phone's RAM, yet it runs in roughly 2 GB. How it works Gemma 4 26B-A4B is a mixture-of-experts model: 26B parameters in total, but each token only uses about 4B of them (the 8 experts the router picks in each layer). So most of the model doesn't need to be in memory at any moment: - The shared core (attention, embeddings, router), about 1.35 GB, stays in RAM along with the KV cache. - The routed experts stay on the SSD, and only the ones each token needs are read with pread. - A small cache (16 slots per layer, least-frequently-used) keeps the popular experts in memory, so most lookups don't touch storage. For Gemma I'm using TurboFieldfare (drumih/turbo-fieldfare) by drumih (Apache 2.0). It's a Swift 6 + Metal 4 engine built for exactly this model, and it compiled for iOS without changes. Its installer downloads only the byte ranges it needs from the MLX weights on Hugging Face, repacks them into its own format, and hashes every file. All credit for the engine goes to them; I wired it into the app, added resumable downloads with iOS 26 background continuation, and handle thinking-mode output. Taking it further with MLX The idea generalizes, so I added the same expert streaming to my own copy of mlx-swift-lm: - When the model loads, stacked expert tensors in SwitchLinear are swapped for placeholders, so they're never read into memory. - At each call, the layer reads which experts the router picked, reads the missing ones from disk in parallel, and keeps a cache per layer. That got these running on the phone too: - Qwen 3.6 35B-A3B: about 6 tok/s, from a 20 GB model - gpt-oss 20B, GLM 4.7 Flash, Qwen 3 30B-A3B, Nemotron 3 Nano, Trinity Mini, Ling mini 2.0, Granite 4.0 H Small and Phi-3.5 MoE, which are all in the build but still need speed testing on more devices Things I learned the hard way - iOS's memory limit is well below the RAM you buy. One phone stopped my app at 6 GB, so size caches from os_proc_available_memory(), not from the RAM total. - Some attention designs (gpt-oss's attention sinks, GLM's latent attention) don't work with a quantized KV cache, and MLX stops the app if you give them one. - The bottleneck right now is likely GPU sync, not storage. Each layer has to read its routing back to the CPU before it can fetch experts, which means 40+ round trips per token. Predicting the next layer's experts and prefetching them is next. It's free on TestFlight if you want to try it on your own phone. I'd love to know what speeds you get, especially on 8 GB iPhones Happy to answer questions about the implementation.

Gemma 4 26B-A4B running on an iPhone at 6–10 tok/s, in about 2 GB of RAM I've been adding on-device models to Vehla for iPhone (the companion to my Mac app), and wanted to share this one because I didn't think it would work. Gemma 4 26B-A4B runs fully offline on my iPhone Duo (12 GB) at 6–7 tokens/s. The 4-bit model is about 14 GB on disk, more than the phone's RAM, yet it runs in roughly 2 GB. How it works Gemma 4 26B-A4B is a mixture-of-experts model: 26B parameters in total, but each token only uses about 4B of them (the 8 experts the router picks in each layer). So most of the model doesn't need to be in memory at any moment: - The shared core (attention, embeddings, router), about 1.35 GB, stays in RAM along with the KV cache. - The routed experts stay on the SSD, and only the ones each token needs are read with pread. - A small cache (16 slots per layer, least-frequently-used) keeps the popular experts in memory, so most lookups don't touch storage. For Gemma I'm using TurboFieldfare (drumih/turbo-fieldfare) by drumih (Apache 2.0). It's a Swift 6 + Metal 4 engine built for exactly this model, and it compiled for iOS without changes. Its installer downloads only the byte ranges it needs from the MLX weights on Hugging Face, repacks them into its own format, and hashes every file. All credit for the engine goes to them; I wired it into the app, added resumable downloads with iOS 26 background continuation, and handle thinking-mode output. Taking it further with MLX The idea generalizes, so I added the same expert streaming to my own copy of mlx-swift-lm: - When the model loads, stacked expert tensors in SwitchLinear are swapped for placeholders, so they're never read into memory. - At each call, the layer reads which experts the router picked, reads the missing ones from disk in parallel, and keeps a cache per layer. That got these running on the phone too: - Qwen 3.6 35B-A3B: about 6 tok/s, from a 20 GB model - gpt-oss 20B, GLM 4.7 Flash, Qwen 3 30B-A3B, Nemotron 3 Nano, Trinity Mini, Ling mini 2.0, Granite 4.0 H Small and Phi-3.5 MoE, which are all in the build but still need speed testing on more devices Things I learned the hard way - iOS's memory limit is well below the RAM you buy. One phone stopped my app at 6 GB, so size caches from os_proc_available_memory(), not from the RAM total. - Some attention designs (gpt-oss's attention sinks, GLM's latent attention) don't work with a quantized KV cache, and MLX stops the app if you give them one. - The bottleneck right now is likely GPU sync, not storage. Each layer has to read its routing back to the CPU before it can fetch experts, which means 40+ round trips per token. Predicting the next layer's experts and prefetching them is next. It's free on TestFlight if you want to try it on your own phone. I'd love to know what speeds you get, especially on 8 GB iPhones: testflight.apple.com Happy to answer questions about the implementation.

Gemma 4 26B-A4B running on an iPhone at 6–10 tok/s, in about 2 GB of RAM I've been adding on-device models to Vehla for iPhone (the companion to my Mac app), and wanted to share this one because I didn't think it would work. Gemma 4 26B-A4B runs fully offline on my iPhone Duo (12 GB) at 6–7 tokens/s. The 4-bit model is about 14 GB on disk, more than the phone's RAM, yet it runs in roughly 2 GB. How it works Gemma 4 26B-A4B is a mixture-of-experts model: 26B parameters in total, but each token only uses about 4B of them (the 8 experts the router picks in each layer). So most of the model doesn't need to be in memory at any moment: - The shared core (attention, embeddings, router), about 1.35 GB, stays in RAM along with the KV cache. - The routed experts stay on the SSD, and only the ones each token needs are read with pread. - A small cache (16 slots per layer, least-frequently-used) keeps the popular experts in memory, so most lookups don't touch storage. For Gemma I'm using TurboFieldfare (drumih/turbo-fieldfare) by drumih (Apache 2.0). It's a Swift 6 + Metal 4 engine built for exactly this model, and it compiled for iOS without changes. Its installer downloads only the byte ranges it needs from the MLX weights on Hugging Face, repacks them into its own format, and hashes every file. All credit for the engine goes to them; I wired it into the app, added resumable downloads with iOS 26 background continuation, and handle thinking-mode output. Taking it further with MLX The idea generalizes, so I added the same expert streaming to my own copy of mlx-swift-lm: - When the model loads, stacked expert tensors in SwitchLinear are swapped for placeholders, so they're never read into memory. - At each call, the layer reads which experts the router picked, reads the missing ones from disk in parallel, and keeps a cache per layer. That got these running on the phone too: - Qwen 3.6 35B-A3B: about 6 tok/s, from a 20 GB model - gpt-oss 20B, GLM 4.7 Flash, Qwen 3 30B-A3B, Nemotron 3 Nano, Trinity Mini, Ling mini 2.0, Granite 4.0 H Small and Phi-3.5 MoE, which are all in the build but still need speed testing on more devices Things I learned the hard way - iOS's memory limit is well below the RAM you buy. One phone stopped my app at 6 GB, so size caches from os_proc_available_memory(), not from the RAM total. - Some attention designs (gpt-oss's attention sinks, GLM's latent attention) don't work with a quantized KV cache, and MLX stops the app if you give them one. - The bottleneck right now is likely GPU sync, not storage. Each layer has to read its routing back to the CPU before it can fetch experts, which means 40+ round trips per token. Predicting the next layer's experts and prefetching them is next. It's free on TestFlight if you want to try it on your own phone. I'd love to know what speeds you get, especially on 8 GB iPhones: testflight.apple.com Happy to answer questions about the implementation.

The closed-source vs. open-source AI divide (October 31, 2026) The Closed-Source vs. Open-Source AI Divide Who controls the models matters. Here's a practical look at the closed-source and open-source AI approaches, their trade-offs, and what they mean for you. In the rapidly evolving landscape of artificial intelligence, the debate between closed-source and open-source AI has become increasingly polarized. On one hand, we have the allure of cutting-edge technology, meticulously curated by tech giants who promise unparalleled performance and innovation. On the other, there's the democratic allure of open-source AI, where transparency, community collaboration, and user control take center stage. But which approach truly serves your needs better? Let's dive deep into this contentious divide. The Current Landscape The AI industry is witnessing a surge in both closed and open-source projects. Giants like OpenAI and Google continue to dominate the closed-source arena with models like GPT-4 and PaLM, respectively. These models are often lauded for their sophisticated capabilities, extensive training data, and robust performance metrics. However, they come with significant caveats: limited transparency, high costs, and restrictive licensing. Meanwhile, the open-source community is bustling with activity. Projects like Gemma, Hob, and Proliferate are gaining traction, offering users the ability to run models locally, customize them, and avoid cloud dependency. These projects are often praised for their transparency and the freedom they provide, but they also face challenges in terms of performance, scalability, and ease of use. The Case for Closed-Source AI Closed-source AI models are the brainchildren of well-funded research labs and corporations. They boast state-of-the-art performance, often outperforming open-source counterparts in benchmarks and real-world applications. The resources poured into these models—ranging from vast datasets to extensive computational power—result in highly refined and capable systems. Moreover, closed-source models often come with user-friendly interfaces and robust support systems. For businesses and individuals seeking reliability and ease of integration, these models offer a compelling solution. The closed nature also ensures a level of security and consistency, as the development and deployment are tightly controlled. However, the trade-offs are significant. Closed-source models lack transparency, making it difficult to understand how decisions are made or to audit for biases and ethical concerns. The high costs associated with these models can be prohibitive, and the reliance on external providers can lead to vendor lock-in and potential service disruptions. The Open-Source Advantage Open-source AI, in contrast, champions transparency and community collaboration. Users have the freedom to inspect, modify, and distribute the code, fostering a culture of innovation and customization. This approach empowers developers and researchers to experiment, iterate, and improve upon existing models without the constraints imposed by proprietary systems. The recent HN discussion on the open-source engine running Gemma 4 26B exemplifies this ethos. HN user giancarlostoro highlights the potential for cost reduction through model optimization: "I wouldn't be surprised if frontier AI could be drastically cheaper if they just finetune and optimize their models." This sentiment is echoed by Keyframe, who emphasizes the ongoing research into efficient model traversal and memory management. However, open-source AI is not without its challenges. Performance and scalability can be significant hurdles, as evidenced by the HN comment by tpurves: "If techniques like OPs here, start to make the RAM situation more amenable, by the time we get to M6 or M7... local AI could be ready to go much more mainstream." Additionally, the open-source community often struggles with fragmentation, as numerous projects compete for attention and resources. What This Means in Practice For businesses and individuals, the choice between closed and open-source AI hinges on specific needs and priorities. If you require cutting-edge performance, seamless integration, and are willing to pay a premium, closed-source models may be the way to go. However, if you value transparency, customization, and cost-effectiveness, open-source AI offers a compelling alternative. Consider the case of Hob, an open-source, on-device alternative to Claude Cowork and ChatGPT. HN user pancomplex appreciates the ability to work with local files without cloud dependency: "Working with local files without anything going to the cloud was the main thing I wanted." This highlights the growing demand for privacy-conscious AI solutions that prioritize user control. Closing Thoughts The closed-source vs. open-source AI debate is not a simple binary. Each approach has its strengths and weaknesses, and the best choice depends on your unique circumstances and priorities. As the AI landscape continues to evolve, the lines between these approaches may blur, with hybrid models and collaborative ecosystems emerging. Ultimately, the key is to remain informed, adaptable, and mindful of the trade-offs inherent in each choice. The future of AI hinges on the balance between innovation and accessibility, and the choices we make today will shape the landscape of tomorrow.

@rafy_xbt Eh sorry gue salah, yang gue coba di mac m4 itu ini drumih/turbo-fieldfare bukan turboquant. nah iya, sebenernya sih kalau mau hasil bagus jgn q4, tapi gimana lagi vga nya mentok wwkwk Sip2, ntar share2 aja bang. btw ini pake gpu apa?