ggml

Stop Guessing Your Context Budget: Practical KV Cache Sizing for Local LLMs You set a 32K context, the weights fit in VRAM, and the model still refuses to load. Or it loads, and generation crawls because the card is one allocation away from paging. The culprit is almost never the model file. It is the KV cache: the per-token key and value tensors that every attention layer keeps so it does not have to recompute the past. It is reserved up front, it scales linearly with context, and on a 7B model at 32K it is measured in gigabytes, not megabytes. This post is the arithmetic I wish I had the first time a "small" context change OOMed a 12 GB card, plus the flags that actually control the allocation in llama.cpp and Ollama. I will also show why the popular "quantize it and it is half the memory" advice is close but not correct, and how to check whether the cheaper cache wrecked your model's recall. The formula, and why GQA decides everything For a standard transformer with full attention, the cache holds two tensors per layer, per token, per KV head. So: bytes_per_token = 2 * n_layers * n_kv_heads * head_dim * bytes_per_element The factor of 2 is K and V. The term that surprises people is n_kv_heads, not the dtype. Take Qwen2.5-7B-Instruct, whose config declares 28 hidden layers, 28 attention heads, 4 key/value heads, and a hidden size of 3584 (so head_dim is 3584 / 28 = 128). Its cache holds: 2 * 28 * 4 * 128 = 28,672 values per token At f16 (2 bytes per value) that is 57,344 bytes, or 56 KiB per token. At 32,768 tokens that is 1.75 GiB of VRAM before a single token is generated. Mistral-7B-Instruct-v0.3 has the same head_dim but 32 layers and 8 KV heads: 2 * 32 * 8 * 128 = 65,536 values per token = 128 KiB per token At 32K that is 4 GiB. Same parameter count as Qwen, more than twice the cache, because it keeps twice as many KV heads per layer. That is the whole GQA story. A 7B model with plain multi-head attention (32 KV heads) would need 512 KiB per token, or 16 GiB at 32K, which is why nobody ships 7B MHA models for consumer GPUs any more. When you compare two models, compare their KV head counts and layer counts first. Parameter count tells you almost nothing about context cost. Two footnotes before the tables: - Some architectures set head_dim explicitly instead of deriving it from hidden size divided by heads. Read the model's config.json rather than assuming. - Sliding-window and hybrid-attention models do not keep the full history in every layer. The formula above is an upper bound for them, not an exact figure. The quantization table, including the part the marketing omits llama.cpp does not store one float per value in a quantized cache. It stores blocks of 32 values with a scale. The ggml block for q8_0 is a 2-byte half plus 32 int8 values, so 34 bytes for 32 values, not 32. q4_0 is a 2-byte scale plus 16 packed bytes, so 18 bytes for 32 values. That overhead is small but it is exactly why "half" and "quarter" are approximations: | Cache type | Bytes per value | Share of f16 | |---|---|---| | f16 / bf16 | 2.0 | 100% | | q8_0 | 34 / 32 = 1.0625 | 53.1% | | q5_1 | 24 / 32 = 0.75 | 37.5% | | q4_1 | 20 / 32 = 0.625 | 31.3% | | q4_0 | 18 / 32 = 0.5625 | 28.1% | So q8_0 saves 47% of the cache, not 50%. q4_0 saves 72%, not 75%. On a 4 GiB cache the difference between the claim and reality is around 120 MiB, which is enough to matter on an 8 GB card running a 14B model. Real numbers for the two models above at 32K, computed with the formula and the table: | Model | f16 | q8_0 | q4_0 | |---|---|---|---| | Qwen2.5-7B (28L / 4 KV) | 1.75 GiB | 952 MiB | 504 MiB | | Mistral-7B v0.3 (32L / 8 KV) | 4.00 GiB | 2.13 GiB | 1.13 GiB | Those figures are pure cache. Add weights, the compute buffer, and the CUDA context on top. On a 12 GB card, a 7B Q4_K_M model plus a 32K f16 cache plus buffers is a genuinely tight fit, which is why the same setup with q8_0 suddenly has headroom. Doing it in llama.cpp Start with an explicit, measurable baseline. Explicit matters, because the defaults are doing more than you think. llama-server \ -m qwen2.5-7b-instruct-q4_k_m.gguf \ --host 127.0.0.1 --port 8080 \ -c 32768 \ -ngl 99 \ -fa on \ -ctk f16 -ctv f16 - -c 32768 is the total context pool. Left at 0, llama.cpp takes the model's trained context, which on a 128K model is a spectacular way to run out of memory. - -fa on forces Flash Attention. The default is auto, which enables it when the backend supports it. - -ctk and -ctv set the K and V cache types independently. Allowed values include f32, f16, bf16, q8_0, q5_1, q5_0, q4_1, q4_0 and iq4_nl. The cache is allocated at load time, so measure before sending traffic: nvidia-smi --query-gpu=memory.used --format=csv,noheader Then restart with the quantized cache and the same everything else: llama-server \ -m qwen2.5-7b-instruct-q4_k_m.gguf \ --host 127.0.0.1 --port 8080 \ -c 32768 \ -ngl 99 \ -fa on \ -ctk q8_0 -ctv q8_0 On the Qwen example the delta should be roughly 800 MiB. If the delta is zero, the cache did not change, and you should read the startup log rather than assume success. Confirm the effective context from the running server: curl -s http://127.0.0.1:8080/props | jq '.default_generation_settings.n_ctx' # if that path is absent in your build, inspect the whole object: # curl -s http://127.0.0.1:8080/props | jq . The Flash Attention dependency Quantizing V without Flash Attention does not silently fall back. It fails at context creation: llama_new_context_with_model: V cache quantization requires flash_attn common_init_from_params: failed to create context with model '...' This is reported in llama.cpp issue #10378, and it is still the first thing to check when a quantized cache will not start. Keep -fa on when you quantize V. If you deliberately disabled Flash Attention for a backend reason, you cannot use a quantized V cache on that backend. Three settings that quietly change your budget --fit is on by default. It adjusts arguments you did not set so the model fits in device memory, including context size, with a floor set by --fit-ctx (default 4096). That is helpful for a quick run and terrible for reproducibility: you can ask for a model default of 128K, get 4K, and never notice. If a specific context is a requirement, pass -c explicitly and watch the load log. --kv-unified changes what parallel slots cost. When slots are auto-sized, llama.cpp uses one unified KV buffer shared by all sequences, so the pool is sized by the total -c, not multiplied by the slot count. If you set --kv-unified-per-slot N without -c, the pool becomes n_parallel * N, which is a very different number. Read the flag you actually passed. --cache-ram is host RAM, not VRAM. It defaults to 8192 MiB and holds the prompt cache so idle slots can be parked. On a 16 GB mini-PC that is a meaningful chunk of system memory. Set it to 0 to disable if RAM, not VRAM, is your constraint. If VRAM is the hard limit and you can afford the latency, --no-kv-offload keeps the cache in system RAM instead of offloading it to the GPU. The model still runs, just slower, and the freed VRAM can go to more layers or a longer context. Doing it in Ollama Ollama has the same physics and two extra traps. First, the default context is 4096 tokens, not the model's trained context. That is why a model can appear to ignore the second half of a long document. Raise it with OLLAMA_CONTEXT_LENGTH, or per request with num_ctx. Second, parallel requests multiply the cache. The official FAQ is explicit: required RAM scales with OLLAMA_NUM_PARALLEL * OLLAMA_CONTEXT_LENGTH. Each slot gets the configured context, so 8K with 4 parallel requests reserves 32K worth of cache, not 8K. Quantized cache in Ollama is an environment variable, and it applies to every model the server loads: sudo mkdir -p /etc/systemd/system/ollama.service.d sudo tee /etc/systemd/system/ollama.service.d/kv-cache.conf >/dev/null <<'EOF' [Service] Environment="OLLAMA_FLASH_ATTENTION=1" Environment="OLLAMA_KV_CACHE_TYPE=q8_0" Environment="OLLAMA_CONTEXT_LENGTH=32768" Environment="OLLAMA_NUM_PARALLEL=1" EOF sudo systemctl daemon-reload sudo systemctl restart ollama Then confirm what actually loaded: ollama ps journalctl -u ollama -n 50 --no-pager The SIZE column in ollama ps includes the cache, so compare it before and after the change with the same context and the same model. If SIZE does not move, the drop-in did not apply: check for a typo in the unit name, and remember that a model already loaded has to be unloaded or restarted before new settings take effect. Ollama documents the trade-off honestly: q8_0 uses about half the memory of f16 with a very small precision loss, q4_0 about a quarter with a loss that can become noticeable at long context. It also notes that models with a high ratio of query heads to KV heads, such as Qwen2, may be more sensitive. That matches the arithmetic: the fewer KV heads a model has, the more each one carries. Test the cache before you trust it Memory is easy to measure. Quality is not, and "it looked fine in one chat" is not a test. A cheap, repeatable check is a needle test: bury a random code deep in a long prompt and ask for it back. Run it at f16, q8_0 and q4_0, several times each, and compare. #!/usr/bin/env python3 import json, random, urllib.request BASE = "http://127.0.0.1:8080" NEEDLE = f"{random.randint(0, 999999):06d}" FILLER = ("The archive index lists routine maintenance notes for the eastern " "warehouse, including shelf counts, inspection dates, and handler " "initials. ") def build(target_sentences=450, depth=0.2): parts = [FILLER] * target_sentences parts.insert(int(len(parts) * depth), f" IMPORTANT: the vault code is {NEEDLE}. ") return "".join(parts) prompt = build() payload = { "model": "qwen2.5-7b-instruct", "messages": [{ "role": "user", "content": prompt + "\n\nAnswer with only the six-digit vault code. " "If it is not present, answer NONE.", }], "temperature": 0, "max_tokens": 16, } req = urllib.request.Request( BASE + "/v1/chat/completions", data=json.dumps(payload).encode(), headers={"Content-Type": "application/json"}, ) with urllib.request.urlopen(req, timeout=600) as resp: out = json.load(resp) answer = out["choices"][0]["message"]["content"].strip() print("needle:", NEEDLE, "answer:", answer) print("prompt_tokens:", out.get("usage", {}).get("prompt_tokens")) print("PASS" if NEEDLE in answer else "FAIL") At roughly 450 sentences of that filler you land near 12K to 14K tokens, which is enough to stress a 32K cache without waiting all afternoon. The prompt_tokens field tells you the real depth; do not guess it, and scale the sentence count if your tokenizer disagrees. A needle test measures retrieval, not reasoning. Add one task that resembles your actual workload and has a checkable answer: extract fields into JSON, answer a question about a specific clause, or produce a diff. Exact-match and structured-output tasks degrade before free-form prose does, so they are the sensitive canaries. If q4_0 passes retrieval but fails your JSON extraction, you have your answer, and it is not the one the memory table suggested. For throughput, compare generation speed at the same context length. The native /completion endpoint returns a timings object with predicted_per_second, which is more useful than a stopwatch: curl -s http://127.0.0.1:8080/completion \ -H 'Content-Type: application/json' \ -d '{"prompt":"Summarize the plot of a heist film in three sentences.","n_predict":128}' \ | jq '.timings' Quantized caches usually help at long context because attention reads less memory, but dequantization costs compute, and the balance depends on your backend. Measure it; do not take a percentage from a blog post, including this one. The short version - Budget the cache with 2 * layers * kv_heads * head_dim * bytes, and read kv_heads from config.json. - q8_0 is 53% of f16 and q4_0 is 28%, because of block scales. Plan with the real number. - In llama.cpp, -ctk and -ctv set the cache, -fa on is required for a quantized V cache, and --fit may shrink a context you did not pin. - In Ollama, the default context is 4096, the cache type is global, and OLLAMA_NUM_PARALLEL multiplies the bill. - A 32K f16 cache on a 7B model is 1.75 to 4 GiB depending on KV heads. That is the line item, not the GGUF. - Test retrieval and one exact-match task before you keep the cheaper cache. The cache is the difference between "the model fits" and "the model works". Once you know its size, context length stops being a guess and becomes a number you choose. Sources - llama.cpp server flags, including -fa, -ctk/-ctv, --kv-unified, --kv-unified-per-slot, --cache-ram, --fit and --fit-ctx: github.com/ggml-org/llama.cpp/blob/master/tools/s… - llama.cpp issue #10378, the "V cache quantization requires flash_attn" failure: llama.cpp#10378 - ggml block definitions for q8_0, q5_1, q4_1 and q4_0: github.com/ggml-org/ggml/blob/master/src/ggml-com… - Ollama FAQ, context default, Flash Attention, KV cache type and parallel scaling: docs.ollama.com - Qwen2.5-7B-Instruct config.json: huggingface.co - Mistral-7B-Instruct-v0.3 config.json: huggingface.co

On-Device LLM Inference on Android with Microsoft Decision-1 and Qwen3.5-9B --- title: "On-Device LLM Inference on Android: Running Qwen3.5-9B with NNAPI and Vulkan Delegate" published: true description: "A production guide to deploying Qwen3.5-9B on Snapdragon 8 Gen 3 — covering Q4_K_M vs Q5_K_S quantization tradeoffs, mmap weight loading, and achieving sub-400ms TTFT." tags: android, kotlin, mobile, architecture canonical_url: https://blog.mvpfactory.co/on-device-llm-android-qwen3-nnapi --- What You Will Build By the end of this tutorial you will have a working Android inference pipeline that runs Qwen3.5-9B on a Snapdragon 8 Gen 3 device, stays under 2.5 GB of LPDDR5 memory, and delivers time-to-first-token under 400ms. Let me show you the pattern I use in every production on-device LLM deployment. Open-weight 9B models are no longer a server-only story. The hardware caught up: Snapdragon 8 Gen 3 ships with ~77 GB/s LPDDR5 bandwidth and a Hexagon NPU rated at 73 TOPS INT8. The remaining question is purely about quantization and delegation strategy — and that is exactly what we are covering here. --- Prerequisites - Android device with Snapdragon 8 Gen 3 (or simulator for architecture work) - llama.cpp Android port with JNI bindings configured - Qwen3.5-9B GGUF weights downloaded (Q4_K_M format — more on why below) - Android GPU Inspector installed for delegation profiling - minSdkVersion 29 or above (required for NNAPI stable paths) --- Step 1 — Choose Q4_K_M, Not Q5 Here is the gotcha that will save you hours: the quantization format is not a quality tradeoff on Android. It is a deployment gate. Your app gets approximately 2.5 GB of LPDDR5 (the OS reserves ~1.5 GB). Here is what the numbers actually look like at runtime: | Format | GGUF File Size | Runtime RSS (2K ctx) | PPL Delta vs FP16 | Avg Token Latency | TTFT | |---|---|---|---|---|---| | Q4_K_M | ~5.3 GB | ~2.1 GB | +0.18 | ~28 ms/tok | ~340 ms | | Q5_K_S | ~6.2 GB | ~2.5 GB | +0.09 | ~34 ms/tok | ~410 ms | | Q5_K_M | ~6.7 GB | ~2.7 GB | +0.07 | ~38 ms/tok | ~460 ms | Q5_K_S consumes the full 2.5 GB budget with zero headroom for KV cache (which costs ~280 MB at 2K context, plus ~350 MB for activation buffers). Q4_K_M leaves ~150 MB of runtime margin — enough to survive background memory pressure without an OOM kill. Do not debate Q5 until you have a device with 12 GB RAM. --- Step 2 — Memory-Map Your Weights Do not load model weights into the heap. Here is the minimal setup to get this working: val weightFile = File(modelDir, "qwen3.5-9b-q4_k_m.gguf") val channel = FileInputStream(weightFile).channel val mappedBuffer = channel.map( FileChannel.MapMode.READ_ONLY, 0L, channel.size() ) // Pass mappedBuffer directly to llama.cpp JNI binding nativeLoadFromMappedBuffer(mappedBuffer) The OS page cache handles eviction gracefully under memory pressure. Cold-start latency drops from 3–4 seconds to under 800ms because the kernel only faults in pages touched during the actual forward pass. --- Step 3 — Split Delegation by Operator Type The docs do not mention this clearly, but llama.cpp's default "auto" mode often leaves performance on the table. Mixed delegation is the right call: - NNAPI — handles quantized INT8 matrix multiplications efficiently on the Hexagon DSP and manages power budget better during sustained sessions. Route MLP layers here. - Vulkan compute — lower driver overhead for attention operations and handles irregular memory access patterns more gracefully. Route the attention mechanism here. Profile with Android GPU Inspector before committing. SoC vendor implementations vary enough that assumptions will cost you latency. --- Step 4 — Chunk-Prefill Batching for Sub-400ms TTFT Time-to-first-token is dominated by prompt prefill. Process the prompt in 64-token batches with double-buffering on weight reads: val CHUNK_SIZE = 64 val promptTokens = tokenizer.encode(userPrompt) promptTokens.chunked(CHUNK_SIZE).forEach { chunk -> nativePrefillChunk(chunk.toIntArray()) } // First decode step now starts from warm KV cache val firstToken = nativeDecode() This trades roughly 10–20% higher total prefill time for dramatically lower perceived latency. Users experience responsiveness; wall-clock throughput is a secondary metric. --- Gotchas android:largeHeap="true" will silently kill your mmap. On 32-bit ABI targets, largeHeap expands the heap address space region and directly competes with your mmap virtual address range. The model mmap can fail with ENOMEM even when physical RAM is available. The default is false — verify yours explicitly rather than assuming. Even on 64-bit targets, the expanded heap increases GC pressure during inference. Thermal throttle window is real. The Snapdragon 8 Gen 3 sustains full performance for roughly 8–12 minutes before throttling. Design your session length expectations accordingly, especially for production chat interfaces. GGUF file size and runtime RSS are not the same. mmap loads pages lazily, but the kernel still maps the full virtual address range. Always measure RSS, not file size, when evaluating your memory budget. --- Conclusion Q4_K_M, mmap loading, mixed NNAPI/Vulkan delegation, and chunk-prefill batching — that is the complete stack for production Qwen3.5-9B on Android. Each decision has a concrete reason behind it rooted in the hardware constraints of Snapdragon 8 Gen 3. Start here, profile on your specific target device, and do not optimize further until the data tells you to. Resources: - llama.cpp Android port - Android GPU Inspector - GGUF format specification

Why Small AI Models Are Becoming Essential Developer Tools: Lessons from Property Deeds, Debugging, and Email Assistants Originally published on tamiz.pro. The narrative surrounding artificial intelligence has long been dominated by the arms race for larger, more parameter-heavy models. However, a significant shift is occurring in the developer ecosystem. Small AI models—those with significantly fewer parameters than their massive counterparts—are emerging as essential, practical tools for everyday engineering tasks. This isn't just about cost savings; it's about efficiency, specificity, and architectural fit. This analysis explores why small AI models are becoming indispensable developer tools, drawing insights from three distinct domains: property deed processing (legal document analysis), debugging assistance, and email productivity assistants. Each case study highlights a core principle: specialization trumps scale when the task is well-defined. The Paradigm Shift: From Generalists to Specialists For years, the prevailing wisdom in machine learning was that more parameters equated to better generalization. While Large Language Models (LLMs) like GPT-4 or Claude 3.5 demonstrate impressive broad knowledge, they often suffer from "hallucinations," high latency, and prohibitive inference costs for routine tasks. Small AI models, typically ranging from 100 million to a few billion parameters, offer a different value proposition: 1. Low Latency & High Throughput: They can run on edge devices or local servers, providing near-instant responses. 2. Cost Efficiency: Inference costs are a fraction of those for massive models, making them viable for high-volume, low-stakes tasks. 3. Domain Specificity: When fine-tuned on narrow datasets, they often outperform generalist models in specific benchmarks. 4. Data Privacy: Local deployment ensures sensitive code or data never leaves the organization's infrastructure. Lesson 1: Property Deeds and Legal Document Analysis The processing of property deeds is a classic example of a domain where large, generalist models are overkill and potentially risky. Property deeds contain structured yet messy natural language, specific legal terminology, and critical numerical data (lot numbers, acreage, dates). Why Small Models Excel Here A generalist LLM might correctly interpret the gist of a deed but could hallucinate a parcel number or misinterpret a boundary description. In contrast, a small, fine-tuned model trained specifically on legal documents and property records can achieve high accuracy in: - Entity Extraction: Pulling out owner names, dates, and legal descriptions with high precision. - Classification: Categorizing deeds by type (warranty, quitclaim, etc.). - Consistency Checking: Ensuring fields match across related documents. Architectural Implication For developers building legal tech or real estate platforms, integrating a small, fine-tuned model (e.g., a distilled BERT or a small Transformer) allows for: - Local Processing: Sensitive client data remains on-premises or in a private cloud. - Predictable Performance: Lower variance in output compared to probabilistic giant models. - Scalability: Handling thousands of document reviews per hour without incurring massive API costs. This lesson underscores that for structured, high-stakes information extraction, a small, specialized model is often superior to a giant, generalist one. Lesson 2: Debugging Assistance Debugging is a core developer activity that requires context-awareness, pattern recognition, and knowledge of specific frameworks or languages. While large models can generate code snippets, they often struggle with deep contextual understanding of a large, complex codebase. The Role of Small Code Models Small AI models designed for code (e.g., CodeLlama-7B, StarCoder2-3B, or fine-tuned variants) are increasingly used for: - Inline Suggestions: Providing real-time code completions that are contextually relevant to the immediate function. - Bug Detection: Identifying common patterns of bugs (e.g., null pointer exceptions, off-by-one errors) within a specific codebase. - Test Generation: Creating unit tests for small, isolated functions based on their signatures and docstrings. Why Not Just Use GPT-4? 1. Context Window Limits: Small models can be optimized to work efficiently within smaller context windows that match the scope of a single function or module, reducing noise. 2. Speed: Developers need debugging help now, not after a 5-second delay. Small models provide sub-second responses. 3. Fine-Tunability: A small model can be fine-tuned on a company's internal codebase, learning its specific conventions, libraries, and anti-patterns. A generalist model cannot easily adapt to this level of specificity without massive, expensive fine-tuning. For developers, this means small AI models act as context-aware pair programmers that live within the IDE, offering immediate, relevant assistance without the overhead of cloud-based giant models. Lesson 3: Email Assistants Email productivity assistants are another area where small AI models shine. Tasks like summarizing long threads, drafting replies, or categorizing emails do not require the broad world knowledge of a massive LLM. They require understanding tone, context, and specific communication patterns. Efficiency in Communication Small models can be deployed locally or in private clouds to handle: - Summarization: Condensing long email chains into key action items. - Drafting: Generating polite, context-appropriate responses based on the incoming message. - Categorization: Sorting emails into folders or flags based on content and sender. The Privacy Advantage For enterprise developers and IT teams, email content is often sensitive. Using a small, on-premises model ensures that email data is never sent to third-party APIs. This is crucial for compliance with regulations like GDPR or HIPAA. Moreover, small models can be fine-tuned on an organization's historical email data to learn its unique communication style, resulting in more natural and brand-consistent auto-replies. Technical Considerations for Developers Integrating small AI models into developer tools requires a different approach than using cloud-based LLM APIs. Key considerations include: 1. Model Selection and Quantization Choosing the right model size is critical. Techniques like quantization (reducing the precision of model weights, e.g., from FP32 to INT8) can significantly reduce memory footprint and inference time with minimal accuracy loss. Tools like GGUF and ONNX Runtime facilitate running small models efficiently on consumer hardware. 2. Fine-Tuning Strategies Small models are easier and cheaper to fine-tune. Techniques like LoRA (Low-Rank Adaptation) allow developers to adapt pre-trained models to specific tasks (e.g., legal document parsing) with minimal computational resources. This makes it feasible for individual teams or even solo developers to create custom AI tools. 3. Hybrid Architectures In many cases, a hybrid approach is best. Use a small model for routine, high-volume tasks (debugging suggestions, email categorization) and reserve a large model for complex, ambiguous problems that require broad reasoning. This balances cost, speed, and capability. Conclusion The rise of small AI models represents a maturation of the AI ecosystem. Developers are moving away from the "one-size-fits-all" giant model approach toward specialized, efficient, and privacy-preserving solutions. Whether it's parsing property deeds, assisting with debugging, or managing email, small models offer the right balance of performance, cost, and control. For software engineers, this means embracing a new toolkit: one where small, fine-tuned models are the first line of defense for common tasks, freeing up human creativity for complex problem-solving. As the saying goes, "The best tool for the job is the right tool, not the biggest tool." Frequently Asked Questions Q: Are small AI models less accurate than large models? A: Not necessarily. For specific, well-defined tasks, small models fine-tuned on relevant data often outperform generalist large models. Their accuracy is higher in their domain of specialization. Q: Can small models run on local hardware? A: Yes. With techniques like quantization and efficient inference engines (e.g., llama.cpp, ONNX), small models can run on consumer-grade GPUs or even CPUs, making them ideal for local deployment. Q: How do I get started with integrating small AI models into my projects? A: Start by identifying a narrow, high-volume task in your workflow. Look for pre-trained small models on platforms like Hugging Face, and experiment with fine-tuning using LoRA. Tools like LangChain or LlamaIndex can help integrate these models into your existing applications. For more insights on practical AI engineering, visit Tamiz's Insights.

Offline speech-to-text for Flutter, with no server and no API key At work I build real-time transcription with cloud speech APIs. They are very good. They also need a network connection, an account, a key, a bill, and a decision about whose servers your users' voices go to. I wanted to know how close a phone could get on its own. So I built transcribe_cpp, a Flutter package that turns speech into text entirely on the device, on iOS and Android. The user talks, your app gets text, and the audio never leaves the phone. It started as an experiment and a way to learn. It is now on pub.dev, MIT licensed, and the code is on GitHub. What it does - Private. No server, no account, no API key. It works in airplane mode. - Hands-free. There is no button to hold. Voice detection decides when each sentence starts and ends. - Live text. Words appear while the person is still talking, and settle in place without flickering. - Fast. On a mid-range phone, the recommended model transcribes speech about 40 times faster than real time. - Survives real life. Phone calls, Siri, headphones plugged in or pulled out, the screen locking, the app going to the background. - Bring your own audio. If your app already records, hand the audio over at any sample rate and skip the microphone handling entirely. Add It flutter pub add transcribe_cpp Using it import 'package:transcribe_cpp/transcribe_cpp.dart'; // Download a model once (90 MB). Later calls return it straight away. final models = await ModelManager.open(); final model = await models.ensure(const ModelId('parakeet-tdt_ctc-110m-Q4_K_M')); // Load it, and listen. final session = await TranscribeSession.open(model); await session.start(); // Show the text as the user speaks. ValueListenableBuilder( valueListenable: session.text, builder: (context, text, _) => Text(text.full), ); await session.stop(); That is the whole thing. Add a microphone permission to Info.plist and the Android manifest, and you have live dictation. If you already capture audio yourself, from a call, a file or your own recorder: await session.transcribeStream( myAudio, format: const AudioFormat(sampleRate: 48000), ); Standing on transcribe.cpp I did not write a speech recognizer. The models run on transcribe.cpp, a C++ engine built on ggml, the library behind llama.cpp and whisper.cpp. It runs a whole catalogue of open speech models, converts them, and publishes their accuracy. transcribe_cpp is the part between that engine and a Flutter app: the microphone, deciding when someone is speaking, splitting speech into sentences, live text, and staying alive through everything a phone throws at a long recording. The model I recommend is NVIDIA's Parakeet 110M: a 90 MB download with a 2.5% word error rate on the LibriSpeech benchmark. Models are downloaded when the app first needs them, so your app stays small; the package itself adds about 6 MB. What turned out to be hard Getting a model to produce text was the easy week. Everything around it took the time. Live text that doesn't flicker. The models I use transcribe a whole sentence at once; they don't stream word by word. To show text while someone is still talking, the package re-reads the unfinished sentence every 350 ms. But the last word or two of each read is a guess, because the model hasn't heard what comes next yet. Show those guesses and the text jitters as words rewrite themselves. So a word only appears once two reads in a row agree on it. The text only grows, and it feels live. Knowing when a sentence ends. With no button, something has to decide when you've finished speaking. Wait too long and the text feels slow; cut too early and you split sentences mid-thought, which costs accuracy. The package runs Silero VAD, a small voice-detection model I ported to plain C, and an endpointer tuned by measuring, not by feel. Most errors came from where sentences were cut, not from the recognition itself. Phones interrupt you. iOS stops your audio for a phone call or Siri and tells you when you can have it back. Headphones change the microphone's sample rate under you. Android doesn't stop the recording at all when a call takes the microphone: it quietly feeds you silence. And iOS refuses GPU work from an app in the background, so when the screen locks, the package moves decoding to the CPU and back to the GPU when the app returns. Each of these needed its own handling, so a long session keeps going without the app babysitting it. Two bugs every Flutter plugin author should know about. Since Flutter 3.29, Dart runs on the platform's main thread, so stopping the microphone froze my UI animations while the audio system shut down. And on iOS, release builds silently removed the C functions Dart calls through FFI, while debug builds hid the problem completely. Both will get their own posts. Try it - Package: pub.dev - Docs: ishangavidusha.github.io - Code and example app: ishangavidusha/transcribe_cpp The example app lets you browse the whole catalogue of 58 models, download a few, and switch between them on your own phone in a few minutes. That beats any number I could give you. It is early. Android runs on the CPU for now, and Bluetooth headset microphones work on iOS but not yet on Android. If you try it, I'd like to hear how it goes: bug reports, questions and ideas are all welcome in GitHub Discussions or the issues. Thanks to the transcribe.cpp authors, the ggml project, the Silero team, NVIDIA and the Moonshine team, whose work this is built on.