curlie

The ARMv9 Speed Paradox: Auditing Real-World Code with Gemma 4 on Bare Metal This article is submitted for the DEV Hacktoberfest 2026 Open-Source AI Challenge: Week 3 ("Slow and Steady"), competing in the Best Use of Gemma category. --- Running automated security audits on real-world codebases with frontier LLMs usually comes with two painful tradeoffs: cloud API bills and headaches or stripped-down context windows that miss cross-file vulnerabilities. For Week 3 of Hacktoberfest 2026, we took the "Slow and Steady" ethos literally: what happens when you run Google's Gemma 4 family on bare-metal ARMv9 silicon with a massive 65,536-token context window, evaluating real open-source targets against known ground-truth vulnerabilities? We deployed a headless test runner across Google Cloud ARM64 instances (c4a-standard-8, 8 vCPUs on Neoverse-V2 silicon, 32 GB RAM) and pitted multiple architectures and quantization schemes against each other: 1. unsloth/gemma-4-12b-it-GGUF (Dense 12B across 4 quantization levels: UD-Q2_K_XL, IQ4_XS, UD-Q4_K_XL, and UD-Q6_K_XL) 2. bottlecapai/ThinkingCap-gemma-4-26B-A4B-it-GGUF (Mixture-of-Experts with 26B total parameters and 4B active: Q4_K_M and Q6_K) 3. prism-ml/Ternary-Bonsai-2-27B-gguf (1.58-bit ternary quantization: PTQ1_0 executed via custom PrismML-Eng/llama.cpp kernels) Here is what we discovered about ARMv9 SIMD execution, context scaling, and vulnerability detection precision. --- The Test Targets: Verified Ground Truth Rather than synthetic benchmarks or toy examples, we scanned two real-world Go projects where our team previously discovered and remediated zero-day security flaws: 1. jhaals/yopass checked out at commit 16bc350a (pre-fix commit before the merged patch in PR #3973). - Scanned files: pkg/server/oidc.go, pkg/server/apitoken.go, and pkg/server/middleware.go (646 lines, 22,181 characters). - Ground truth flaw: Missing validation of the email_verified claim in the OIDC userinfo handler, allowing unverified identities to bypass access controls. 2. rs/curlie checked out at commit 5dfcbb17 (pre-fix commit before patches PR #104 and PR #105). - Scanned files: main.go, args/httpie.go, args/parse.go, formatter/json.go, formatter/color.go, and formatter/binaryfilter.go (723 lines, 15,312 characters). - Ground truth flaws: Arbitrary local file read in args/httpie.go via HTTPie-compatible @filename multipart form syntax, and terminal ANSI escape sequence injection in formatter/json.go. --- The Full Benchmark Dataset Across dozens of hours of headless execution on Neoverse-V2 cores, here are the complete measurements: | Model & Quantization | Size (GB) | Prompt Eval | Token Gen | yopass (Time / Triage) | curlie (Time / Triage) | | :--- | :--- | :--- | :--- | :--- | :--- | | Gemma-4-12B UD-Q2_K_XL | 4.34 GB | 8.30 t/s | 8.38 t/s | 4,040 s (0 valid / 0 rejected) | 8,720 s (2 valid / 0 rejected) | | Gemma-4-12B IQ4_XS | 5.94 GB | 10.83 t/s | 11.16 t/s | 9,260 s (1 valid / 3 rejected) | 18,260 s (6 valid / 7 rejected) | | Gemma-4-12B UD-Q4_K_XL | 6.86 GB | 15.12 t/s | 11.08 t/s | 7,375 s (1 valid / 3 rejected) | 12,715 s (4 valid / 6 rejected) | | Gemma-4-12B UD-Q6_K_XL | 9.95 GB | 14.92 t/s | 8.06 t/s | 8,730 s (1 valid / 3 rejected) | 19,739 s (2 valid / 11 rejected) | | ThinkingCap-26B-A4B Q4_K_M | 15.84 GB | 55.24 t/s | 21.78 t/s | 3,170 s (0 valid / 5 rejected) | 6,404 s (2 valid / 12 rejected) | | ThinkingCap-26B-A4B Q6_K | 21.70 GB | 54.37 t/s | 21.04 t/s | 3,213 s (1 valid / 4 rejected) | OOM crash at 2,500 s (32 GB RAM ceiling) | | Ternary-Bonsai-27B PTQ1_0 | 5.54 GB | 6.85 t/s | 3.68 t/s | 22,970 s (1 valid / 4 rejected) | 27,090 s (1 valid / 3 rejected) | | Gemma-4-26B-A4B (Vertex MaaS, Cloud) | Cloud API | ~650 t/s | 55.10 t/s | 62 s (0 valid / 4 rejected) | 90 s (4 valid / 9 rejected) | Bold values indicate best performance among local bare-metal CPU runs. Managed cloud API is included as an external reference baseline. --- Four Engineering Insights from Bare-Metal Execution 1. The ARMv9 Hardware Paradox: 4-bit Beats 2-bit Conventional intuition assumes smaller quants run faster because of reduced memory traffic. On 8 DDR5-backed Neoverse-V2 cores, the opposite occurred: - UD-Q4_K_XL (6.86 GB) achieved 15.12 tokens/second on prompt evaluation—nearly double the throughput of UD-Q2_K_XL (8.30 tokens/second), despite reading 58% more weight data per token! - Neoverse-V2 features dual 128-bit vector pipelines with native ARMv9 i8mm and dotprod instructions. Byte-aligned 4-bit and 6-bit blocks unpack with minimal instruction overhead. By contrast, non-linear 2-bit packing (Q2_K) and variable-rate lookup grids (IQ4_XS) force the CPU into heavy bit-shifting, masking, and table indirection on scalar/integer ALUs. On modern CPU architectures, SIMD unpacking overhead easily dwarfs memory bus contention. 2. The Mixture-of-Experts Supremacy on CPU The sparse ThinkingCap-Gemma-4-26B-A4B architecture routed every token to only 4 billion active parameters out of 26 billion total. On bare CPU, it sustained over 55 tokens/second during prompt ingestion and 21.8 tokens/second during generation. It completed the full repository audits in half the wall-clock time of 12B dense models, while retaining the reasoning breadth of a 26B parameter model. 3. The 1.58-Bit Quantization Floor and Triage Immunity At 1.58 bits per parameter, Ternary-Bonsai-2-27B fits 27 billion parameters into 5.54 GB of RAM. However, during deep 65k context code analysis on CPU: - It generated at 3.68 tokens/second via custom PrismML bit-emulation routines. - More importantly, extreme quantization degraded cross-domain semantic coherence: when analyzing Go source code (oidc.go), it hallucinated C functions (void parse_packet with memcpy) from its pre-training data. - The system immune response: Our multi-round skeptical triage layer immediately caught the hallucination. The verifier grepped the target codebase, confirmed no such C construct existed, and rejected the finding with a confidence of 0% ([III→I]). A disciplined multi-round pipeline successfully neutralized model hallucinations even under severe quantization breakdown. 4. 65k Context Sizing on 32 GB RAM When deploying a 65,536-token context window with 8-bit KV caching (-ctk q8_0 -ctv q8_0), the KV cache expands by 8 to 10 GB under deep reasoning chains: - Models up to 16 GB (12B dense and 26B-A4B-Q4_K_M) operated with comfortable safety margins on a 32 GB RAM instance. - The 21.70 GB 26B-A4B-Q6_K model operated smoothly during single-turn evaluations, but hit the Linux kernel OOM limit during recursive multi-turn triage on larger files. For 65k contexts on 26B+ models, 48 GB to 64 GB of RAM is the recommended hardware baseline. --- Taming the 65,536 Context Window in llama-server Feeding entire repository components into local LLMs requires specific inference server tuning: nice -n 10 llama-server \ -m ./models/ThinkingCap-gemma-4-26B-A4B-it-Q4_K_M.gguf \ -c 65536 \ -n 4096 \ -np 1 \ -fa on \ -ctk q8_0 -ctv q8_0 \ -t 8 \ --reasoning-format deepseek \ --reasoning-budget 1536 \ --host 127.0.0.1 \ --port 8080 Three configuration flags proved essential: - -np 1: By default, llama-server divides -c 65536 across multiple concurrent slots (e.g. 4 slots of 16k tokens). Passing -np 1 guarantees the full 64k window belongs to the security pipeline. - -fa on -ctk q8_0 -ctv q8_0: Flash Attention combined with 8-bit quantized KV cache reduced context memory overhead by 48%, keeping RAM usage predictable even past 45,000 prompt tokens. - --reasoning-format deepseek --reasoning-budget 1536: Gemma 4 outputs structured thought tokens (<|thought|>). Separating thought tokens from the final JSON payload prevented grammar parsing errors when enforcing strict JSON schemas. --- Two Pipeline Fixes for Local 65k Models Transitioning our scanner pipeline (AISLE nano-analyzer) to local 65k inference revealed two practical bugs: 1. Greedy Markdown Code Block Parsing The original parser used a greedy regex to locate JSON payloads: # Buggy: matches the first code fence, which may be target code inside thought reasoning match = re.search(r'```(?:json)?\s*\n?(.*?)```', text, re.DOTALL) When Gemma 4 analyzed code, its internal reasoning frequently quoted snippets of Go source wrapped in ``` `go ``` fences. The parser matched the first snippet, failed JSON deserialization, and dropped the audit finding. We resolved this by iterating backward through all fenced blocks to find the true payload: # Fixed: inspect blocks from bottom to top to identify the valid JSON payload for block in reversed(re.findall(r'```(?:json)?\s*\n?(.*?)```', text, re.DOTALL)): try: return json.loads(block.strip()) except json.JSONDecodeError: continue 2. Runaway Generation Loops Without an explicit max_tokens cap in client payloads and server flags (-n 4096), repetition loops after exhausting the reasoning budget can cause the model to generate tokens until filling the 65,536-token ceiling. Enforcing explicit caps prevented multi-hour hangs on edge-case files. --- Compute Economics: Bare-Metal CPU vs. Cloud APIs Benchmarking deep context windows often raises concerns about cloud infrastructure costs. For this study on Google Cloud Platform, our hardware setup consisted of c4a-standard-8 instances (8 vCPUs on ARM Neoverse-V2 silicon, 32 GB RAM) billed at $0.282 / hour with standard hyperdisk storage. The total compute cost across all experimental worker instances: - gemma-arm-26b (ThinkingCap-26B-A4B MoE on yopass and curlie across Q4_K_M and Q6_K): ~5.5 hours runtime = $1.55 - gemma-arm-c4a (Gemma-4-12B dense across four quantization levels): ~13.5 hours runtime = $3.80 - gemma-arm-bonsai (Ternary-Bonsai-27B 1.58-bit execution): ~15.0 hours runtime = $4.23 Running the full local benchmark across all seven model configurations cost under $10.00 in total GCP compute expenses. For comparison, running the same audit against Google Cloud's managed Vertex AI Model Garden (google/gemma-4-26b-a4b-it-maas at $0.15/M input and $0.60/M output) completed both repository audits in 2.5 minutes with a combined API cost of $0.06. By contrast, running the same 65k multi-round skeptical triage through commercial frontier closed-source APIs regularly reaches $25 to $60 per repository. For teams working within restricted networks or private VPCs where source code cannot leave the boundary, running sparse MoE models on self-hosted ARMv9 CPU hardware offers high-depth context inspection for pennies per run. --- Key Takeaways for Local Security Auditing 1. Do not default to 2-bit quants on modern CPUs: On ARMv9 Neoverse cores with DDR5 memory, integer SIMD execution makes standard 4-bit quants faster and substantially more capable than 2-bit variants. 2. MoE architectures excel on CPU: Sparse routing provided the best throughput-to-parameter ratio, sustaining over 21 tokens/second generation and 55 tokens/second prompt ingestion. 3. Multi-round skeptical triage is mandatory: Candidate scans produce false alarms; pairing discovery with an automated skeptical review round filters out benign patterns and neutralizes model hallucinations. The test runner, inference configurations, and reproduction instructions are available in the open-source repository at Huge/nano-analyzer. For the interactive web report and case studies, visit BudgetScan Stories.