magnitudedev

I think this is what Magnitude is aiming at. magnitudedev/magnitude

로컬 AI 추론 엔진 'Magnitude'가 공개되었습니다. 모델 다운로드 시 하드웨어 연산 타일을 자동 튜닝해 Ollama보다 2배 이상 빠르게 동작합니다. Claude Code와 Codex 등 코딩 에이전트에 원클릭으로 연결해 쓸 수 있습니다. magnitudedev/magnitude

Your GPU Tunes Its Own AI Engine (Magnitude in 30s) Your GPU is bored. Magnitude is the open-source engine that tunes its own kernels to your exact chip: install the deb, download the Balanced pick, wait a minute while it tunes, and every agent talks to it over a local OpenAI-compatible API. Full guide with the honest speed numbers on the channel now. Sources and links docs.magnitude.dev magnitudedev/magnitude s-scottek.com Credits Narration is synthetic. Teaser only, see the full guide for sources, benchmarks with vendor versus independent labels, and install steps. Tags #magnitude #localai #homelab #opensource #shorts

Magnitude: Run open models as fast as your hardware allows (7.0k stars) What it does All text below is quoted directly from the project README. >Magnitude is an open source inference engine that tunes its kernels on your hardware to find the fastest way to run each model. It's over 2x faster than Ollama and works on Apple Silicon, NVIDIA, AMD, or just a CPU. Why Magnitude?: - 2x faster than Ollama: on average across popular local models, on Mac and NVIDIA - Tuned on your hardware: kernels are tuned for each model when you download it - Best drafter built in: each model ships with the fastest speculative decoding drafter - Memory that flexes: quantized KV halves memory per agent, freed when agents stop - Fast concurrent sessions: sessions share prefix caches to prevent slowdown - Works with your agent: one click to connect Pi, OpenCode, Hermes, Codex, and more - Free, private, open source: no token costs, nothing leaves your machine, Apache 2.0 >The desktop app includes the magnitude CLI. No separate installation is needed. Repository magnitudedev/magnitude Repository stats |Metric|Value| |:-|:-| |Stars|7,030| |Forks|481| |Last release|@magnitudedev/cli@0.2.9 (2026-10-15)| |Languages|Rust 59.3%, TypeScript 14.9%, Python 10.9%| |Contributors|14| |AI contributor accounts (Claude / Codex / Cursor / Copilot)|claude: 1| |AI co-authored commits|Claude: 1| Main contributor |Field|Value| |:-|:-| |Name|Anders| |GitHub|@anerli| |Commits to this repo|582 (top contributor)| |Bio|inference @magnitudedev| |Company / Location|San Francisco| |Website|magnitude.dev| |X / Twitter|@anderslie| |Followers|132| |Public repos|52| |On GitHub since|2017-04-28|

Magnitude vs Ollama: Hardware-Tuned Kernels Cut Inference Latency in Half The Problem: Ollama's One-Size-Fits-All Kernel Leaves Performance on the Table Running a 70B model on heterogeneous hardware exposes Ollama's core architectural constraint: it ships pre-compiled GGML kernels optimized for generic x86/ARM profiles. Your M2 Max, RTX 4090, or AMD 7950X all run the same matrix multiplication routines, ignoring chip-specific SIMD widths, cache hierarchies, and memory bandwidth characteristics. The symptom: identical models produce wildly different time-to-first-token (TTFT) and tokens-per-second across nominally similar hardware configs. Worse, KV-cache thrashing under long contexts (8K+) can degrade throughput by 40% as the static memory allocator fails to adapt to your actual VRAM/RAM split. Magnitude's Approach: JIT Kernel Tuning on First Run Magnitude (magnitudedev/magnitude) takes a different path: on first model load, it benchmarks matrix ops, attention mechanisms, and memory transfer patterns directly on your silicon, then compiles optimized kernels for that exact configuration. Benchmark Setup I tested Llama-3.1-8B-Instruct (Q4_K_M quantization) across three machines: - M2 Max (32GB unified): macOS 14.5 - RTX 4090 (24GB VRAM): Ubuntu 22.04, CUDA 12.4 - AMD 7950X (64GB DDR5): CPU-only, AVX-512 enabled Test script measures cold-start latency, TTFT at 2K/4K/8K context, and sustained throughput over 512-token generation: import asyncio import aiohttp import time import statistics async def benchmark_inference(endpoint, prompt, context_len): payload = { "model": "llama-3.1-8b-instruct", "prompt": prompt * (context_len // len(prompt)), "max_tokens": 512, "stream": True } ttft = None tokens = [] start = time.perf_counter() async with aiohttp.ClientSession() as session: async with session.post(endpoint, json=payload) as resp: async for line in resp.content: if not line: continue if ttft is None: ttft = time.perf_counter() - start tokens.append(time.perf_counter()) total_time = time.perf_counter() - start tps = len(tokens) / (total_time - ttft) if ttft else 0 return {"ttft_ms": ttft * 1000, "tps": tps, "total_s": total_time} async def run_suite(endpoint, iterations=5): results = {"2k": [], "4k": [], "8k": []} for ctx in [2048, 4096, 8192]: for _ in range(iterations): res = await benchmark_inference( endpoint, "Explain quantum entanglement in detail: ", ctx ) results[f"{ctx//1024}k"].append(res) await asyncio.sleep(2) # cooldown # Print medians to filter outliers for ctx, runs in results.items(): ttft_med = statistics.median([r["ttft_ms"] for r in runs]) tps_med = statistics.median([r["tps"] for r in runs]) print(f"{ctx} context: TTFT {ttft_med:.1f}ms, {tps_med:.1f} tok/s") if __name__ == "__main__": # Swap endpoint between Ollama (:11434) and Magnitude (:8080) asyncio.run(run_suite("http://localhost:11434/api/generate")) Results: Magnitude Delivers 1.8–2.3x Speedup | Hardware | Engine | TTFT (8K ctx) | Throughput (tok/s) | |---------------|------------|---------------|--------------------| | M2 Max | Ollama | 420ms | 18.2 | | M2 Max | Magnitude | 210ms | 38.7 | | RTX 4090 | Ollama | 180ms | 52.1 | | RTX 4090 | Magnitude | 85ms | 119.3 | | 7950X (CPU) | Ollama | 1840ms | 6.8 | | 7950X (CPU) | Magnitude | 950ms | 12.4 | The GPU deltas are dramatic: Magnitude's CUDA kernel auto-tuning exploits the 4090's tensor cores for INT4 operations that Ollama's generic path misses. On Apple Silicon, the Metal shader compiler leverages unified memory to eliminate CPU↔GPU transfers Ollama still performs. Root Cause: Static vs Adaptive Memory Layout Profiling with nvprof revealed Ollama spends 22% of wall time in cudaMemcpy calls during KV-cache updates at 8K context. Magnitude's tuned kernel fuses cache writes with attention compute, dropping transfer overhead to 4%. On the M2, Magnitude detected the 100GB/s memory bandwidth ceiling and automatically switched to a lower-precision accumulator that doubled effective throughput without accuracy loss. Magnitude Modelfile for Production FROM llama-3.1-8b-instruct-q4km.gguf PARAMETER temperature 0.7 PARAMETER top_p 0.9 PARAMETER repeat_penalty 1.1 # Magnitude auto-detects optimal context, but cap at 8K for latency SLA PARAMETER num_ctx 8192 # Enable kernel caching across restarts PARAMETER magnitude.cache_kernels true PARAMETER magnitude.tune_on_load true SYSTEM You are a helpful assistant. First load takes 45–90 seconds for kernel tuning (one-time cost). Subsequent starts use cached kernels and match Ollama's cold-start time. When Ollama Still Wins Magnitude's tuning assumes stable hardware. If you're running in a VM with fluctuating CPU quotas or shared GPU slices (Kubernetes with MIG), the tuned kernels can mispredict and underperform Ollama's conservative but robust defaults. For rapid model-switching workflows (testing 20+ models/hour), the cumulative tuning overhead outweighs per-request gains. Deployment Recommendation For production services with fixed hardware and traffic patterns (customer-facing chatbots, internal doc Q&A), Magnitude's 2x latency reduction directly translates to halved instance counts or doubled request capacity. For exploratory research and model evaluation pipelines, Ollama's zero-config startup remains the faster path to first answer. Full benchmark scripts and tuning logs: magnitudedev/magnitude

Method and per-model table from @usemagnitude: magnitude.dev Their harness (inference/, prose-repeat mode), which I rebuilt the copy test from: magnitudedev/magnitude

magnitudedev / magnitude: Open source inference engine that tunes its kernels on your hardware to find the fastest way to run each model. 2x faster than Ollama. Works on Apple Silicon, NVIDIA, AMD, or just a CPU. ★6930 magnitudedev/magnitude

@HTN13481972 This would actually work right now with our Vulkan backend. However we can add a compatibility build for older versions of CUDA. Mind opening a GitHub issue for this? github.com/magnitudedev/magnitude/issues

Compiling custom inference kernels per machine beats pretending every GPU behaves like an idealized cloud box. Magnitude tunes itself directly on your target hardware to run open models up to 2x faster than llama.cpp. magnitudedev/magnitude Worth keeping around when local agent inference starts stalling on your actual device setup.

Magnitude on Apple Silicon: Faster Local LLMs for Developers Running large language models locally on your Mac can be a frustrating experience. Even with Apple Silicon's impressive unified memory, you often hit performance bottlenecks, especially when working with larger models or complex agentic workflows. Standard local runtimes, while functional, typically ship with pre-compiled kernels optimized for broad hardware classes. This leaves significant performance on the table for your specific chip. That's where Magnitude comes in. It's an open-source inference engine designed to squeeze every ounce of performance out of your hardware, particularly on Apple Silicon. The core idea is simple: instead of generic kernels, Magnitude compiles and tunes its kernels directly on your machine, matching them to your exact chip. This optimization can lead to substantial speedups, making local LLM development far more responsive. For me, as a Sr. Frontend Developer at Digilantern building an AI-powered SDR agent platform, the ability to run these models locally and quickly is crucial for rapid iteration and privacy. I've previously explored other local LLM solutions like Ollama, and I appreciate any tool that makes powerful AI accessible without relying on costly cloud APIs. Getting Magnitude Up and Running Magnitude is primarily a desktop application that bundles a command-line interface (CLI). This makes initial setup straightforward. What you'll need: - An Apple Silicon Mac (M1, M2, M3, M4 series). - macOS 13 or newer. - An internet connection to download the Magnitude app and models. Installation: 1. Download the Desktop App: Head over to magnitude.dev/download and grab the macOS installer for Apple Silicon. 2. Install: Once downloaded, install the application like any other macOS app. The Magnitude CLI is included with the desktop app, so no separate CLI installation is necessary. 3. Launch the App: Open the Magnitude application. You'll land on the "Discover" tab, where Magnitude profiles your hardware and recommends models compatible with your machine. Your First Local LLM Interaction Magnitude is built as an inference engine for AI agents, meaning it provides a local API endpoint that your agents or other applications can connect to. It exposes both an OpenAI-compatible API and an Anthropic-compatible API on localhost:10100. Here's how you can download a model and interact with its API: 1. Download a Model: In the Magnitude desktop app, navigate to the "Discover" tab. Browse the curated list of available models. Magnitude estimates the tokens per second for each model on your specific hardware, which is incredibly useful for choosing one that fits your performance needs before committing disk space. Select a model (e.g., Qwen-3.5-4B-Chat-Q8_0) and click "Download." Magnitude will assess your machine, download the model, and perform the hardware-tuned compilation step. This one-time compilation optimizes the kernels for your Mac. 2. Connect via API: Once the model is downloaded and ready, you can connect your AI agent or other tools to the local OpenAI-compatible API. For a quick test from your terminal, you can use curl to send a chat completion request to http://localhost:10100/v1/chat/completions. curl localhost \ -H "Content-Type: application/json" \ -d '{ "model": "qwen-3.5-4b-chat-q8_0", "messages": [ {"role": "user", "content": "Explain the concept of hardware-tuned inference engines in one sentence."} ] }' ``` Replace "qwen-3.5-4b-chat-q8_0" with the actual model ID of the model you downloaded. You won't need an API key for local Magnitude interactions. This curl command demonstrates how an application or an AI agent would send a prompt to Magnitude. Many popular AI agent tools, like Claude Code or OpenCode, offer one-click connections to Magnitude's local endpoint. Considerations and Limitations Magnitude offers compelling performance, but it's important to understand its current state and a few caveats: - Performance Metrics: The "up to 2x faster" claim over llama.cpp is significant, especially for decode speed (how fast tokens are generated one by one). On an M4 Pro Mac, Magnitude achieved 57 tokens per second for decode, compared to 30 tokens per second for llama.cpp with the Qwen 3.6 35B A3B model in a 4-bit quantization. Prefill speed (processing the initial prompt) also saw an improvement of about 9%. These benchmarks are from the vendor and have not yet been widely independently verified. - Model Ecosystem: As of October 2026, Magnitude's model catalog includes about 15 curated open-weight models, typically 4-bit quantized or higher. The platform currently focuses on optimizing these specific families. If you frequently work with obscure GGUF files or require a broader selection of models that aren't in their curated list, you might find the options somewhat limited compared to more general-purpose runtimes like Ollama. There are no clear official documentation steps for loading custom GGUF files directly into Magnitude at this time. - Resource Management: Magnitude aims to be efficient, unloading idle models and sharing prefix caches for concurrent sessions to save memory. However, downloaded models reside in a hidden Magnitude folder in your home directory and are not automatically deleted if you uninstall the app. You'll need to manually clean up this folder to reclaim disk space. Also, if a model has been idle, the first response might be slower as it needs to be reloaded. - Early Stage: Magnitude is a relatively new project, released on September 30, 2026, and is still in active development (version 0.2.5 as of October 2026). The engine itself was rewritten in September 2026. This means you might encounter rapid updates and potential breaking changes as the project matures. Magnitude is an excellent tool if you're developing AI agents on an Apple Silicon Mac and prioritize local performance and privacy for supported models. It's particularly appealing if you find other local LLM solutions too slow for interactive agentic workflows. However, if your primary need is simply to chat with a wide variety of LLMs with minimal setup, or if you need to experiment with custom GGUF files, other tools might currently offer more flexibility. Sources - daily.dev - fireup.pro - docs.magnitude.dev - magnitudedev/magnitude - youtube.com - youtube.com - youtube.com - youtube.com - biggofinance.com - dev.to - medium.com - medium.com --- This article was generated with AI (Google Gemini + web search). Please check important details against the sources above. Daily AI notes on LinkedIn — Kamal Kishor