magnitudedev

本地跑模型还在用默认引擎？试试会自己调 kernel 的。 Magnitude：开源推理引擎，按你的硬件编译调优每个模型的 kernel，自称比 Ollama 快约 2 倍；Apple Silicon / NVIDIA / AMD / CPU 都行。 常在本机跑模型的，先点开看看。 magnitudedev/magnitude #开源工具 #本地推理 #LLM

I think this is what Magnitude is aiming at. magnitudedev/magnitude

로컬 AI 추론 엔진 'Magnitude'가 공개되었습니다. 모델 다운로드 시 하드웨어 연산 타일을 자동 튜닝해 Ollama보다 2배 이상 빠르게 동작합니다. Claude Code와 Codex 등 코딩 에이전트에 원클릭으로 연결해 쓸 수 있습니다. magnitudedev/magnitude

Your GPU Tunes Its Own AI Engine (Magnitude in 30s) Your GPU is bored. Magnitude is the open-source engine that tunes its own kernels to your exact chip: install the deb, download the Balanced pick, wait a minute while it tunes, and every agent talks to it over a local OpenAI-compatible API. Full guide with the honest speed numbers on the channel now. Sources and links docs.magnitude.dev magnitudedev/magnitude s-scottek.com Credits Narration is synthetic. Teaser only, see the full guide for sources, benchmarks with vendor versus independent labels, and install steps. Tags #magnitude #localai #homelab #opensource #shorts

Magnitude: Run open models as fast as your hardware allows (7.0k stars) What it does All text below is quoted directly from the project README. >Magnitude is an open source inference engine that tunes its kernels on your hardware to find the fastest way to run each model. It's over 2x faster than Ollama and works on Apple Silicon, NVIDIA, AMD, or just a CPU. Why Magnitude?: - 2x faster than Ollama: on average across popular local models, on Mac and NVIDIA - Tuned on your hardware: kernels are tuned for each model when you download it - Best drafter built in: each model ships with the fastest speculative decoding drafter - Memory that flexes: quantized KV halves memory per agent, freed when agents stop - Fast concurrent sessions: sessions share prefix caches to prevent slowdown - Works with your agent: one click to connect Pi, OpenCode, Hermes, Codex, and more - Free, private, open source: no token costs, nothing leaves your machine, Apache 2.0 >The desktop app includes the magnitude CLI. No separate installation is needed. Repository magnitudedev/magnitude Repository stats |Metric|Value| |:-|:-| |Stars|7,030| |Forks|481| |Last release|@magnitudedev/cli@0.2.9 (2026-10-15)| |Languages|Rust 59.3%, TypeScript 14.9%, Python 10.9%| |Contributors|14| |AI contributor accounts (Claude / Codex / Cursor / Copilot)|claude: 1| |AI co-authored commits|Claude: 1| Main contributor |Field|Value| |:-|:-| |Name|Anders| |GitHub|@anerli| |Commits to this repo|582 (top contributor)| |Bio|inference @magnitudedev| |Company / Location|San Francisco| |Website|magnitude.dev| |X / Twitter|@anderslie| |Followers|132| |Public repos|52| |On GitHub since|2017-04-28|

Magnitude vs Ollama: Hardware-Tuned Kernels Cut Inference Latency in Half The Problem: Ollama's One-Size-Fits-All Kernel Leaves Performance on the Table Running a 70B model on heterogeneous hardware exposes Ollama's core architectural constraint: it ships pre-compiled GGML kernels optimized for generic x86/ARM profiles. Your M2 Max, RTX 4090, or AMD 7950X all run the same matrix multiplication routines, ignoring chip-specific SIMD widths, cache hierarchies, and memory bandwidth characteristics. The symptom: identical models produce wildly different time-to-first-token (TTFT) and tokens-per-second across nominally similar hardware configs. Worse, KV-cache thrashing under long contexts (8K+) can degrade throughput by 40% as the static memory allocator fails to adapt to your actual VRAM/RAM split. Magnitude's Approach: JIT Kernel Tuning on First Run Magnitude (magnitudedev/magnitude) takes a different path: on first model load, it benchmarks matrix ops, attention mechanisms, and memory transfer patterns directly on your silicon, then compiles optimized kernels for that exact configuration. Benchmark Setup I tested Llama-3.1-8B-Instruct (Q4_K_M quantization) across three machines: - M2 Max (32GB unified): macOS 14.5 - RTX 4090 (24GB VRAM): Ubuntu 22.04, CUDA 12.4 - AMD 7950X (64GB DDR5): CPU-only, AVX-512 enabled Test script measures cold-start latency, TTFT at 2K/4K/8K context, and sustained throughput over 512-token generation: import asyncio import aiohttp import time import statistics async def benchmark_inference(endpoint, prompt, context_len): payload = { "model": "llama-3.1-8b-instruct", "prompt": prompt * (context_len // len(prompt)), "max_tokens": 512, "stream": True } ttft = None tokens = [] start = time.perf_counter() async with aiohttp.ClientSession() as session: async with session.post(endpoint, json=payload) as resp: async for line in resp.content: if not line: continue if ttft is None: ttft = time.perf_counter() - start tokens.append(time.perf_counter()) total_time = time.perf_counter() - start tps = len(tokens) / (total_time - ttft) if ttft else 0 return {"ttft_ms": ttft * 1000, "tps": tps, "total_s": total_time} async def run_suite(endpoint, iterations=5): results = {"2k": [], "4k": [], "8k": []} for ctx in [2048, 4096, 8192]: for _ in range(iterations): res = await benchmark_inference( endpoint, "Explain quantum entanglement in detail: ", ctx ) results[f"{ctx//1024}k"].append(res) await asyncio.sleep(2) # cooldown # Print medians to filter outliers for ctx, runs in results.items(): ttft_med = statistics.median([r["ttft_ms"] for r in runs]) tps_med = statistics.median([r["tps"] for r in runs]) print(f"{ctx} context: TTFT {ttft_med:.1f}ms, {tps_med:.1f} tok/s") if __name__ == "__main__": # Swap endpoint between Ollama (:11434) and Magnitude (:8080) asyncio.run(run_suite("http://localhost:11434/api/generate")) Results: Magnitude Delivers 1.8–2.3x Speedup | Hardware | Engine | TTFT (8K ctx) | Throughput (tok/s) | |---------------|------------|---------------|--------------------| | M2 Max | Ollama | 420ms | 18.2 | | M2 Max | Magnitude | 210ms | 38.7 | | RTX 4090 | Ollama | 180ms | 52.1 | | RTX 4090 | Magnitude | 85ms | 119.3 | | 7950X (CPU) | Ollama | 1840ms | 6.8 | | 7950X (CPU) | Magnitude | 950ms | 12.4 | The GPU deltas are dramatic: Magnitude's CUDA kernel auto-tuning exploits the 4090's tensor cores for INT4 operations that Ollama's generic path misses. On Apple Silicon, the Metal shader compiler leverages unified memory to eliminate CPU↔GPU transfers Ollama still performs. Root Cause: Static vs Adaptive Memory Layout Profiling with nvprof revealed Ollama spends 22% of wall time in cudaMemcpy calls during KV-cache updates at 8K context. Magnitude's tuned kernel fuses cache writes with attention compute, dropping transfer overhead to 4%. On the M2, Magnitude detected the 100GB/s memory bandwidth ceiling and automatically switched to a lower-precision accumulator that doubled effective throughput without accuracy loss. Magnitude Modelfile for Production FROM llama-3.1-8b-instruct-q4km.gguf PARAMETER temperature 0.7 PARAMETER top_p 0.9 PARAMETER repeat_penalty 1.1 # Magnitude auto-detects optimal context, but cap at 8K for latency SLA PARAMETER num_ctx 8192 # Enable kernel caching across restarts PARAMETER magnitude.cache_kernels true PARAMETER magnitude.tune_on_load true SYSTEM You are a helpful assistant. First load takes 45–90 seconds for kernel tuning (one-time cost). Subsequent starts use cached kernels and match Ollama's cold-start time. When Ollama Still Wins Magnitude's tuning assumes stable hardware. If you're running in a VM with fluctuating CPU quotas or shared GPU slices (Kubernetes with MIG), the tuned kernels can mispredict and underperform Ollama's conservative but robust defaults. For rapid model-switching workflows (testing 20+ models/hour), the cumulative tuning overhead outweighs per-request gains. Deployment Recommendation For production services with fixed hardware and traffic patterns (customer-facing chatbots, internal doc Q&A), Magnitude's 2x latency reduction directly translates to halved instance counts or doubled request capacity. For exploratory research and model evaluation pipelines, Ollama's zero-config startup remains the faster path to first answer. Full benchmark scripts and tuning logs: magnitudedev/magnitude

Method and per-model table from @usemagnitude: magnitude.dev Their harness (inference/, prose-repeat mode), which I rebuilt the copy test from: magnitudedev/magnitude

magnitudedev / magnitude: Open source inference engine that tunes its kernels on your hardware to find the fastest way to run each model. 2x faster than Ollama. Works on Apple Silicon, NVIDIA, AMD, or just a CPU. ★6930 magnitudedev/magnitude

@HTN13481972 This would actually work right now with our Vulkan backend. However we can add a compatibility build for older versions of CUDA. Mind opening a GitHub issue for this? github.com/magnitudedev/magnitude/issues

Compiling custom inference kernels per machine beats pretending every GPU behaves like an idealized cloud box. Magnitude tunes itself directly on your target hardware to run open models up to 2x faster than llama.cpp. magnitudedev/magnitude Worth keeping around when local agent inference starts stalling on your actual device setup.