candle

One `.contiguous()` call makes Qwen2 decode 2.4× faster in Candle on Metal: 42 → 102 tok/s at 2k context. After prefill, V is a transposed view, so each `Tensor::cat` copies the whole KV cache. Candle PR #4077: candle#4077 Numbers, repro: inertialabs.xyz

When "Just Enable the CUDA Feature" Met Two Free GPUs If a Rust LLM inference engine already runs on CPU and Apple Metal, and its tensor framework supports CUDA, how hard could it be to run it on NVIDIA GPUs? I expected a simple recompile. Instead, moving mini-vllm-rs, written in Rust and built on top of Candle, from my M2 Mac mini to Linux and CUDA exposed two separate F16 failures on the GPU, a profiler return value that broke my scope guards, a Mac-shaped assumption in my CLI, and a CPU optimization that had been silently leaking onto the GPU. My motivation for the move was simple: Kaggle Notebooks give you two free NVIDIA Tesla T4 GPUs. On my Mac mini, heterogeneous serving was limited to one CPU worker and one Metal GPU worker (see my previous post). With a dual-T4 notebook, I could test two GPU workers concurrently, or combine them with a CPU worker into a larger heterogeneous system. Before I could test three workers together, however, a single CUDA worker had to pass my end-to-end smoke test. F16 failed twice on CUDA: first loudly, then silently The smoke test requests F16 activations. On macOS, Candle's quantized Metal matmul does not support F16 activations, so GPU inference fell back to F32 while CPU inference used F16. On CUDA, I initially let F16 run through. It failed during model loading with unsupported dtype for dequantize F32. Even in a 4-bit GGUF model, small tensors such as normalization weights are typically left unquantized, so their stored type may be F32, F16, or BF16 depending on the source model and conversion. When converting weights to match F16 activations, Candle 0.11.0 takes different paths by device: non-CUDA backends dequantize to F32 and cast to F16, whereas CUDA dispatches to a specialized kernel table that handles a subset of quantized formats and rejects the rest, including unquantized F32 weights. I simply added a check to route F32 weights through the generic dequantize-then-cast path. That got the model past initialization, and Qwen2.5 0.5B started producing the right results. Hooray! That was the first victory. However, when I tried a larger model, Qwen2.5 7B, it failed the smoke test in a worse way: F16 activations produced corrupted text on CUDA. The prompt asked why leaves change color in autumn, and the output was: >   In!!! the!!!!!! autumn! season!! !,!!!!!!!! leaves!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !change!!!!!! !!their!! !!colors!! !from!! !green!! !!to!!!! !!!yellow! !,! !!orange! !,! !and!!! !red! !. !!This!! !is!!!!!!!!! !!!!!!!!!a!!! !wonder!! !!of!!!!! !!!!nature!! !!!that!! !!happens!!!!! !!!!!!!!!!! !!!!!!!!! !!!! !! !! !! [...] Beneath the exclamation marks, the model was still emitting relevant words, which pointed to numerical errors rather than a broken tokenizer or weight mapping. When I fell back to F32 activations on CUDA, the output returned to normal immediately, and both Llama 3.1 8B and Llama 3.2 1B worked out of the box as well. Given more time to evolve this engine, I would dig further into that numerical issue and test more data formats. For now, scoping CUDA GPU backends to fall back F32 with a warning got end-to-end inference working reliably, while F16 is still usable on the CPU backend. Why my profiler ranges popped as soon as they were pushed On CPU and Metal, specifying --trace-directory exports Chrome traces for Perfetto. On CUDA, Chrome traces show host-side scopes, but not GPU activity such as kernel execution. To see both without duplicating instrumentation, I hooked NVTX ranges into the same model, layer, attn, mlp, and qmatmul trace scopes, using an RAII guard to end each range automatically. In the first capture, NVTX labels such as qmatmul operation=attention-query appeared on the timeline, but each range lasted only up to 600 nanoseconds, and even the longest ones never covered the corresponding CUDA kernel launch call. !GPU trace with NVTX ranges My first suspicion was that Candle handed the CUDA API work off to another host thread, so the launches were happening outside the thread-local NVTX range. My coding agent chased the QMatMul call path into Candle, but found no evidence of a thread handoff. The agent next suggested making the thread sleep briefly after the range starts, to see whether the range would stretch to cover the CUDA dispatch. Instead of trying that, my intuition told me something was wrong with our RAII guard: the range always closed right before the CUDA dispatch, which looked too deterministic to be a race condition. So I went back to check our code: impl NvtxRangeGuard { fn enter(name: &str) -> Option<Self> { (cudarc::nvtx::result::range_push(name) >= 0).then_some(Self) } } impl Drop for NvtxRangeGuard { fn drop(&mut self) { cudarc::nvtx::result::range_pop(); } } I asked the agent: would enter() ever return None? Checking the CUDA Profiler documentation turned up this warning: > Some of the NVTX functions are defined to have return values. For example, the nvtxRangeStart() function returns a unique range identifier and nvtxRangePush() function outputs the current stack level. It is recommended not to use the returned values as part of conditional code in the instrumented application. The returned values can differ between various implementations of the NVTX library and, consequently, having added dependencies on the return values might work with one tool, but may fail with another. The agent had written that check, and I had missed it during review. When we logged the return value under Nsight Systems, it was -1 on every call while still recording the range, matching reports from other developers. Why did that return value produce a tiny range instead of leaving the range open forever? 1. range_push() recorded the start of the range in Nsight Systems and returned -1. 2. Because the .then_some(...) helper evaluates its argument eagerly, it constructed the NvtxRangeGuard before checking whether range_push(name) >= 0 was true. 3. Because -1 >= 0 was false, .then_some(...) discarded that newly created guard and returned None, running Drop immediately inside enter() and calling range_pop() a few hundred nanoseconds after the push. The range wasn't ending early because of async GPU work. It was being pushed and popped inside enter() before the caller even received the guard. As the agent put it in its retrospective: > I treated the push return value as a success flag: nonnegative meant “push succeeded, so create a guard”; negative meant “don’t create one.” I hadn’t verified that assumption against the NVTX implementation we were using. It was a mistake to make range behavior depend on that value. It is a mistake I could easily have made myself, and a good reminder to be careful with external APIs. Keeping the guard alive regardless of the return value and assigning a distinct color to each trace scope produced the nested hierarchy below: the blue model span breaks down into orange layer blocks, green attn and mlp phases, and individual qmatmul and attention operations right above the CUDA API launches: !GPU trace with NVTX labels From "mixed" mode to one CPU and two GPUs Alongside the CUDA fixes, I made a few other changes for the multi-GPU environment: - Make --inference-device flag repeatable: My old mixed inference mode assumed one CPU plus Metal GPU 0, each managing their own KV cache and scheduler. Replacing it with a repeatable --inference-device flag lets each gpu entry claim the next accelerator index (0, 1, and so on), supporting one CPU plus multiple GPU workers without changing the orchestrator. - Fix a leaked CPU optimization: While chasing down CUDA dispatch, my coding agent noticed that the contiguous attention path enabled a CPU grouped-query matmul layout by default without checking whether the tensor actually lived on the CPU, so this leaked onto the GPU path. - Fix model cache: In my Kaggle sessions, the model downloader used Hugging Face Hub’s default cache path (~/.cache/huggingface/hub), which wasn’t retained between sessions. I updated it to honor the environment variable HF_HOME, to choose where the cache lives. Once those were in place, I ran a smoke test sending 3 requests to a heterogeneous setup of one CPU worker and two GPU workers, with each worker running a copy of Qwen2.5 0.5B and round-robin routing sending one request to each backend: | Backend | Input tokens | Output tokens | Prefill throughput (tok/s) | Decode throughput (tok/s) | |---|---:|---:|---:|---:| | GPU 0 | 1,068 | 873 | 2,067.39 | 14.99 | | GPU 1 | 998 | 1,024 | 2,485.12 | 15.35 | | CPU | 1,047 | 1,024 | 3.94 | 2.87 | The CPU backend was definitely much slower than the GPUs, as expected, but all three workers processed their requests concurrently end to end. What I learned Moving from a Mac mini to dual T4s tested every boundary I had drawn earlier. The process split and per-backend threads scaled cleanly to a larger heterogeneous system, while the bugs showed where single-device and Metal-specific assumptions had been hiding in plain sight. There are definitely more optimizations to do. For example, when the paged KV cache is used, my current attention implementation concatenates cache pages before self-attention, which was tolerable on my Mac mini's unified memory but more significantly hurt decode performance on the T4s (smoke-tested with Qwen2.5 7B): | KV cache | Input tokens | Output tokens | Prefill throughput (tok/s) | Decode throughput (tok/s) | |---|---:|---:|---:|---:| | Contiguous | 128 | 421 | 482.1 | 63.0 | | Paged | 128 | 479 | 501.9 | 38.6 | Dedicated GPU memory also opens the door to larger models: on my Mac mini the CPU and GPU share 16 GB of RAM, whereas each T4 GPU has its own 16 GB of VRAM. The orchestration and scheduling algorithms will need to be recalibrated for multi-GPU setups as well. There is always more work to do, but this marks a steady first step for my inference engine in proving its cross-platform support.

In an attempt to learn a bit more about Transformers and Rust, I asked ChatGPT Sol to port mistral.ai to huggingface/candle 🦀 pims/timestral-rs

AI Agents Are Quietly Being Rewritten in Rust OpenAI, Block, and Perplexity all rebuilt key parts of their AI stack in Rust. In this video I break down why the layer around the model (agents, sandboxes, and tooling) is moving to Rust, and the one catch most people skip. I’m a fresh grad learning Rust and harness engineering in public, so every claim here comes from sources I checked, linked below. Chapters 0:00 Elon’s Larp 00:36 Codex CLI: TypeScript to Rust 01:46 Goose: Python to Rust 02:22 Sandboxes: Firecracker and Perplexity SPACE 03:14 uv, Ruff, and OpenAI buying Astral 03:50 Hugging Face tokenizers and Candle 04:13 The catch 04:45 Advance Sources Elon post : x.com openai/codex block/builderbot perplexityai/SPACE x.com codex#1174 news.ycombinator.com reddit.com codex#1174 news.ycombinator.com reddit.com github.com/aaif-goose/goose/blob/main/crates/goos… goose-docs.ai github.com/aaif-goose/goose/blob/main/crates/goos… firecracker-microvm.github.io x.com anthropics/claude-code reddit.com huggingface.co huggingface/candle astral-sh/uv astral.sh github.com/perplexityai/SPACE/tree/main/crates/ap… firecracker-microvm/firecracker github.com/aaif-goose/goose/tree/main Follow along Rust playlist: youtube.com Building Harness playlist: youtube.com Portfolio: vkgeed.in X post for this video: x.com #rust #aiengineering #softwareengineering

When LLM Inference on CPU Beats Waiting for the GPU mini-vllm-rs is a single-node LLM inference engine I built in Rust on top of Candle for macOS. It can run CPU and Metal GPU backends concurrently, routing whole requests between them. My first request routing benchmark favored sending almost everything to GPU: the fastest policies assigned it 11 of 12 requests. That left a more useful question unanswered: could selective CPU offload improve completion time compared with serving the entire workload on GPU alone? Separating prefill from decode suggested where to look: requests with warm prefixes, arriving behind a growing GPU backlog. A follow-up experiment tested that idea. Routing 1 of 8 requests to CPU reduced the burst's completion time by 9.8% compared with forcing all 8 onto GPU. With 48 requests, offloading 6 to CPU increased aggregate output throughput by 12.4%. The phase gap suggested the next experiment The experiments used a quantized Qwen2.5 0.5B model with F32 activations on an Apple M2. For a 512-input/128-output request, isolated backend profiles showed: | Phase | CPU | Metal GPU | GPU advantage | |---|---:|---:|---:| | Prefill | 73 tok/s | 2,551 tok/s | About 35× | | Decode | 45 tok/s | 293 tok/s | About 6.5× | A long, uncached prompt exposed the CPU's largest disadvantage. Even when GPU already had work, avoiding CPU prefill could justify waiting for it. A warm prefix could remove most of that cost, leaving the smaller decode disadvantage for GPU backlog to outweigh. Each backend owns its own scheduler and KV cache, and a request stays on the backend selected for it. I seeded an approximately 512-token prefix into both caches. Each measured request used the same 516-token prompt and generated exactly 128 tokens. Warming both sides gave CPU a chance to avoid expensive prefill. I compared forced-GPU routing against mixed routing, where the router chose between CPU and Metal using estimated pending and incoming work. Both cases used the same scheduler settings, with up to 4 active requests per backend. Requests arrived 50 ms apart without waiting for earlier responses. Each case started with a fresh server and seeded caches, and ran 3 times. What changed between requests 7 and 8? The router priced prefill and decode separately: score = remaining prefill tokens / prefill rate + remaining decode-token budget / decode rate The totals included existing work plus the incoming request. Decode work was approximated using each existing request's unused max_new_tokens budget and the incoming request's full budget, so the router did not predict actual output lengths or early stopping. The rates came from isolated backend profiles, making this a work-based ranking rather than an exact prediction of batched completion time. The candidate diagnostic logs show the boundary in run 1: | Incoming request | CPU active / queued | GPU active / queued | CPU score | GPU score | Selection | |---|---:|---:|---:|---:|---| | 7 | 0 / 0 | 4 / 2 | 3.118 s | 3.027 s | GPU | | 8 | 0 / 0 | 4 / 3 | 3.118 s | 3.444 s | CPU | Estimated routing costs from mixed-routing run 1. All 3 runs made the same choices, with slightly different GPU scores. Both candidates advertised a 496-token reusable prefix for the incoming 516-token prompt. The idle CPU therefore had only 20 estimated prefill tokens and a 128-token decode budget to account for. Its score stayed at 20/73 + 128/45 = 3.118 seconds. At request 7, GPU already had 4 active requests and 2 queued requests. Each queued prompt also carried a 496-token cache estimate, leaving 20 prefill tokens apiece. Including the incoming request, GPU's estimated prefill work was therefore 2 × 20 + 20 = 60 tokens. Its decode budget was 256 queued + 496 active + 128 incoming = 880 tokens. The score was 60/2551 + 880/293 = 3.027 seconds, still below CPU's. Occupied GPU slots alone were not enough to justify offload. About 50 ms later, request 8 reached the router. Request 7 had joined the GPU queue, while the active requests’ published remaining decode budget had fallen by 8 tokens. Including the incoming request, the GPU now had 80 estimated prefill tokens and a combined decode budget of 384 queued + 488 active + 128 incoming = 1,000 tokens. Its score rose to 3.444 seconds, above the idle CPU’s unchanged 3.118 seconds, so the router chose CPU. The warm prefixes kept prefill work small, and the accumulated decode work tipped the decision. While helping me organize the earlier benchmark data, AI surfaced an accounting bug: queued requests lost their cache discount, inflating GPU prefill work. I decided to fix it and rerun the benchmark. The first CPU assignment moved from request 7 to request 8, and the capacity results here use the corrected runs. Queued requests now retain their routing-time cache estimates, though actual reuse may decrease or increase as the cache changes before execution. Output-length estimation remains unfinished. For comparison, SGLang's scheduler scales remaining token budgets with an adjustable new_token_ratio to estimate future KV-cache demand for running requests, a heuristic rather than an exact output-length prediction. Did the decision help in practice? Run 1 produced the following measurements for request 8 and the burst as a whole: | Measured outcome | Forced-GPU control | Mixed routing | |---|---:|---:| | Request 8: time to first token | 5.561 s | 0.339 s | | Request 8: end-to-end latency | 11.116 s | 3.681 s | | Whole-burst completion time | 11.486 s | 10.361 s | Whole-burst time starts at the first submission, while request latency starts at that request's submission. The offloaded request finished well before the mixed burst ended. CPU could start useful work while GPU continued serving its assigned requests, and the entire burst also finished sooner. Across all 3 runs, mean completion time fell from 11.47 to 10.35 seconds, a 9.8% reduction. !Completion time for forced-GPU and mixed routing at 4, 8, and 48 requests Mixed routing assigned 0, 1, and 6 requests to CPU, respectively. Wall time excludes startup and prefix seeding. The benefit continued under a longer GPU backlog. In a 48-request burst, CPU received just 6 requests in every run. Mean completion time fell from 68.43 to 60.88 seconds. With the same total output, this 11.0% reduction in completion time translated into 12.4% higher aggregate output throughput. !Routing decisions for all 48 requests in saturated mixed-routing run 1 Cells show routing estimates, not measured latencies. Latency gains were not uniform. In the 8-request case, median TTFT improved, but p95 TTFT increased by 3.5%. Under the longer 48-request backlog, both median and p95 TTFT improved, alongside median and p95 end-to-end latency. Selective offload helped capacity without guaranteeing a win for every latency statistic. A GPU concurrency sweep remains the next baseline check, since these gains were measured against the fixed 4-active-request GPU configuration. Sustained arrivals and varied cache-hit rates would test how far the result carries. What the experiments changed The first benchmark suggested where to look: the CPU’s disadvantage was much smaller in decode than in prefill. The follow-up showed that warm prefixes and GPU backlog could make selective CPU offload useful. Backend speed alone did not determine placement: the router also needed to account for the work already waiting and the prefill that cache reuse could avoid. For a broader discussion of the tradeoffs behind heterogeneous execution, see When Does Heterogeneous Inference Pay Off?. The benchmark report contains the complete aggregate measurements and reproduction details.

Beyond Bigger Models: Building a Rust-Based AI Platform with Specialized Nano-LLMs --- description: "Exploring Rust-native AI, specialized nano-models, LoRA adapters, knowledge distillation, RAG, and efficient inference." tags: ai, rust, machinelearning, llm --- Why the future of efficient AI might be an intelligent network of specialized models rather than one enormous neural network. Introduction: Do We Really Need Billions of Parameters for Every Task? The AI industry has spent years pursuing increasingly powerful language models. More parameters. Larger context windows. More compute. More expensive infrastructure. But what if the next major efficiency improvement doesn't come from building another massive model? What if it comes from using smaller, highly specialized models that are exceptionally good at a narrow range of tasks? Consider a public administration system. A citizen submits a question about the deadline for appealing an administrative decision. Does this task really require a general-purpose model with nine billion parameters? Not necessarily. The system needs to: 1. Recognize the administrative procedure. 2. Extract relevant dates and information. 3. Identify the applicable legal framework. 4. Calculate the deadline correctly. 5. Explain the result in understandable language. Most of these operations do not require the full capabilities of a large language model. This observation leads to an architectural concept I want to explore at A:Team Digital Work Solutions: A modular, Rust-based AI platform combining specialized nano-models, shared base models, LoRA adapters, knowledge graphs, and deterministic execution. The objective: Use the smallest amount of computational intelligence necessary to solve a task reliably. --- 1. The Architecture: Specialized Intelligence Instead of One Giant Model The proposed architecture consists of several cooperating components. At the entry point, a semantic router analyzes incoming requests and identifies: - User intent - Domain and subject area - Task complexity - Required knowledge sources - Risk level - Appropriate model or tool The router then selects the most suitable execution path. | Domain | Specialized Component | Responsibilities | |---|---|---| | Administrative Law | Nano-Legal | Appeals, deadlines, administrative documents | | Public Finance | Nano-Budget | Municipal budgets, financial terminology, budget classifications | | General Reasoning | General LLM | Complex analysis, general questions, cross-domain reasoning | | Deterministic Tasks | Rust Tools | Date calculations, financial calculations, validation | The general-purpose model remains available but is not automatically involved in every request. An important architectural decision is that specialized models do not necessarily need independent sets of model weights. Instead, several specialization strategies can coexist. --- 2. Shared Base Models and LoRA Adapters One particularly interesting approach involves using a common base model with multiple specialized LoRA adapters. Low-Rank Adaptation (LoRA) allows us to fine-tune a relatively small number of additional parameters instead of retraining every parameter in the original model. Imagine a shared language model with separate adapters for: - Administrative appeals and procedural law - Municipal budgeting - Taxation and local fees - Public-sector document classification The runtime selects an adapter based on the incoming task. Conceptually: Incoming Request | Semantic Router | Shared Base Model | +----+----+ | | | Legal Budget General LoRA LoRA Mode | | | +----+----+ | Validated Output This architecture could reduce the storage and maintenance overhead of maintaining completely independent fine-tuned models. However, there is an important limitation. LoRA adapters do not automatically reduce the inference cost of the underlying base model. A 9B model with a small adapter still requires the computational resources of approximately a 9B model. That is why the platform should also support genuinely smaller models. --- 3. Nano-LLMs: Small Models for Narrow Problems The second strategy involves specialized models with significantly fewer parameters. Potential experimental model sizes could range from 100 million to one billion parameters, depending on task complexity. Nano-Legal A compact model specialized in: - Identifying administrative procedures - Extracting relevant dates - Recognizing missing information - Classifying administrative documents - Generating structured requests for legal validation - Explaining validated results Nano-Budget A model trained to: - Understand municipal budget terminology - Classify financial information - Identify budget categories - Interpret structured financial datasets - Explain financial terminology - Interact with deterministic financial calculation tools These models would not necessarily replace larger models. Instead, they would handle well-defined tasks independently and escalate difficult cases when required. The actual model size would be determined through benchmarking rather than arbitrary parameter targets. --- 4. Training Specialized Models I see three complementary approaches. Approach A: Fine-Tuning Existing Models The most practical starting point is to specialize an existing small language model. Training data could include: - Structured administrative examples - Domain-specific questions - Document classifications - Validated procedural workflows - Structured input/output examples - Expert-reviewed explanations Depending on model size and available hardware, training could use LoRA or full fine-tuning. Approach B: Knowledge Distillation Another promising approach is knowledge distillation. A larger teacher model generates examples, explanations, and structured task outputs. These are independently validated and used to train a smaller student model. The objective is not to reproduce every capability of the teacher. Instead, the student should learn a narrowly defined set of skills. For example, a legal nano-model might become highly effective at recognizing the information required for an administrative deadline calculation. The actual calculation would still be performed by deterministic software. Approach C: Training from Scratch A longer-term research direction involves developing custom small Transformer architectures. This would require implementing and experimenting with: - Tokenization - Attention mechanisms - Model architecture - Optimization algorithms - Training pipelines - Evaluation datasets - Quantization - Memory management Training from scratch offers greater architectural freedom, but it also introduces substantial data and compute requirements. For that reason, it should follow initial experiments with existing pretrained models. --- 5. Why Rust? Rust is particularly interesting for building the execution platform. It offers: - Native performance - Strong memory-safety guarantees - Efficient concurrency - Low runtime overhead - Predictable resource management - Cross-platform deployment - Integration with existing GPU libraries Existing projects already demonstrate its potential: - Burn — a Rust deep-learning framework. - Candle — a lightweight machine-learning framework. - mistral.rs — a high-performance LLM inference engine. Rather than recreating the entire machine-learning ecosystem, the proposed platform could build on these technologies. A possible modular structure: ateam-ai/ ├── crates/ │ ├── ateam-tensor/ │ ├── ateam-train/ │ ├── ateam-runtime/ │ ├── ateam-router/ │ ├── ateam-adapters/ │ ├── ateam-knowledge/ │ ├── ateam-optimize/ │ └── ateam-eval/ ├── models/ │ ├── nano-legal/ │ ├── nano-budget/ │ └── general/ └── benchmarks/ The first implementation would focus on inference, adapter management, evaluation, and fine-tuning integration. Custom GPU kernels and deeper training-framework development would follow only where measurements demonstrate clear advantages. Rust itself does not automatically make neural networks faster than optimized PyTorch or CUDA implementations. The advantage must come from the complete system architecture and measurable improvements. --- 6. A Practical Example: Administrative Appeal Deadlines Consider a request: > I received an administrative notice dated October 1, 2026. When does my appeal period end? A conventional LLM might attempt to calculate the deadline directly. That creates several risks. The applicable legal framework may differ by jurisdiction, notification method, and procedural context. Instead, the proposed system would use a hybrid architecture. Step 1: Intent Recognition The nano-model identifies the request as an administrative appeal deadline question. Step 2: Information Extraction It extracts: - Document date - Jurisdiction - Notification method - Relevant procedural information - Missing or ambiguous details Step 3: Legal Knowledge Retrieval The system retrieves the relevant legal provisions from a versioned knowledge base. Step 4: Deterministic Calculation A Rust function calculates the deadline using the applicable rules, including calendar adjustments and exceptions. Step 5: Validation The result is checked against required information and legal constraints. Step 6: Explanation The specialized model explains the calculation and cites the applicable legal sources. The critical principle is: Language models should interpret and explain. Deterministic software should perform calculations that require exactness. This separation is especially important in legal, financial, and administrative applications. --- 7. Knowledge Graphs and RAG Specialized training alone is not sufficient. Legal provisions change. Municipal financial data changes. Administrative regulations evolve. Continuously retraining models to reflect every update would be inefficient and potentially unreliable. Therefore, the architecture includes a shared intelligence layer combining: - Retrieval-Augmented Generation (RAG) - Knowledge graphs - Structured databases - Versioned legal documents - Deterministic tools - Output validation Fine-tuning teaches a model how to perform a task. RAG and knowledge graphs provide the current information needed to perform that task. This separation allows specialized models to remain compact while accessing up-to-date knowledge. --- 8. Reducing GPU Time and Inference Costs The primary goal is not merely to reduce parameter counts. It is to reduce the total computational cost of completing useful tasks correctly. Several techniques could contribute. Quantization Quantization reduces model weight precision and memory requirements, potentially with some quality trade-offs. Prefix Caching Prefix caching avoids repeatedly processing identical prompt prefixes. This can be particularly useful in administrative systems where many requests share the same system instructions and procedural context. Continuous Batching Continuous batching improves GPU utilization when multiple requests are processed concurrently. Speculative Decoding Speculative decoding can accelerate generation by using a smaller draft model whose proposed tokens are verified by a larger model. Performance depends on the hardware, model combination, and token acceptance rate. Adaptive Model Routing Simple requests are directed to smaller models. More complex tasks are escalated to larger models. Adapter Sharing Multiple specialized adapters can share the same underlying model. This reduces the need to store and maintain multiple complete fine-tuned models, although it does not inherently reduce base-model computation. The platform should evaluate these techniques individually and in combination. --- 9. How Should We Measure Success? Tokens per second are useful, but they do not tell the complete story. A faster model that produces more incorrect answers is not necessarily more efficient. I would evaluate the platform using the following metrics: | Metric | Purpose | |---|---| | Task accuracy | Measures correctness | | Time to First Token | Measures responsiveness | | Tokens per second | Measures generation throughput | | GPU-seconds per request | Measures compute consumption | | Peak VRAM | Measures memory requirements | | Energy per completed task | Measures energy efficiency | | Cost per validated task | Measures practical operating efficiency | | Escalation rate | Measures dependence on larger models | The central optimization target should be: Minimize computational cost per correctly completed task while maintaining defined quality and reliability thresholds. For legal and administrative applications, incorrect answers must be explicitly measured and cannot be hidden inside an average quality score. --- 10. A Practical Development Roadmap I would approach the project in four stages. Phase 1: Establish the Baseline Run existing small and general-purpose models on a fixed collection of administrative tasks. Measure: - Accuracy - Latency - GPU utilization - Memory consumption - Cost per completed task Phase 2: Build the Rust Runtime Implement: - Semantic routing - Model execution - Adapter management - Deterministic tools - Evaluation and monitoring Reuse established Rust machine-learning libraries wherever practical. Phase 3: Train Specialized Models Develop a nano-model for administrative appeals and another for municipal budgeting. Compare: - Shared-base LoRA specialization - Standalone small models - Knowledge distillation - Different quantization levels Phase 4: Optimize the Complete System Experiment with: - Quantization - Caching - GPU scheduling - Model routing - Memory management - Speculative decoding Measure whether each change actually reduces costs without compromising task quality. --- 11. The Bigger Picture I believe the future of AI infrastructure may involve more than simply deploying increasingly large models. We may see architectures where multiple specialized components collaborate, share knowledge, and execute tasks according to their actual computational requirements. A small model might classify an administrative document. A deterministic Rust function might calculate a deadline. A knowledge graph might identify the relevant legal provisions. A larger language model might only become involved when deeper reasoning is required. The result could be an AI platform that is: - Faster - More resource-efficient - Easier to validate - More economical to operate - More flexible - Better suited to specialized domains This is the direction I want to explore with A:Team Digital Work Solutions. Not because smaller models are automatically better. But because the right model, combined with the right tools and the right knowledge, may be substantially more effective than using the largest available model for everything. The future of efficient AI might not be one enormous brain. It might be an intelligent system of specialized capabilities working together. --- This article describes a proposed research and development architecture. Performance improvements and model-size targets are hypotheses that require experimental validation. What do you think? Could specialized nano-models, shared adapters, and Rust-native inference become a practical alternative to monolithic AI deployments?

Your Python AI Code Is Secretly Running Rust You write three lines of Python, and a sentence becomes nine token IDs. But the code that cut that sentence apart is compiled Rust. In this video we follow one request down the stack: from a real Hugging Face Tokenizers call, across the PyO3 boundary into Rust, and on to the C++ and CUDA code that still does the GPU math. Then we look at why an expensive GPU can sit waiting on CPU-side work, what actually helps, and what Rust costs. Every code output on screen is real (tokenizers 0.23.3, safetensors 0.8.0, rustc 1.97.1). Timelines of idle GPUs and request-time bars are labelled illustrations, not benchmarks. Chapters 0:00 Python calling Rust (real output) 0:18 The stack under your Python 0:37 Text becomes token IDs 1:08 Why tokenization is real work 1:33 Crossing the Python ↔ Rust boundary 2:10 Why Rust? A real compiler example 2:56 Where Rust stops: C++, CUDA, Candle 3:33 Why a fast GPU can wait 3:56 Overlap first, then measure 4:24 What Rust costs 4:46 Takeaway: follow the work Sources - Hugging Face Tokenizers: huggingface/tokenizers - PyO3 (Rust bindings for Python): pyo3.rs - The Rust Book, References and Borrowing: doc.rust-lang.org - safetensors: huggingface/safetensors - PyTorch: pytorch/pytorch - PyTorch Performance Tuning Guide (data loading): docs.pytorch.org - Candle: huggingface/candle Narration is AI-generated (Gemini TTS, voice Alnilam); script, research and graphics are original. Sound effects (CC0): "Amplifier Snap Button 1_4" by Joao_Janz, "Deep Whoosh #1" by Kinoton, "SeriousError01e.aif" by Kuru23 (via Openverse/Freesound). Rust logo © Rust Foundation (CC BY 4.0); Python logo is a trademark of the Python Software Foundation; both used to refer to the languages. #Python #Rust #MachineLearning

Why AI Infrastructure Is Moving to Rust ? GitHub just rewrote its Copilot agent runtime in Rust - and you are probably running Rust inside your Python AI code already. So is Rust replacing Python in AI? No: Python stays the language you drive AI with, while Rust replaces the C and C++ engine underneath and the servers around it. In this video: why Python only gives the orders, the memory bug C++ lets through and Rust refuses to compile, four reasons Rust fits AI systems (no garbage collector, real parallel threads, memory layout control, PyO3), real Rust projects in the AI stack, the honest downsides, and who should actually learn Rust. The C++ and Rust snippets are our own examples; the PyO3 snippet is the official example from the PyO3 README (Apache-2.0 / MIT). Copilot performance figures are GitHub's own measurements, excluding model and network time. Chapters: 0:00 Intro 0:13 GitHub's Rust rewrite 0:48 Rust you already run 1:20 Who really does the math 1:42 Two different jobs 2:00 The problem with C++ 2:42 What Rust does instead 3:07 Why Rust fits AI systems 4:13 Rust inside Python 4:35 Rust around AI 4:59 The catch 5:23 Should you learn Rust? Stock footage: Video by Atakan Ozkan on Pexels; Video by Nothing Ahead on Pexels. Sources (checked 2026-10-17): github.blog huggingface/tokenizers safetensors/safetensors openai/tiktoken github.com/pydantic/pydantic/tree/main/pydantic-c… pydantic.dev pola-rs/polars astral-sh/uv astral-sh/ruff qdrant/qdrant lancedb/lancedb huggingface/candle tracel-ai/burn PyO3/pyo3 PyO3/maturin chromium.org microsoft.com docs.python.org doc.rust-lang.org doc.rust-lang.org docs.pytorch.org huggingface.co docs.lancedb.com docs.python.org raw.githubusercontent.com Independent explainers of AI creative tools. Not affiliated with, endorsed by, or sponsored by any company or tool shown. Prices and features change fast - always check the official sites. Narration uses an AI-generated voice. Script and facts are written and checked by the creator. #rust #python #ai #AItools #AIart #AIvideo

Why Continuous Batching Made My LLM Inference Engine Slower I had been adding optimizations to mini-vllm-rs, my Rust LLM inference engine for macOS, including prefix caching and speculative decoding. Continuous batching looked like the next easy win. Process requests together, reuse model weights across more tokens, and watch throughput climb. What could go wrong? I tested two concurrent requests on my M2 Mac mini with 16 GB of unified memory, running a 4-bit quantized Qwen2.5 7B model. Each request generated 512 tokens, with speculative decoding disabled. Guess what? Boom. Aggregate throughput dropped from 16.8 tokens/s unbatched to 8.7 tokens/s batched, almost cut in half. Fixing that slowdown led to another surprise: the same kernel-selection threshold could also change generated text, even with greedy decoding. Two rows, a different kernel The clue was in Candle, the framework I use for backend execution. Its quantized Metal path contained this tiny decision: // Candle 0.11.0: candle-core/src/quantized/metal.rs, line 372 if src_shape.dim(D::Minus2)? == 1 { return self.fwd_mv(self_shape, storage, layout); } // Line 414 calls into call_quantized_matmul_mm_t Source. The condition checks the number of input rows. One row takes the GEMV path, multiplying the weights by one vector. More than one row continues into GEMM, which handles multiple input vectors together. Packing two requests changed the input from one row to two. That was enough to switch kernels. GEMM can reuse weights across rows, but this implementation's overhead outweighed that benefit at such a small batch. My scheduler was doing what I wanted. The backend was choosing a slower way to execute it. !One input row takes the GEMV path, while two packed rows take GEMM and slow down in this small-batch test. Getting the speed back I added a fallback: split small inputs into rows, run GEMV for each, then concatenate the results. At a batch size of two, the fallback brought throughput back to 16.25 tokens/s, compared with an unbatched baseline of 16.15 tokens/s. Relief! But would GEMV still win with larger batches? | Concurrent requests | Unbatched | Batched GEMM | Batched row-wise GEMV | GEMV / GEMM throughput | | --- | ---: | ---: | ---: | ---: | | 3 | 17.41 | 13.05 | 17.52 | 134.3% | | 4 | 17.22 | 16.55 | 18.07 | 109.2% | | 5 | 17.44 | 19.92 | 18.06 | 90.7% | Aggregate output throughput in tokens/s. At five requests, the ratio fell below 100%, so GEMM pulled ahead. Separate dispatches and concatenation may have become more expensive than one GEMM call. I chose four rows as the default switching threshold for this workload. See the benchmark report for details. Problem solved? Not quite. Same prompt, different story Later, I tested speculative decoding with two requests, using Qwen2.5 0.5B to propose four tokens at a time. Their overall draft acceptance rate was 32.9% when processed one at a time, versus 37.1% when allowed to run concurrently. Same requests. Same models. Same proposal policy. Greedy decoding in both runs. Why? A separate check made the numerical issue easier to see. I asked Qwen to write a science-fiction scene about a mechanic and an archivist investigating a distress signal in an abandoned greenhouse on the Moon. Both runs also used greedy decoding. I changed the threshold so that prefill used GEMM in one run and row-wise GEMV in the other. The first 597 characters matched exactly. Then the continuations split: | Prefill kernel | First different continuation | | --- | --- | | GEMM | “Mara moved to the second door, where an emergency terminal stood…” | | GEMV | “Mara’s flashlight beam cut through the gloom, illuminating rows of dry hydroponic beds…” | Both responses were coherent and followed the same characters and setting. Nothing looked like a completely broken model. That made small numerical differences a plausible suspect. Tracing the speculative test gave me stronger evidence. At the first differing token, the drafter proposed the same token IDs in both runs, but two leading target scores had swapped order: | Execution | Token 476 | Token 323 | Selected token | | --- | ---: | ---: | --- | | Serial admission | 25.571232 | 25.565775 | 476 | | Continuous batching | 25.549618 | 25.562220 | 323 | That tiny change was enough to flip the greedy choice. Every later token then followed a different prefix, which could change draft acceptance too. !The same lunar-greenhouse prompt produces two coherent continuations: Mara approaches a terminal in one and explores with a flashlight in the other. The other request had already finished Here was the part that caught me off guard: when the token choice flipped, the other request was already gone. So why did its earlier presence still matter? Here, rows count token positions, not requests. An earlier batch had combined 120 prefill tokens from the new request with four verification tokens from the existing request. Those 124 rows crossed my threshold and selected GEMM instead of GEMV, introducing small numerical differences into the cached state. By the time the output diverged, the other request had finished, and the remaining request's four-token verification was back on the GEMV path. But switching back did not undo the differences already stored in its KV cache. Looking only at the iteration where the text diverged would have missed the cause. I forced all relevant inputs through GEMV in both execution modes. In the shortened comparison, output hashes and acceptance totals matched. This did not establish which kernel was more accurate, but it supported the explanation. The numerical report contains the investigation. A batching decision changed the kernel selection, and its effects survived in cached state long after the batch changed again. My longer article follows how a generation loop grew into a serving engine with clear boundaries between scheduling, model execution, and request state. It also explores how AI helped with architecture, coding, and performance optimization, and where I had to challenge its decisions.

From a Rust Inference Engine to Heterogeneous Serving My background spans ML accelerator compilers and OS virtualization, and I now work on accelerator fleet efficiency. To explore how compilers, runtimes, and hardware interact in LLM serving, I built mini-vllm-rs, a single-node inference engine written entirely in Rust for macOS. It supports continuous batching, paged KV caching, prefix caching, and speculative decoding. CPU and Metal GPU workers can run concurrently, each handling complete requests. The project and two articles document that learning process (with more to come): - mini-vllm-rs: code, setup, and benchmarks - Building an LLM Inference Engine with AI: The Code Was the Easy Part - When Does Heterogeneous Inference Pay Off? What building the engine taught me To give you a glimpse of the problems I encountered, two requests generating 512 tokens each achieved an aggregate throughput of 16.8 output tokens/s when run sequentially on an Apple GPU. However, continuous batching reduced it to 8.7 tokens/s. Why? Batching changed the matrix shape, making Candle, the framework I use for backend execution, switch from a GEMV kernel to a GEMM kernel. GEMM was slower for this small batch, so I added a GEMV fallback for small batches and benchmarked the crossover point to choose the switching threshold. The first article follows how a simple generation loop grew into an engine with clear boundaries between scheduling, model execution, and request state. It examines how continuous batching required each request’s KV state to be managed independently of model execution, and how prefix caching complicated when that memory could be reused or released. Those boundaries later made adding concurrent CPU and GPU workers possible without a major redesign. It also reflects on using AI to implement the engine, and the design reviews and experiments I needed to decide which changes to keep. Why split work when a GPU can run the whole model? In my project, I considered GPU prefill with CPU decode, or a GPU target model with a CPU drafter. Apple Silicon's shared memory looked helpful, but Candle's device-based memory model complicated cross-device state access. Synchronization and low request concurrency also made keeping both devices busy difficult. Hence, I implemented neither split. Meanwhile, disaggregation is being explored at a much larger scale: NVIDIA has announced configurations that split inference work between Vera Rubin GPUs and Groq LPX, while AWS has announced prefill on Trainium paired with decode on Cerebras. The second article asks when specialization can repay communication costs. In one simplified prefill/decode example, transferring 2 GiB of KV cache at 25 GB/s takes about 86 ms. Without compute–communication overlap, specialized accelerator decode must beat GPU decode by more than 0.67 ms/token for a 128-token output to repay that transfer. For a 512-token output, the threshold falls to 0.17 ms/token. The same accelerator could make a long answer faster but a short one slower. Beyond this example, it compares prefill/decode, attention/FFN, and target/draft separation. It also asks why production might choose a slower path: more KV capacity, larger batches, and scaling each hardware pool independently can improve service capacity even when one request finishes later. !“What would you try?” If you are coming to inference from another part of the stack, or mostly work with GPUs and want to understand the alternatives, you may be asking similar questions. I'm sharing the code, experiments, and articles as I learn, and I'd be interested to hear which trade-offs you've encountered.