unsloth

Introducing DeepSeek-V4.1-Flash-1017 Dynamic v3.1 Unsloth GGUFs Hey everyone! We’re releasing new DeepSeek-V4.1-1017 Dynamic V3.1 (preview) GGUFs which showcases how Unsloth’s GGUFs retain the most accuracy for its size across 7 different benchmarks. We benchmarked ISTA's GSQ-RCO GGUFs but could not reproduce their reported results, even when using the highest scores. The biggest discrepancy was on AIME25 where they claimed 100% accuracy recovery to baseline, but was only ~91.88% in benchmark tests. Across AIME25, LiveCodeBench, GPQA Diamond, Aider Polyglot, and more, Unsloth Dynamic V3.1 outperforms GSQ-RCO GGUFs on nearly all metrics. Even our old v3 does better on nearly all metrics than ISTA. We ran each benchmark at least 3 times (AIME25 was 16 times) and averaged the results to ensure a fair, reliable comparison. More results coming next week! All benchmark data, methodologies and results are in our blog. Separately, for KLD, there were queries whether it’s a reliable measure of accuracy. The primary goal of quantization is to minimize KLD but also minimize disk space. If you get a KLD of 0, this means your quantized model == 16bit model, as logits are perfectly modelled. Our benchmarks show on KLD and top-1% across multiple domains (coding, agentic coding, chat etc), UDv3.1 does much better than GSQ-RCO across every domain. We show that lower bit quants do think somewhat longer than the unquantized model - we plan to investigate and make quants even better. However, we're still much ahead vs GSQ-RCO on even reasoning length as well, and provide plots for truncated answers in our blog. The model will be able to run on 128GB RAM/unified memory devices or even on 64GB RAM with engram offloading. Blog with all details and more benchmarks:unsloth.aiunsloth.ai GGUF:huggingface.cohuggingface.co We also introduced lots of new things to Unsloth Desktop including Browser Use, auto compaction, a whole new TTS pipeline (with voice cloning, separation etc), sandboxing, and background customization and more:unslothai/unsloth Hope you all have a Happy Halloween! Also table of numbers below which is also described with more plots and data in our blog: |Quant|Size GB|AIME25|HMMT Feb26|GPQA-D|MMLU-Pro|IFEval|LCB v6 (32K)|Avg of 6|Aider pass@1| |:-|:-|:-|:-|:-|:-|:-|:-|:-|:-| |UD-Q8_K_XL|508.4|97.08|68.94|90.47|88.61|95.10|85.90|87.68|-| |UD-Q2_K_XL|334.8|97.50|69.89|89.58|88.28|94.92|84.76|87.49|-| |UD-IQ2_XXS|300.5|96.88|67.61|88.95|87.53|95.19|83.62|86.63|44.38| |UD-IQ1_M|250.4|95.00|62.88|88.38|87.53|95.10|84.38|85.55|42.13| |UD-IQ1_S|233.0|92.92|57.01|87.63|86.62|94.92|82.10|83.53|36.52| |GSQ-RCO IQ2_XS|265.8|95.00|64.39|88.64|86.62|94.27|84.95|85.64|41.57| |GSQ-RCO IQ2_XXS|248.8|95.21|57.95|85.80|86.53|93.72|82.86|83.68|36.52| |GSQ-RCO IQ1_M|231.3|91.88|53.60|85.10|85.49|91.77|82.86|81.78|29.78|

How do you fine-tune with Unsloth on a cloud GPU? Pick the smallest GPU that fits your model in Unsloth's VRAM table, save the trainer's checkpoints to a directory that survives a restart, and run your script as a batch job with a spending cap. Nodus runs that script on an RTX 4090 listed at $0.29/hr or an H100 at $2.60/hr, and restores the checkpoint directory on a new machine if the GPU is reclaimed. A preemption then costs you minutes of training, not the whole run. We build Nodus, so weigh that. Unsloth facts below link to Unsloth's own docs, other platforms' prices link to their pricing pages, and every cost figure shows its math. What does Unsloth change about fine-tuning? Unsloth makes the same fine-tune need a much smaller GPU. Its GitHub README says it trains LLMs 2x faster with 70% less VRAM and no accuracy loss. The core package is Apache 2.0, so commercial use is fine. The memory saving matters more than the speed for your bill. By Unsloth's own table, a 14B model needs 33 GB for 16-bit LoRA but 8.5 GB with 4-bit QLoRA. That moves it from a 48 GB L40S, listed at $0.793/hr on the Nodus pricing page, to a 24 GB RTX 4090 at $0.29/hr. Unsloth plugs into Hugging Face TRL's SFTTrainer. That detail is what makes cloud runs easy to recover, because the Trainer already knows how to save and resume. Which GPU do you need for your model size? Use Unsloth's own VRAM requirements table and add headroom. Unsloth calls these numbers "the absolute minimum", and longer sequences or bigger batches need more. | Model size | QLoRA (4-bit) minimum | LoRA (16-bit) minimum | A GPU that fits QLoRA with headroom | |---|---|---|---| | 3B | 3.5 GB | 8 GB | NVIDIA L4 or RTX 4090 (24 GB) | | 8B | 6 GB | 22 GB | RTX 4090 or L4 (24 GB) | | 14B | 8.5 GB | 33 GB | RTX 4090 or L4 (24 GB) | | 32B | 26 GB | 76 GB | L40S or RTX A6000 (48 GB) | | 70B | 41 GB | 164 GB | L40S (48 GB) for short sequences, A100 80 GB or H100 for room | !Unsloth minimum VRAM by model size: QLoRA needs 3.5 GB for 3B, 6 GB for 8B, 8.5 GB for 14B, 26 GB for 32B and 41 GB for 70B; 16-bit LoRA needs 8, 22, 33, 76 and 164 GB. QLoRA keeps every model up to 70B on a single GPU; 16-bit LoRA needs an 80 GB card at 32B and several cards at 70B. Our rule: start with QLoRA on a 24 GB card for anything up to 14B. Switch to 16-bit LoRA only when you have measured a quality gap that matters to you. On Nodus you can also skip naming a GPU and set resources.gpu.minMemory in the Job spec, and Nodus picks any accelerator with at least that much memory. Is a bigger GPU ever cheaper per finished run? Yes, but only if it finishes proportionally faster. Divide the bigger card's hourly price by the smaller card's to get the speedup it must deliver just to break even. Here is that ratio against an RTX 4090 at Nodus list prices: | GPU | Listed on the Nodus pricing page | Speedup needed vs RTX 4090 | |---|---|---| | RTX 4090 (24 GB) | $0.29/hr | 1.0x | | L4 (24 GB) | $0.43/hr | 1.5x | | L40S (48 GB) | $0.793/hr | 2.7x | | A100 80 GB | $1.09/hr | 3.8x | | H100 SXM 80 GB | $2.60/hr | 9.0x | Prices checked October 28, 2026 on the Nodus pricing page. !Speedup a GPU needs over an RTX 4090 to cost the same per finished run: L4 1.5x, L40S 2.7x, A100 80 GB 3.8x, H100 9.0x. An H100 has to train 9x faster than an RTX 4090 before it saves money on the same job. We have not benchmarked Unsloth across these cards, so measure it yourself. Run 50 steps on two GPUs, compare seconds per step, and multiply by the hourly price. Unless the H100 comes close to 9x faster on your job, the cheaper card finishes it for less. Pay for the H100 when you need the result sooner or the model does not fit anywhere smaller. How do you make an Unsloth script resume after a restart? Point the Trainer's output_dir at the Nodus state directory and resume from the newest checkpoint on start. The script below is Unsloth's Llama 3.1 8B Alpaca notebook with those two changes, plus a callback that saves when Nodus asks. from unsloth import FastLanguageModel # import first so its patches apply import os from datasets import load_dataset from transformers.trainer_utils import get_last_checkpoint from trl import SFTConfig, SFTTrainer import nodus STATE = os.environ.get("NODUS_STATE_DIR", "/nodus/state") OUTPUTS = os.environ.get("NODUS_OUTPUTS_DIR", "/nodus/outputs") model, tokenizer = FastLanguageModel.from_pretrained( model_name="unsloth/Llama-3.1-8B", max_seq_length=2048, load_in_4bit=True) model = FastLanguageModel.get_peft_model( model, r=16, lora_alpha=16, lora_dropout=0, bias="none", target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"], use_gradient_checkpointing="unsloth", random_state=3407) prompt = "### Instruction:\n{}\n\n### Input:\n{}\n\n### Response:\n{}" def fmt(ex): return {"text": [prompt.format(i, x, o) + tokenizer.eos_token for i, x, o in zip(ex["instruction"], ex["input"], ex["output"])]} dataset = load_dataset("unsloth/alpaca-cleaned", split="train").map(fmt, batched=True) trainer = SFTTrainer( model=model, tokenizer=tokenizer, train_dataset=dataset, dataset_text_field="text", max_seq_length=2048, callbacks=[nodus.checkpoint.HFTrainerCallback()], args=SFTConfig( output_dir=STATE, save_steps=200, save_total_limit=2, per_device_train_batch_size=2, gradient_accumulation_steps=4, num_train_epochs=1, learning_rate=2e-4, warmup_steps=5, optim="adamw_8bit", logging_steps=10, seed=3407, report_to="none")) last = get_last_checkpoint(STATE) if os.path.isdir(STATE) else None trainer.train(resume_from_checkpoint=last) model.save_pretrained(f"{OUTPUTS}/lora-adapter") tokenizer.save_pretrained(f"{OUTPUTS}/lora-adapter") Three lines do the recovery work. output_dir=STATE puts Trainer's checkpoints where Nodus snapshots them. get_last_checkpoint finds the newest checkpoint-N folder after a restore, and stays None on a first start. HFTrainerCallback comes from the Nodus SDK and makes Trainer save at the end of the current step when Nodus asks, including when the capacity gives notice that it is about to be reclaimed. Outside Nodus the SDK calls do nothing, so the same file runs on your laptop or in Colab. save_total_limit=2 keeps the directory small. With QLoRA each checkpoint holds the adapter and optimizer state, not a copy of the 8B base model. How do you launch it as a Nodus Job? Install the CLI, sign in, and run from the folder that holds train.py. These flags come from the Nodus checkpoints guide and the nodus run reference: $ pip install nodus-compute $ nodus login $ nodus run --name unsloth-llama8b --gpu L4 --image nodus/pytorch \ --checkpoint /nodus/state --interruptible \ --max-cost 10 --timeout 24h -d \ -- sh -c "pip install unsloth nodus-compute && python train.py" $ nodus logs -f job/unsloth-llama8b $ nodus cp job/unsloth-llama8b:outputs/lora-adapter/adapter_model.safetensors . An L4 fits this 8B QLoRA run with room to spare; nodus get gpus lists the other types and their current rates, including the RTX 4090. --interruptible lets Nodus use interruptible capacity, which is exactly the case the checkpoint setup covers. Add --dry-run first to see the cost estimate without starting anything. --max-cost 10 is a cap, not a quote: at the cap the Job saves its state and suspends until you raise it. Everything under /nodus/outputs is collected when the Job succeeds, so copy adapter_config.json the same way. Installing Unsloth at start adds install time to every attempt, including restored ones. For repeat runs, pin the versions, push your own image to a registry and pass it to --image. What does a preemption cost with and without checkpoints? Say a 10-hour QLoRA run on one H100 is reclaimed at hour 6. At $2.60/hr the whole run costs $26.00. Without checkpoints you start over and pay the first 6 hours twice, an extra $15.60. With save_steps set so Trainer saves about every 30 minutes, the most you redo is 30 minutes, or $1.30. The Nodus callback usually shrinks that further, because Nodus asks for a save when the capacity gives notice. The checkpointing comparison covers which other platforms restore a run like this for you. What does the same run cost on other platforms? At list prices, 10 hours on one H100 SXM is cheapest on Nodus, with RunPod's Community Cloud close behind: | Platform | H100 SXM hourly rate | 10 hours | |---|---|---| | Nodus | $2.60/hr | $26.00 | | RunPod Community Cloud | $2.69/hr | $26.90 | | Modal | $3.949/hr | $39.49 | | RunPod Secure Cloud | $3.99/hr | $39.90 | Prices checked October 28, 2026: Nodus pricing page, RunPod pricing, Modal pricing. !Cost of 10 hours on one H100 SXM: Nodus $26.00, RunPod Community Cloud $26.90, Modal $39.49, RunPod Secure Cloud $39.90. The rate gap is small next to the GPU choice: the same 10 hours on an RTX 4090 at $0.29/hr is $2.90. The hourly rate is only part of it. A RunPod Pod gives you a machine where you run and restart the script yourself. Modal restarts a preempted function on the same input and asks you to add your own checkpoints. If you already work in either, Unsloth runs there too, and the script above works there apart from the state path and the Nodus callback. When is Colab or your own GPU enough? For learning Unsloth and short experiments, start in Google Colab. Unsloth's README links a free Colab notebook for each of its featured models. The catch is that Colab's FAQ says free resources are not guaranteed, idle VMs are deleted, and lifetimes and GPU types vary over time. A 24 GB card you already own is enough for QLoRA up to 14B by Unsloth's table. It costs nothing per hour and your data never leaves the machine. Move to a cloud job when the run has to finish overnight without you watching it, when the model outgrows your card, or when you want several runs at once. If you would rather not maintain a training script, the managed nodus/sft runtime in Nodus Training (Beta) does full, LoRA or QLoRA fine-tuning from a JSONL file. It is not Unsloth, so you give up Unsloth's kernels for less code. FAQ Does Unsloth work on multiple GPUs? Unsloth's README says it supports multi-GPU setups. For most QLoRA jobs up to 70B, one GPU is enough by its own VRAM table, and a single card avoids the communication overhead entirely. Do I need a Hugging Face token? Only for gated models. Unsloth publishes copies of many open models under its Hugging Face organization. For a gated one, store your token as a Nodus Secret and pass it with --secret. Can I use Unsloth's Docker image instead of pip? Unsloth publishes an official Docker image, unsloth/unsloth, built mainly for running Jupyter and Unsloth Studio locally. Nodus runs prebuilt public registry images, but test the image's startup command before a long run. How do I keep a large dataset from uploading on every run? Put it in a Nodus Volume once with nodus volume put, then mount it with --volume NAME:/data:ro. Only the code folder uploads with each run. What do I do with the adapter afterwards? Load it back with FastLanguageModel.from_pretrained pointed at the adapter folder, or merge it into 16-bit weights with Unsloth's save_pretrained_merged. The VRAM guide covers what serving the merged model needs. Originally published on the Nodus blog.

Your local LLM can now take control of a browser and complete tasks autonomously via Unsloth! Qwen3.8-27B successfully controlled the browser to complete tasks like chess, painting, with no human intervention. GitHub: unslothai/unsloth Guide: unsloth.ai

Local models can now perform Browser Use in Unsloth! Hey guys, your local LLM can now take control of a browser & complete tasks autonomously via Unsloth. Yes, its full browser use so your model can control your browser to search, click things and do stuff. Qwen3.8-27B successfully completed a wide range of tasks, including chess, painting, pool and more with no human intervention. We improved the usability and workflow of the browser use since a week ago. Try Browser Use now via Unsloth Desktop on Windows, Mac and Linux! Also new Halloween release today so be sure to update Unsloth! Btw for those unaware, yes browser use for local LLMs was always possible, however what makes Unsloth's extra special is that it particularly works well for small or medium sized local models like Qwen3.8-27B or even smaller like Gemma 4 E4B where otherwise, using something else would cause it to loop, cancel abruptly etc. This implementation is integrated with all of Unsloth's self healing toolcalls, looping prevention etc GitHub: unslothai/unsloth Guide: unsloth.ai

Your local LLM can now take control of a browser and complete tasks autonomously via Unsloth! Qwen3.8-27B successfully controlled the browser to complete tasks like chess, painting, with no human intervention. GitHub: unslothai/unsloth Guide: unsloth.ai t.co

Who gets to build the next internet? | Open Source AI Awards Who gets to build the next internet? A field film from the Open Source AI Awards at Founders Inc., Fort Mason, October 23, 2026. Article, photographs and links to the projects: chloestarr.me A room full of builders, a flying shirt, and a question that lasts beyond the prizes: who gets to participate when agents start choosing what people see? Featuring the work discussed by Matt White, Ian Ballantyne, the Google AI Edge team, Daniel Han, Alex Newman, Peter Walker and Irene Solaiman. Alex's advice about editing with sound waveforms travelled from an earlier event at Pebblebed into making this film. PROJECTS TO TRY Alex Newman / Claude-Mem: thedotmack/claude-mem Matt White / Model Openness Framework: isitopen.ai Gemma: deepmind.google Google AI Edge: ai.google.dev Daniel Han / Unsloth: unslothai/unsloth Peter Walker / OpenRouter: openrouter.ai Irene Solaiman: irenesolaiman.com Hugging Face investigation: huggingface.co CREDITS AND SOURCE CONTEXT A film by Chloe Starr. Direction, narration and edit decisions: Chloe Starr. Original event recordings supplied by Chloe Starr, with Chris Starr AI: x.com Local assembly: Codex, Python and FFmpeg. Waveform-editing advice: Alex Newman, conversation at Pebblebed's October 17 gathering. The flashback is from that earlier visit, not footage of the advice exchange. Alex and Claude-Mem did not generate or render this film. Music: Sweet You by Loyalty Freak Music, from MELODIES WITH A BEAT. CC0. Excerpt edited with scene shaping and dialogue ducking. Published work appears separately from original event footage: Claude-Mem repository animation: thedotmack/claude-mem Google Developers application demo: developers.googleblog.com OpenRouter chart and analysis: openrouter.ai The agentic token-use comparison is specifically on OpenRouter, not a measurement of all internet traffic. Hugging Face security team's published investigation and replay: huggingface.co The wider campaign lasted 4.5 days; the replay is a published reconstruction, not live awards-night footage. The investigation was team work. Personalities and attendee photographs, editorial excerpts: Ian Ballantyne's October 6 cat-meow search post: x.com Daniel Han's October 21 sloth-game post: x.com Irene Solaiman's March 17 community post and illustration: x.com Jason / @jasonjias, event photograph: x.com Jason's gift-card photograph and prize post: x.com Organizer's Best OSAI Week Post contest: x.com Jason reports winning a $100 Apple gift card; the exact contest-winning photo and prize category are not confirmed. Official program, hosts and supporters: opensourceaiweek.com The Google AI Edge guest is credited by team because the individual identity has not been confirmed. Alex's award is based on Chloe's account; its category is not asserted. More films and writing: chloestarr.me #OpenSourceAIWeek #OpenIntelligence

Unsloth + Ollama: Fine-Tune and Run Local LLMs Unsloth + Ollama: Fine-Tune Open-Source LLMs 2x Faster and Run Them Locally Stop burning thousands of dollars on cloud LLM API tokens. Fine-tuning an 8B parameter model no longer requires a cluster of enterprise A100 GPUs or endless cloud compute budgets. By pairing Unsloth with Ollama, you can perform QLoRA local fine tuning on a consumer GPU using under 8GB of VRAM, convert the model directly to GGUF format, and serve it on your local machine with a single CLI command. Here is how to set up the entire end-to-end pipeline in under 30 minutes. --- Why Unsloth + Ollama? Standard Hugging Face pipelines often encounter CUDA Out-Of-Memory (OOM) crashes on 16GB or 24GB GPUs when training 8B parameter models. Unsloth rewrites PyTorch attention layers and cross-entropy loss functions into hand-crafted OpenAI Triton kernels. This optimization yields concrete advantages: - 2x Faster Training: Bypasses PyTorch autograd overhead. - 70% to 80% Less VRAM: Train Llama 3 8B or Mistral 7B in 6.6GB to 8GB VRAM with QLoRA. - Zero Conversion Overhead: Exports directly to quantized GGUF format and auto-generates the Ollama Modelfile. - Zero API Bills: Serve your proprietary tuned model locally on macOS, Linux, or Windows via Ollama. --- Where to Get It (Official Links, GitHub & Hugging Face) Before starting, bookmark and pull resources from the official repositories: - Unsloth GitHub Repository: unslothai/unsloth — Core training framework and Triton kernels. - Unsloth Official Documentation: unsloth.ai — Setup guides, memory benchmarks, and architecture details. - Unsloth Hugging Face Hub: huggingface.co — Official 4-bit pre-quantized base models (Llama 3, Mistral, Qwen, Gemma). - Ollama Download: ollama.com — Standalone installers for macOS, Linux, and Windows. - Ollama GitHub Repository: ollama/ollama — Open-source local runtime repository. --- Quick Install & Setup Set up Ollama and an isolated Python virtual environment on your Linux/WSL2 system with CUDA installed. 1. Install Ollama Runtime curl -fsSL https://ollama.com/install.sh | sh 2. Configure Python Virtual Environment Create a dedicated environment and install Unsloth alongside PyTorch dependencies: python3 -m venv finetune-env source finetune-env/bin/activate pip install --upgrade pip pip install 'unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git' pip install --no-deps trl peft accelerate bitsandbytes datasets --- How to Use It in Real Life (Step-by-Step Walkthrough) In this walkthrough, we will fine-tune Llama 3 8B Instruct on a customer support dataset, export it to GGUF, and query it locally via Ollama. Step 1: Initialize Model and Adapters Create a training script named train.py. Load the pre-quantized 4-bit model directly from Hugging Face and inject LoRA parameters: import torch from unsloth import FastLanguageModel from datasets import load_dataset from trl import SFTTrainer from transformers import TrainingArguments max_seq_length = 2048 dtype = None # Auto-detect Float16 / Bfloat16 load_in_4bit = True # 1. Load 4-bit base model model, tokenizer = FastLanguageModel.from_pretrained( model_name = 'unsloth/llama-3-8b-Instruct-bnb-4bit', max_seq_length = max_seq_length, dtype = dtype, load_in_4bit = load_in_4bit, ) # 2. Attach LoRA adapters model = FastLanguageModel.get_peft_model( model, r = 16, target_modules = ['q_proj', 'k_proj', 'v_proj', 'o_proj', 'gate_proj', 'up_proj', 'down_proj'], lora_alpha = 16, lora_dropout = 0, bias = 'none', use_gradient_checkpointing = 'unsloth', random_state = 3407, ) Step 2: Format Data and Train Train on instruction-response pairs using SFTTrainer for 60 steps: # 3. Format dataset using Llama 3 template prompt_template = '''<|start_header_id|>system<|end_header_id|> You are a specialized internal support bot.<|eot_id|><|start_header_id|>user<|end_header_id|> {}<|eot_id|><|start_header_id|>assistant<|end_header_id|> {}<|eot_id|>''' def formatting_prompts_func(examples): texts = [] for instruction, output in zip(examples['instruction'], examples['output']): text = prompt_template.format(instruction, output) texts.append(text) return {'text': texts} dataset = load_dataset('philschmid/dolly-15k-oai-style', split='train[:1000]') dataset = dataset.map(formatting_prompts_func, batched=True) trainer = SFTTrainer( model = model, tokenizer = tokenizer, train_dataset = dataset, dataset_text_field = 'text', max_seq_length = max_seq_length, dataset_num_proc = 2, packing = False, args = TrainingArguments( per_device_train_batch_size = 2, gradient_accumulation_steps = 4, warmup_steps = 5, max_steps = 60, learning_rate = 2e-4, fp16 = not torch.cuda.is_bf16_supported(), bf16 = torch.cuda.is_bf16_supported(), logging_steps = 10, optim = 'adamw_8bit', weight_decay = 0.01, lr_scheduler_type = 'linear', seed = 3407, output_dir = 'outputs', ), ) trainer.train() Step 3: Export to GGUF and Ollama Unsloth eliminates manual llama.cpp compilation. Export directly to 4-bit Medium Quantization (q4_k_m) at the end of train.py: # 4. Export directly to GGUF model.save_pretrained_gguf('custom_model', tokenizer, quantization_method = 'q4_k_m') Run the training file: python train.py This writes the quantized GGUF weights and an automatically configured Modelfile into the ./custom_model directory. Step 4: Register and Run in Ollama Point Ollama to the generated folder and launch the model: # 5. Build and serve inside Ollama ollama create my-assistant -f ./custom_model/Modelfile ollama run my-assistant Example Interaction: >>> How do I reset my company VPN token? To reset your company VPN token: 1. Open the Okta Verify dashboard on your mobile device. 2. Navigate to Settings > Enrolled Factors > VPN Gateway. 3. Click 'Regenerate Token' and confirm your biometric prompt. Your new session token will be active within 60 seconds. --- Key Takeaways Combining Unsloth and Ollama creates a self-contained local fine-tuning workflow: train locally in 15–30 minutes, eliminate external API dependencies, and keep all proprietary data within your own infrastructure.

8. Unsloth + Ollama = fine-tune + self-host open LLMs on consumer GPUs. Cut VRAM 70-80%. Train 2x faster. Kill cloud API costs. Start: unslothai/unsloth | ollama.com #AI #LLMs #OpenSource #LocalAI

3. Step 1—Install Ollama: curl -fsSL ollama.com | sh Step 2—Python env: python3 -m venv finetune && source finetune/bin/activate Step 3—Install Unsloth: pip install 'unsloth[colab-new] @ git+unslothai/unsloth'

LLM Quantization Hands-On Guide: Four Routes from Zero to Production Introduction: What Problem Does This Article Solve? If you've read my other article Quantization Precision Guide, you already know how much "intelligence" quantization costs (spoiler: 8-bit is nearly free, 4-bit is the sweet spot). But there's a huge gap between "knowing you should quantize" and "actually doing it": - AutoGPTQ's GitHub page screams "🚨 Unmaintained"—what do you use instead? - How do you prepare calibration datasets for AWQ and GPTQ? Are 128 samples enough? - What exactly does vLLM's --quantization fp8 do behind the scenes? Do you need pre-calibration? - How do you verify the model hasn't gone stupid after quantization? This article fills that gap. Four practical routes, each with complete code you can copy and run. What do you want to do? │ ├─ Route 1: I just want to pull a model and play (Zero barrier) │ → Download a community pre-quantized model, run it with Ollama / LM Studio │ ├─ Route 2: I want to quantize a model myself (Geek essential) │ → Use AWQ / GPTQ / GGUF tools for hands-on quantization │ ├─ Route 3: I need high-throughput inference serving (Production grade) │ → vLLM + FP8 full quantization, squeeze every drop from H100 │ └─ Route 4: I need to fine-tune a large model on limited VRAM (QLoRA) → BitsAndBytes NF4 quantized base + LoRA adapter, fine-tune 32B on 24GB --- Route 1: Using Community Pre-Quantized Models (Zero Barrier) For 90% of individual users, you don't need to quantize anything yourself. The community has already done it for you. Finding Reliable Quantized Models on HuggingFace Go to HuggingFace Models, search for your desired model (e.g., Qwen3-32B), then filter by quantization format in the left sidebar. Trusted quantization contributors—these names mean quality: | Contributor | Specialty | Notes | |------------|-----------|-------| | Unsloth | GGUF, AWQ | Excellent quality with complete benchmark comparisons | | bartowski | GGUF | Covers almost every popular model, fastest turnaround | | ModelCloud | GPTQ | Official GPTQModel team | | neuralmagic | FP8, INT8 | Neural Magic official, focused on server-side quantization | Decoding Naming Conventions Secrets hidden in model names: Qwen3-32B-AWQ → AWQ format, default 4-bit weight quantization Qwen3-32B-GPTQ-Int4 → GPTQ format, 4-bit integer weight quantization Qwen3-32B-GGUF → GGUF format (multiple precision versions inside) Qwen3-32B-FP8 → FP8 full quantization (weights + activations) Secrets in GGUF filenames: Q2_K → 2-bit (extreme compression, noticeable quality loss) Q3_K_S → 3-bit small (space-optimized) Q4_K_M → 4-bit medium (⭐ most recommended balance) Q5_K_M → 5-bit medium (higher precision) Q6_K → 6-bit (near lossless) Q8_0 → 8-bit (virtually lossless, but larger) Quick Start Option A: Ollama (Simplest) # Install Ollama curl -fsSL https://ollama.com/install.sh | sh # Pull and run a quantized model in one command ollama run qwen3:32b # Want a specific precision? Search for tags ollama run qwen3:32b-q4_K_M Option B: LM Studio (GUI) Download LM Studio, search for your model, pick a quantization level your VRAM can handle, download → load → chat. Option C: llama.cpp (CLI geek) # Download GGUF from HuggingFace (e.g., Qwen3-8B-Q4_K_M for lightweight edge deployment) huggingface-cli download bartowski/Qwen3-8B-GGUF \ --include "Qwen3-8B-Q4_K_M.gguf" \ --local-dir ./models # Run local CLI inference with Qwen3-8B-Q4_K_M.gguf ./llama-cli \ -m ./models/Qwen3-8B-Q4_K_M.gguf \ -p "System: You are an expert AI assistant.\nUser: Explain 4-bit quantization in two sentences.\nAssistant:" \ -n 256 -ngl 99 # Or launch high-performance OpenAI-compatible API server (port 8080) llama-server \ -m ./models/Qwen3-8B-Q4_K_M.gguf \ --port 8080 \ -ngl 99 # offload all layers to GPU VRAM > Route 1 summary: If you're a personal user, downloading GGUF Q4_K_M and running Ollama is the optimal solution. --- Route 2: Quantizing Weights Yourself (Geek Essential) You need to do this yourself when: - You've fine-tuned a private model with no community-quantized version - You need domain-specific calibration for maximum accuracy - You want to fully understand what quantization actually does 2.1 AWQ Quantization (Recommended First Choice) AWQ (Activation-Aware Weight Quantization) analyzes activation statistics to protect the most critical 1% of weights—currently the best precision-preserving weight-only quantization method. pip install autoawq from awq import AutoAWQForCausalLM from transformers import AutoTokenizer from datasets import load_dataset # ====== Configuration ====== model_path = "Qwen/Qwen3-8B" quant_path = "./Qwen3-8B-AWQ" quant_config = { "zero_point": True, # Enable zero-point quantization (higher precision) "q_group_size": 128, # 128 weights share one scaling factor "w_bit": 4, # Quantize to 4-bit "version": "GEMM", # GEMM kernel (best compatibility) } # ====== Load model and tokenizer ====== model = AutoAWQForCausalLM.from_pretrained(model_path) tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True) # ====== Prepare calibration data ====== # AWQ is very efficient—128-256 samples are enough! def get_calibration_data(): data = load_dataset( "databricks/databricks-dolly-15k", split="train[:128]" ) return [ f"{x['instruction']}\n{x['context']}\n{x['response']}" for x in data ] calib_data = get_calibration_data() # ====== Quantize ====== model.quantize( tokenizer=tokenizer, quant_config=quant_config, calib_data=calib_data, ) # ====== Save ====== model.save_quantized(quant_path) tokenizer.save_pretrained(quant_path) print(f"✅ AWQ quantization complete, saved to {quant_path}") Calibration data tips: - Quantity: 128-256 samples are sufficient. AWQ only needs activation distribution statistics - Representativeness: Calibration data should match your actual use case. Chinese model → Chinese data - OOM fallback: Add max_calib_seq_len=512 to model.quantize() if you run out of memory 2.2 GPTQ Quantization (GPTQModel) > ⚠️ Important: AutoGPTQ is unmaintained! Since 2025, the official recommendation is GPTQModel (maintained by ModelCloud), a drop-in replacement. pip install -U gptqmodel --no-build-isolation from datasets import load_dataset from gptqmodel import GPTQModel, QuantizeConfig # ====== Configuration ====== model_id = "Qwen/Qwen3-8B" quant_path = "./Qwen3-8B-GPTQ-Int4" # ====== Calibration data ====== # GPTQ typically needs more data (512-1024) for Hessian matrix computation calibration_dataset = load_dataset( "allenai/c4", data_files="en/c4-train.00001-of-01024.json.gz", split="train" ).select(range(1024))["text"] # ====== Quantization config ====== quant_config = QuantizeConfig( bits=4, group_size=128, ) # ====== Load and quantize ====== model = GPTQModel.load(model_id, quant_config) model.quantize(calibration_dataset, batch_size=2) # ====== Save ====== model.save(quant_path) print(f"✅ GPTQ quantization complete, saved to {quant_path}") AWQ vs GPTQ: Practical Differences | Dimension | AWQ | GPTQ | |-----------|-----|------| | Calibration data | 128-256 samples | 512-1024 for best results | | Quantization speed | Faster (only analyzes activation distributions) | Slower (per-layer Hessian optimization) | | Precision retention | Best (protects salient weights) | Excellent | | Inference speed | Fast | Faster (Marlin kernel acceleration) | | Best for | Precision-critical scenarios | Server-side throughput | | vLLM support | ✅ | ✅ | One-liner: Pick AWQ for precision, GPTQ for serving speed. 2.3 GGUF Quantization (llama.cpp) GGUF is the cross-platform king—Mac, Windows, Linux, CPU, GPU, everything works. Step 1: Setup git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp # Build with CUDA for faster conversion mkdir build && cd build cmake .. -DGGML_CUDA=ON cmake --build . --config Release -j$(nproc) # Install Python dependencies for conversion pip install -r ../requirements.txt Step 2: Convert HuggingFace model to GGUF (FP16 baseline) python convert_hf_to_gguf.py ../models/Qwen3-8B/ \ --outtype f16 \ --outfile ../models/Qwen3-8B-f16.gguf Step 3: Quantize # Quantize to Q4_K_M (recommended sweet spot) ./bin/llama-quantize \ ../models/Qwen3-8B-f16.gguf \ ../models/Qwen3-8B-Q4_K_M.gguf \ Q4_K_M Advanced: Using imatrix for better low-bit quality When doing Q3 or Q2 extreme compression, imatrix (importance matrix) is a lifesaver: # Generate importance matrix ./bin/llama-imatrix \ -m ../models/Qwen3-8B-f16.gguf \ -f ../calibration_data.txt \ -o ../models/imatrix.dat # Quantize with imatrix guidance ./bin/llama-quantize \ --imatrix ../models/imatrix.dat \ ../models/Qwen3-8B-f16.gguf \ ../models/Qwen3-8B-Q3_K_M-imat.gguf \ Q3_K_M > Key tip: imatrix has minimal impact on Q4_K_M and above, but provides perceptible improvements for Q3 and Q2 levels. --- Route 3: Server-Side Full Quantization (Production FP8) If you're running high-concurrency inference on H100/H800 in a data center, stop worrying about GPTQ/AWQ. FP8 full quantization is the only way. 3.1 vLLM FP8 Zero-Calibration (Fastest Setup) vllm serve Qwen/Qwen3-32B-Instruct \ --quantization fp8 \ --dtype auto \ --gpu-memory-utilization 0.90 \ --enable-prefix-caching What happens behind the scenes: 1. vLLM loads BF16 original weights 2. Dynamically computes min/max range for each tensor on first inference 3. Converts weights and activations to FP8 E4M3 in real-time 4. Leverages H100's native FP8 Tensor Cores for acceleration Advantage: Zero calibration, zero data, zero wait. Disadvantage: Minor runtime overhead from dynamic scaling factor computation. 3.2 Offline Static Calibration (Maximum Performance) For ultimate throughput, use llm-compressor to pre-compute static scaling factors: pip install llmcompressor from llmcompressor.modifiers.quantization import QuantizationModifier from llmcompressor import oneshot from datasets import load_dataset from transformers import AutoTokenizer model_id = "Qwen/Qwen3-32B-Instruct" output_dir = "./Qwen3-32B-FP8-Static" tokenizer = AutoTokenizer.from_pretrained(model_id) ds = load_dataset("HuggingFaceFW/fineweb-edu", split="train", streaming=True) calibration_data = [tokenizer.apply_chat_template( [{"role": "user", "content": x["text"][:2000]}], tokenize=False ) for x in list(ds.take(512))] recipe = QuantizationModifier( targets="Linear", scheme="FP8", ignore=["lm_head"], ) oneshot( model=model_id, dataset=calibration_data, recipe=recipe, output_dir=output_dir, max_seq_length=4096, num_calibration_samples=512, ) Then serve with vLLM: vllm serve ./Qwen3-32B-FP8-Static \ --dtype auto \ --gpu-memory-utilization 0.90 # No --quantization fp8 needed—vLLM auto-detects FP8 config in the checkpoint 3.3 FP8 Mode Comparison | Mode | Calibration | Runtime Overhead | Accuracy | Best For | |------|------------|-----------------|----------|----------| | Online Dynamic (--quantization fp8) | None | Highest | Good | Quick testing | | Offline Dynamic Activations | None | Moderate | Good | Memory savings | | Offline Static (llm-compressor) | 512 samples | Lowest | Best | Production | 3.4 KV Cache Quantization—The Hidden Accelerator vllm serve Qwen/Qwen3-32B-Instruct \ --quantization fp8 \ --kv-cache-dtype fp8_e5m2 \ --enable-prefix-caching Effect: KV Cache memory usage halved. Particularly impactful for long-context scenarios (8K-128K), enabling more concurrent requests on the same GPU. --- Route 4: Quantization for Fine-Tuning (QLoRA) When you want to fine-tune a 32B model on a consumer GPU (e.g., RTX 4090 with 24GB VRAM), QLoRA is the only viable option. Core Concept QLoRA's genius: 1. Quantize base model with NF4 → 32B shrinks from 64GB to ~16GB 2. Freeze quantized base weights (don't train!) 3. Attach tiny LoRA adapters (typically just tens of MB) 4. Only train the LoRA adapters 5. During training, quantized weights are temporarily dequantized to BF16 Complete Code from transformers import ( AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig, TrainingArguments, ) from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training from trl import SFTTrainer from datasets import load_dataset # ====== 4-bit quantization config ====== bnb_config = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_quant_type="nf4", # NF4 (information-theoretically optimal) bnb_4bit_compute_dtype="bfloat16", # Dequantize to BF16 for computation bnb_4bit_use_double_quant=True, # Double quantization (saves ~0.4 bit/param) ) # ====== Load quantized base model ====== model = AutoModelForCausalLM.from_pretrained( "Qwen/Qwen3-8B", quantization_config=bnb_config, device_map="auto", ) tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-8B") # ====== Prepare for training ====== model = prepare_model_for_kbit_training(model) # ====== LoRA config ====== lora_config = LoraConfig( r=16, lora_alpha=32, target_modules=[ "q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj", ], lora_dropout=0.05, bias="none", task_type="CAUSAL_LM", ) model = get_peft_model(model, lora_config) model.print_trainable_parameters() # → trainable params: 13,631,488 || all params: 8,043,235,328 # → trainable%: 0.1695% ← Only 0.17% of parameters are trained! # ====== Load training data ====== dataset = load_dataset("your-dataset-here", split="train") # ====== Train ====== trainer = SFTTrainer( model=model, train_dataset=dataset, tokenizer=tokenizer, args=TrainingArguments( output_dir="./qlora-output", per_device_train_batch_size=4, gradient_accumulation_steps=4, num_train_epochs=3, learning_rate=2e-4, bf16=True, logging_steps=10, save_strategy="epoch", ), ) trainer.train() > Performance tip: Unsloth can boost QLoRA training speed by 2-5x with 60% less memory through Triton kernel optimizations. --- Post-Quantization Quality Validation Deploying without validation is gambling. Three levels of verification: Level 1: Quick Perplexity Test (30 seconds) ./bin/llama-perplexity \ -m ./models/Qwen3-8B-Q4_K_M.gguf \ -f ./test_data.txt \ --chunks 32 | Quant Level | Typical PPL Degradation | Verdict | |-------------|------------------------|---------| | Q8_0 | < 0.5% | ✅ Near lossless | | Q6_K | < 1% | ✅ Very safe | | Q4_K_M | 1-3% | ✅ Sweet spot | | Q3_K_M | 3-8% | ⚠️ Monitor closely | | Q2_K | 8-20%+ | ❌ Experimental only | Level 2: Benchmark Scoring (lm-evaluation-harness) pip install lm-eval lm_eval --model hf \ --model_args pretrained=./Qwen3-8B-AWQ \ --tasks mmlu,hellaswag,arc_challenge \ --batch_size auto \ --output_path ./eval_results/ Level 3: Business Data Testing (Most Critical) High benchmark scores don't guarantee performance on your specific use case. Always test with 50-100 real business questions. > Rule of thumb: If your workload involves code generation or mathematical reasoning, anything below 4-bit may have perceptible degradation. Always validate with real data. --- Troubleshooting Guide Pitfall 1: Model collapses after quantization Causes: Mismatched calibration data, bit-width too low for small models (<7B), unsupported architecture. Pitfall 2: Memory usage didn't decrease Cause: Weight-only quantization (GPTQ/AWQ) still uses FP16 activations + KV Cache. Enable --kv-cache-dtype fp8_e5m2 and reduce --max-model-len. Pitfall 3: Inference isn't faster Cause: Weight-only quantization saves memory, not compute. For actual speedup, use FP8 full quantization with high concurrency. Pitfall 4: AutoGPTQ errors AutoGPTQ is unmaintained. Migrate to GPTQModel: pip uninstall auto-gptq pip install -U gptqmodel --no-build-isolation --- 2026 Quantization Tool Landscape | Tool | Type | Precision | Status | Best Paired With | |------|------|-----------|--------|-----------------| | AutoAWQ | Weight-only | INT4 | ✅ Active | vLLM / Transformers | | GPTQModel | Weight-only | INT4, INT8 | ✅ Active | vLLM / Transformers | | llama.cpp | Weight-only | Q2-Q8 | ✅ Very active | Ollama / LM Studio | | BitsAndBytes | Weight-only | NF4, INT8 | ✅ Active | QLoRA fine-tuning | | llm-compressor | Full quant | FP8, INT8 | ✅ Active | vLLM production | | torchao | Full quant | INT4, INT8, FP8 | ✅ Active | PyTorch native | | HQQ | Weight-only | INT2-INT8 | ✅ Active | Ultra-low-bit research | | AQLM | Weight-only | 2-bit | ⚡ Frontier | Extreme compression | | AutoGPTQ | Weight-only | INT4, INT8 | ❌ Deprecated | Replaced by GPTQModel | --- Final Takeaways 1. Download before you quantize: Community pre-quantized models are excellent unless you have special needs 2. AWQ for precision, GPTQ for speed, GGUF for universality: They're not interchangeable—pick by scenario 3. FP8 is the endgame for production: If you have H100s, --quantization fp8 is the optimal answer 4. Always validate after quantization: Perplexity takes 30 seconds, benchmarks take 30 minutes, business testing may take 3 days—but those 3 days prevent 3 months of nightmares --- Frequently Asked Questions (FAQ) Q1: How should I choose between AWQ, GPTQ, and GGUF in production? For Linux inference servers running high-throughput engines, choose AWQ (best perplexity retention by protecting salient weights) or GPTQ; for local workstations, Macs, or edge inference, choose GGUF paired with llama.cpp or Ollama. In modern cloud data centers with Hopper or Ada GPUs, native FP8 is the definitive choice. Check our vLLM Production Serving Guide for deployment recipes. Q2: Why did my 4-bit quantized model save memory but fail to run faster? Weight-only quantization (W4A16) dequantizes weights back to FP16/BF16 in SRAM/registers during matrix multiplication, so compute latency is not reduced. The speedup from weight-only quantization comes primarily from overcoming memory-bandwidth bottlenecks in single-batch or small-batch regimes. For multi-batch acceleration, use full FP8 or W8A8 quantization to exploit specialized Tensor Cores. Q3: How can I verify that my quantized model hasn't suffered accuracy collapse? Follow a three-stage validation pipeline: first, measure PPL (Perplexity) on a clean benchmark such as WikiText-2 (a degradation delta under 0.2 is standard); second, run generic benchmark evaluations (e.g. MMLU, GSM8k); third, execute automated regression tests on domain-specific prompts to prevent formatting regressions. See our LLM Evaluation Guide for test harnesses. Q4: How do I download and run Qwen3-8B-Q4_K_M.gguf locally? To download and run Qwen3-8B-Q4_K_M.gguf, use huggingface-cli download bartowski/Qwen3-8B-GGUF --include "Qwen3-8B-Q4_K_M.gguf" --local-dir ./models. You can then launch it with llama.cpp using ./llama-cli -m ./models/Qwen3-8B-Q4_K_M.gguf -p "Hello" -n 256 or import it directly into Ollama via a custom Modelfile (FROM ./models/Qwen3-8B-Q4_K_M.gguf). With an active VRAM footprint around 5.2GB, the Q4_K_M quantization level retains over 99% of dense FP16 perplexity while fitting comfortably on modern 8GB GPUs and Apple Silicon. --- Originally published at LLM Systems Architecture on blog.llmgo.top.