FreeToken: Run Frontier MoE Models on Your Own GPU, and the RAM It Takes
---
title: "FreeToken: Run Frontier MoE Models on Your Own GPU, and the RAM It Takes"
published: true
tags: opensource, localllm, mixtureofexperts, llminference
canonical_url: devpik.com
---
FreeToken runs frontier-scale Mixture-of-Experts models on personal hardware by keeping experts in system RAM and caching the busy ones on the GPU. Its paper reports a 284B model on a gaming desktop. Here is how it works, how much RAM that claim implies, and how to serve your first model.
What Is FreeToken?
FreeToken is an open-source serving engine for running frontier-scale Mixture-of-Experts (MoE) language models on personal and consumer hardware. Instead of assuming a datacenter GPU with room for every weight, it treats the GPU, the CPU, system memory and the links between them as one elastic inference platform, and it exposes OpenAI and Anthropic compatible APIs so coding agents can use a local model the way they use a cloud one.
It comes from FlashML-org, and the accompanying paper, "FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Execution," was posted to arXiv on August 17, 2026, with authors including Ion Stoica, Matei Zaharia, Song Han and Kurt Keutzer. The abstract reports more than 20 MoE models served on hardware ranging from an 8 GB laptop GPU to a single workstation GPU: a 35B model on a laptop, a 284B model on a gaming desktop, and the 753B GLM-5.2 on one workstation card.
The repository was created on July 20, 2026, is licensed under Apache 2.0, and had 15,782 stars and 1,571 forks when we checked on November 2, 2026. The project credits mini-sglang as its main inspiration and says it reuses designs and code from SGLang, vLLM, FlashInfer, flash-linear-attention, LightLLM and llama.cpp.
Why a Huge MoE Model Fits on One Consumer GPU
The trick is in how MoE models work. Each layer holds many small expert networks, but a router sends each token to only a few of them, so most weights sit idle on any given step. Qwen3.6-35B-A3B, the model FreeToken's quick start uses, has 35 billion parameters in total but only about 3 billion active per token, which is what the A3B in its name means.
FreeToken builds on that. In its default offload strategy the experts live in host RAM and the GPU keeps a least-recently-used cache of expert slots: a token whose expert is already cached runs at GPU speed, and a miss is streamed over PCIe. The cpu strategy computes misses on the CPU instead of fetching them, and hybrid splits each step's misses between the two, overlapped, using bandwidth figures you measure once per machine with ft bench bw. If the experts fit in VRAM, fused keeps them all resident. Left on auto, dense models resolve to fused and MoE models to offload, except on unified-memory machines such as DGX Spark, where copying between CPU and GPU memory would only move data between two names for the same RAM.
Around that core sit the features the README highlights: double-buffered prefill streaming, one global LRU pool of experts shared across layers, FreeToken's own FTW fast-load weight format, VRAM that can move between the expert cache and the KV cache at runtime without a restart, and semantic anchor checkpoints that let agent edits such as tool calls and thinking blocks reuse context instead of recomputing it.
The Real Requirement Is System RAM
Offloading moves the problem rather than removing it: every expert still has to live somewhere, and that somewhere is your system memory. A back-of-envelope estimate shows what the paper's headline implies. A 284-billion-parameter model stored at 4 bits per weight is about 142 GB. In NVFP4, which adds an 8-bit scale for every 16 weights, it is closer to 160 GB, before the operating system, the KV cache and anything else you run. By the same arithmetic, the 753B GLM-5.2 result implies well over 400 GB of RAM.
So the "gaming desktop" in that claim is a PC with a GPU you can buy off the shelf and around 192 GB of RAM, which is the top end of consumer builds rather than a typical one. The laptop result scales the same way: a 35B model at 4.5 bits per weight is roughly 20 GB, far more than an 8 GB laptop GPU holds, so most of it sits in system RAM and a 16 GB laptop would not fit it. These are our estimates from the parameter counts, not figures FreeToken publishes, and the real footprint depends on each checkpoint's formats.
Storage matters too. DeepSeek-V4.1-Flash's two 98 GiB Engram tables stream from the checkpoint shards on demand, and the docs recommend keeping them on a fast NVMe drive. Qwen3.8-Flash-Next keeps a 47.7 GiB n-gram table pinned in host RAM. On Windows and WSL, CUDA caps pinned memory at around half of RAM, so FreeToken pins what it can from the middle layers outward and OS-locks the rest, which works but copies more slowly.
Long Prompts Are the Bottleneck
Generating a token touches only a few experts per layer, but reading a long prompt does not. FreeToken's prefill guide explains that with offloaded experts, every MoE layer copies all of its experts from host to GPU for every prefill chunk, because a large chunk routes to nearly every expert anyway. Copy time per chunk is therefore roughly the total expert bytes divided by PCIe bandwidth, and the guide measures about 55.6 GB/s on PCIe 5.0. The default chunk is 8,192 tokens.
Put the earlier estimate through that formula. If a 284B model carries about 150 GB of expert weights, each chunk needs about 2.7 seconds of copying, which caps prefill at roughly 3,000 tokens per second while the copy is the bottleneck. A 100,000-token coding-agent context is 13 chunks, about 35 seconds before the first output token, and roughly twice that on a PCIe 4.0 slot with about half the bandwidth. This is our arithmetic using the project's formula, not a published benchmark.
That number explains several design choices. Because the copy cost is per chunk, not per token, raising --max-prefill-length gives the GPU more work to hide each copy behind, until compute catches up. --moe-prefill-hit-d2d (CUDA 13 or newer) skips experts already in the GPU cache so only misses cross PCIe. Overlap, on by default, copies layer N+1 while layer N computes, and the server logs a warning at startup if the expert cache is too small to hold the two staging layers it needs. And the semantic anchor checkpoints exist so an agent's tool call does not send the whole context back through this pipeline.
Install FreeToken and Serve a Model
The command-line engine needs Linux on x86_64, an NVIDIA GPU with driver r580 or newer (CUDA 13), and Python 3.10 or later, and the README lists native support for RTX 30, 40 and 50 series cards. If you would rather avoid the terminal, the desktop app for Windows and Linux at flashml.ai sets up the engine and gives you a GUI for running models, chatting and tuning. For the CLI, install with uv:
uv venv && source .venv/bin/activate
uv pip install "freetoken[accel]"
The PyPI package JIT-compiles its CUDA kernels on first use, so it needs a CUDA 13 toolkit with nvcc on your PATH. Nightly wheels with the kernels prebuilt are published on the repository's rolling nightly release; pin a wheel URL rather than the tag, because the tag moves every night.
Then start a server. --model takes a local checkpoint directory or a Hugging Face repo id, and the dtype, attention and MoE backends, cache sizes and tool-call and reasoning parsers all resolve from the checkpoint and the GPU:
ft serve --model ~/models/Qwen3.6-35B-A3B
The server is ready when the log says it is serving on 127.0.0.1:1919. It speaks the OpenAI API (/v1/chat/completions, /v1/responses, /v1/models) and the Anthropic API (/v1/messages, /v1/messages/count_tokens), so a client library for either works once you point its base URL at the server:
curl http://127.0.0.1:1919/v1/chat/completions -H 'Content-Type: application/json' -d '{"model": "Qwen3.6-35B-A3B", "messages": [{"role": "user", "content": "What is a Mixture-of-Experts model?"}], "max_tokens": 256}'
The reply comes back as one long line of JSON, and pasting it into a JSON formatter makes the choices and usage blocks readable. For a quick conversation without writing requests, ft shell opens a terminal chat attached to the running server, and ft shell --model starts an engine and chats in one process.
Point a Coding Agent at It
Local serving is most useful when the tools you already use can call it, and FreeToken has a shortcut for that. ft launch claude writes Claude Code's provider config, installs the CLI if it is missing and starts it against your server. The same command works for codex, opencode, openclaw, hermes and dsh, and --dry-run previews the changes first. The README names Codex, Claude Code, OpenCode, OpenClaw and DeepSeek Harness among the agents it works with.
Tool calling runs through parsers inferred from the model family, and reasoning models get their chain of thought split into a separate reasoning_content field. When you debug an agent, remember that in OpenAI-style responses a tool call's arguments arrive as a JSON-encoded string inside the JSON reply, full of escaped quotes; a JSON unescape tool turns them back into readable JSON.
By default the server binds to 127.0.0.1 and accepts any request, which is fine on a single machine. If you expose it to your network with --host, set --api-key (or the FREETOKEN_API_KEY environment variable) as well, so every route except /health requires the key as a Bearer token or an x-api-key header. A long value from a random string generator does the job. The server runs four requests at once by default, adjustable with --max-running-requests.
What to Watch Out For
FreeToken is very young. The repository is about three and a half months old, it had 473 open issues and pull requests on November 2, 2026, and its docs show a weight format still in motion: several notes say FTW checkpoints converted by older builds no longer load, and one warns that an old Qwen3.5, Qwen3.6 or Qwen3.8-27B conversion loads without an error but gives wrong output. If you convert checkpoints with ft checkpoint, plan to reconvert after upgrading, and run the bundled checkpoint diagnosis script when a model refuses to load.
Hardware support is narrower than "your gaming PC" suggests. The CLI is Linux x86_64 and NVIDIA only, there is no macOS build, Windows users get the desktop app or WSL with its pinned-memory cap, and AMD support for RDNA3 and RDNA4 cards on ROCm 7.14 is labelled experimental, without the CUDA-only kernel backends. Not every feature covers every model either: DeepSeek-V4 experts have no resident path, and DSpark speculative decoding is not served for DeepSeek-V4.1-Flash.
Finally, the abstract makes capability claims, not speed claims. It says which models run on which class of machine, not how many tokens per second you will get, so measure on your own hardware. ft bench bw reports your real PCIe and CPU bandwidth, and --moe-collect-stats logs the expert-cache hit rate. Together they tell you whether a model will feel interactive on your machine before you buy more RAM for it.
FAQs
What is FreeToken?
An open-source serving engine from FlashML-org, licensed under Apache 2.0, for running large Mixture-of-Experts language models on consumer hardware. It keeps experts in system RAM, caches the busiest ones on the GPU, and serves OpenAI and Anthropic compatible APIs. It had 15,782 stars in November 2026.
Can FreeToken really run a 284B model on a gaming PC?
Its paper reports a 284B model on a gaming desktop, but the limit is system RAM, not the GPU. At about 4.5 bits per weight such a model is roughly 160 GB, so by our estimate you need around 192 GB of RAM, plus a fast NVMe drive for some models.
What hardware does FreeToken need?
The CLI needs Linux on x86_64, an NVIDIA GPU with driver r580 or newer for CUDA 13 (RTX 30, 40 and 50 series are supported), and Python 3.10 or later. The desktop app runs on Windows and Linux. AMD RDNA3 and RDNA4 GPUs are experimental under ROCm 7.14.
Does FreeToken work on a Mac?
No. There is no macOS build: the engine targets Linux with NVIDIA GPUs, the desktop app is for Windows and Linux, and AMD support is still experimental.
Which models does FreeToken support?
Its known-good list includes DeepSeek-V4 and V4.1-Flash, GLM-4.7, GLM-5.2 and GLM-5.3-Flash, Qwen3, Qwen3.6 and Qwen3.8 models, gpt-oss, Gemma-4, and MiniMax-M2.5 and M3, among others. It loads Hugging Face safetensors directly and also reads GGUF files, in formats such as NVFP4, MXFP4, FP8 and BF16.
Is FreeToken compatible with the OpenAI API?
Yes. ft serve exposes /v1/chat/completions, /v1/responses and /v1/models in the OpenAI format and /v1/messages and /v1/messages/count_tokens in the Anthropic format on 127.0.0.1:1919, so existing client libraries work once you change the base URL.
Can I use FreeToken with Claude Code or Codex?
Yes. ft launch claude configures Claude Code to use your local server, installs its CLI if needed and starts it, and codex, opencode, openclaw, hermes and dsh work the same way. Add --dry-run to preview the config changes first.
Does FreeToken cost anything?
The engine is open source under the Apache License 2.0, so there is no licence fee or subscription; the costs are your own hardware and electricity. The desktop app for Windows and Linux is downloaded from flashml.ai.
---
Originally published at devpik.com. Repo: FlashML-org/FreeToken.
Mentions
FreeToken: Run Frontier MoE Models on Your Own GPU, and the RAM It Takes --- title: "FreeToken: Run Frontier MoE Models on Your Own GPU, and the RAM It Takes" published: true tags: opensource, localllm, mixtureofexperts, llminference canonical_url: devpik.com --- FreeToken runs frontier-scale Mixture-of-Experts models on personal hardware by keeping experts in system RAM and caching the busy ones on the GPU. Its paper reports a 284B model on a gaming desktop. Here is how it works, how much RAM that claim implies, and how to serve your first model. What Is FreeToken? FreeToken is an open-source serving engine for running frontier-scale Mixture-of-Experts (MoE) language models on personal and consumer hardware. Instead of assuming a datacenter GPU with room for every weight, it treats the GPU, the CPU, system memory and the links between them as one elastic inference platform, and it exposes OpenAI and Anthropic compatible APIs so coding agents can use a local model the way they use a cloud one. It comes from FlashML-org, and the accompanying paper, "FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Execution," was posted to arXiv on August 17, 2026, with authors including Ion Stoica, Matei Zaharia, Song Han and Kurt Keutzer. The abstract reports more than 20 MoE models served on hardware ranging from an 8 GB laptop GPU to a single workstation GPU: a 35B model on a laptop, a 284B model on a gaming desktop, and the 753B GLM-5.2 on one workstation card. The repository was created on July 20, 2026, is licensed under Apache 2.0, and had 15,782 stars and 1,571 forks when we checked on November 2, 2026. The project credits mini-sglang as its main inspiration and says it reuses designs and code from SGLang, vLLM, FlashInfer, flash-linear-attention, LightLLM and llama.cpp. Why a Huge MoE Model Fits on One Consumer GPU The trick is in how MoE models work. Each layer holds many small expert networks, but a router sends each token to only a few of them, so most weights sit idle on any given step. Qwen3.6-35B-A3B, the model FreeToken's quick start uses, has 35 billion parameters in total but only about 3 billion active per token, which is what the A3B in its name means. FreeToken builds on that. In its default offload strategy the experts live in host RAM and the GPU keeps a least-recently-used cache of expert slots: a token whose expert is already cached runs at GPU speed, and a miss is streamed over PCIe. The cpu strategy computes misses on the CPU instead of fetching them, and hybrid splits each step's misses between the two, overlapped, using bandwidth figures you measure once per machine with ft bench bw. If the experts fit in VRAM, fused keeps them all resident. Left on auto, dense models resolve to fused and MoE models to offload, except on unified-memory machines such as DGX Spark, where copying between CPU and GPU memory would only move data between two names for the same RAM. Around that core sit the features the README highlights: double-buffered prefill streaming, one global LRU pool of experts shared across layers, FreeToken's own FTW fast-load weight format, VRAM that can move between the expert cache and the KV cache at runtime without a restart, and semantic anchor checkpoints that let agent edits such as tool calls and thinking blocks reuse context instead of recomputing it. The Real Requirement Is System RAM Offloading moves the problem rather than removing it: every expert still has to live somewhere, and that somewhere is your system memory. A back-of-envelope estimate shows what the paper's headline implies. A 284-billion-parameter model stored at 4 bits per weight is about 142 GB. In NVFP4, which adds an 8-bit scale for every 16 weights, it is closer to 160 GB, before the operating system, the KV cache and anything else you run. By the same arithmetic, the 753B GLM-5.2 result implies well over 400 GB of RAM. So the "gaming desktop" in that claim is a PC with a GPU you can buy off the shelf and around 192 GB of RAM, which is the top end of consumer builds rather than a typical one. The laptop result scales the same way: a 35B model at 4.5 bits per weight is roughly 20 GB, far more than an 8 GB laptop GPU holds, so most of it sits in system RAM and a 16 GB laptop would not fit it. These are our estimates from the parameter counts, not figures FreeToken publishes, and the real footprint depends on each checkpoint's formats. Storage matters too. DeepSeek-V4.1-Flash's two 98 GiB Engram tables stream from the checkpoint shards on demand, and the docs recommend keeping them on a fast NVMe drive. Qwen3.8-Flash-Next keeps a 47.7 GiB n-gram table pinned in host RAM. On Windows and WSL, CUDA caps pinned memory at around half of RAM, so FreeToken pins what it can from the middle layers outward and OS-locks the rest, which works but copies more slowly. Long Prompts Are the Bottleneck Generating a token touches only a few experts per layer, but reading a long prompt does not. FreeToken's prefill guide explains that with offloaded experts, every MoE layer copies all of its experts from host to GPU for every prefill chunk, because a large chunk routes to nearly every expert anyway. Copy time per chunk is therefore roughly the total expert bytes divided by PCIe bandwidth, and the guide measures about 55.6 GB/s on PCIe 5.0. The default chunk is 8,192 tokens. Put the earlier estimate through that formula. If a 284B model carries about 150 GB of expert weights, each chunk needs about 2.7 seconds of copying, which caps prefill at roughly 3,000 tokens per second while the copy is the bottleneck. A 100,000-token coding-agent context is 13 chunks, about 35 seconds before the first output token, and roughly twice that on a PCIe 4.0 slot with about half the bandwidth. This is our arithmetic using the project's formula, not a published benchmark. That number explains several design choices. Because the copy cost is per chunk, not per token, raising --max-prefill-length gives the GPU more work to hide each copy behind, until compute catches up. --moe-prefill-hit-d2d (CUDA 13 or newer) skips experts already in the GPU cache so only misses cross PCIe. Overlap, on by default, copies layer N+1 while layer N computes, and the server logs a warning at startup if the expert cache is too small to hold the two staging layers it needs. And the semantic anchor checkpoints exist so an agent's tool call does not send the whole context back through this pipeline. Install FreeToken and Serve a Model The command-line engine needs Linux on x86_64, an NVIDIA GPU with driver r580 or newer (CUDA 13), and Python 3.10 or later, and the README lists native support for RTX 30, 40 and 50 series cards. If you would rather avoid the terminal, the desktop app for Windows and Linux at flashml.ai sets up the engine and gives you a GUI for running models, chatting and tuning. For the CLI, install with uv: uv venv && source .venv/bin/activate uv pip install "freetoken[accel]" The PyPI package JIT-compiles its CUDA kernels on first use, so it needs a CUDA 13 toolkit with nvcc on your PATH. Nightly wheels with the kernels prebuilt are published on the repository's rolling nightly release; pin a wheel URL rather than the tag, because the tag moves every night. Then start a server. --model takes a local checkpoint directory or a Hugging Face repo id, and the dtype, attention and MoE backends, cache sizes and tool-call and reasoning parsers all resolve from the checkpoint and the GPU: ft serve --model ~/models/Qwen3.6-35B-A3B The server is ready when the log says it is serving on 127.0.0.1:1919. It speaks the OpenAI API (/v1/chat/completions, /v1/responses, /v1/models) and the Anthropic API (/v1/messages, /v1/messages/count_tokens), so a client library for either works once you point its base URL at the server: curl http://127.0.0.1:1919/v1/chat/completions -H 'Content-Type: application/json' -d '{"model": "Qwen3.6-35B-A3B", "messages": [{"role": "user", "content": "What is a Mixture-of-Experts model?"}], "max_tokens": 256}' The reply comes back as one long line of JSON, and pasting it into a JSON formatter makes the choices and usage blocks readable. For a quick conversation without writing requests, ft shell opens a terminal chat attached to the running server, and ft shell --model starts an engine and chats in one process. Point a Coding Agent at It Local serving is most useful when the tools you already use can call it, and FreeToken has a shortcut for that. ft launch claude writes Claude Code's provider config, installs the CLI if it is missing and starts it against your server. The same command works for codex, opencode, openclaw, hermes and dsh, and --dry-run previews the changes first. The README names Codex, Claude Code, OpenCode, OpenClaw and DeepSeek Harness among the agents it works with. Tool calling runs through parsers inferred from the model family, and reasoning models get their chain of thought split into a separate reasoning_content field. When you debug an agent, remember that in OpenAI-style responses a tool call's arguments arrive as a JSON-encoded string inside the JSON reply, full of escaped quotes; a JSON unescape tool turns them back into readable JSON. By default the server binds to 127.0.0.1 and accepts any request, which is fine on a single machine. If you expose it to your network with --host, set --api-key (or the FREETOKEN_API_KEY environment variable) as well, so every route except /health requires the key as a Bearer token or an x-api-key header. A long value from a random string generator does the job. The server runs four requests at once by default, adjustable with --max-running-requests. What to Watch Out For FreeToken is very young. The repository is about three and a half months old, it had 473 open issues and pull requests on November 2, 2026, and its docs show a weight format still in motion: several notes say FTW checkpoints converted by older builds no longer load, and one warns that an old Qwen3.5, Qwen3.6 or Qwen3.8-27B conversion loads without an error but gives wrong output. If you convert checkpoints with ft checkpoint, plan to reconvert after upgrading, and run the bundled checkpoint diagnosis script when a model refuses to load. Hardware support is narrower than "your gaming PC" suggests. The CLI is Linux x86_64 and NVIDIA only, there is no macOS build, Windows users get the desktop app or WSL with its pinned-memory cap, and AMD support for RDNA3 and RDNA4 cards on ROCm 7.14 is labelled experimental, without the CUDA-only kernel backends. Not every feature covers every model either: DeepSeek-V4 experts have no resident path, and DSpark speculative decoding is not served for DeepSeek-V4.1-Flash. Finally, the abstract makes capability claims, not speed claims. It says which models run on which class of machine, not how many tokens per second you will get, so measure on your own hardware. ft bench bw reports your real PCIe and CPU bandwidth, and --moe-collect-stats logs the expert-cache hit rate. Together they tell you whether a model will feel interactive on your machine before you buy more RAM for it. FAQs What is FreeToken? An open-source serving engine from FlashML-org, licensed under Apache 2.0, for running large Mixture-of-Experts language models on consumer hardware. It keeps experts in system RAM, caches the busiest ones on the GPU, and serves OpenAI and Anthropic compatible APIs. It had 15,782 stars in November 2026. Can FreeToken really run a 284B model on a gaming PC? Its paper reports a 284B model on a gaming desktop, but the limit is system RAM, not the GPU. At about 4.5 bits per weight such a model is roughly 160 GB, so by our estimate you need around 192 GB of RAM, plus a fast NVMe drive for some models. What hardware does FreeToken need? The CLI needs Linux on x86_64, an NVIDIA GPU with driver r580 or newer for CUDA 13 (RTX 30, 40 and 50 series are supported), and Python 3.10 or later. The desktop app runs on Windows and Linux. AMD RDNA3 and RDNA4 GPUs are experimental under ROCm 7.14. Does FreeToken work on a Mac? No. There is no macOS build: the engine targets Linux with NVIDIA GPUs, the desktop app is for Windows and Linux, and AMD support is still experimental. Which models does FreeToken support? Its known-good list includes DeepSeek-V4 and V4.1-Flash, GLM-4.7, GLM-5.2 and GLM-5.3-Flash, Qwen3, Qwen3.6 and Qwen3.8 models, gpt-oss, Gemma-4, and MiniMax-M2.5 and M3, among others. It loads Hugging Face safetensors directly and also reads GGUF files, in formats such as NVFP4, MXFP4, FP8 and BF16. Is FreeToken compatible with the OpenAI API? Yes. ft serve exposes /v1/chat/completions, /v1/responses and /v1/models in the OpenAI format and /v1/messages and /v1/messages/count_tokens in the Anthropic format on 127.0.0.1:1919, so existing client libraries work once you change the base URL. Can I use FreeToken with Claude Code or Codex? Yes. ft launch claude configures Claude Code to use your local server, installs its CLI if needed and starts it, and codex, opencode, openclaw, hermes and dsh work the same way. Add --dry-run to preview the config changes first. Does FreeToken cost anything? The engine is open source under the Apache License 2.0, so there is no licence fee or subscription; the costs are your own hardware and electricity. The desktop app for Windows and Linux is downloaded from flashml.ai. --- Originally published at devpik.com. Repo: FlashML-org/FreeToken.
15524 个星，今天涨了 242。FreeToken 把数据中心规模的模型服务做到你的桌面上，让超大模型在本地跑得快，也跑得高效。 在自己电脑上跑大模型，快指的是你的电脑跑着数据中心规模的服务，超大模型留在本地。 FreeToken 的仓库，定位是把数据中心规模的模型服务做到桌面。 FlashML-org/FreeToken
647 new stars this week. Instead of renting datacenter GPUs for 290B+ MoE models, FreeToken streams experts across consumer CPU and GPU memory locally. FlashML-org/FreeToken
「遊戲電腦跑 2900 億參數模型」FreeToken，今天 GitHub +654 星，總星 1.57 萬。 翻文檔：專家權重放主機記憶體，GPU 只快取一部分，靠 PCIe 來回搬。卡你的是記憶體和頻寬，不是顯卡。只支援 N 卡 RTX 30 到 50，AMD 還是 WIP。 下模型前，先跑 ft bench bw 測頻寬。 FlashML-org/FreeToken
Source: FlashML-org/FreeToken > FreeToken is an edge-native Mixture-of-Experts (MoE) serving engine designed for running frontier-scale open-weight models on personal and consumer hardware. It treats heterogeneous edge resources—GPUs, CPUs, host memory, and interconnects—as a unified, elastic inference platform.
太牛了，FreeToken 居然可以在家用电脑上跑 290B 的大模型。 再也不用租数据中心显卡了，它的边缘推理引擎把显存、内存和 CPU 统筹起来跑前沿开源 MoE 模型，这波真的可以冲。 FlashML-org/FreeToken
FreeVideo: Make videos on the computer you already own (2.5k stars) What it does All text below is quoted directly from the project README. >Make videos on the computer you already own. FreeVideo runs MiniMax H3 in as little as 8 GB of VRAM and 16 GB of RAM, and adapts its acceleration path to your hardware. About: - Hardware-adaptive execution: Chooses the weight format for each GPU, int8 on GeForce and RTX 30-series cards and FP8 on workstation and datacenter cards, and automatically probes the available attention kernels. - Low-memory inference: Weight streaming, asynchronous prefetching and chunked computation keep peak memory low, enabling inference with as little as 8 GB of VRAM and 16 GB of RAM. - Multimodal inputs: Text prompts, first and last frames, and image, video and audio references. - Community LoRAs: Use MiniMax H3 LoRAs in your workflow. - ComfyUI integration: A dedicated creative workspace inside ComfyUI that supports two-pass sampling and batch generation and keeps a history of past creations. - One-click deployment: The Windows, Mac and Linux launchers set up ComfyUI, the runtime environment and the models, reuse existing models, and support offline installation. >FreeVideo runs as a ComfyUI plugin, with launchers for Windows, macOS and Linux. Repository FlashML-org/FreeVideo Repository stats |Metric|Value| |:-|:-| |Stars|2,482| |Forks|291| |Last release|v0.4.25 (2026-10-29)| |Languages|Python 71.5%, JavaScript 24.8%, QML 2%| |Contributors|3| |AI contributor accounts (Claude / Codex / Cursor / Copilot)|None| |AI co-authored commits|None| Main contributor |Field|Value| |:-|:-| |Name|Bowen Xue| |GitHub|@KBRASK| |Commits to this repo|872 (top contributor)| |Bio|a third-year undergraduate student| |Followers|54| ────────────────────────────── 📌 If you want to browse more repos like this, we've organized them by category and rankings here: bestgithubrepositories.com 🗂️ We're also archiving repos where possible, so useful projects don't disappear when they're removed from GitHub.
290B 级 MoE 大模型，家用电脑真能跑？ FreeToken 是边端 MoE 推理引擎：CPU、GPU 协同算，专家按需缓存换入，290B+ 模型能在游戏 PC 上本地跑。 适合想本地跑大模型的人，先看显卡在不在支持列表。 先在本地跑哪个 MoE，DeepSeek 或 Qwen？ #大模型 #推理 @grok FlashML-org/FreeToken
A 284B-parameter model on a single gaming GPU. 🖥️ FreeToken, from a Berkeley team incl. Ion Stoica and @matei_zaharia, parks MoE experts in RAM and pulls only what each token needs over PCIe. Catch: 192GB+ of RAM, and the API key is off by default. FlashML-org/FreeToken t.co
How to Generate AI Video on Your Gaming PC | FreeVideo Tutorial Powered by Minimax H3 FreeVideo is a free AI video generator that runs on your own gaming GPU. This video shows you how to install it on Windows, make a first clip, and show what each setting does to quality and waiting time. DOWNLOAD FREEVIDEO GitHub: FlashML-org/FreeVideo Windows: github.com/FlashML-org/FreeVideo/releases/latest/… macOS (Apple silicon preview, not notarized yet): github.com/FlashML-org/FreeVideo/releases/latest/… Mac guide: github.com/FlashML-org/FreeVideo/blob/main/docs/M… Linux (AppImage, or setup.sh for a terminal or server): github.com/FlashML-org/FreeVideo/releases/latest/… Linux guide: github.com/FlashML-org/FreeVideo/blob/main/docs/L… I only tested the Windows version. CHAPTERS 0:00 Intro: free AI video on a gaming GPU 0:47 Before you start: GPU, memory and disk 1:30 Before you start: the model license 1:53 Install FreeVideo on Windows 2:50 The FreeVideo workspace 3:36 First clip: a simple prompt 3:47 How it fits on a regular GPU 4:18 First clip: the result 4:31 Settings: the prompt 5:08 Settings: frame and duration 5:42 Settings: quality levels explained 6:41 Speed and quality on my RTX 4070 7:26 Two-pass switch and Preview first 8:25 Media: first and last frame 9:17 Media: reference image 9:43 Media: generate the image with AI 10:15 Seed and history 10:37 Limitations 11:09 Wrap-up WHAT YOU NEED • Official minimum: 8 GB of video memory and 16 GB of system memory. I didn't test a card that small. • About 75 GB of free disk space, more if you add models later. The generation cache also grows over time. • My PC: RTX 4070 with 12 GB of video memory and 64 GB of system memory. A 5-second clip at 960 × 544 took about 3 to 5 minutes on every quality level. Your times will differ. WHAT "FREE" MEANS HERE FreeVideo's code is open source (Apache 2.0), and generating on your own PC has no per-video fees. The model weights come with their own license, the MiniMax H3 Community License. It excludes some territories (it lists the European Union, the United Kingdom, the Republic of Korea and the United States), has rules for commercial use, and asks that content you publish is clearly disclosed as machine-generated. Read it before you use FreeVideo or share what you make: huggingface.co ABOUT THIS VIDEO Recorded with FreeVideo v0.4.19 on Windows. FreeVideo updates often (v0.4.21 is already out), so some labels may have moved. Every example clip in this video, and the waterfall scene in the thumbnail, was generated locally with FreeVideo. On a Mac? My earlier video runs MiniMax H3 there another way: youtube.com CREDITS FreeVideo by FlashML: FlashML-org/FreeVideo Video DeltaNet (VDN) by OpenVDN: openvdn.github.io MiniMax H3 base model: huggingface.co Which GPU are you running? Tell me your card and how long your first clip took. #FreeVideo #AIVideo #LocalAI