karpathy

Over the last 3 years we made a lot of progress in efficient training methods. On text models, you can now train GPT-2 levels of coherent English from scratch for about $50 (e.g. using Karpathy's nanochat [1]) 1: karpathy/nanochat

nanochat: Train Your Own GPT-2 Grade ChatGPT in About Two Hours --- title: "nanochat: Train Your Own GPT-2 Grade ChatGPT in About Two Hours" published: true tags: opensource, llmtraining, machinelearning, pytorch canonical_url: devpik.com --- nanochat trains a ChatGPT-style model from scratch, tokenizer to chat, on a single 8XH100 node. The GPT-2 speedrun record is 1.65 hours, about 1,000 times cheaper than GPT-2 cost in 2019. Here is how it works, what a run really costs, and how to try it. What Is nanochat? nanochat is Andrej Karpathy's experimental harness for training a large language model from scratch, end to end, on a single GPU node. One codebase covers every stage: training the tokenizer, pretraining, supervised finetuning, evaluation and inference, ending in a chat model you can talk to from the command line. The repository's own tagline is "The best ChatGPT that $100 can buy." The name comes from Karpathy's earlier nanoGPT, which only covered pretraining. nanochat goes the rest of the way to a chatbot and deliberately stays small. Its contributing notes promise no giant configuration objects, no model factories and no if-then-else monsters, just a single, cohesive, minimal, readable, hackable and maximally forkable strong baseline. Most of the logic sits in a handful of files, such as nanochat/gpt.py for the Transformer, nanochat/engine.py for inference with a KV cache and nanochat/optim.py for the AdamW and Muon optimizers. It is written in Python on PyTorch, licensed under MIT, and was first published on October 13, 2025. When we checked on November 2, 2026 it had 59,646 stars and 8,388 forks. For questions, the README points to DeepWiki, the repository's Discussions tab and a #nanochat channel on Discord. One Dial Sets the Whole Model The design idea that makes nanochat easy to use is a single complexity dial: --depth, the number of Transformer layers. Everything else is derived from it automatically, including the width of the model, the number of attention heads, learning rate adjustments, the training horizon and weight decay, all chosen so the result comes out compute optimal. You ask for a smaller or bigger model and the rest follows. Sweeping that one dial gives what Karpathy calls a miniseries of compute-optimal models at different sizes. GPT-2 capability currently sits between depth 24 and depth 26. For quick experiments he likes depth 12, roughly GPT-1 sized, where a pretraining run takes about five minutes: change something, rerun d12 and watch validation bits per byte, the CORE score and training throughput to see whether it helped. Odd depths such as 25 are not recommended, because the head-dimension math stops coming out neatly. The dial is also the bar for contributions. A change has to be principled enough to work at every depth rather than being tuned to one model, which is what keeps the codebase small while it improves. Training length has its own knob, --target-param-data-ratio, the number of training tokens per non-embedding parameter. Compute optimal is currently 10.5. The GPT-2 runs deliberately go lower and slightly undertrain a d24, because the goal there is to cross the GPT-2 line as fast as possible rather than to get the best model for the compute spent. The Record: GPT-2 in 1.65 Hours The main focus of development is a "time to GPT-2" speedrun, inspired by the modded-nanogpt project: the wall-clock time an 8XH100 node needs to train a model that beats GPT-2 (1.6B) on the DCLM CORE score, an ensemble of 22 evaluations such as ARC and MMLU. GPT-2's CORE score is 0.256525, and a run has to land above it. The record has fallen quickly. The first entry, on January 29, 2026, took 3.04 hours with a slightly overtrained d24. FP8 training brought it to 2.91 hours on February 2, doubling the batch to about 1M tokens got it to 2.76 hours on February 5, and switching the pretraining data to NVIDIA ClimbMix cut it to 2.02 hours on March 4, after FineWeb, DCLM and Olmo data had all failed to help. The last two records, 1.80 hours on March 9 and 1.65 hours on March 14, 2026, came from autonomous research: improvements found by a private version of Karpathy's autoresearch on a d12 model, which carried over to d24. The current record, 1.65 hours or 99 minutes, is the average of five runs with a mean CORE of 0.262634. Whatever holds the record is always the recipe in runs/speedrun.sh, so cloning the repository gives you the current best rather than last year's tutorial. The Cost Math: About 1,000 Times Cheaper Than 2019 The leaderboard notes give enough numbers to check the headline claim. OpenAI trained GPT-2 in 2019 on 32 TPU v3 chips for 168 hours at about $8 per chip-hour: 32 x 168 x $8 = $43,008, the "approx. $43K" the README quotes. At the README's assumed $3 per GPU-hour, an 8XH100 node costs about $24 an hour, so the 1.65-hour record is about $39.60 of training time. That is roughly 1,086 times cheaper and about 100 times faster in wall-clock time, and the hardware tells the same story: 5,376 TPU-hours then, 13.2 H100-hours now. Run 1's log puts its total compute at 4.330784131228946e+19 floating point operations; a scientific notation converter turns that into a plain number, about 43.3 quintillion. The catch is that the leaderboard measures only the training iterations. Its notes say the reported time excludes evaluations and logging, while the full speedrun script also downloads data, trains a tokenizer, evaluates the base model and runs supervised finetuning. So the repository quotes three numbers for one job: about 1.5 hours in the script's header, 1.65 hours on the leaderboard, and about 2 hours, or $48, in the README's introduction. The 1.5-hour figure is already shorter than the training phase alone, so plan on the last one. The README adds that a spot instance can bring the total closer to $15. Run the Speedrun Yourself nanochat uses uv for dependencies. On a CUDA machine run uv sync --extra gpu, or uv sync --extra cpu for CPU-only and Apple Silicon, then activate the environment with source .venv/bin/activate. For the real thing, rent an 8XH100 node (the README mentions Lambda, which provided compute for the project), clone the repository and start the whole pipeline in one command, ideally inside screen so it survives a dropped connection: screen -L -Logfile runs/speedrun.log -S speedrun bash runs/speedrun.sh The script is short enough to read before you pay for the node. It downloads 8 data shards, about 2 billion characters, and trains a BPE tokenizer with a vocabulary of 32,768 on them while the download continues in the background up to 170 shards. About 150 are needed for GPT-2 capability, and at roughly 100 MB each that is around 17 GB of text to pull before pretraining starts; a unit converter helps when you are sizing the disk. Then it pretrains a d24 with FP8 and a data ratio of 8, evaluates the CORE score, and runs supervised finetuning to teach conversation tokens, tool use and multiple choice, with evaluations on tasks such as ARC, GSM8K, MMLU and a simple Python coding set. When it finishes, chat with your model: python -m scripts.chat_cli Add -p "Why is the sky blue?" for a single prompt instead of an interactive session. Logging to Weights & Biases is optional: run wandb login and set WANDB_RUN before starting the script, otherwise it skips wandb entirely. Smaller Hardware Works, Just Slower You do not need eight H100s to use nanochat, only to match the leaderboard. The README says the code runs fine on an 8XA100 node, a bit slower. On a single GPU, drop torchrun and the code switches to gradient accumulation automatically and produces nearly identical results, but it takes eight times longer, so a run measured in hours on a node becomes most of a day. If your GPU has less than 80 GB of memory, lower --device-batch-size from its default of 32 to 16, 8, 4, 2 or even 1 until it fits, keeping it a power of two. The training script compensates with more gradient accumulation steps, so the total batch the optimizer sees stays the same and only the number of passes changes. Precision is set explicitly rather than through autocast. A single COMPUTE_DTYPE defaults to bfloat16 on A100, H100 and newer cards, and to float32 on older ones such as the V100 and T4, where float16 is available with NANOCHAT_DTYPE=float16 and a gradient scaler (reinforcement learning does not support it yet). CPU and Apple MPS default to float32, and recent macOS also runs bfloat16 with about 25% less memory at similar speed. The --fp8 flag is optional: without it training runs in bf16 and produces a slightly stronger model. For a laptop, runs/runcpu.sh shrinks the model until training fits in a few tens of minutes on CPU or Apple Silicon. The README is blunt that you will not get strong results that way. It is for learning the pipeline, not for building a useful chatbot. What to Watch Out For Set expectations for the model itself. The speedrun trains with about 4e19 FLOPs, which the README compares to talking to a kindergartener: it writes stories and poems, explains why the sky is blue with confident nonsense, and hallucinates if you ask it who you are. That is the nature of a model this size, not a flaw in your run. A single run is a sample, not a guarantee. When Karpathy ran the Run 4 recipe seven times, CORE scores ranged from 0.25119 to 0.26765, and three of the seven landed below GPT-2's 0.256525. Later records were set on averages with a larger safety margin, and the leaderboard notes recommend tracking validation bits per byte as a smoother signal, with the warning that it is not comparable across runs on different datasets. Some documentation has drifted behind the code. The README's file tree still describes speedrun.sh as training "the ~$100 nanochat d20" while the script now trains a d24, and the original October 2025 announcement is marked as containing deprecated information. Non-CUDA backends such as XPU and MPS should work because the code is mostly plain PyTorch, but the README says not all of those paths have been exercised. If you plan to contribute, read the AI policy first: pull requests must disclose any parts with substantial LLM contribution that you did not write or do not fully understand. The latest leaderboard record dates from March 14, 2026 and the last push to the repository was on September 7, 2026, so check the Discussions tab for current activity before building on it. FAQs What is nanochat? Andrej Karpathy's minimal, open-source harness for training a ChatGPT-style model from scratch on one GPU node. It covers tokenizer training, pretraining, finetuning, evaluation and inference in one small PyTorch codebase, licensed under MIT, and had 59,646 stars in November 2026. How much does it cost to train a GPT-2 grade model with nanochat? The README puts a full run at about 2 hours on an 8XH100 node, roughly $48 at $3 per GPU-hour, and closer to $15 on a spot instance. The 1.65-hour leaderboard record counts only the training iterations, about $40 of node time. Training GPT-2 originally cost about $43,000 in 2019. How long does the nanochat speedrun take? The leaderboard record is 1.65 hours (99 minutes) of training, set on March 14, 2026. The full speedrun.sh pipeline also downloads data, trains the tokenizer, evaluates and finetunes, so budget about 2 hours of node time. Can I run nanochat on a single GPU or a Mac? Yes. On one GPU, omit torchrun and it uses gradient accumulation with nearly identical results, eight times slower; lower --device-batch-size if the GPU has less than 80 GB. runs/runcpu.sh trains a much smaller model on CPU or Apple Silicon in tens of minutes, with weak results. What does the --depth flag do in nanochat? It is nanochat's single complexity dial: the number of Transformer layers. Width, attention heads, learning rate, training horizon and weight decay are derived from it automatically. GPT-2 capability is around depth 24 to 26, and depth 12 suits five-minute experiments. What is the CORE score? An ensemble metric from the DCLM paper over 22 evaluations such as ARC and MMLU. nanochat uses it to decide whether a run beats GPT-2, whose CORE score is 0.256525. How is nanochat different from nanoGPT? nanoGPT, Karpathy's earlier project, covered only pretraining. nanochat goes all the way to a chat model: tokenizer, pretraining, supervised finetuning, evaluation and inference with a KV cache, plus a command-line chat to talk to the result. Is the model nanochat trains actually useful? As a learning tool and research baseline, yes. As an assistant, no: the README compares the speedrun model to a kindergartener that writes short stories and poems but hallucinates freely. Its value is that you trained every stage yourself and can change any of it. --- Originally published at devpik.com. Repo: karpathy/nanochat.

一个模型回答问题，容易把偏见写得很自信。 Karpathy 的 LLM Council 换了个思路：让多个模型分别回答、互相评审，最后再综合结果。 它不会自动产生“真相”，但能让分歧和薄弱点更早暴露。适合复杂判断，不适合每个小问题都开一桌模型会议——成本和时间都更高。 karpathy/llm-council #GitHub #大模型 #AI工具

想亲手训个 GPT-2 级模型，又不想烧几万刀？ Karpathy 的 nanochat 是极简 LLM 训练框架：分词到推理全包，只调 --depth，GPT-2 级别约 2 小时、48 美元跑完。 适合想搞懂 LLM 训练、又没大算力预算的人。 你会先拿它训个什么小模型玩玩？ #LLM #开源模型 @grok karpathy/nanochat

Build a BPE Tokenizer in 80 Lines of TypeScript and Watch JSON Eat Your Tokens - Book: AI That Answers - The series: AI in TypeScript — 5 books, from your first LLM call to agents in production — all five here - My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools - Me: xgabriel.com | GitHub --- The maintainers of the TOON format ran three flat datasets through OpenAI's o200k_base encoding and counted 164,643 tokens for the pretty-printed JSON and 104,908 for the same data minified. Nothing was dropped. The minified version holds every key and every value. It stops paying for indentation and line breaks, and that alone takes about 36% off the count. Tokens are the unit your LLM bill is written in. Anthropic's pricing table, for one, quotes input and output separately, per million tokens. Yet most developers who pay that bill have never watched a string turn into tokens. You can build the machine that does it in 80 lines of TypeScript and train it on your own JSON. What happens to your prompt before the model sees it OpenAI's tokenizers use byte-pair encoding. BPE began as a compression algorithm, and Sennrich, Haddow and Birch adapted it to split words into subword units for neural machine translation (ACL 2016). The tiktoken README says the encoding is reversible and lossless and works on any text, even text the tokenizer never saw in training. On average, a token covers about 4 bytes. Encoding runs in three steps. 1. A regex cuts the text into chunks, so letters, digits, punctuation and whitespace end up in separate runs. Karpathy's minbpe credits this to the GPT-2 paper and notes it "ensures that no merges will happen across category boundaries." 2. Each chunk becomes its UTF-8 bytes. The starting vocabulary is the 256 possible byte values, so nothing is ever out of vocabulary. 3. Learned merges glue adjacent byte pairs into bigger tokens, in the order they were learned. Training is where the merges come from. Count every adjacent pair across the corpus, take the pair with the highest count, give it the next free id (256, then 257, and so on), replace it everywhere, repeat. The minbpe README walks through aaabdaaabac: three merges (aa, then ab, then the pair of those two new tokens) turn 11 bytes into five tokens, [258, 100, 258, 97, 99]. !Rows of small square byte tiles fusing pairwise into larger tiles, layer by layer Byte-pair encoding: start from single bytes, then fuse the most frequent neighbours, one merge at a time. The tokenizer in 80 lines This is the whole trainer and encoder. It uses GPT-2's pre-split pattern, minus the contraction rules ('s, 'll, and friends), copied from minbpe's regex.py. // bpe.ts: a toy byte-level BPE trainer and encoder type Pair = [number, number]; // GPT-2's pre-split pattern, minus its contraction rules const SPLIT = / ?\p{L}+| ?\p{N}+| ?[^\s\p{L}\p{N}]+|\s+(?!\S)|\s+/gu; const utf8 = new TextEncoder(); const text8 = new TextDecoder(); const key = (a: number, b: number) => `${a},${b}`; const show = (b: Uint8Array) => JSON.stringify(text8.decode(b)); const chunksOf = (text: string) => (text.match(SPLIT) ?? []).map((c) => Array.from(utf8.encode(c))); function countPairs(ids: number[], counts: Map<string, number>) { for (let i = 0; i < ids.length - 1; i++) { const k = key(ids[i], ids[i + 1]); counts.set(k, (counts.get(k) ?? 0) + 1); } } function merge(ids: number[], [a, b]: Pair, id: number) { const out: number[] = []; for (let i = 0; i < ids.length; i++) { if (ids[i] === a && ids[i + 1] === b) { out.push(id); i++; // the pair's second half is consumed too } else out.push(ids[i]); } return out; } export function train(text: string, numMerges: number) { let chunks = chunksOf(text); const merges = new Map<string, number>(); const vocab = new Map<number, Uint8Array>(); for (let b = 0; b < 256; b++) vocab.set(b, Uint8Array.of(b)); for (let m = 0; m < numMerges; m++) { const counts = new Map<string, number>(); for (const ids of chunks) countPairs(ids, counts); let best = ""; let bestCount = 1; // a pair seen once earns no token for (const [k, c] of counts) { if (c > bestCount) { best = k; bestCount = c; } } if (!best) break; const [a, b] = best.split(",").map(Number) as Pair; const id = 256 + m; chunks = chunks.map((ids) => merge(ids, [a, b], id)); merges.set(best, id); const l = vocab.get(a)!, r = vocab.get(b)!; const bytes = new Uint8Array([...l, ...r]); vocab.set(id, bytes); console.log(`#${m}`, show(l), "+", show(r), "->", show(bytes), `${bestCount}x`); } return { merges, vocab }; } export function encode(text: string, merges: Map<string, number>) { const out: number[] = []; for (let ids of chunksOf(text)) { while (ids.length > 1) { // earliest-learned merge first, the order training used let pair: Pair | null = null; let rank = Infinity; for (let i = 0; i < ids.length - 1; i++) { const r = merges.get(key(ids[i], ids[i + 1])); if (r !== undefined && r < rank) { rank = r; pair = [ids[i], ids[i + 1]]; } } if (!pair) break; ids = merge(ids, pair, rank); } out.push(...ids); } return out; } Pairs are keyed as "a,b" strings because a Map compares array keys by reference, so [1, 2] would never find [1, 2]. Ties go to the pair seen first, since Map iterates in insertion order and the comparison is strict. Training works chunk by chunk, so a merge can never span a letter and a digit, or a word and the punctuation after it, which matches the real thing. And encode always applies the lowest-ranked merge available, the same rule minbpe's _encode_chunk uses: replay the merges in the order training learned them. The show helper decodes a token's bytes for printing. When a token holds only part of a multi-byte character, TextDecoder prints �, because the token boundary falls inside the character. The Japanese sample further down does the same. Train it on JSON and read the merges Feed it the kind of text your app actually sends: an array of records, pretty-printed. // demo.ts import { train, encode } from "./bpe"; const records = Array.from({ length: 40 }, (_, i) => ({ customer_id: 1000 + i, status: i % 4 ? "active" : "suspended", country: ["DE", "BR", "JP", "US"][i % 4], })); const pretty = JSON.stringify(records, null, 2); const { merges } = train(pretty, 40); const one = records[0]; const asPretty = encode(JSON.stringify(one, null, 2), merges); const asMin = encode(JSON.stringify(one), merges); console.log({ pretty: asPretty.length, minified: asMin.length }); Run it with tsx: npx tsx demo.ts. Each log line is one merge, with the two parts, the new token, and how often the pair appeared. Read all 40 and sort them in your head: how many are fragments of the keys customer_id, status and country, how many are values and digits, and how many are quotes, colons and runs of spaces? On a corpus made of nothing but JSON, every merge has to come from one of those. The keys repeat in every single record, and that repetition is what earns them merges. A production tokenizer learns its merges from a huge general corpus instead, so common English words have a good chance of getting a token of their own (the next script shows you whether customer did). Your acctBalanceAdjFlagV2 pays for every piece. Then try the experiments the script makes cheap. Train on JSON.stringify(records) instead of the pretty version and compare the merge log. Rename customer_id to cid and see how many merges it frees up. Compare the two counts in the last line, and remember that this toy only knows 40 merges, so the ratio between them tells you more than either number. Ask a real encoder: o200k_base Your toy has 40 merges. o200k_base has a vocabulary of about 200,000 tokens, roughly double GPT-4's ~100,000, according to Simon Willison's notes from the GPT-4o launch. tiktoken's model table maps gpt-4o, gpt-4.1, o3, o4-mini and gpt-5 to it. js-tiktoken loads it in Node; the lite import pulls in only the one rank file you need. // compare.ts import { Tiktoken } from "js-tiktoken/lite"; import o200k_base from "js-tiktoken/ranks/o200k_base"; const enc = new Tiktoken(o200k_base); const pieces = (s: string) => enc.encode(s).map((id) => enc.decode([id])); const samples = [ '{\n "customer_id": 1000\n}', '{"customer_id":1000}', "customerShippingAddressLine2", "1730505600000", "Your order has been cancelled.", "ご注文はキャンセルされました。", ]; for (const s of samples) { const p = pieces(s); console.log(p.length, JSON.stringify(p)); } You can predict part of what it prints by reading the o200k pre-split regex in tiktoken's openai_public.py. Every chunk the regex produces costs at least one token, so the regex sets a floor. - camelCase splits at every capital. The letter rule matches a run of uppercase letters (possibly none) followed by lowercase, so customerShippingAddressLine2 cuts into customer, Shipping, Address, Line and 2 before BPE starts. Five chunks, at least five tokens. - Digits go in groups of three. The number rule is \p{N}{1,3}. A 13-digit millisecond timestamp is five chunks, so at least five tokens, every time it appears in a log line you paste. - Whitespace is its own chunk. Indentation runs, minus their last space, are cut out separately from the keys they sit in front of. One small quirk: the space after "customer_id": in the pretty version cannot attach to a following number, because the digit rule (\p{N}{1,3}) takes no leading character and the punctuation rule's optional space needs a symbol after it. It becomes a chunk on its own. Before a string value it rides along with the opening quote instead. - Non-English text gets fewer merges. Byte-level BPE can encode anything, but the merges go to whatever was common in training data. Petrov et al. (NeurIPS 2023) found the same text translated into different languages could differ in token length by up to 15 times on the tokenizers they studied. OpenAI's larger vocabulary cut non-English counts: its launch figures, as quoted in Willison's notes, put Gujarati at 4.4x fewer tokens and Japanese at 1.4x fewer than the previous tokenizer. When a token boundary still lands inside a Japanese character, pieces prints � for those fragments. !A long accordion of curly braces and indentation blocks next to a compact stack of the same blocks Same data, same keys and values; only the whitespace and repetition change, and that is where the tokens go. One payload, three shapes Now measure the thing you actually send. This script builds one payload and counts it as pretty JSON, minified JSON and TOON, a tabular encoding of the JSON data model that declares the keys of a uniform array once (@toon-format/toon on npm). // shapes.ts import { Tiktoken } from "js-tiktoken/lite"; import o200k_base from "js-tiktoken/ranks/o200k_base"; import { encode as toToon } from "@toon-format/toon"; const enc = new Tiktoken(o200k_base); const orders = Array.from({ length: 50 }, (_, i) => ({ orderId: 5000 + i, customerId: 70 + (i % 9), status: i % 3 ? "shipped" : "pending", totalCents: 1999 + i * 37, })); const payload = { orders }; const shapes: Record<string, string> = { pretty: JSON.stringify(payload, null, 2), minified: JSON.stringify(payload), toon: toToon(payload), }; for (const [name, text] of Object.entries(shapes)) { console.log(name.padEnd(9), enc.encode(text).length); } Swap payload for a real request body from your logs and you have your own numbers. I can't give you numbers for your data, so here are published ones for theirs. TOON's benchmark, counted with o200k_base: | Track | Pretty JSON | Minified JSON | TOON | |---|---|---|---| | Flat data, 3 datasets | 164,643 | 104,908 | 68,030 | | Mixed structure, 5 datasets | 393,637 | 260,451 | 264,734 | On flat, uniform records TOON came in 58.7% under pretty JSON and 35.2% under minified. On mixed and nested data the picture flips: minified JSON beat TOON by 1.6% overall, and the README says plainly that "for deeply nested or non-uniform data, JSON may be more efficient." It also notes that absolute counts are tokenizer-specific, since other providers tokenize differently, and that you have to measure latency yourself. Fewer tokens is also not the same as better answers, so run your evals on the new shape before you ship it. One win held on both tracks: minifying the JSON cut about a third of the tokens, so do that to anything a program produced before it goes into a prompt. Dropping keys the model does not need helps whatever the shape. Save the tabular format for long arrays of identical objects, where TOON won, and keep JSON for nested data, where it lost. Then put shapes.ts next to your prompt builder and check the count every time someone adds a field. --- If this was useful Token cost is the first bill an LLM app runs up, and it is one of the threads in AI That Answers, the first book of my AI in TypeScript series. It covers your first LLM app in TypeScript end to end: prompts, structured output, and keeping an eye on what each call costs before the invoice does it for you.  The other four books go from retrieval and tool calling to agents in production: the whole AI in TypeScript series is here.

How long does it take to train an AI model on cloud GPUs? Training an AI model takes anywhere from a few minutes to several months, depending on model size, how many tokens it trains on and how many GPUs run in parallel. On Nodus a small fine-tune can finish in under five minutes, while by our math a 7B language model trained from scratch on 140 billion tokens takes about 3.6 days on 64 NVIDIA H100s. The sections below show the formula, check it against published runs, and list what makes the real clock run longer than the math. How do you estimate how long a training run will take? Work out the total math the run needs, then divide by how fast your GPUs actually do math. Training a transformer takes about 6 floating-point operations per parameter per token, the rule from Kaplan et al.'s scaling-laws paper. An NVIDIA H100 SXM peaks at about 989 teraFLOPS of dense BF16 math, half the sparse figure on NVIDIA's H100 page. No real run gets the full peak. The share you get is called model FLOPs utilization, and 30% is a fair planning number for code you have not tuned. That gives: GPU-hours = (6 × parameters × tokens) / (989e12 × utilization × 3,600) wall-clock = GPU-hours / number of GPUs (if scaling were perfect) At 30% utilization, one H100-hour delivers about 1.07 × 10^18 useful operations. The second line is the optimistic part, and we come back to it below. How long does it take to train a language model from scratch? Days to weeks for small and mid-size models, if you have enough GPUs. Here is the formula at 30% utilization, using a compute-optimal budget of about 20 tokens per parameter from DeepMind's Chinchilla paper, plus one longer 7B run. | Model and tokens | H100 GPU-hours | Fewer GPUs | More GPUs | | --- | ---: | ---: | ---: | | 1B on 20B tokens | 112 | 1 GPU: 4.7 days | 8 GPUs: 14 hours | | 7B on 140B tokens | 5,505 | 8 GPUs: 28.7 days | 64 GPUs: 3.6 days | | 7B on 2T tokens | 78,643 | 64 GPUs: 51.2 days | 512 GPUs: 6.4 days | | 70B on 1.4T tokens | 550,500 | 512 GPUs: 44.8 days | 2,048 GPUs: 11.2 days | Our arithmetic: 6 × parameters × tokens, 989 teraFLOPS dense BF16, 30% utilization, perfect scaling. Real runs land later; see the overheads section. Token count moves the clock as much as model size. The same 7B model takes 14 times longer on 2 trillion tokens than on 140 billion. Many open models now train far past the Chinchilla ratio, so settle the token budget first. Our post on what it costs to train an LLM prices the same GPU-hours in dollars. How long do real published training runs take? Small reproductions take hours, and frontier models take months. Andrej Karpathy's llm.c reproduction of GPT-2 124M trained on 10 billion tokens in about 90 minutes on one 8 × NVIDIA A100 80GB node. His GPT-2 1.6B run took about 24 hours on one 8 × H100 node: 32,000 steps of roughly 1 million tokens each, at about 2.75 seconds per step. That 1.6B run is a good check on the formula. It works out to 192 GPU-hours for about 33.6 billion tokens, which back-solves to roughly 46% utilization. Karpathy's own logs show 42% to 47% at the start, so well-tuned code beats our 30% planning figure by about half. At the top end, Meta's Llama 3.1 model card lists 30.84 million H100 GPU-hours for the 405B model. The Llama 3 paper says it trained on up to 16,384 H100s. Divide one by the other and even that cluster needs about 78 days of pure compute, a floor on the real calendar time. Do more GPUs make training proportionally faster? Only roughly, and never for free. Adding GPUs splits the same GPU-hours across more machines, so the bill stays about the same while the calendar shrinks. In practice the bill creeps up, because GPUs spend more time talking to each other. Meta's own numbers show the effect. The Llama 3 paper reports 43% utilization on 8,192 GPUs and 41% on 16,384 GPUs for the same model, a small but real loss from doubling the cluster. !Days to train a 7B model on 2 trillion tokens: 409.6 days on 8 H100s, 51.2 days on 64, 6.4 days on 512, at 30% utilization and perfect scaling. The same 78,643 GPU-hours: one 8-GPU node needs 409.6 days, a 512-GPU cluster about a week. Our decision rule: pick the smallest GPU count that finishes before your deadline, not the largest you can get. Single nodes of 8 GPUs scale best because the GPUs share fast NVLink links. Multi-node runs add network cost and more machines that can fail. How long does fine-tuning take? Usually minutes to a few hours, because fine-tuning sees far fewer tokens than pretraining. Fine-tuning a 7B model on 50 million tokens is about 2 H100-hours of math at 30% utilization by the same formula, and LoRA does less than that because it skips most weight gradients. For small jobs, the work around training often takes longer than training itself. In our published fine-tuning benchmark, Qwen2.5-0.5B on one NVIDIA L4 spent 112.8 seconds in the training loop on Nodus. From submission to a verified exported model took 4.8 minutes, or 288 seconds. !Small fine-tune on Nodus: 112.8 seconds in the training loop versus 288 seconds from submission to exported model. For a small fine-tune, startup, evaluation and export took longer than the training loop. So for fine-tuning, the honest answer to "how long" is mostly startup, model download, evaluation and export. On Nodus, a TrainingJob imports a pinned Hugging Face model revision into a read-only Volume once, and later runs reuse it, so billed GPU time is not spent downloading weights. Why does the real clock run longer than the math? Because the formula counts only useful training steps. Four things add time on most projects: - Waiting for capacity. A large multi-GPU request may queue until enough machines are free together. - Startup. Pulling the image, loading the model and reading the first batch of data. - Failures and restarts. Over a 54-day stretch of Llama 3 405B pretraining, Meta logged 466 job interruptions, 419 of them unexpected. It still reported more than 90% effective training time, meaning at least a tenth of the elapsed time did no useful training. - Evaluation and export. Validation passes and writing out the final weights. Our planning rule is to add at least 10% to the formula for multi-day runs. Go higher if your checkpoints are far apart, since every restart redoes the work since the last save. How does Nodus show the time before you launch? A dry run returns the estimate before anything is charged. For a managed TrainingJob, the training guide says the estimate is the number of steps times the measured seconds per step on the slowest GPU type you allowed, plus evaluation time. You can set spec.expectedDuration to use your own figure. $ nodus apply -f sft.yaml --dry-run -o yaml # status.estimate: cost to completion, first hold, start time For your own script, nodus run takes the same idea from flags. This prints the estimate only, using your expected duration: $ nodus run --dry-run --gpu H100:8 --expected-duration 24h -- torchrun train.py If the run must finish by a date, --complete-by (or placement.completeByTime) keeps it off any capacity whose p90 finish is later, per our placement docs. H100s are listed at $2.60/hr on the Nodus pricing page, checked October 31, 2026. If a machine is lost mid-run, Nodus starts a new attempt from the latest committed checkpoint, as described in attempts and recovery. Our post on how often spot GPUs are interrupted shows how much that matters on cheaper capacity. FAQ How long does it take to train a model like GPT-4? Nobody outside the lab knows, because the GPU-hours were never published. The closest public figure is Llama 3.1 405B at 30.84 million H100 GPU-hours, which is months even on a cluster of more than 16,000 GPUs. Can I train a useful model on a single GPU? Yes, if it is small or a fine-tune. Karpathy notes his GPT-2 124M run would take roughly 4 to 24 hours on a single GPU, depending on the card. Is a faster GPU always worth it for training time? Not always. A GPU that costs twice as much per hour but runs the job 2.5 times faster is cheaper overall, and one that runs it 1.5 times faster is not. Compare per-step time on your own model. Nodus managed training runtimes publish measured step times for their presets, which is what their estimates use. Does training time include data preparation? No, none of the figures here do. Tokenizing and cleaning a large dataset is usually CPU work that you can finish before you pay for any GPU time. Originally published on the Nodus blog.

How much does it cost to train an LLM from scratch? Training an LLM from scratch costs from a few hundred dollars for a small model to tens of millions for a frontier model, and almost all of it is GPU-hours times the hourly rate. At the $2.60/hr Nodus lists for an NVIDIA H100 on its pricing page, a compute-optimal 1B model costs about $292 and a 7B model about $14,300 in compute. The rest of this post shows the formula, checks it against what Meta and DeepSeek actually published, and lists the costs the headline numbers leave out. How do you estimate what an LLM training run will cost? You need three numbers: model size, training tokens and how much of the GPU's peak speed you will really get. Training a transformer takes about 6 floating-point operations per parameter per token, the rule from Kaplan et al.'s scaling-laws paper. So a run needs roughly 6 × parameters × tokens operations in total. An NVIDIA H100 SXM peaks at about 989 teraFLOPS of dense BF16 math. NVIDIA's H100 page lists 1,979 teraFLOPS, but that figure assumes sparsity, so halve it for normal training. No real run gets the full peak. The share you do get is called utilization, and it is the number that moves your budget most. Put together, the estimate is: GPU-hours = (6 × parameters × tokens) / (989e12 × utilization × 3,600) cost = GPU-hours × hourly rate At 30% utilization, one H100-hour delivers about 1.07 × 10^18 useful operations. Every cost in this post comes from that line and a rate. How many tokens should a model train on? A common starting point is about 20 tokens per parameter. DeepMind's Chinchilla paper found that model size and training tokens should grow together for the best model at a fixed compute budget, which works out to roughly that ratio: Chinchilla itself paired 70 billion parameters with 1.4 trillion tokens. A 1B model gets 20 billion tokens and a 7B model gets 140 billion. Most open models now train far past that point. Meta trained every Llama 3.1 model on about 15 trillion tokens, which is over 1,800 tokens per parameter for the 8B model. Extra tokens cost more to train but give a better small model that is cheaper to serve. That choice can multiply your bill by ten or more, so decide it before you price anything. What did real LLM training runs actually cost? Published GPU-hour counts are the best reality check, and they run from about 1.5 million to over 30 million for recent open models. Meta's Llama 3.1 model card lists the H100-80GB hours for each size. DeepSeek's V3 technical report lists 2.788 million H800 GPU-hours for its full training, which it prices at $5.576 million assuming $2 per GPU-hour. | Model | Published GPU-hours | At $2.60/GPU-hour | | --- | ---: | ---: | | Llama 3.1 8B | 1.46M H100 | $3.80M | | Llama 3.1 70B | 7M H100 | $18.2M | | Llama 3.1 405B | 30.84M H100 | $80.2M | | DeepSeek-V3 (37B active) | 2.788M H800 | $7.25M | GPU-hours from the Llama 3.1 model card and the DeepSeek-V3 report; Nodus H100 list rate checked October 30, 2026. The last column is for scale only: runs this size need thousands of GPUs held for weeks, which is a reserved-capacity contract, not an hourly bill. !Published GPU-hours: Llama 3.1 8B 1.46 million, DeepSeek-V3 2.788 million, Llama 3.1 70B 7.0 million, Llama 3.1 405B 30.84 million. DeepSeek-V3 used fewer GPU-hours than Llama 3.1 70B because only 37B of its parameters are active per token. DeepSeek-V3 is a mixture-of-experts model, so each token only touches 37 billion of its parameters. That is why its compute bill sits between the 8B and 70B dense Llama models even though it is much larger in total. What utilization should you plan for? Plan for 30% unless you have measured better on your own code. We back-solved the effective utilization of each published run from its parameters, tokens and GPU-hours, using the formula above. The results were about 14% for Llama 3.1 8B, 25% for Llama 3.1 70B and 33% for Llama 3.1 405B. The best small-scale result we found is higher. Andrej Karpathy's llm.c reproduction of the 1.6B GPT-2 trained on 33.6 billion tokens in 24 hours on one 8 × H100 node. That is 192 GPU-hours, which back-solves to about 46% utilization, close to the roughly 50% he reports. !Effective utilization back-solved from published runs: Llama 3.1 8B 14%, Llama 3.1 70B 25%, Llama 3.1 405B 33%, llm.c GPT-2 1.6B 46%. The same formula on four real runs: plan for 30% and treat 45% as a well-tuned best case. The gap between running speed and the full bill is the useful part. The Llama 3 paper reports 38% to 43% utilization while the 405B model was training, yet the whole GPU-hour count back-solves to 33%. Over a 54-day stretch of that run, Meta logged 466 job interruptions, 419 of them unexpected. Restarts, redone work and idle time all land on the bill, and that is why we plan at 30%, not at peak. How much does it cost to train a small LLM from scratch? A small model is cheap enough to train for learning or for a narrow domain. Here is the formula at 30% utilization and the Nodus H100 list rate of $2.60/hr. | Model and tokens | Total operations | H100 GPU-hours | Compute cost | | --- | ---: | ---: | ---: | | 1B on 20B tokens | 1.2 × 10^20 | 112 | $292 | | 7B on 140B tokens | 5.9 × 10^21 | 5,505 | $14,313 | | 7B on 2T tokens | 8.4 × 10^22 | 78,643 | $204,471 | | 70B on 1.4T tokens | 5.9 × 10^23 | 550,500 | $1.43M | Our arithmetic from 6 × parameters × tokens, 989 teraFLOPS dense BF16, 30% utilization and $2.60 per H100-hour (checked October 30, 2026). To check the method against a real bill, take the llm.c GPT-2 run. Karpathy priced its 192 H100-hours at $672, which is $3.50 per GPU-hour. At $2.60, the same 192 GPU-hours cost $499.20. If your code reaches 45% instead of 30%, every row above falls by a third. What costs do these numbers leave out? The formula prices one clean run, and real projects are never one clean run. DeepSeek says plainly that its $5.576 million covers only the official training run and excludes prior research and ablation experiments on architecture, algorithms and data. Expect to spend real money on small test runs before the big one. Four other costs show up on most projects: - Lost work after an interruption. If an 8-node, 64-GPU run loses 6 hours of progress, that is 384 GPU-hours, or $998.40 at $2.60. With a checkpoint no more than 30 minutes old, the same failure costs 32 GPU-hours, or $83.20. - Idle GPUs. A node that sits waiting for data, a fix or a person bills the same as one that trains. - Data preparation and evaluation. Cleaning and tokenizing a large corpus and running benchmarks takes its own compute. - Storage for checkpoints and datasets, which grows with model size. The first two are the ones you control with tooling. That is where most of the gap between the formula and the final invoice comes from. Should you train from scratch at all? Usually not. Train from scratch only when you need your own architecture or tokenizer, a model with no inherited weights, or research results about pretraining itself. For most teams, continuing to pretrain an open model on your own text, or fine-tuning it, gets a better model for a small fraction of the cost. Our own continued-pretraining benchmark shows the scale difference. Continuing to train Qwen2.5-0.5B on raw text on one NVIDIA L4 cut held-out perplexity by 21.8% for a recorded charge of $0.121164. That was a small pilot on a tiny model. Still, it shows the order of magnitude: well under a dollar to adapt a small model, against hundreds to train even a 1B model from nothing. If you are fine-tuning instead, the cheapest way to fine-tune Llama on an H100 walks through that cost. How do you run a from-scratch pretraining job on Nodus? Nodus runs pretraining as a TrainingJob with task: Pretrain and initialization: Scratch, which keeps only the model's architecture and tokenizer. This manifest follows the training guide, with a spending cap and two 8-GPU nodes added: apiVersion: nodus.dev/v1beta1 kind: TrainingJob metadata: name: scratch-pretrain spec: runtime: nodus/sft task: Pretrain model: {uri: hf://Qwen/Qwen3-0.6B@c1899de289a04d12100db370d81485cdf75e47ca} data: {volume: corpus, format: JSONL} parameters: {steps: 10000, initialization: Scratch, packing: true} resources: {gpu: {type: [h100-sxm-80g], count: 8}, nodes: 2} maxCostUSD: "500.00" Price it before anything runs: $ nodus apply -f scratch-pretrain.yaml --dry-run -o yaml $ nodus apply -f scratch-pretrain.yaml The dry run's estimate is the number of steps times the measured seconds per step on the slowest GPU type you allowed. maxCostUSD caps the whole run, including retries; when it is reached the run is suspended, not failed, and raising the cap resumes it from its checkpoint. If a node is lost, the gang restarts from the latest checkpoint every rank committed, up to three times. Multi-node TrainingJobs are in Beta and go up to 8 nodes, as the multi-node guide explains. FAQ How much did it cost to train GPT-4? OpenAI has not published GPT-4's training compute or cost, so any figure you see is an outside estimate. The best public anchors are open models that publish GPU-hours, such as Llama 3.1 405B at 30.84 million H100-hours. Can you train an LLM on one GPU? Yes, for small models. Karpathy notes the 1.6B GPT-2 run would take about 8 days on one GPU instead of 1 day on eight, for the same GPU-hours. Nodus lists prices per GPU-hour, so the GPU-hours, not the GPU count, set the compute cost. Is a B200 cheaper than an H100 for pretraining? Only if it finishes the job more than proportionally faster. Nodus lists the NVIDIA B200 at $5.50/hr against $2.60/hr for the H100, so the B200 needs over 2.1 times the H100's throughput on your model to cost less per run. Measure seconds per step on both before you commit. Does a mixture-of-experts model cost less to train? Per token, yes, because compute scales with active parameters, not total ones. DeepSeek-V3 has 37B active parameters and needed 2.788 million GPU-hours, fewer than the dense Llama 3.1 70B at 7.0 million. You still need enough GPU memory to hold all the experts. Originally published on the Nodus blog.

Change 3 letters and your AI reads a whole GitHub repo (free) Change three letters in a GitHub link (hub becomes ingest) and your AI gets the whole repository as one text. Gitingest is an open-source tool (MIT) by Coderamp that turns any Git repository into text ready to paste into ChatGPT, Claude or any AI: a summary with estimated tokens, the folder tree and the content of every file. In the video we try it on karpathy/nanoGPT at gitingest.com and with its CLI in a container. 📦 Find it on GitHub: coderamp-labs/gitingest ⚙️ Commands: Install: pipx install gitingest Repo → digest.txt: gitingest karpathy/nanoGPT Only Python files: gitingest karpathy/nanoGPT -i '*.py' -o py.txt See options: gitingest --help Update: pipx upgrade gitingest Uninstall: pipx uninstall gitingest ⚠️ On gitingest.com the repo is processed on their servers; with the CLI it runs on your computer. Private repos need a GitHub token (--token or GITHUB_TOKEN). Token counts are estimates. 💬 Questions? Ask me in the comments. 🔔 Subscribe: one useful repo every day. Data as of Oct 25, 2026 · Not affiliated with the project. #ai #github #opensource #coding #freetools

LLM Council (free, open source): karpathy/llm-council Anthropic's study: anthropic.com

Cambia 3 letras y tu IA lee un repo de GitHub completo (gratis) Cambia tres letras en un enlace de GitHub (hub por ingest) y tu IA recibe el repositorio completo en un solo texto. Gitingest es una herramienta de código abierto (licencia MIT) de Coderamp que convierte cualquier repositorio de Git en un texto listo para pegar en ChatGPT, Claude u otra IA: resumen con tokens estimados, árbol de carpetas y contenido de cada archivo. En el video la probamos con karpathy/nanoGPT en gitingest.com y con su CLI en un contenedor. 📦 Búscalo en GitHub: coderamp-labs/gitingest ⚙️ Comandos: Instalar: pipx install gitingest Repo → digest.txt: gitingest karpathy/nanoGPT Solo archivos de Python: gitingest karpathy/nanoGPT -i '*.py' -o py.txt Ver opciones: gitingest --help Actualizar: pipx upgrade gitingest Desinstalar: pipx uninstall gitingest ⚠️ En gitingest.com el repo se procesa en sus servidores; con el CLI se procesa en tu computadora. Para repos privados necesitas un token de GitHub (--token o GITHUB_TOKEN). Los tokens son estimados. 💬 ¿Dudas? Pregúntame en los comentarios. 🔔 Suscríbete: un repo útil cada día. Datos al 25 oct 2026 · No afiliado al proyecto. #ia #github #opensource #programacion #herramientasgratis