meta-llama

GSM8K on Hugging Face: 4 of the Top 12 Rows Show a Sibling Checkpoint's Score, and 20 of 28 Rows Are Unmerged PRs TL;DR - On November 3, 2026 the Hugging Face GSM8K board lists 28 rows from 28 repos. 20 come from pull requests that are still open, 8 sit on the repo's main branch, and 0 are verified. - Four of the top 12 rows show a number that the model card reports for a different checkpoint of the same family. Rank 1 (99.6) is the MiMo-V2.5-Pro Base score, registered on the post-trained repo. Rank 2 (96.8) is the Llama 3.1 405B Instruct score, registered on the base repo. Ranks 7 and 12 are DeepSeek base scores on the post-trained repos. - The numbers are real self-reports. The labels are what is off, and the board has no column that would show it. - GSM8K has 1,319 test problems, so one problem is 0.076 points. The rank 2 to rank 3 gap (0.74) is 10 problems, and a 95% interval at 95% accuracy is about 1.2 points wide on each side. - A score of 99.6 means about 5 misses. The GSM8K-Platinum project relabeled 10 answers in the same test set. If those relabels are right, a score that high implies the model matched at least 5 of the 10 keys they call wrong. !Top 12 rows of the GSM8K board on Hugging Face, with the four rows whose number belongs to a sibling checkpoint in red In the chart, red marks rows where the model card gives this number for a sibling checkpoint, blue marks a row whose card table matches the repo (LLaDA2.0-flash), and grey marks rows I did not check against the card. What does the GSM8K board on Hugging Face contain? GSM8K is a set of grade-school math word problems with a train split of 7,473 and a test split of 1,319. Hugging Face lists openai/gsm8k as a benchmark dataset, so a model repo can publish a result in a .eval_results/*.yaml file and the dataset's board collects it. I pulled the board on November 3, 2026 (KST) from the JSON behind the leaderboard view. It has 28 rows. Each row is from a different repo, verified is false on every row, and the top of the page reads like a clean ranking: 99.6, 96.8, 96.06, 94.16, 93.93. The dataset's own eval.yaml describes a specific protocol: an inspect-ai task, a zero-shot prompt template that asks for a boxed final answer, four epochs reduced to pass@1, and a model-graded scorer. The rows do not follow one protocol. Of the 28 notes, 15 are empty. Where a note gives a shot count it ranges across 0, 4, 5 and 8. Two rows from one lab give a complete setup (8-shot, temperature 0, n=1319, lm-evaluation-harness), and one row reports a strict-match score over 1,075 of the 1,319 problems. So I treat each row as a claim with a source link, not as a measurement the board produced. Why does rank 1 belong to a different checkpoint? The top row is XiaomiMiMo/MiMo-V2.5-Pro at 99.6. Its model card has a section titled "Base Model Evaluation", and the GSM8K row there reads 8-shot with these values: 99.6 for MiMo-V2.5-Pro Base, 83.3 for MiMo-V2.5 Base, 92.6 for DeepSeek-V4-Pro Base. That is the only place GSM8K appears in the card. The card's own download table lists two repos: MiMo-V2.5-Pro and MiMo-V2.5-Pro-Base, and the second one exists on the Hub. So the board shows the base-model score on the post-trained repo. The pull request that registered it says the results "were extracted from the model card". It was opened on April 27, 2026 and is still open. Nothing in the YAML says "base", and the 8-shot setting is not in the note, which is empty. The card is not hiding anything. It labels its table as Base. The distinction is lost only in the step that turns a card into a board row. Is that one slip, or does it repeat? I checked the sources for the top rows. It repeats in four of the top 12, and in both directions. | Board rank | Repo on the board | Score | Where the card puts this number | Sibling repo | |---|---|---|---|---| | 1 | MiMo-V2.5-Pro | 99.6 | "Base Model Evaluation" table, Pro Base column, 8-shot | MiMo-V2.5-Pro-Base | | 2 | Llama-3.1-405B | 96.8 | "Instruction tuned models" table, 405B Instruct column, 8-shot CoT, em_maj1@1 | Llama-3.1-405B-Instruct | | 7 | DeepSeek-V4-Pro | 92.6 | Base table, DeepSeek-V4-Pro-Base column, 8-shot | DeepSeek-V4-Pro-Base | | 12 | DeepSeek-V3 | 89.3 | "Base Model" section, DeepSeek-V3 column, 8-shot | DeepSeek-V3-Base | Rank 2 is the mirror image. The Llama 3.1 model card covers every size and both variants. Its "Base pretrained models" table has no GSM8K row. Its "Instruction tuned models" table has one, and the 405B Instruct value is 96.8. The pull request is titled "Add GSM8K eval result (96.8, 8-shot CoT)" and attaches that number to the base repo meta-llama/Llama-3.1-405B. For DeepSeek-V3 the card has a Base Model section with GSM8K at 89.3 and a Chat Model section with no GSM8K row. For DeepSeek-V4-Pro the table header names the three columns DeepSeek-V3.2-Base, DeepSeek-V4-Flash-Base and DeepSeek-V4-Pro-Base, and 92.6 is the last one. Why does it matter if the number is the same? Because the board is read by repo name. Someone choosing a model for math word problems sees MiMo-V2.5-Pro at the top and downloads that repo. The card reports no GSM8K number for that checkpoint. If the four rows were moved to the sibling repos, the post-trained MiMo, DeepSeek-V4-Pro and DeepSeek-V3 would have no GSM8K row at all, and the Llama 405B base repo would lose its rank 2. The correct placement would not change the ordering of the numbers: 99.6 would still lead, from the right repo. It would change which model each number tells you about. !Where the 28 GSM8K rows come from: open pull requests, main branch, verified Where do the rows come from, and what does "verified" mean here? I fetched the discussion for every row that carries a pullRequest number, and listed .eval_results on the main branch of every repo. | Source | Rows | |---|---| | Open PR from an account flagged as Hugging Face (julien-c 9, nielsr 6, SaylorTwift 3, burtenshaw 1) | 19 | | Open PR from a bot account (EvalEvalBot) | 1 | | File on the repo's main branch | 8 | | Verified | 0 | All 20 pull requests are open, and for all 20 the YAML file is absent from main. For all 8 other rows it is present. Four of the 20 pull requests were opened on November 2, one day before I pulled the board, and one of them produced rank 3 (LLaDA2.0-flash, 96.06). The board moves when someone opens a pull request, with no action from the model's owner. The Hugging Face documentation defines the badges. "Community" means a result submitted via an open PR that is not merged to main. "Verified" means a valid verify token, which requires the evaluation to have run in HF Jobs with inspect-ai. So 20 of 28 rows are community results by definition, and no row meets the verified bar. The documentation says community scores are visible while the PR is open. That is a reasonable design for bootstrapping a board. It also means a reader has to look at the badge to know which rows the model's owner stands behind, and a screenshot of the ranking does not show it. The eight main-branch rows are not uniformly clean either. Two Mellum2 repos show the same 81.73. The YAML on the Base repo states that its numbers were measured on the pre-extension Base-Pretrain checkpoint, so it is one measurement displayed twice. This one is disclosed in the file, and it still occupies two ranks. How much can 1,319 problems separate? GSM8K is nearly saturated at the top, so the ordering is decided by a handful of problems. | Quantity | Value | |---|---| | One problem | 0.076 points | | Rank 1 to rank 2 gap (99.6 to 96.8) | 2.8 points, about 37 problems | | Rank 2 to rank 3 gap (96.8 to 96.06) | 0.74 points, about 10 problems | | Spread of ranks 3 to 6 (96.06 to 93.10) | 2.96 points | | 95% half-width at 95% accuracy, n=1,319 | 1.18 points (illustrative) | | 95% half-width at 99% accuracy, n=1,319 | 0.54 points (illustrative) | The interval rows are a sampling calculation for a single run on a fixed test set. They ignore prompt variation, which is larger when rows use different shot counts. Even so, ranks 3 to 6 sit inside a band that a single run cannot separate, and the rows within it come from different protocols: 8-shot CoT for some, no note for others. One other check is exact arithmetic. The two rows with a full setup report 84.9886 and 78.4685, and with n=1319 these are 1,121 and 1,035 correct answers. ProKope-7B's note says 1,075 of 1,319, which is 81.50. Those rows can be audited back to a count. Most rows cannot. !How big is each effect: one problem, intervals, ceiling lost to relabeled keys, gaps between ranks Can a 99.6 be right if some answer keys are wrong? This part is arithmetic with one assumption, so I mark it as a question to check, not a finding. The GSM8K-Platinum project (MIT, Gradient Science) reviewed the 1,319 test problems. Its dataset card reports 219 flagged by models, 110 rejected as poorly written, 10 relabeled, 99 verified, leaving 1,209 questions. The 10 relabeled items are answer keys that the authors found to be wrong. Take a score of 99.6 on the original test set. 1,314 of 1,319 rounds to 99.6, so that is about 5 misses. Suppose the 10 relabels are correct and scoring is exact match against the original key. A model that solves an item correctly then loses it, because the key is wrong. A model that is right on all 10 would lose all 10 and top out at 99.24. To score 99.6 it must have matched the key on at least 5 of the 10, which means it gave the answer the key says, not the answer the authors say is right. Three explanations fit: the relabels are wrong for some of those items, the scoring is more lenient than exact match, or the model has seen the original keys. I have not run the model, and a base model's 8-shot number with no note on answer extraction does not let me tell these apart. What I can say is that for a benchmark with known key errors, a score above the ceiling is a reason to read the setup, not a reason to rank it first. Platinum also reports that cleaning the set separates models that look tied on the original, which is the same point from the other side. How do I pull the board and classify rows myself? This script needs only the standard library. It counts rows by source and prints the sampling arithmetic used above. import json, urllib.request, collections def get(url): req = urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0"}) return json.load(urllib.request.urlopen(req, timeout=60)) DATASET = "openai/gsm8k" rows = get(f"https://huggingface.co/api/datasets/{DATASET}/leaderboard") by_source = collections.Counter() for r in rows: pr = r.get("pullRequest") if pr: disc = get(f"https://huggingface.co/api/models/{r['modelId']}/discussions/{pr}") by_source[("open PR" if disc["status"] == "open" else disc["status"])] += 1 else: by_source["main branch"] += 1 print("rows:", len(rows), "| verified:", sum(bool(r["verified"]) for r in rows)) print(dict(by_source)) n = 1319 print("one problem = %.3f points" % (100 / n)) for acc in (0.95, 0.99): se = (acc * (1 - acc) / n) ** 0.5 * 100 print("accuracy %.2f: 95%% half-width = %.2f points" % (acc, 1.96 * se)) print("ceiling if 10 keys are wrong: %.2f" % (100 * (n - 10) / n)) On November 3, 2026 it printed: rows: 28 | verified: 0 {'open PR': 20, 'main branch': 8} one problem = 0.076 points accuracy 0.95: 95% half-width = 1.18 points accuracy 0.99: 95% half-width = 0.54 points ceiling if 10 keys are wrong: 99.24 The board endpoint is undocumented, so the field names may change. To check a row, open its pullRequest discussion, read the YAML, then open the source card and find the table the number came from. Checklist for reading a saturated board 1. Open the badge before the rank. Community means an open PR, and verified means a run in HF Jobs. Neither is a statement from the model's owner. 2. Match the checkpoint, not the family. Find the card table the number came from and read its column header. Base, instruct and chat checkpoints sit in different tables. 3. Ask how many problems separate two rows. On GSM8K, 10 problems is 0.76 points. 4. Prefer rows that state their setup. Shot count, temperature, n and the answer-extraction rule decide the score more than rank does. 5. Check for the same measurement on two repos. A copied number is one result, not two. 6. Distrust a score above the key ceiling. If a benchmark has known label errors, read the evaluation setup before you trust the lead. FAQ How many GSM8K results on Hugging Face are verified? None. On November 3, 2026 all 28 rows on the board show verified as false. Verified requires a valid verify token from an inspect-ai run in HF Jobs. Is the 99.6 at rank 1 of the GSM8K board wrong? Nothing here shows the number is wrong. It is the MiMo-V2.5-Pro Base score from the model card's Base Model table, 8-shot. The problem is that it is registered on the post-trained repo, which has its own separate row in the card's download table. What does "community" mean on a Hugging Face leaderboard row? It means the result was submitted through an open pull request that has not been merged to the repo's main branch. In the GSM8K board, 20 of 28 rows are in that state. Why is the Llama 3.1 405B base repo ranked second on GSM8K? Its row carries 96.8, which the shared model card gives for Llama 3.1 405B Instruct, 8-shot CoT. The card has no GSM8K row in its base table. The instruct repo is a separate repo. How many problems does one point of GSM8K accuracy represent? GSM8K has 1,319 test problems, so one point is about 13 problems and one problem is 0.076 points. What is GSM8K-Platinum? It is a revised version of the GSM8K test set from Gradient Science. Of 1,319 test problems it rejected 110, relabeled 10 and kept 1,209. It exists because the original set has ambiguous questions and wrong keys. Further reading - GPQA Diamond Has 198 Questions: What the Hugging Face Leaderboard's 115 Rows Can and Cannot Rank - How Hugging Face Official Benchmark Leaderboards Actually Work: .eval_results YAML, the base_model Filter, and the 30% the Default View Hides Sources - GSM8K dataset and board: huggingface.co - GSM8K evaluation definition (eval.yaml): huggingface.co - Hugging Face documentation, Evaluation Results (badges, community PRs): huggingface.co - GSM8K-Platinum dataset card: huggingface.co - GSM8K-Platinum write-up: gradientscience.org - Rank 1 model card (Base Model Evaluation table): huggingface.co - Rank 1 registering pull request: huggingface.co - Llama 3.1 model card (base and instruction tuned tables): github.com/meta-llama/llama-models/blob/main/mode… - Rank 2 registering pull request: huggingface.co - DeepSeek-V4-Pro model card: huggingface.co - DeepSeek-V3 model card: huggingface.co All counts in this article are my own measurements from the board JSON, pull request discussions, dataset cards and the cited model cards on November 3, 2026 (KST); I have no affiliation with any model, lab or benchmark named here. Which table a number came from is my reading of each card, and the board endpoint is undocumented and may change.

How long does it take to train an AI model on cloud GPUs? Training an AI model takes anywhere from a few minutes to several months, depending on model size, how many tokens it trains on and how many GPUs run in parallel. On Nodus a small fine-tune can finish in under five minutes, while by our math a 7B language model trained from scratch on 140 billion tokens takes about 3.6 days on 64 NVIDIA H100s. The sections below show the formula, check it against published runs, and list what makes the real clock run longer than the math. How do you estimate how long a training run will take? Work out the total math the run needs, then divide by how fast your GPUs actually do math. Training a transformer takes about 6 floating-point operations per parameter per token, the rule from Kaplan et al.'s scaling-laws paper. An NVIDIA H100 SXM peaks at about 989 teraFLOPS of dense BF16 math, half the sparse figure on NVIDIA's H100 page. No real run gets the full peak. The share you get is called model FLOPs utilization, and 30% is a fair planning number for code you have not tuned. That gives: GPU-hours = (6 × parameters × tokens) / (989e12 × utilization × 3,600) wall-clock = GPU-hours / number of GPUs (if scaling were perfect) At 30% utilization, one H100-hour delivers about 1.07 × 10^18 useful operations. The second line is the optimistic part, and we come back to it below. How long does it take to train a language model from scratch? Days to weeks for small and mid-size models, if you have enough GPUs. Here is the formula at 30% utilization, using a compute-optimal budget of about 20 tokens per parameter from DeepMind's Chinchilla paper, plus one longer 7B run. | Model and tokens | H100 GPU-hours | Fewer GPUs | More GPUs | | --- | ---: | ---: | ---: | | 1B on 20B tokens | 112 | 1 GPU: 4.7 days | 8 GPUs: 14 hours | | 7B on 140B tokens | 5,505 | 8 GPUs: 28.7 days | 64 GPUs: 3.6 days | | 7B on 2T tokens | 78,643 | 64 GPUs: 51.2 days | 512 GPUs: 6.4 days | | 70B on 1.4T tokens | 550,500 | 512 GPUs: 44.8 days | 2,048 GPUs: 11.2 days | Our arithmetic: 6 × parameters × tokens, 989 teraFLOPS dense BF16, 30% utilization, perfect scaling. Real runs land later; see the overheads section. Token count moves the clock as much as model size. The same 7B model takes 14 times longer on 2 trillion tokens than on 140 billion. Many open models now train far past the Chinchilla ratio, so settle the token budget first. Our post on what it costs to train an LLM prices the same GPU-hours in dollars. How long do real published training runs take? Small reproductions take hours, and frontier models take months. Andrej Karpathy's llm.c reproduction of GPT-2 124M trained on 10 billion tokens in about 90 minutes on one 8 × NVIDIA A100 80GB node. His GPT-2 1.6B run took about 24 hours on one 8 × H100 node: 32,000 steps of roughly 1 million tokens each, at about 2.75 seconds per step. That 1.6B run is a good check on the formula. It works out to 192 GPU-hours for about 33.6 billion tokens, which back-solves to roughly 46% utilization. Karpathy's own logs show 42% to 47% at the start, so well-tuned code beats our 30% planning figure by about half. At the top end, Meta's Llama 3.1 model card lists 30.84 million H100 GPU-hours for the 405B model. The Llama 3 paper says it trained on up to 16,384 H100s. Divide one by the other and even that cluster needs about 78 days of pure compute, a floor on the real calendar time. Do more GPUs make training proportionally faster? Only roughly, and never for free. Adding GPUs splits the same GPU-hours across more machines, so the bill stays about the same while the calendar shrinks. In practice the bill creeps up, because GPUs spend more time talking to each other. Meta's own numbers show the effect. The Llama 3 paper reports 43% utilization on 8,192 GPUs and 41% on 16,384 GPUs for the same model, a small but real loss from doubling the cluster. !Days to train a 7B model on 2 trillion tokens: 409.6 days on 8 H100s, 51.2 days on 64, 6.4 days on 512, at 30% utilization and perfect scaling. The same 78,643 GPU-hours: one 8-GPU node needs 409.6 days, a 512-GPU cluster about a week. Our decision rule: pick the smallest GPU count that finishes before your deadline, not the largest you can get. Single nodes of 8 GPUs scale best because the GPUs share fast NVLink links. Multi-node runs add network cost and more machines that can fail. How long does fine-tuning take? Usually minutes to a few hours, because fine-tuning sees far fewer tokens than pretraining. Fine-tuning a 7B model on 50 million tokens is about 2 H100-hours of math at 30% utilization by the same formula, and LoRA does less than that because it skips most weight gradients. For small jobs, the work around training often takes longer than training itself. In our published fine-tuning benchmark, Qwen2.5-0.5B on one NVIDIA L4 spent 112.8 seconds in the training loop on Nodus. From submission to a verified exported model took 4.8 minutes, or 288 seconds. !Small fine-tune on Nodus: 112.8 seconds in the training loop versus 288 seconds from submission to exported model. For a small fine-tune, startup, evaluation and export took longer than the training loop. So for fine-tuning, the honest answer to "how long" is mostly startup, model download, evaluation and export. On Nodus, a TrainingJob imports a pinned Hugging Face model revision into a read-only Volume once, and later runs reuse it, so billed GPU time is not spent downloading weights. Why does the real clock run longer than the math? Because the formula counts only useful training steps. Four things add time on most projects: - Waiting for capacity. A large multi-GPU request may queue until enough machines are free together. - Startup. Pulling the image, loading the model and reading the first batch of data. - Failures and restarts. Over a 54-day stretch of Llama 3 405B pretraining, Meta logged 466 job interruptions, 419 of them unexpected. It still reported more than 90% effective training time, meaning at least a tenth of the elapsed time did no useful training. - Evaluation and export. Validation passes and writing out the final weights. Our planning rule is to add at least 10% to the formula for multi-day runs. Go higher if your checkpoints are far apart, since every restart redoes the work since the last save. How does Nodus show the time before you launch? A dry run returns the estimate before anything is charged. For a managed TrainingJob, the training guide says the estimate is the number of steps times the measured seconds per step on the slowest GPU type you allowed, plus evaluation time. You can set spec.expectedDuration to use your own figure. $ nodus apply -f sft.yaml --dry-run -o yaml # status.estimate: cost to completion, first hold, start time For your own script, nodus run takes the same idea from flags. This prints the estimate only, using your expected duration: $ nodus run --dry-run --gpu H100:8 --expected-duration 24h -- torchrun train.py If the run must finish by a date, --complete-by (or placement.completeByTime) keeps it off any capacity whose p90 finish is later, per our placement docs. H100s are listed at $2.60/hr on the Nodus pricing page, checked October 31, 2026. If a machine is lost mid-run, Nodus starts a new attempt from the latest committed checkpoint, as described in attempts and recovery. Our post on how often spot GPUs are interrupted shows how much that matters on cheaper capacity. FAQ How long does it take to train a model like GPT-4? Nobody outside the lab knows, because the GPU-hours were never published. The closest public figure is Llama 3.1 405B at 30.84 million H100 GPU-hours, which is months even on a cluster of more than 16,000 GPUs. Can I train a useful model on a single GPU? Yes, if it is small or a fine-tune. Karpathy notes his GPT-2 124M run would take roughly 4 to 24 hours on a single GPU, depending on the card. Is a faster GPU always worth it for training time? Not always. A GPU that costs twice as much per hour but runs the job 2.5 times faster is cheaper overall, and one that runs it 1.5 times faster is not. Compare per-step time on your own model. Nodus managed training runtimes publish measured step times for their presets, which is what their estimates use. Does training time include data preparation? No, none of the figures here do. Tokenizing and cleaning a large dataset is usually CPU work that you can finish before you pay for any GPU time. Originally published on the Nodus blog.

How much does it cost to train an LLM from scratch? Training an LLM from scratch costs from a few hundred dollars for a small model to tens of millions for a frontier model, and almost all of it is GPU-hours times the hourly rate. At the $2.60/hr Nodus lists for an NVIDIA H100 on its pricing page, a compute-optimal 1B model costs about $292 and a 7B model about $14,300 in compute. The rest of this post shows the formula, checks it against what Meta and DeepSeek actually published, and lists the costs the headline numbers leave out. How do you estimate what an LLM training run will cost? You need three numbers: model size, training tokens and how much of the GPU's peak speed you will really get. Training a transformer takes about 6 floating-point operations per parameter per token, the rule from Kaplan et al.'s scaling-laws paper. So a run needs roughly 6 × parameters × tokens operations in total. An NVIDIA H100 SXM peaks at about 989 teraFLOPS of dense BF16 math. NVIDIA's H100 page lists 1,979 teraFLOPS, but that figure assumes sparsity, so halve it for normal training. No real run gets the full peak. The share you do get is called utilization, and it is the number that moves your budget most. Put together, the estimate is: GPU-hours = (6 × parameters × tokens) / (989e12 × utilization × 3,600) cost = GPU-hours × hourly rate At 30% utilization, one H100-hour delivers about 1.07 × 10^18 useful operations. Every cost in this post comes from that line and a rate. How many tokens should a model train on? A common starting point is about 20 tokens per parameter. DeepMind's Chinchilla paper found that model size and training tokens should grow together for the best model at a fixed compute budget, which works out to roughly that ratio: Chinchilla itself paired 70 billion parameters with 1.4 trillion tokens. A 1B model gets 20 billion tokens and a 7B model gets 140 billion. Most open models now train far past that point. Meta trained every Llama 3.1 model on about 15 trillion tokens, which is over 1,800 tokens per parameter for the 8B model. Extra tokens cost more to train but give a better small model that is cheaper to serve. That choice can multiply your bill by ten or more, so decide it before you price anything. What did real LLM training runs actually cost? Published GPU-hour counts are the best reality check, and they run from about 1.5 million to over 30 million for recent open models. Meta's Llama 3.1 model card lists the H100-80GB hours for each size. DeepSeek's V3 technical report lists 2.788 million H800 GPU-hours for its full training, which it prices at $5.576 million assuming $2 per GPU-hour. | Model | Published GPU-hours | At $2.60/GPU-hour | | --- | ---: | ---: | | Llama 3.1 8B | 1.46M H100 | $3.80M | | Llama 3.1 70B | 7M H100 | $18.2M | | Llama 3.1 405B | 30.84M H100 | $80.2M | | DeepSeek-V3 (37B active) | 2.788M H800 | $7.25M | GPU-hours from the Llama 3.1 model card and the DeepSeek-V3 report; Nodus H100 list rate checked October 30, 2026. The last column is for scale only: runs this size need thousands of GPUs held for weeks, which is a reserved-capacity contract, not an hourly bill. !Published GPU-hours: Llama 3.1 8B 1.46 million, DeepSeek-V3 2.788 million, Llama 3.1 70B 7.0 million, Llama 3.1 405B 30.84 million. DeepSeek-V3 used fewer GPU-hours than Llama 3.1 70B because only 37B of its parameters are active per token. DeepSeek-V3 is a mixture-of-experts model, so each token only touches 37 billion of its parameters. That is why its compute bill sits between the 8B and 70B dense Llama models even though it is much larger in total. What utilization should you plan for? Plan for 30% unless you have measured better on your own code. We back-solved the effective utilization of each published run from its parameters, tokens and GPU-hours, using the formula above. The results were about 14% for Llama 3.1 8B, 25% for Llama 3.1 70B and 33% for Llama 3.1 405B. The best small-scale result we found is higher. Andrej Karpathy's llm.c reproduction of the 1.6B GPT-2 trained on 33.6 billion tokens in 24 hours on one 8 × H100 node. That is 192 GPU-hours, which back-solves to about 46% utilization, close to the roughly 50% he reports. !Effective utilization back-solved from published runs: Llama 3.1 8B 14%, Llama 3.1 70B 25%, Llama 3.1 405B 33%, llm.c GPT-2 1.6B 46%. The same formula on four real runs: plan for 30% and treat 45% as a well-tuned best case. The gap between running speed and the full bill is the useful part. The Llama 3 paper reports 38% to 43% utilization while the 405B model was training, yet the whole GPU-hour count back-solves to 33%. Over a 54-day stretch of that run, Meta logged 466 job interruptions, 419 of them unexpected. Restarts, redone work and idle time all land on the bill, and that is why we plan at 30%, not at peak. How much does it cost to train a small LLM from scratch? A small model is cheap enough to train for learning or for a narrow domain. Here is the formula at 30% utilization and the Nodus H100 list rate of $2.60/hr. | Model and tokens | Total operations | H100 GPU-hours | Compute cost | | --- | ---: | ---: | ---: | | 1B on 20B tokens | 1.2 × 10^20 | 112 | $292 | | 7B on 140B tokens | 5.9 × 10^21 | 5,505 | $14,313 | | 7B on 2T tokens | 8.4 × 10^22 | 78,643 | $204,471 | | 70B on 1.4T tokens | 5.9 × 10^23 | 550,500 | $1.43M | Our arithmetic from 6 × parameters × tokens, 989 teraFLOPS dense BF16, 30% utilization and $2.60 per H100-hour (checked October 30, 2026). To check the method against a real bill, take the llm.c GPT-2 run. Karpathy priced its 192 H100-hours at $672, which is $3.50 per GPU-hour. At $2.60, the same 192 GPU-hours cost $499.20. If your code reaches 45% instead of 30%, every row above falls by a third. What costs do these numbers leave out? The formula prices one clean run, and real projects are never one clean run. DeepSeek says plainly that its $5.576 million covers only the official training run and excludes prior research and ablation experiments on architecture, algorithms and data. Expect to spend real money on small test runs before the big one. Four other costs show up on most projects: - Lost work after an interruption. If an 8-node, 64-GPU run loses 6 hours of progress, that is 384 GPU-hours, or $998.40 at $2.60. With a checkpoint no more than 30 minutes old, the same failure costs 32 GPU-hours, or $83.20. - Idle GPUs. A node that sits waiting for data, a fix or a person bills the same as one that trains. - Data preparation and evaluation. Cleaning and tokenizing a large corpus and running benchmarks takes its own compute. - Storage for checkpoints and datasets, which grows with model size. The first two are the ones you control with tooling. That is where most of the gap between the formula and the final invoice comes from. Should you train from scratch at all? Usually not. Train from scratch only when you need your own architecture or tokenizer, a model with no inherited weights, or research results about pretraining itself. For most teams, continuing to pretrain an open model on your own text, or fine-tuning it, gets a better model for a small fraction of the cost. Our own continued-pretraining benchmark shows the scale difference. Continuing to train Qwen2.5-0.5B on raw text on one NVIDIA L4 cut held-out perplexity by 21.8% for a recorded charge of $0.121164. That was a small pilot on a tiny model. Still, it shows the order of magnitude: well under a dollar to adapt a small model, against hundreds to train even a 1B model from nothing. If you are fine-tuning instead, the cheapest way to fine-tune Llama on an H100 walks through that cost. How do you run a from-scratch pretraining job on Nodus? Nodus runs pretraining as a TrainingJob with task: Pretrain and initialization: Scratch, which keeps only the model's architecture and tokenizer. This manifest follows the training guide, with a spending cap and two 8-GPU nodes added: apiVersion: nodus.dev/v1beta1 kind: TrainingJob metadata: name: scratch-pretrain spec: runtime: nodus/sft task: Pretrain model: {uri: hf://Qwen/Qwen3-0.6B@c1899de289a04d12100db370d81485cdf75e47ca} data: {volume: corpus, format: JSONL} parameters: {steps: 10000, initialization: Scratch, packing: true} resources: {gpu: {type: [h100-sxm-80g], count: 8}, nodes: 2} maxCostUSD: "500.00" Price it before anything runs: $ nodus apply -f scratch-pretrain.yaml --dry-run -o yaml $ nodus apply -f scratch-pretrain.yaml The dry run's estimate is the number of steps times the measured seconds per step on the slowest GPU type you allowed. maxCostUSD caps the whole run, including retries; when it is reached the run is suspended, not failed, and raising the cap resumes it from its checkpoint. If a node is lost, the gang restarts from the latest checkpoint every rank committed, up to three times. Multi-node TrainingJobs are in Beta and go up to 8 nodes, as the multi-node guide explains. FAQ How much did it cost to train GPT-4? OpenAI has not published GPT-4's training compute or cost, so any figure you see is an outside estimate. The best public anchors are open models that publish GPU-hours, such as Llama 3.1 405B at 30.84 million H100-hours. Can you train an LLM on one GPU? Yes, for small models. Karpathy notes the 1.6B GPT-2 run would take about 8 days on one GPU instead of 1 day on eight, for the same GPU-hours. Nodus lists prices per GPU-hour, so the GPU-hours, not the GPU count, set the compute cost. Is a B200 cheaper than an H100 for pretraining? Only if it finishes the job more than proportionally faster. Nodus lists the NVIDIA B200 at $5.50/hr against $2.60/hr for the H100, so the B200 needs over 2.1 times the H100's throughput on your model to cost less per run. Measure seconds per step on both before you commit. Does a mixture-of-experts model cost less to train? Per token, yes, because compute scales with active parameters, not total ones. DeepSeek-V3 has 37B active parameters and needed 2.788 million GPU-hours, fewer than the dense Llama 3.1 70B at 7.0 million. You still need enough GPU memory to hold all the experts. Originally published on the Nodus blog.

Exploring Other Large Language Models for Improving Kinyarwanda Text Generation Choosing the Best Open-Source Model for Text Generation 1. Introduction Text generation is an important task in Natural Language Processing (NLP). It allows computers to generate text from a prompt. For example, when we write “The people of Kicukiro say”, a language model can predict what words should come next. In our project, we studied GPT-2 and compared it with a tiny GPT model that our group built from scratch. Our tiny GPT was trained on Kinyarwanda news from the KINNEWS dataset. Our experiments showed that GPT-2 can generate fluent English, but it struggles to continue a sentence correctly in Kinyarwanda. Our tiny GPT generates text that looks more like Kinyarwanda, but its sentences do not always make sense. GPT-2 has 124 million parameters and was trained mainly on English text. Our tiny GPT has about 0.82 million parameters and was trained on approximately 24 million characters of Kinyarwanda news. This showed us that both model size and training data are important for text generation. The next step is to find a better language model that can generate useful text and follow instructions. In this blog, I compare four open-weight models and choose the one that is most suitable for improving our project. 2. Models We Considered Model 1: Qwen3-4B-Instruct-2507 Qwen3-4B-Instruct-2507 was developed by the Qwen team at Alibaba. It has 4 billion parameters and supports a context length of up to 262,144 tokens. It uses the Apache 2.0 licence. This model is designed to follow instructions, understand text, and generate helpful responses. It also has multilingual capabilities and can handle many different tasks. One advantage is that it does not require a separate thinking mode to generate its answers. This makes its output easier to use when we want a direct response. However, it is larger than our original GPT-2 model, so it needs more memory and computing power. We also need to test its Kinyarwanda output instead of assuming that it will always generate correct Kinyarwanda. Model 2: Qwen3-1.7B Qwen3-1.7B is another model from Alibaba's Qwen team. It has approximately 1.7 billion parameters and uses the Apache 2.0 licence. Its standard context length is around 32,768 tokens. Its main advantage is its smaller size compared with Qwen3-4B. This can make it easier to run and faster to test on a computer with limited memory. It can follow instructions and generate text, making it useful for small NLP applications. However, its smaller size may affect the quality of some responses. This model would be a good choice if our main concern was reducing memory use. We would still need to compare its Kinyarwanda generation with the output from the larger model. Model 3: Llama 3.2-3B-Instruct Llama 3.2-3B-Instruct was developed by Meta. It has approximately 3 billion parameters and a context length of up to 128,000 tokens. It uses the Llama 3.2 Community Licence. This model is designed for tasks such as text generation, summarisation, and following instructions. It is relatively small compared with many larger language models. One advantage is that it can generate conversational text and follow user instructions. However, its licence has specific terms, and access to its model files may require accepting Meta's conditions. Its officially supported languages also do not include Kinyarwanda. This means it is worth considering, but its licence and language limitations make it less convenient for our project. Model 4: SmolLM3-3B SmolLM3-3B was developed by Hugging Face and its contributors. It has approximately 3 billion parameters and supports a standard context length of 65,536 tokens, with extensions available for longer contexts. It uses the Apache 2.0 licence. The model is designed for instruction following, text generation, and reasoning. It is also released with information about its training process, which makes it interesting for learning and research. Its size makes it a possible option for running on a GPU with limited memory. However, its documented multilingual support focuses on six languages: English, French, Spanish, German, Italian, and Portuguese. Kinyarwanda is not among them. For our project, we would need to test whether it produces useful Kinyarwanda text before choosing it. 3. Comparison Table | Model | Developer | Parameters | Context length | Licence | Main strength | |---|---|---:|---:|---|---| | Qwen3-4B-Instruct-2507 | Alibaba Qwen team | 4B | 262,144 tokens | Apache 2.0 | Instruction following and multilingual text generation | | Qwen3-1.7B | Alibaba Qwen team | 1.7B | 32,768 tokens | Apache 2.0 | Smaller size and lower memory needs | | Llama 3.2-3B-Instruct | Meta | 3B | 128,000 tokens | Llama Community Licence | Conversational text and instruction following | | SmolLM3-3B | Hugging Face and contributors | 3B | 65,536 tokens by default | Apache 2.0 | Efficient text generation and transparent training details | Context lengths are the models' documented limits. A longer context does not automatically mean better text generation. 4. The Model Chosen I would choose Qwen3-4B-Instruct-2507 to improve our text-generation project. The first reason is its instruction-following ability. We want a model that can receive a prompt and generate a response in the requested format. For example, we could ask it to continue a Kinyarwanda news sentence, produce a short news paragraph, or rewrite text more clearly. The second reason is its multilingual capability. Our notebook showed that GPT-2 struggles with Kinyarwanda partly because its tokenizer breaks many Kinyarwanda words into smaller pieces. The Kinyarwanda example required about 1.7 times as many tokens as the English example. Qwen3 may handle this task better, but we must test it because multilingual support does not guarantee accurate Kinyarwanda. The third reason is that it uses the Apache 2.0 licence and its Hugging Face model page provides access to the model files without the gated-access process required by some other models. We can load it directly in a notebook rather than depending on a paid text-generation API. Can it run on free Google Colab? Qwen3-4B-Instruct-2507 is a realistic model to test on a free Google Colab T4 GPU with 16 GB of GPU memory. Its 4-billion-parameter size makes local inference more practical than using a much larger model. The exact memory use depends on the precision, context length, and generation settings. For our first test, we should use a moderate prompt length and generate a limited number of new tokens. If memory becomes a problem, we can try a quantized version. The model itself does not require a paid API key. However, free Colab GPU access is not guaranteed at all times, and Colab still requires a Google account. We should also check the model's licence terms before using it in a published application. How will it improve our project? We can compare Qwen3 with GPT-2 and our tiny GPT using the same Kinyarwanda prompts. For example, we can use the prompt: "Abaturage bo mu Kicukiro bavuga ko..." We can then compare the generated text for language quality, sentence meaning, grammar, repetition, and whether it follows the prompt. We should keep the prompts and generation settings as similar as possible. We can also test English prompts to compare performance across languages. Our tiny GPT is still valuable because it helps us understand how a transformer works and how training data affects the results. Qwen3 would be an additional model for comparison, not a replacement for the lessons we learned from building our own model. 5. Conclusion Our experiments showed that generating text is not only about having a large model. The model's training data, tokenizer, and ability to follow instructions also affect the result. Qwen3-4B-Instruct-2507 is my preferred option because it combines a manageable model size, instruction following, multilingual capabilities, and an Apache 2.0 licence. It is also practical to test on a free Colab T4 GPU, although actual memory use must be checked. Most importantly, we should not assume that it will automatically produce perfect Kinyarwanda. We need to test it with real Kinyarwanda prompts and compare the results with GPT-2 and our tiny GPT. These tests will help us decide whether it is a useful improvement for our project. 6. References 1. OpenAI. Language Models are Unsupervised Multitask Learners (2019). openai.com 2. Qwen Team. Qwen3-4B-Instruct-2507 model card. Hugging Face. huggingface.co 3. Qwen Team. Qwen3-1.7B model card. Hugging Face. huggingface.co 4. Meta. Llama 3.2 model card and licence information. github.com/meta-llama/llama-models/blob/main/mode… 5. Hugging Face. SmolLM3-3B model card. huggingface.co 6. Hugging Face. Transformers documentation. huggingface.co 7. Vaswani, A. et al. Attention Is All You Need (2017). arxiv.org

@nikks_techie example: Llama 3.1 70B, bf16 KV, no prefix sharing: 320 KiB/token × 8,192 tokens × 100 active sequences = 250 GiB of cache alone. 70B isn't enough to size it. the model's KV heads and the cache dtype matter. github.com/meta-llama/llama-models/blob/main/mode…

What Every AI Engineer (Builder) Needs to Know About the Data Behind AI Table of Contents Chapter 1 of 4, deep dive. Click any line to jump straight to it: - [Introduction](#introduction) - [Module 1.1: The Big Picture: What a Model Actually Learns](#module-11-the-big-picture-what-a-model-actually-learns) - [1.1.1 The One Sentence Everything Builds On](#111-the-one-sentence-everything-builds-on) - [1.1.2 The Pipeline Loop as Three Mathematical Operations](#112-the-pipeline-loop-as-three-mathematical-operations) - [1.1.3 How Big Is "Big"?](#113-how-big-is-big) - [1.1.4 The Quantity, Quality, Diversity Trade-off, Measured](#114-the-quantity-quality-diversity-trade-off-measured) - [Module 1.2: The Mathematics of the Training Objective](#module-12-the-mathematics-of-the-training-objective) - [1.2.1 Entropy, Cross-Entropy and KL Divergence](#121-entropy-cross-entropy-and-kl-divergence) - [1.2.2 Training Is Maximum Likelihood](#122-training-is-maximum-likelihood) - [1.2.3 The Cost of Spam, Derived](#123-the-cost-of-spam-derived) - [1.2.4 Proof in Code: Clean vs Spam-Polluted Training](#124-proof-in-code-clean-vs-spam-polluted-training) - [1.2.5 The ln(V) Sanity Check](#125-the-lnv-sanity-check) - [Module 1.3: What the Model Is Truly Optimizing For](#module-13-what-the-model-is-truly-optimizing-for) - [1.3.1 From Tokens to Loss to Perplexity](#131-from-tokens-to-loss-to-perplexity) - [1.3.2 How the Data Pushes on the Model: the Gradient](#132-how-the-data-pushes-on-the-model-the-gradient) - [1.3.3 Scaling Laws, With Real Numbers](#133-scaling-laws-with-real-numbers) - [1.3.4 Compute-Optimal vs Over-Trained](#134-compute-optimal-vs-over-trained) - [1.3.5 Where the Numbers From the Intro Come Back](#135-where-the-numbers-from-the-intro-come-back) - [Chapter 1 Cheat Sheet](#chapter-1-cheat-sheet) - [References](#references) Chapters 2, 3 and 4 (cleaning the web, teaching filters to think, and the final mile into tokens and batches) publish as their own posts. --- Introduction I have been quiet for a while, not because I had nothing to say, but because I kept turning one question over in my head: how do people like us, without a fraction of the money the big labs spend, actually contribute to building AI that reflects who we are. The honest answer took time to arrive at, and the number that finally pushed me to write this down is too large to sit with quietly. > Did you know? Meta's own published model card for Llama 3.1 states that training the 405-billion-parameter version alone used 30.84 million GPU-hours on H100 chips drawing 700 watts each. Run that math and it comes out to roughly 21.6 million kilowatt-hours of electricity, enough to power about 2,000 average households for a full year. Meta's own figures put the location-based emissions from that one training run at 8,930 tons of CO2, close to what 1,900 cars emit in a year. Spread across the 16,384 GPUs used, that workload still took about 78 days of continuous training, and at a modest $2 per GPU-hour cloud rate, the compute bill alone comes to roughly $62 million, before a single researcher, dataset, or building is paid for. Sit with that $62 million for a second, then sit with this. In 2021, OpenAI contracted a company called Sama to have Kenyan workers label disturbing content so ChatGPT would not reproduce it, paying those workers roughly $1.50 to $2 an hour while internal documents showed OpenAI itself was billed $12.50 an hour for that same work. Separately, Scale AI, the company Alexandr Wang led as CEO until Meta bought 49 percent of it for $14.3 billion in 2025, has faced its own wave of US lawsuits after reports that its Remotasks platform paid some workers as little as a few cents per task. Two different companies, two separate scandals, the same pattern underneath: the humans doing the most psychologically demanding part of building this technology are paid the least, while the compute bill for the model they helped clean runs into the tens of millions. !exploting Kenyan people to label data Who this is for. This series is written for builders, students, self-taught engineers, researchers and curriculum designers anywhere who want to understand how large language models are actually built, not just how to call an API, especially those of us working with limited compute, limited funding and limited institutional backing. You do not need a GPU cluster or a research lab to read this. You need curiosity, a laptop, and the willingness to look closely at data most people scroll past. And while that gap sits unresolved, the people steering this industry are moving at a different altitude entirely. In September 2025, Donald Trump told the United Nations General Assembly that the United States would start calling artificial intelligence "superintelligence," or SI, later signing an executive order requiring federal agencies to use the new term. That is the altitude at which this race is being run, nations rewriting policy and vocabulary around who leads next, while much of the data that trains these systems, our languages, our conversations, our culture, keeps flowing outward for free. This is where the series begins, not with despair at the size of the gap, but with the one part of it we can actually close ourselves. Over the next four chapters we move from the big picture of why training data matters so completely, through where real data comes from and how it gets cleaned, into the filtering and quality techniques that separate a frontier lab's dataset from a pile of scraped pages, and finally into how that data becomes the tokens and batches a model actually trains on. None of it requires a data center. All of it requires showing up. --- How to read this post. Every section follows the same rhythm: the idea in plain words, the math (each equation is followed by a "read it as" sentence), a worked example with real arithmetic you can check with a calculator, and why it matters for data. All code lives in one repository so this post stays readable: !Source of truth > 💻 Companion repo: iws3/before-we-train. Every script in chapter-01/ runs on a laptop CPU with plain Python and NumPy. No GPU needed. Notation used throughout: | Symbol | Meaning | |---|---| | x_t | the token at position t | | x_{<t} | all tokens before position t (the context) | | V | vocabulary size (number of distinct tokens) | | p or p_data | the true distribution of your dataset | | q or q_θ | the model's distribution, with parameters θ | | n(·) | a count, e.g. n(the, cat) is how many times "cat" follows "the" | | N | number of model parameters (used from 1.3.3 on) | | D | number of training tokens (used from 1.3.3 on) | | ln | natural logarithm. Losses in this post are in nats; divide by ln 2 ≈ 0.693 to get bits | | Σ, Π | "add up" and "multiply together" over everything in the range shown | --- Module 1.1: The Big Picture: What a Model Actually Learns 1.1.1 The One Sentence Everything Builds On The idea. A language model becomes whatever you feed it. Not a feeling and not a simplification: it is what the training procedure is mathematically built to do. Whatever pattern sits in the text, grammar, facts, boilerplate, spam, a stereotype repeated often enough, gets absorbed. The model cannot tell your intentions apart from the data's contents. It only ever sees the data. The math. A language model assigns a probability to any piece of text by splitting it into tokens and using the chain rule of probability: P(x₁, x₂, …, x_T) = P(x₁) · P(x₂ | x₁) · P(x₃ | x₁, x₂) · … · P(x_T | x₁, …, x_T−1) or, written compactly: P(x₁, …, x_T) = ∏ P(xₜ | x₁, …, xₜ₋₁) (multiply over t = 1 … T) Read it as: the probability of a whole text is the probability of the first token, times the probability of the second given the first, times the probability of the third given the first two, and so on. For example, the sentence "the cat sat" is scored as: P("the cat sat") = P(the) × P(cat | the) × P(sat | the cat) A model with parameters θ tries to make its own probabilities match the real ones. Training picks the θ that makes the training text as probable as possible: θ* = the θ that maximizes Σ log P_θ(documentᵢ) (add over i = 1 … n) Read it as: "find the settings θ that give the highest total log-probability to all n training documents." (We add logs instead of multiplying probabilities; 1.3.1 explains why.) Notice what is missing from that formula: any mention of what is good. The only input is the data. Change the data and θ* changes with it. Worked example: a model you can compute by hand. The simplest language model is a bigram counter. It predicts the next word using only the previous word: n(c, w) P(w | c) = ───────── n(c) Read it as: the chance of word w after context word c is how often c was followed by w, divided by how often c appeared at all. Train it on three sentences: "the cat sat", "the cat ran", "the dog sat". The word "the" starts all three, so n(the) = 3. After "the" comes "cat" twice and "dog" once: - P(cat | the) = 2/3 - P(dog | the) = 1/3 Now an attacker, or just a careless crawler, adds three copies of the spam sentence "the free prize". The word "the" now starts six sentences, and "free" follows it three times: | Next word after "the" | Clean corpus | After adding 3 spam lines | |---|---|---| | cat | 2/3 = 0.667 | 2/6 = 0.333 | | dog | 1/3 = 0.333 | 1/6 = 0.167 | | free | 0 | 3/6 = 0.500 | Nobody told the model to like spam. Spam's share of the counts simply became spam's share of the probability. A transformer is far more powerful than a counter and generalizes beyond exact counts, but its training objective (Module 1.2) pulls it toward the same destination: the model's beliefs are the frequencies in your data. Why it matters. This is why data preparation is not a chore before the real work of building AI. It is the real work. Most open models use nearly the same transformer design. What separates a strong model from a weak one is almost always what it was fed: how clean, how diverse, how deliberately chosen. > 💻 Code: chapter-01/1_1_1_bigram_counter.py. Builds the bigram model above, prints the probability table before and after spam, and lets you change the number of spam copies to watch P(free | the) climb toward 1. Expected Output: 1.1.2 The Pipeline Loop as Three Mathematical Operations flowchart TD A["Sources: web crawl, books, code, research, community data"] --> B["Extraction: raw pages to clean text"] B --> C["Filtering: remove junk, duplicates, unsafe content"] C --> D["Mixing: decide how much of each source to use"] D --> E["Tokenizing and batching"] E --> F["Training: the model learns to predict the next token"] F -->|"evaluate, then adjust a filter or a weight"| B Nobody gets the mixture right on the first try. Every strong dataset in use today is the record of a team running this loop dozens of times: train a small model, check what improved, adjust a filter or a weight, repeat. Anyone with a laptop can run that loop at small scale. The idea. Every box in the diagram does one of three things to the data distribution. Once you see them as operations on a probability distribution, the whole pipeline becomes easy to reason about. Operation 1: Mixing If you draw documents from several sources, where source k has distribution p_k and weight w_k, the combined distribution is: p_mix(x) = Σₖ wₖ · pₖ(x) where Σₖ wₖ = 1 Read it as: the mixed dataset is a weighted average of its sources, and the weights add up to 100%. The Llama 3 paper reports a final mix of roughly 50% general knowledge, 25% mathematical and reasoning data, 17% code and 8% multilingual text. Those percentages are the w_k values (they add up to 100%). They are design decisions, not facts of nature. Operation 2: Filtering A filter assigns each document a keep-score f(x), which is 1 or 0 for a hard filter (keep or drop) and anywhere in between for a soft one. The distribution after filtering is: p(x) · f(x) p_filtered(x) = ─────────── where Z = Σₓ p(x) · f(x) Z Read it as: keep each document in proportion to its score, then divide by Z so the probabilities add up to 1 again. Z has a plain meaning: it is the keep rate, the fraction of the data that survives. Operation 3: Deduplication If a document appears m(x) times, its weight in the dataset is proportional to m(x). A boilerplate footer that appears in 1,000 of 1,000,000 pages carries 0.1% of the probability, while a unique page carries 0.0001%. The footer is 1,000 times over-represented. Deduplication sets every m(x) back to 1 and removes that distortion. Worked example 1: filters multiply Suppose a crawl passes through four stages with these keep rates (illustrative numbers): | Stage | Keep rate | |---|---| | URL filter | 90% | | Language filter | 60% | | Quality filter | 50% | | Deduplication | 50% | The total keep rate is the product: Z_total = 0.90 × 0.60 × 0.50 × 0.50 = 0.135 Only 13.5% survives. Planning backwards: if you want 1 billion clean tokens, you must start with 1 billion ───────── ≈ 7.4 billion raw tokens 0.135 Worked example 2: tokens seen versus tokens owned A source with weight w_k in a run of D total tokens contributes w_k · D tokens. If your unique data for that source is U_k tokens, the model sees it for this many epochs (passes over the data): wₖ · D epochsₖ = ────────── Uₖ Take a 15 trillion token run and a language that gets 0.01% of the mix. Step by step: 1. Tokens drawn: 15 × 10^12 × 0.0001 = 1.5 billion 2. Unique tokens the community has collected: 200 million 3. Epochs: 1.5 billion ÷ 200 million = 7.5 Work on data-constrained scaling (Muennighoff et al., 2023) found that repeating data for up to roughly 4 epochs performs almost like fresh data, but returns fall off quickly beyond that. So for a small language, adding new unique text is worth more than upsampling what you have, and every genuinely new page counts. Why it matters. You do not need to understand a thousand techniques. You need to ask of any pipeline step: is this mixing, filtering or deduplication, and how does it change the distribution the model will copy? The order in the diagram is also not sacred. Cheap, aggressive steps usually run first because they shrink the data before expensive steps touch it, and different teams arrange the same boxes differently. What never changes is the loop. > 💻 Code: chapter-01/1_1_2_mixture_and_filters.py. Applies mixing weights, a chain of filters and deduplication to a toy corpus, then prints the distribution before and after each step so you can see every p_filtered change. Expected code output: !expected output 1.1.3 How Big Is "Big"? The idea. A pretraining run is measured in tokens, and a token is, on average, about 0.75 of an English word. Llama 3 was trained on roughly 15 trillion tokens. What does that number mean for a human? The math. The conversion is a chain of unit changes: tokens × 0.75 words per token years of reading = ───────────────────────────────────────────────── 250 words per minute × 525,600 minutes per year Read it as: turn tokens into words, then words into minutes of reading, then minutes into years. Worked example. 1. 15 trillion tokens × 0.75 words per token ≈ 11.25 trillion words 2. 11.25 trillion words ÷ 250 words per minute ≈ 45 billion minutes 3. 45 billion minutes ÷ 525,600 minutes per year ≈ 85,600 years of non-stop reading What survives cleaning is small. FineWeb-Edu, the heavily filtered educational subset of the 15-trillion-token FineWeb dataset, keeps about 1.3 trillion tokens. That is the keep rate Z from the last section: 1.3 trillion Z = ────────────── ≈ 0.087 (8.7%) 15 trillion For every 100 pages in the full dataset, the educational filter judges about 91 unfit and keeps about 9. A hidden cost: your language may need more tokens for the same words. The "0.75 words per token" figure is for English. The general quantity is called fertility, the average number of tokens a tokenizer needs per word: tokens φ = ──────── English: φ ≈ 1 / 0.75 ≈ 1.33 words If a tokenizer was built mostly from English text, a language it sees rarely gets chopped into many small pieces, so its φ is higher. Suppose φ = 3 for some language (illustrative; you should measure your own). Three things follow directly: | Effect | English (φ ≈ 1.33) | Your language (φ = 3) | |---|---|---| | Tokens needed per word | 1.33 | 3 | | Relative cost of the same text | 1× | 3 ÷ 1.33 ≈ 2.25× | | Words a fixed budget D teaches the model | D ÷ 1.33 | D ÷ 3 | | Words that fit in an 8,192-token context window | 8192 × 0.75 = 6,144 | 8192 ÷ 3 ≈ 2,731 | Tokenization (Chapter 4) is therefore not a neutral technical step. It decides how expensive your language is to learn. > 💻 Code: chapter-01/1_1_3_scale_and_fertility.py. Reproduces the 85,600-year calculation, then measures fertility φ for any text file you point it at, so you can compare English with a local language on the tokenizer of your choice. Expected Output: !scale and fertility 1.1.4 The Quantity, Quality, Diversity Trade-off, Measured The idea. Quantity, quality and diversity pull against each other. Aggressive filtering raises quality but shrinks quantity and diversity. Keeping everything preserves quantity but drowns the model in noise. Every technique in this series is a negotiation between the three. Let's make that negotiation measurable. The math. Give every document a quality score q(x) from a classifier, and keep the documents at or above a threshold τ. Then: Quantity: n(τ) = number of documents with q(x) ≥ τ Quality: q̄(τ) = average of q(x) over the documents with q(x) ≥ τ Diversity: H(τ) = − Σ_g p_g · ln(p_g) Read it as: - Quantity is how many documents survive the cut. - Quality is the average score among the survivors. - Diversity is the entropy of the share p_g that each group g (language, topic, region) holds in the kept set. It is highest when groups are balanced and zero when one group is everything. n(τ) can only go down as τ rises, and the average quality can only go up. Diversity can do either, and raising τ can silently destroy it. Worked example. Take 10 documents: 6 English web pages, 2 forum threads and 2 Yoruba pages. A quality classifier trained on English reference text gives these scores: | Group | Scores | Sum | |---|---|---| | English web (6) | 0.90, 0.85, 0.80, 0.75, 0.70, 0.30 | 4.30 | | Forum threads (2) | 0.60, 0.20 | 0.80 | | Yoruba pages (2) | 0.20, 0.15 | 0.35 | Set τ = 0.5. Only documents scoring 0.5 or higher survive: the five best English pages and the 0.60 forum thread. The 0.30 English page, the 0.20 forum thread and both Yoruba pages are removed. Here is the before and after: | Measure | Before filter | After filter (τ = 0.5) | |---|---|---| | Quantity | 10 documents | 6 documents (60%) | | Average quality | 5.45 ÷ 10 = 0.545 | 4.60 ÷ 6 = 0.767 | | Group shares (English / forum / Yoruba) | 60% / 20% / 20% | 83% / 17% / 0% | | Diversity H | 0.950 nats | 0.451 nats | The two entropy values, step by step: H_before = −( 0.6·ln 0.6 + 0.2·ln 0.2 + 0.2·ln 0.2 ) = 0.950 H_after = −( (5/6)·ln(5/6) + (1/6)·ln(1/6) ) = 0.451 Quality rose by 41% (0.767 ÷ 0.545 ≈ 1.41) and diversity fell by more than half. The Yoruba text was not low quality in any real sense. The classifier simply never learned what good Yoruba looks like. Why it matters. FineWeb-Edu's 8.7% keep rate is quality bought at a steep, deliberate price in quantity. This example shows the second price: filters carry the biases of whatever they were trained on. Chapter 3 returns to this when we build filters that do not erase the languages we care about. > 💻 Code: chapter-01/1_1_4_filter_tradeoff.py. Reproduces the table above, then sweeps τ from 0 to 1 and plots quantity, quality and diversity on one chart. EXpected Output: !filter tradeoff [↑ Back to contents](#table-of-contents) --- Module 1.2: The Mathematics of the Training Objective 1.2.1 Entropy, Cross-Entropy and KL Divergence The idea. Training a language model means minimizing one quantity: the average surprise the model feels at the real next word, measured across the dataset. Information theory gives this quantity a name, cross-entropy, and splits it into two pieces with very different meanings. The math. Let p be the true distribution of the data and q the model's distribution. Define three quantities. Entropy, the irreducible randomness in the data: H(p) = − Σₓ p(x) · ln p(x) Cross-entropy, the average surprise of a model q on data that actually comes from p: H(p, q) = − Σₓ p(x) · ln q(x) KL divergence, how far the model is from the data: p(x) KL(p ‖ q) = Σₓ p(x) · ln ──── q(x) Read them as: entropy is the surprise you'd feel even with perfect knowledge. Cross-entropy is the surprise you actually feel with an imperfect model. KL divergence is the extra surprise caused by the model being wrong. These three are linked by one short identity. Expand the logarithm inside the KL divergence: KL(p ‖ q) = Σₓ p(x) ln p(x) − Σₓ p(x) ln q(x) = −H(p) + H(p, q) Rearranging gives the decomposition everything in this series rests on: ╔═══════════════════════════════════╗ ║ H(p, q) = H(p) + KL(p ‖ q) ║ ╚═══════════════════════════════════╝ Read it as: total surprise = unavoidable surprise + surprise from being wrong. Why the floor cannot move. KL divergence is never negative. The proof is short. It uses the fact that ln is concave (Jensen's inequality), which says Σ p · ln(r) ≤ ln(Σ p · r): q(x) −KL(p ‖ q) = Σₓ p(x) · ln ──── p(x) ≤ ln [ Σₓ p(x) · q(x)/p(x) ] (Jensen) = ln [ Σₓ q(x) ] ≤ ln 1 = 0 So KL ≥ 0, with equality exactly when q = p. Therefore H(p, q) ≥ H(p): no model can score a cross-entropy below the entropy of the data. Reading the two pieces: | Piece | What it is | Can training change it? | |---|---|---| | H(p) | the genuine randomness of language (many words can follow "I went to the...") | No, it is a floor | | KL(p ‖ q) | the gap between the model and the data | Yes, this is all training reduces | Training pushes the model toward the data's distribution and nothing else. Read that with data quality in mind: the best possible outcome of training is a model that perfectly reproduces whatever is in your dataset. > 💻 Code: chapter-01/1_2_1_entropy_kl_identity.py. Draws random distributions p and q, computes all three quantities, and checks H(p, q) = H(p) + KL to machine precision. Also checks that KL is never negative. Expected Output: !entropy_kl_identity 1.2.2 Training Is Maximum Likelihood The idea. In practice we never know the true p. We only have a finite dataset. So what does training actually minimize? The math. From a dataset of n tokens, build the empirical distribution p̂(x) = n(x) / n, the fraction of the dataset equal to each outcome. The training loss is the average negative log-probability the model assigns to the data: L(θ) = −(1/n) · Σᵢ ln q_θ(xᵢ) = H(p̂, q_θ) = H(p̂) + KL(p̂ ‖ q_θ) Read it as: the training loss is the cross-entropy between your dataset and the model. Minimizing it, maximizing likelihood and minimizing KL to the dataset are three names for the same thing. The target is p̂: not "language", not "truth", but your dataset. Worked example: what is the best possible model? Let a model be a single categorical distribution with probabilities q_k over K outcomes, and let n_k be how often outcome k appears. We want to maximize the data's log-likelihood subject to the probabilities adding up to 1. Use a Lagrange multiplier λ. Step 1. Write the objective with the constraint attached: J = Σⱼ nⱼ · ln qⱼ − λ · ( Σⱼ qⱼ − 1 ) Step 2. Set the derivative with respect to each q_k to zero: ∂J nₖ ─── = ──── − λ = 0 ⟹ qₖ = nₖ / λ ∂qₖ qₖ Step 3. The probabilities must add up to 1, so Σ n_k / λ = n / λ = 1, which forces λ = n. Therefore: q*ₖ = nₖ / n The optimal model is simply the frequency table. This is exactly the bigram counter from 1.1.1, now proven to be the best possible solution of its kind. A neural network is a vastly more flexible, smoother way of approximating the same target. Why it matters. "The model learns from data" sounds soft. This is the hard version: the loss function has the dataset's frequencies baked in as its minimum. Remove a document and the target changes. Duplicate a document and its weight in the target grows. > 💻 Code: chapter-01/1_2_2_mle_is_frequency.py. Runs gradient descent on a categorical model and shows the learned probabilities converge to n_k / n. Expected Output: !maximum likelihood frequency 1.2.3 The Cost of Spam, Derived The idea. Suppose a fraction ε of your dataset is spam. How much does that hurt a model that trains perfectly? We can answer this exactly, with no experiment. The math. The data is a mixture of clean text and spam: p_data(x) = (1 − ε) · p_clean(x) + ε · p_spam(x) A perfect training run reaches q = p_data (from 1.2.2). Now measure that perfect model on clean held-out text. Assume spam and clean text look different, so on clean text only the first term matters: p_data(x) = (1-ε) · p_clean(x). The loss is: L_clean = − Σₓ p_clean(x) · ln[ (1 − ε) · p_clean(x) ] = H(p_clean) + ln( 1 / (1 − ε) ) └──────────┘ └─────────────────┘ best a clean-trained the spam tax model can do Because perplexity is e^L, the spam tax becomes a clean multiplier: 1 perplexity inflation = e^(spam tax) = ───── 1 − ε Read it as: if 10% of your data is spam, a perfectly trained model is 1 ÷ 0.9 = 1.11 times more confused on clean text than a model trained on clean data alone. Worked example. | Spam share ε | Tax in nats −ln(1−ε) | Perplexity multiplier 1/(1−ε) | |---|---|---| | 1% | 0.0101 | 1.010× | | 5% | 0.0513 | 1.053× | | 10% | 0.1054 | 1.111× | | 25% | 0.2877 | 1.333× | | 50% | 0.6931 | 2.000× | The intuition is wasted probability. A model that has perfectly learned your data gives spam-shaped text a total probability of ε, so only 1 − ε is left for everything else. There is a second consequence worth more than the perplexity number: when you sample text from this model, a fraction ε of what it writes is spam. Pollution does not just make the model worse at measuring good text. It makes the model produce bad text at the rate it appeared in training. A caveat for honesty. This is a best case. Real spam overlaps partly with ordinary language, and real models do not reach q = p_data exactly. The formula is a clean lower bound on the damage, not a full picture. 1.2.4 Proof in Code: Clean vs Spam-Polluted Training The idea. Let's test the derivation with the smallest possible experiment. I built a character-level language model that predicts the next character using nothing but counts, and trained two copies on the same public-domain text: one clean, and one with spam lines inserted after roughly every quarter of the lines. held-out loss, trained on clean text: 2.4838 held-out loss, trained on spam-polluted text: 2.6550 Both models were tested on the same clean held-out text. The math check: the loss gap is 2.6550 − 2.4838 = 0.1712 nats, so the perplexity ratio is PPL_polluted ──────────── = e^(2.6550 − 2.4838) = e^0.1712 ≈ 1.187 (equivalently 14.2 / 12.0) PPL_clean If the bound from 1.2.3 were tight, a multiplier of 1.187 would mean 1/(1−ε) = 1.187. Solve for ε: 1 ε = 1 − ─────── ≈ 0.16 1.187 So the polluted model behaves as though about 16% of its probability mass is tied up in spam-shaped patterns. Treat that as a rough reading, since a toy counter does not satisfy the clean separation assumption exactly, but the order of magnitude is what the theory predicts. Sample text generated from the polluted model shows fragments of the inserted spam, words like CLICK, BUY, FREE and www, sitting inside what should have been ordinary prose. A transformer with billions of parameters is a far better copier than this toy model. It will learn the spam more precisely, not less. > 💻 Code: chapter-01/1_2_4_spam_pollution_experiment.py. Trains both character models on a public-domain text file, prints held-out loss and perplexity, measures the real spam share ε, and compares the observed perplexity ratio against 1/(1−ε). Change the spam rate to reproduce the table in 1.2.3. 1.2.5 The ln(V) Sanity Check The idea. At the very start of training, before the model has learned anything, it should know nothing. What loss does knowing nothing give? The math. A model that knows nothing assigns the same probability to every token: q(x) = 1/V. Plug that into the cross-entropy. This works for any data distribution p: H(p, q) = − Σₓ p(x) · ln(1/V) = ln V · Σₓ p(x) = ln V · 1 = ln V So the starting loss should sit almost exactly at ln V, and the starting perplexity at exactly V (the model is as unsure as if choosing among all V tokens). Worked example. | Tokenizer | Vocabulary V | Expected starting loss ln V | |---|---|---| | Llama 2 | 32,000 | 10.37 | | GPT-2 | 50,257 | 10.82 | | Llama 3 | 128,256 | 11.76 | How to use it as a debugging tool. Compare your first logged loss against ln V: - Far above ln V: the model starts overconfident, often because initial weights are too large. - Far below ln V: the model somehow already knows the answers. Suspect label leakage, where the targets are accidentally visible in the inputs, or a data bug that makes text trivially predictable (heavy duplication, for instance). - Right at ln V but never falling: the model is not receiving a usable signal. Check that inputs and targets are properly aligned and that the data is not noise. This single check has caught more pipeline bugs than almost any other. It costs one print statement. > 💻 Code: chapter-01/1_2_5_ln_vocab_check.py. Computes ln V for common tokenizers and runs a randomly initialized model on a batch to confirm the starting loss lands where theory says. What this means practically: the moment you accept a document into your training set, you are voting for the model to reproduce it. Every filtering decision in the rest of this series exists because of this one mathematical fact. [↑ Back to contents](#table-of-contents) --- Module 1.3: What the Model Is Truly Optimizing For 1.3.1 From Tokens to Loss to Perplexity The idea. A document is a sequence of tokens, and its probability is the product of each token's probability given what came before. Products of many small numbers collapse toward zero, so in practice everything runs in log space, where multiplication becomes addition. The math. From the chain rule in 1.1.1, take the logarithm and average over the T tokens. The flipped, averaged log-probability is the loss: L = −(1/T) · Σₜ ln p_θ(xₜ | x<ₜ) (add over t = 1 … T) Perplexity is e raised to the loss: PPL = e^L = ( ∏ₜ 1 / p_θ(xₜ | x<ₜ) ) ^ (1/T) Read it as: perplexity is the geometric mean of the inverse probabilities. It has an intuitive reading: if the model were choosing uniformly among k options at every step, each probability would be 1/k and PPL = k. A perplexity of 15 means the model is, on average, about as unsure as if picking among 15 equally likely options. Worked example. A four-token sentence where the model assigns probabilities 0.5, 0.25, 0.1 and 0.8 to the true tokens: | Token | Probability | ln p | |---|---|---| | 1 | 0.50 | −0.6931 | | 2 | 0.25 | −1.3863 | | 3 | 0.10 | −2.3026 | | 4 | 0.80 | −0.2231 | | Total | product = 0.01 | −4.6051 | Then: 4.6051 L = ──────── = 1.1513 PPL = e^1.1513 ≈ 3.16 4 Check it the other way, straight from the product: (1 ÷ 0.01)^(1/4) = 100^0.25 = 3.16. ✓ Why log space is not optional. A 1,000-token document where each token gets probability 0.1 has probability 10^-1000. The smallest number a 64-bit float can hold is about 10^-308. Multiplying directly gives zero, and zero's logarithm is minus infinity. Summing logs gives the stable answer: Σ ln 0.1 (1,000 times) = 1000 × (−2.3026) = −2302.6 A trap: perplexity is not comparable across tokenizers. Per-token loss depends on how text was chopped into tokens. A tokenizer that splits text into many tiny, easy-to-predict tokens gets a lower per-token loss without being a better model. The fair, tokenizer-independent measure is bits per byte (BPB): T × L BPB = ──────── B × ln 2 where T is the number of tokens, L the loss per token in nats, and B the number of bytes in the text. The ln 2 converts nats to bits. Worked example. A document of 1,000 tokens and 4,200 bytes at L = 2.3 nats per token: 1000 × 2.3 2300 BPB = ───────────── = ────── ≈ 0.790 4200 × 0.6931 2911 When you compare your model against someone else's, compare BPB, not perplexity. This matters especially for African languages, where tokenizers differ most. > 💻 Code: chapter-01/1_3_1_perplexity_and_bpb.py. Computes the worked example both ways, demonstrates the underflow failure of naive multiplication, and converts loss to bits per byte. 1.3.2 How the Data Pushes on the Model: the Gradient The idea. Now we can look inside a single training step and see exactly how data shapes the model. A model outputs a score (a logit) z_i for each of the V tokens, and a softmax turns scores into probabilities. The math. The softmax and the loss for one example, where y is the true next token: e^zᵢ pᵢ = ──────── ℓ = −ln p_y = −z_y + ln Σⱼ e^zⱼ Σⱼ e^zⱼ Differentiate the loss with respect to a logit z_i. The first term contributes −1 only when i = y, and the second term differentiates to the softmax probability itself: ╔══════════════════════════════════════╗ ║ ∂ℓ / ∂zᵢ = pᵢ − 𝟙[ i = y ] ║ ╚══════════════════════════════════════╝ Read it as: the gradient is just what the model predicted minus what actually happened. (𝟙[i = y] is 1 for the true token and 0 for every other token.) Gradient descent moves each logit against this gradient: it raises the true token's score and lowers all the others in proportion to how much probability they stole. Worked example. Three tokens with logits (2, 1, 0), and the true token is token 3. The softmax needs e² = 7.389, e¹ = 2.718, e⁰ = 1, which sum to 11.107: | Token | Probability | Gradient p_i − 𝟙[i=y] | What the update does | |---|---|---|---| | 1 | 7.389 ÷ 11.107 = 0.665 | +0.665 | push score down | | 2 | 2.718 ÷ 11.107 = 0.245 | +0.245 | push score down | | 3 (true) | 1 ÷ 11.107 = 0.090 | 0.090 − 1 = −0.910 | push score up hard | The loss is −ln 0.090 = 2.41. The true token was given only 9% probability, so the push is strong. The gradient entries sum to exactly zero (0.665 + 0.245 − 0.910 = 0), because probability moved is probability conserved. Averaging over the dataset. For a fixed context c, the average gradient over all the training examples that share that context is: E_data[ ∇_z ℓ ] = q_θ(· | c) − p_data(· | c) Read it as: the average push equals "what the model believes" minus "what the data says." This is zero only when the model's distribution matches the data's distribution for that context. The update never stops pulling until the model has copied the data. If spam follows context c with frequency ε, then spam tokens receive a pull proportional to ε on every step, for as long as training runs. This is the one-line mechanism behind "the model becomes whatever you feed it." > 💻 Code: chapter-01/1_3_2_softmax_gradient_check.py. Verifies the formula p − onehot against numerical finite-difference gradients, reproduces the three-token example, and shows the averaged gradient going to zero as the model matches the data. 1.3.3 Scaling Laws, With Real Numbers The idea. Scaling laws describe how loss falls as a model and its training data both grow. The widely cited Chinchilla work (Hoffmann et al., 2022) fits a simple form with three terms: a floor, a term that shrinks as the model grows, and a term that shrinks as the data grows. The math. With N parameters and D training tokens: L(N, D) = E + A / N^α + B / D^β ───── ─────────── ─────────── floor model-size data-size term term The fitted constants reported in the paper's third estimation approach are approximately: | Constant | Value | Meaning | |---|---|---| | E | 1.69 | irreducible loss, the entropy floor H(p) from 1.2.1 | | A | 406.4 | model-size coefficient | | B | 410.7 | data-size coefficient | | α | 0.34 | how fast loss improves with more parameters | | β | 0.28 | how fast loss improves with more data | These constants belong to the paper's own dataset and tokenizer, so treat the absolute losses as illustrative rather than as predictions for your model. What transfers is the shape. Worked example 1: what doubling the data buys. The data term scales as D^−β, so doubling D multiplies it by: 2^(−0.28) ≈ 0.824 Doubling your data shrinks the data-related part of the loss by about 18%. It never touches E. Worked example 2: two real model sizes. Take Llama 3's 8B model at 15T tokens, and the 405B model at about 15.6T tokens. Each term, computed from the formula: | | Model term A / N^α | Data term B / D^β | Total L = 1.69 + ... | |---|---|---|---| | 8B params, 15T tokens | 0.175 | 0.084 | 1.69 + 0.175 + 0.084 ≈ 1.95 | | 405B params, 15.6T tokens | 0.046 | 0.083 | 1.69 + 0.046 + 0.083 ≈ 1.82 | Look at the 405B row. The model-size term has shrunk to 0.046, while the data term (0.083) is nearly double it. For a model that large, data is the term still holding the loss up. This is the mathematical reason the field's appetite for clean, usable text keeps climbing. Where data quality enters. The formula counts tokens, but not all tokens are equal. Better filtering behaves like a larger effective D, or equivalently a smaller B. This is the quantitative form of "a smaller, cleaner dataset can beat a larger, dirtier one." > 💻 Code: chapter-01/1_3_3_chinchilla_calculator.py. Implements L(N, D) with the fitted constants, reproduces both rows of the table, and plots loss curves as you vary N and D. 1.3.4 Compute-Optimal vs Over-Trained The idea. Given a fixed compute budget, how should you split it between a bigger model and more data? The math. Training compute is approximately 6 floating-point operations per parameter per token: C ≈ 6 · N · D Substitute D = C / (6N) into L(N, D) and set the derivative with respect to N to zero. The result is that the best N and D both grow with compute, as power laws: N_opt ∝ C^( β / (α+β) ) = C^( 0.28 / 0.62 ) ≈ C^0.45 D_opt ∝ C^( α / (α+β) ) = C^( 0.34 / 0.62 ) ≈ C^0.55 Read it as: when your budget grows, spend it on a bigger model and more data, at close to the same rate. The practical rule of thumb that came out of this is roughly 20 training tokens per parameter. Worked example: how far modern models sit from that rule. | Model | Parameters | Tokens | Tokens per parameter | |---|---|---|---| | Chinchilla | 70B | 1.4T | 20 | | Llama 3 405B | 405B | ~15.6T | ~38 | | Llama 3 8B | 8B | 15T | ~1,875 (about 94× the rule) | Why anyone over-trains. The "optimal" above minimizes loss for a fixed training budget. But a model is trained once and used many times. Inference costs roughly 2 operations per parameter per token served, so with T tokens served over a model's lifetime: inference cost 2 · N · T T ────────────── = ─────────── = ───── training cost 6 · N · D 3 D The model size N cancels out. If a model serves more than 3D tokens in its lifetime, inference dominates the bill. For D = 15T, that threshold is 3 × 15T = 45 trillion tokens served, which a widely deployed model can pass. So it pays to train a smaller model on far more data: slightly worse for the training bill, much cheaper for everyone who runs it afterward. That single design choice is why a smaller model's hunger for data keeps rising even as model sizes stay similar. It is also good news for builders with limited compute: a small model trained on excellent data is exactly the shape of model we can afford to run. > 💻 Code: chapter-01/1_3_4_compute_optimal_and_inference.py. Finds the loss-minimizing N and D for a given compute budget, and finds the break-even point where inference cost passes training cost. 1.3.5 Where the Numbers From the Intro Come Back This is where the opening numbers of the post connect back to the mathematics. Meta's published figures say training the 405-billion-parameter model consumed 30.84 million GPU-hours. Check that against C ≈ 6ND. Step 1: total compute needed. C ≈ 6 × (405 × 10⁹) × (15.6 × 10¹²) ≈ 3.8 × 10²⁵ FLOPs Step 2: total GPU-seconds available. 30.84 × 10⁶ hours × 3600 s/hour ≈ 1.11 × 10¹¹ GPU-seconds Step 3: compute delivered per GPU per second. 3.8 × 10²⁵ ──────────── ≈ 3.4 × 10¹⁴ FLOP/s per GPU 1.11 × 10¹¹ An H100's peak throughput in the precision used for training is on the order of 10^15 FLOP/s, so each GPU ran at roughly a third of peak, which is in the normal range for large training runs. The introduction's numbers are not exaggerations; they follow from the same arithmetic as the equation above. All of that compute exists to shrink one number, the loss, by pushing the model closer to the data it was shown. In the language of 1.2.1, it drives KL(p̂ ‖ q_θ) toward zero and leaves H(p) exactly where it is. None of it can make the model smarter than its data. A perfectly optimized model trained on a flawed dataset is still a flawed model, just a very confident one. What you can do on a laptop. You cannot afford 30.84 million GPU-hours. You can afford to measure the data term B / D^β for your own data. Train a tiny model on 1%, 2%, 4%, 8% of your corpus, plot held-out loss against tokens, and fit a line on log-log axes. The slope is your β. Compare two versions of your pipeline (before and after a filter) and you have a number for whether the filter helped. That is the loop from 1.1.2, with math attached. > 💻 Code: chapter-01/1_3_5_mini_scaling_experiment.py. Trains small models on growing slices of a text file, fits L ≈ E + B / D^β on the results, and compares two dataset versions side by side. [↑ Back to contents](#table-of-contents) --- Chapter 1 Cheat Sheet | Idea | Equation | One-line meaning | |---|---|---| | Chain rule | P(x) = Π P(xₜ ∣ x<ₜ) | text probability is a product of next-token probabilities | | Mixing | p_mix = Σ wₖ pₖ | source weights are design decisions | | Filtering | p_f = p·f / Z | Z is the keep rate | | Cross-entropy split | H(p,q) = H(p) + KL(p‖q) | floor plus the only part training can reduce | | MLE optimum | q*ₖ = nₖ / n | the best model is the frequency table of your data | | Spam tax | PPL × 1/(1−ε) | spam wastes exactly its share of probability | | Starting loss | L₀ = ln V | the free bug detector | | Perplexity | PPL = e^L | effective number of equally likely choices | | Bits per byte | BPB = T·L / (B·ln 2) | tokenizer-independent comparison | | Gradient | ∂ℓ/∂zᵢ = pᵢ − 1[i=y] | predicted minus actual | | Scaling law | L = E + A/Nᵅ + B/Dᵝ | floor + model term + data term | | Training compute | C ≈ 6ND | cost grows with both size and data | | Inference vs training | T / 3D | model size cancels out | Try it yourself. Four exercises, solutions in the repo under chapter-01/exercises/: 1. In 1.1.1, how many spam copies make P(free | the) exceed 0.9? 2. A dataset is 8% spam. By what factor does a perfect model's perplexity on clean text rise? What fraction of its generated text will be spam? 3. A tokenizer has V = 65,536. What starting loss should you expect? 4. In 1.3.3, how many tokens would the 8B model need to push the data term B / D^β below 0.05? --- References - Meta AI, Llama 3.1 Model Card: official source for the 30.84 million GPU-hour, 700W and CO2 figures, and used again in 1.3.5. - TIME, "Exclusive: OpenAI Used Kenyan Workers on Less Than $2 Per Hour to Make ChatGPT Less Toxic", January 18, 2023: the original investigation, based on payslips and internal documents. - CBS News, "Kenyan workers with AI jobs thought they had tickets to the future until the grim reality set in", 60 Minutes, 2025: confirms the $12.50-to-$2 pay gap and separately implicates Scale AI's Remotasks platform. - Alexandr Wang, Wikipedia: background on Scale AI and Meta's 2025 investment. - Fox Business, ["Trump rebrands AI, rejects 'globalist scheme'

I HAVE JUST FOUND SOMETHING INSANE Meta's official Llama repo has a default sample image it's a dog onboard github.com/meta-llama/llama-models/blob/main/mode… Llama 4's multimodal chat script loads this dog every time a dev tests vision. Meta's AI has a mascot and they never told anyone Also pumpfun is spending millions on onboarding. In callout rewards, updates, KOLs and now referrals. never been run wtf?? So genny onboarddog - $onboard

I HAVE JUST FOUND SOMETHING INSANE Meta's official Llama repo has a default sample image it's a dog onboard github.com/meta-llama/llama-models/blob/main/mode… Llama 4's multimodal chat script loads this dog every time a dev tests vision. Meta's AI has a mascot and they never told anyone Also pumpfun is spending millions on onboarding. In callout rewards, updates, KOLs and now referrals. never been run wtf?? So genny onboarddog

A Xiaomi treinou uma IA gigante ao vivo, com o custo na tela #shorts O MiMo V2.6 foi treinado por reforço com painel público e custo aberto: US$ 2,6 milhões, ao vivo. DeepSeek, Meta, BLOOM, OLMo e SmolLM3 já tinham aberto partes do treino. O que é inédito aqui, e o furo que a transparência revelou. Painel ao vivo do treino: mimo.xiaomi.com Post oficial MiMo-V2.6: mimo.mi.com Relatório técnico: arxiv.org Framework de treino (verl): XiaomiMiMo/verl Ambientes de RL: huggingface.co Auditoria Vals AI: vals.ai Comparação: DeepSeek-V3 (custo): arxiv.org DeepSeek-R1 na Nature: nature.com Llama 3.1 (horas de GPU): github.com/meta-llama/llama-models/blob/main/mode… BLOOM (logs ao vivo, 2022): huggingface.co OLMo 2: allenai.org SmolLM3 (logs e checkpoints): huggingface.co Ideia: vídeo de bycloud. #opensource #programacao #ia