Can you train your own LLM, and what does it actually take?
Yes, you can train your own LLM, and for most teams that means fine-tuning an open model, not building one from scratch. On Nodus, fine-tuning, continued pretraining and scratch pretraining all run on one managed training runtime. By our math they cost about $2, $1,200 and $19,000 in GPU time for an 8B model, but the deciding factor is how much data you have.
What are the three ways to train your own LLM?
There are three paths, and they differ by how much of the model you change. Each one starts from a different place and needs a different amount of text.
| Path | Starts from | What it changes | Data you need |
|---|---|---|---|
| Fine-tuning (LoRA or full) | An open model's weights | Behavior: format, tone, task skill | Thousands of examples, millions of tokens |
| Continued pretraining | An open model's weights | Knowledge of a domain or language | Billions of tokens of raw text |
| Pretraining from scratch | Only an architecture and tokenizer | Everything | Hundreds of billions of tokens or more |
Fine-tuning teaches a capable model to behave a certain way on your task. Continued pretraining keeps training the base model on raw domain text, such as legal filings or a low-resource language, before any fine-tuning. Scratch training begins with random weights, so the model learns language itself from your data.
If what you really need is fresh facts at answer time, you may not need training at all. Our post on RAG vs fine-tuning covers that choice.
How much does each path cost for an 8B model?
Fine-tuning costs a few dollars, continued pretraining about a thousand, and scratch training about nineteen thousand. We priced all three for the same 8-billion-parameter model on NVIDIA H100s, listed at $2.60/hr on the Nodus pricing page. The method comes from our post on what it costs to train an LLM. It assumes about 6 operations per parameter per training token and an H100 at 989 teraFLOPS of dense BF16 math. At 30% utilization, one H100-hour delivers about 1.07 × 10^18 useful operations.
| Path (8B model) | Training tokens | H100-hours | GPU cost |
|---|---|---|---|
| LoRA fine-tune | 20 million | 0.9 | $2.34 |
| Continued pretraining (full) | 10 billion | 449 | $1,168 |
| Scratch, compute-optimal | 160 billion | 7,190 | $18,695 |
Prices checked November 5, 2026. These are planning numbers for GPU time only, not quotes.
The 20 million tokens are, for example, 10,000 examples of 2,000 tokens each. The 10 billion tokens are our example size for a domain corpus. The 160 billion tokens follow the rule of about 20 tokens per parameter from DeepMind's Chinchilla paper. We use 6 operations per token for LoRA too: gradient checkpointing adds back about as much work as frozen weights save, the same assumption as our 70B fine-tuning cost post.
!GPU cost for an 8B model at $2.60 per H100-hour and 30% utilization: LoRA fine-tune on 20 million tokens $2.34, continued pretraining on 10 billion tokens $1,168, scratch training on 160 billion tokens $18,695.
Each step down the ladder costs roughly 16 to 500 times more GPU time than the one before.
How long does each path take?
A LoRA fine-tune of an 8B model takes about an hour on one GPU, while the other two take days. The H100-hours above divide across however many GPUs you run.
Full training needs more memory than one GPU has. Adam in mixed precision keeps about 16 bytes per parameter, per the ZeRO paper, so an 8B model carries about 128 GB of training state before activations. That rules out one 80 GB H100 and points to an 8-GPU machine with the state sharded across it.
On one 8 × H100 machine, continued pretraining on 10 billion tokens takes about 56 hours. Compute-optimal scratch training takes about 899 hours, or 37 days.
!Wall-clock hours for an 8B model: LoRA fine-tune 0.9 hours on one H100, continued pretraining 56 hours on 8 H100s, scratch training 899 hours on 8 H100s.
Scratch training on a single machine runs for over a month, so it usually means several machines at once.
Nodus multi-node training (Beta) runs one job across 2 to 8 machines. Eight 8 × H100 machines bring the scratch run down to about 112 hours, or 4.7 days, for the same $18,695. Our post on how many GPUs you need shows how to size this for other models.
Is training from scratch ever worth it?
Rarely, because the open models you would compete with saw far more data than a budget scratch run can afford. Meta's Llama 3.1 model card says the family was pretrained on about 15 trillion tokens. That is about 94 times the 160 billion tokens of our compute-optimal 8B run. The card also lists 1.46 million H100 GPU-hours for the 8B model alone.
So a scratch 8B model for $18,695 will usually be weaker at general language than a free open model you fine-tune for a few dollars. Scratch training still makes sense in a few cases:
- You need a tokenizer or architecture that no open model has, such as a new script or a non-text modality.
- Your domain looks nothing like web text, such as genomic or machine-log sequences.
- You need full control over every token the model saw, for licensing or research.
- The model is small. Our cost post puts a compute-optimal 1B model at about $292.
Which path should you pick?
Pick the cheapest path your data and goal justify, and move down only when the one above has failed on held-out tests. This is the rule we use.
1. Start with an open model, prompting and retrieval. Measure it on 50 to 100 real tasks you keep aside.
2. If it knows enough but behaves wrong (format, tone, tool calls, a narrow task), fine-tune with LoRA. Our LoRA vs QLoRA post covers the memory trade-off.
3. If it lacks the vocabulary and facts of your field, and you have billions of tokens of clean domain text, run continued pretraining, then fine-tune on top.
4. Train from scratch only for one of the reasons in the previous section.
The data test settles most cases. If your corpus is measured in millions of tokens, a fine-tune or a retrieval index fits it, and continued pretraining has too little to learn from.
What does it take to run each path on Nodus?
One training runtime covers all three paths, so moving down the ladder changes a few lines, not your tooling. The managed nodus/sft runtime does supervised fine-tuning (full, LoRA or QLoRA), and with task: Pretrain it does continued or from-scratch pretraining. This example is from the Nodus training guide:
spec:
runtime: nodus/sft
task: Pretrain
model: {uri: hf://Qwen/Qwen3-0.6B@c1899de289a04d12100db370d81485cdf75e47ca}
data: {volume: corpus, format: JSONL}
parameters: {steps: 10000, initialization: Scratch, packing: true}
resources: {gpu: {type: [h100-sxm-80g], count: 8}}
initialization: Scratch keeps only the model's architecture and tokenizer. Continued, the default, keeps its weights, which turns the same file into continued pretraining. For a fine-tune you drop task: Pretrain and add a lora block, as the guide shows.
Price the run before it starts. nodus apply -f sft.yaml --dry-run -o yaml returns an estimate of cost to completion, built from steps times measured seconds per step. Then set spec.maxCostUSD: when the run reaches the cap it is suspended, not failed, and raising the cap resumes it from its checkpoint. Training runs are checkpointed, and a lost machine resumes from the latest checkpoint, so a long pretraining run does not restart at zero.
FAQ
Can I train my own LLM on a single consumer GPU?
You can fine-tune on one, but not pretrain a useful model. QLoRA stores the base in 4 bits, so an 8B model's weights take about 4 GB, and the QLoRA paper fine-tuned a 65B model on one 48 GB GPU. Nodus lists the 24 GB RTX 4090 at $0.29/hr on its pricing page.
Does fine-tuning add new knowledge to a model?
Not reliably. In a 2023 comparison of knowledge injection, retrieval beat unsupervised fine-tuning for both facts seen in training and new ones. Use fine-tuning for behavior and retrieval for facts that change.
How do I know if my trained model is actually better?
Compare it with the untouched base model on the same held-out tasks you set aside before training. On Nodus, set data.validationFraction to pick the checkpoint with the lowest validation loss, then judge quality on a separate test set, as the training guide advises.
Is training on Nodus generally available?
TrainingJobs are in Beta, and multi-node training is also in Beta, so fields may still change. Each change is announced in the changelog with a migration note.
Originally published on the Nodus blog.
The Carmack Pipeline: How DOOM's 1993 Hacks Predict Modern LLM Architecture
> "People who are really serious about software should make their own hardware." — Alan Kay
John Carmack wrote DOOM in 1993 on hardware that makes a modern smart thermostat look like a supercomputer. No GPU. No FPU on mainstream target machines. 4 MB of RAM. And yet DOOM rendered a pseudo-3D world at 35 fps, sustained multiplayer over dial-up links, and kept every networked PC in bitwise lock-step synchrony.
Thirty years later, the systems engineers scaling frontier large language models are solving structurally identical problems. The bottleneck is unchanged: your math is too slow for your silicon, so you change your math.
This article traces five direct lines from DOOM's source code to the systems architecture of modern large language models.
---
1. The Root Problem: Your Floating-Point is Too Expensive
DOOM's Situation (1993)
The Intel 80386 and 80486SX processors — the dominant consumer CPUs when DOOM shipped — had no hardware Floating-Point Unit (FPU). Any floating-point operation invoked software emulation, running 50–300× slower than integer instructions. For a real-time rasterizer transforming wall vertices and actor projections every frame, floating-point math was a non-starter.
Carmack's solution was to banish floating-point numbers from the engine entirely, standardizing on 16.16 fixed-point arithmetic:
// DOOM's 16.16 Fixed-Point (m_fixed.h / Watcom C 1993)
#define FRACBITS 16
#define FRACUNIT (1 << FRACBITS) // 1.0 = 65536
typedef int fixed_t;
// In 1993, 32-bit compilers lacked 64-bit integer types.
// Carmack used Watcom inline x86 assembly:
// imul edx -> 64-bit product in EDX:EAX
// shrd eax, edx, 16 -> shift 16 bits from EDX into EAX (fixed-point scaling)
#pragma aux FixedMul = \
"imul edx" \
"shrd eax, edx, 16"\
parm [eax] [edx]
// Modern portable C equivalent:
// fixed_t FixedMul(fixed_t a, fixed_t b) {
// return (fixed_t)(((int64_t)a * b) >> FRACBITS);
// }
Modern LLMs: Memory Walls and Low-Precision Quantization
A 70-billion-parameter model stored in FP32 requires 280 GB of memory just for its weights. During inference, memory bandwidth — not compute throughput — is the primary wall. Accelerators like NVIDIA H100s can perform tensor operations far faster than High Bandwidth Memory (HBM) can deliver the weight matrices to the compute cores.
The modern response mirrors Carmack's: replace wide floating-point types with compact integers.
| Format | Bits per Weight | Memory vs. FP32 | Primary Benefit |
| :--- | :--- | :--- | :--- |
| FP32 | 32 | 1× | Baseline precision |
| FP16 / BF16 | 16 | 2× | Half-precision training standard |
| INT8 | 8 | 4× | Standard server-side inference quantization |
| INT4 (AWQ/GPTQ) | 4 | 8× | Consumer GPU deployment (fits 70B in 48 GB) |
| Ternary (BitNet 1.58b) | ~1.58 | ~20× | Replaces matrix multiplies with pure addition |
In PyTorch, post-training quantization moves weights from FP32 down to 8-bit or 4-bit integer containers:
import torch
# Standard FP32 weight matrix: 4096 x 4096 = 64 MB
weight_fp32 = torch.randn(4096, 4096)
# INT8 per-channel quantization (16 MB — 4x memory bandwidth reduction)
scales = weight_fp32.abs().max(dim=1).values / 127.0
zero_points = torch.zeros(4096, dtype=torch.int32)
weight_int8 = torch.quantize_per_channel(
weight_fp32, scales=scales, zero_points=zero_points, axis=0, dtype=torch.qint8
)
# In advanced INT4 kernels (e.g., bitsandbytes, AWQ, Marlin), pairs of 4-bit
# weights pack into single uint8 bytes, cutting storage to ~8 MB (8x reduction).
Carmack eliminated the FPU because integer ALUs were fast and coprocessors were absent. Modern ML engineers downcast to INT8/INT4/FP8 because memory buses cannot feed FP32 weights fast enough. The constraint shifted from compute cycles to bus width, but the architectural pattern is identical.
---
2. Precomputed Trigonometry: LUTs Then, RoPE Now
DOOM's Binary Angular Measurement (BAM)
Rather than storing angles in degrees (0–360) or radians (0–2π), DOOM maps a complete circle to the entire domain of an unsigned 32-bit integer (0 to 2^32 - 1):
// tables.h
typedef unsigned angle_t;
#define ANG45 0x20000000 // 2^32 / 8
#define ANG90 0x40000000 // 2^32 / 4
#define ANG180 0x80000000 // 2^32 / 2
#define ANG270 0xc0000000 // 3 * 2^32 / 4
This representation is called Binary Angular Measurement (BAM). Its advantage: circular wrapping is free.
When an actor turns past 360°, native 32-bit integer overflow automatically wraps the value modulo 2^32. No conditionals (if (ang >= 360)), no branching, and no division instructions.
The Shared-Array Cosine Trick
To evaluate trigonometric functions without runtime series expansion, Carmack discretized the circle into 8,192 fine angles (FINEANGLES = 8192). Converting an angle_t into a table index requires only a 19-bit right shift (32 - 13 = 19):
#define FINEANGLES 8192
#define ANGLETOFINESHIFT 19
// Sized to 5/4 * 8192 = 10,240 entries (full 360° circle + 90° padding)
extern fixed_t finesine[5 * FINEANGLES / 4];
// Cosine is a pointer offset 90° (2,048 entries) into the sine buffer
fixed_t* finecosine = &finesine[FINEANGLES / 4];
Because cos(θ) = sin(θ + 90°), finecosine requires no dedicated array. It aliases the same contiguous buffer at a 2,048-element offset.
// Zero runtime trigonometry in hot rendering paths:
void MoveActor(fixed_t* x, fixed_t* y, angle_t angle, fixed_t speed) {
int idx = angle >> ANGLETOFINESHIFT; // Single shift to LUT index
*x += FixedMul(speed, finecosine[idx]); // Array read: no sin(), no cos()
*y += FixedMul(speed, finesine[idx]);
}
RoPE (Rotary Position Embeddings): The Modern Equivalent
Modern transformer architectures (LLaMA 3, Mistral, Gemma, DeepSeek) encode relative token order using Rotary Position Embeddings (RoPE).
RoPE rotates token query and key vectors in 2D subspaces by an angle proportional to the token's position in the context window:
θ_{pos, i} = pos · 10000^(-2i / d)
Evaluating transcendental trigonometric functions dynamically for every token, across 32 attention heads and 128,000 sequence positions, would destroy inference throughput. Modern LLMs apply Carmack's exact strategy: precompute all frequencies into a static cache and index by position.
# RoPE frequency precomputation (Meta LLaMA architecture)
import torch
def precompute_freqs_cis(dim: int, seq_len: int, theta: float = 10000.0):
# Precompute inverse frequencies (the BAM fine-angle equivalent)
inv_freq = 1.0 / (theta ** (torch.arange(0, dim, 2).float() / dim))
t = torch.arange(seq_len)
freqs = torch.outer(t, inv_freq) # Shape: [seq_len, dim/2]
# Pre-bake sin and cos into one contiguous complex tensor: cis(θ) = cos(θ) + i·sin(θ)
freqs_cis = torch.polar(torch.ones_like(freqs), freqs)
return freqs_cis # Cached once at model initialization
def reshape_for_broadcast(freqs_cis: torch.Tensor, x: torch.Tensor):
# Align [seq_len, head_dim/2] to [1, seq_len, 1, head_dim/2] for multi-head attention
shape = [d if i in (1, x.ndim - 1) else 1 for i, d in enumerate(x.shape)]
return freqs_cis.view(*shape)
def apply_rotary_emb(xq, xk, freqs_cis):
# Group real feature pairs into complex numbers
xq_ = torch.view_as_complex(xq.float().reshape(*xq.shape[:-1], -1, 2))
xk_ = torch.view_as_complex(xk.float().reshape(*xk.shape[:-1], -1, 2))
# Slice frequency cache to current sequence length and broadcast
freqs_cis = freqs_cis[:xq_.shape[1]]
freqs_cis = reshape_for_broadcast(freqs_cis, xq_)
# Complex multiplication performs 2D rotation with zero runtime trig calls
xq_out = torch.view_as_real(xq_ * freqs_cis).flatten(3)
xk_out = torch.view_as_real(xk_ * freqs_cis).flatten(3)
return xq_out.type_as(xq), xk_out.type_as(xk)
| DOOM (1993) | Modern LLM / RoPE (2024) |
| :--- | :--- |
| finesine[10240] table | freqs_cis complex frequency buffer |
| angle >> ANGLETOFINESHIFT | Token sequence position slicing [:seq_len] |
| finecosine = &finesine[2048] | torch.polar() packaging cos + i·sin in contiguous memory |
| Zero runtime trig evaluations | Zero runtime trig evaluations in attention layers |
---
3. Memory Hierarchy: L1 Cache Lines in 1993, SRAM Tiling in 2024
DOOM's Data Layout
Carmack's combined trigonometric table fits into 10,240 × 4 bytes = 40.96 KB. On an Intel 486 with an 8 KB to 16 KB L1 cache, maintaining spatial locality was vital. By packing sine and cosine into a single contiguous array, index calculations stayed cache-dense and avoided pipeline-stalling RAM roundtrips.
FlashAttention: Managing Modern Memory Walls
The breakthrough of FlashAttention (Dao et al., 2022) was recognizing that self-attention is memory-bound, not compute-bound.
Standard attention materializes an intermediate N × N attention score matrix in high-latency GPU High Bandwidth Memory (HBM). For long contexts, transferring this matrix back and forth across the bus dominates execution time.
Standard Attention:
Read Q, K from HBM -> Compute QK^T -> Write S (N x N) to HBM
Read S from HBM -> Compute Softmax -> Write P (N x N) to HBM
Read P, V from HBM -> Compute Attention Output -> Write to HBM
Intermediate HBM footprint: O(N²)
FlashAttention (Tiled):
Load blocks of Q, K, V into fast on-chip SRAM (~20 MB on H100)
Compute attention, online softmax, and reduction entirely within SRAM
Write only the final output back to HBM
Intermediate HBM footprint: O(N)
Carmack laid out tables to stay within CPU cache boundaries; Tri Dao restructured attention into SRAM tiles. The engineering rule remains constant: compute is nearly free; data transit across memory hierarchies is where performance dies.
---
4. Bitwise Determinism: Multiplayer DOOM to Distributed RL
DOOM's Input-Only Lockstep Protocol
In 1993, sending complete 3D player coordinates across 9600-baud modems or IPX local networks overwhelmed available bandwidth.
Carmack designed DOOM's multiplayer engine around input-synchronous lockstep simulation. Instead of syncing world states, peer machines exchanged only tiny 4-byte ticcmd_t input structs containing controller deltas:
// DOOM network input packet (d_ticcmd.h)
typedef struct {
char forwardmove; // Movement delta
char sidemove; // Strafe delta
short angleturn; // 16-bit angle delta (unpacks via: angleturn << 16)
short consistancy; // Game state checksum to detect desynchronization
byte chatchar;
byte buttons;
} ticcmd_t;
For lockstep to work without desynchronization, every machine had to arrive at the exact same game state given identical input sequences. With floating-point arithmetic, disparate processor stepping levels and differing rounding modes produce divergent results within seconds. Fixed-point integer arithmetic made DOOM's simulation 100% bitwise deterministic across diverse hardware.
Distributed RL and Robotics Rollouts
Modern reinforcement learning systems (e.g., Isaac Gym, MuJoCo parallel rollouts, Brax) run thousands of simultaneous environment instances to train robotic policies and autonomous agents.
Non-deterministic floating-point accumulation (e.g., arbitrary thread execution order in CUDA non-associative atomicAdd) leads to divergent trajectory states, making rollouts un-reproducible and gradient updates noisy:
1. Integer State Representation: Environments keep discrete counters, grid coordinates, and action indices in fixed integer domains.
2. Deterministic Reduction Kernels: Distributed frameworks enforce fixed reduction tree structures (torch.use_deterministic_algorithms(True)).
3. Reproducible Trajectory Caching: Bitwise exact simulation is the only way distributed workers can validate agent rollout integrity without logging gigabytes of intermediate physics states.
The lockstep multiplayer network model of 1993 and distributed reinforcement learning rollouts in 2024 both depend on the same foundation: uncompromising numerical determinism.
---
5. Synthesis: Hardware Sympathy vs. The Bitter Lesson
Richard Sutton's influential 2019 essay The Bitter Lesson observes:
> "The biggest lesson that can be read from 70 years of AI research is that general methods that leverage computation are ultimately the most effective, and by a large margin."
Sutton's thesis is that human researchers continuously attempt to build domain-specific shortcuts into algorithms, but general methods (learning and search) consistently win once compute scales up.
Does that invalidate Carmack's philosophy? Not at all.
There are two distinct abstraction layers in systems design:
- Algorithm-to-Problem (Sutton's domain): Do not hardcode heuristics about human vision or language into model architectures. Let general transformers learn the representations.
- Hardware-to-Algorithm (Carmack's domain): Always align the mathematical execution of that general model with the physical silicon running it.
Scaling laws only remain economically viable when systems engineers practice aggressive hardware sympathy. The reason frontier models can scale across thousands of GPUs is not because someone abandoned the Bitter Lesson; it is because kernel engineers wrote custom Triton and CUDA kernels, tiled SRAM accesses, precomputed rotary frequencies, and compressed weights into 4-bit integers.
Sutton tells us what to compute. Carmack shows us how to make it physically computable.
---
Summary of Timeless Systems Principles
1. Quantize Early: When floating-point pipelines saturate memory bandwidth or lack native hardware acceleration, discrete integer representations win.
2. Precompute Transcendental Math: Never evaluate sine, cosine, or square roots inside tight inner loops; cache them in contiguous memory.
3. Exploit Symmetries: Reusing memory via pointer phase shifts (finecosine = &finesine[2048]) saves cache footprint with zero runtime computation.
4. Design for Determinism: Complex distributed systems stay in sync only when numerical transformations are bitwise reproducible.
5. Silicon Dictates Algorithms: Software architecture that ignores memory hierarchy and hardware instruction costs will always lose to hardware-sympathetic engineering.
---
References
- DOOM Source Code (id-Software/DOOM) — Original tables.h, m_fixed.h, r_main.c, d_ticcmd.h
- RoFormer: Enhanced Transformer with Rotary Position Embedding — Su et al. (2021)
- FlashAttention: Fast and Memory-Efficient Exact Attention — Dao et al. (2022)
- BitNet b1.58: 1-bit LLMs for All — Ma et al. (2024)
- The Bitter Lesson — Richard S. Sutton (2019)
- LLaMA Architecture Source Code — Meta Platforms
- Original DOOM Trig Note by @TrisH0x2A
GSM8K on Hugging Face: 4 of the Top 12 Rows Show a Sibling Checkpoint's Score, and 20 of 28 Rows Are Unmerged PRs
TL;DR
- On November 3, 2026 the Hugging Face GSM8K board lists 28 rows from 28 repos. 20 come from pull requests that are still open, 8 sit on the repo's main branch, and 0 are verified.
- Four of the top 12 rows show a number that the model card reports for a different checkpoint of the same family. Rank 1 (99.6) is the MiMo-V2.5-Pro Base score, registered on the post-trained repo. Rank 2 (96.8) is the Llama 3.1 405B Instruct score, registered on the base repo. Ranks 7 and 12 are DeepSeek base scores on the post-trained repos.
- The numbers are real self-reports. The labels are what is off, and the board has no column that would show it.
- GSM8K has 1,319 test problems, so one problem is 0.076 points. The rank 2 to rank 3 gap (0.74) is 10 problems, and a 95% interval at 95% accuracy is about 1.2 points wide on each side.
- A score of 99.6 means about 5 misses. The GSM8K-Platinum project relabeled 10 answers in the same test set. If those relabels are right, a score that high implies the model matched at least 5 of the 10 keys they call wrong.
!Top 12 rows of the GSM8K board on Hugging Face, with the four rows whose number belongs to a sibling checkpoint in red
In the chart, red marks rows where the model card gives this number for a sibling checkpoint, blue marks a row whose card table matches the repo (LLaDA2.0-flash), and grey marks rows I did not check against the card.
What does the GSM8K board on Hugging Face contain?
GSM8K is a set of grade-school math word problems with a train split of 7,473 and a test split of 1,319. Hugging Face lists openai/gsm8k as a benchmark dataset, so a model repo can publish a result in a .eval_results/*.yaml file and the dataset's board collects it.
I pulled the board on November 3, 2026 (KST) from the JSON behind the leaderboard view. It has 28 rows. Each row is from a different repo, verified is false on every row, and the top of the page reads like a clean ranking: 99.6, 96.8, 96.06, 94.16, 93.93.
The dataset's own eval.yaml describes a specific protocol: an inspect-ai task, a zero-shot prompt template that asks for a boxed final answer, four epochs reduced to pass@1, and a model-graded scorer. The rows do not follow one protocol. Of the 28 notes, 15 are empty. Where a note gives a shot count it ranges across 0, 4, 5 and 8. Two rows from one lab give a complete setup (8-shot, temperature 0, n=1319, lm-evaluation-harness), and one row reports a strict-match score over 1,075 of the 1,319 problems. So I treat each row as a claim with a source link, not as a measurement the board produced.
Why does rank 1 belong to a different checkpoint?
The top row is XiaomiMiMo/MiMo-V2.5-Pro at 99.6. Its model card has a section titled "Base Model Evaluation", and the GSM8K row there reads 8-shot with these values: 99.6 for MiMo-V2.5-Pro Base, 83.3 for MiMo-V2.5 Base, 92.6 for DeepSeek-V4-Pro Base. That is the only place GSM8K appears in the card. The card's own download table lists two repos: MiMo-V2.5-Pro and MiMo-V2.5-Pro-Base, and the second one exists on the Hub.
So the board shows the base-model score on the post-trained repo. The pull request that registered it says the results "were extracted from the model card". It was opened on April 27, 2026 and is still open. Nothing in the YAML says "base", and the 8-shot setting is not in the note, which is empty.
The card is not hiding anything. It labels its table as Base. The distinction is lost only in the step that turns a card into a board row.
Is that one slip, or does it repeat?
I checked the sources for the top rows. It repeats in four of the top 12, and in both directions.
| Board rank | Repo on the board | Score | Where the card puts this number | Sibling repo |
|---|---|---|---|---|
| 1 | MiMo-V2.5-Pro | 99.6 | "Base Model Evaluation" table, Pro Base column, 8-shot | MiMo-V2.5-Pro-Base |
| 2 | Llama-3.1-405B | 96.8 | "Instruction tuned models" table, 405B Instruct column, 8-shot CoT, em_maj1@1 | Llama-3.1-405B-Instruct |
| 7 | DeepSeek-V4-Pro | 92.6 | Base table, DeepSeek-V4-Pro-Base column, 8-shot | DeepSeek-V4-Pro-Base |
| 12 | DeepSeek-V3 | 89.3 | "Base Model" section, DeepSeek-V3 column, 8-shot | DeepSeek-V3-Base |
Rank 2 is the mirror image. The Llama 3.1 model card covers every size and both variants. Its "Base pretrained models" table has no GSM8K row. Its "Instruction tuned models" table has one, and the 405B Instruct value is 96.8. The pull request is titled "Add GSM8K eval result (96.8, 8-shot CoT)" and attaches that number to the base repo meta-llama/Llama-3.1-405B.
For DeepSeek-V3 the card has a Base Model section with GSM8K at 89.3 and a Chat Model section with no GSM8K row. For DeepSeek-V4-Pro the table header names the three columns DeepSeek-V3.2-Base, DeepSeek-V4-Flash-Base and DeepSeek-V4-Pro-Base, and 92.6 is the last one.
Why does it matter if the number is the same? Because the board is read by repo name. Someone choosing a model for math word problems sees MiMo-V2.5-Pro at the top and downloads that repo. The card reports no GSM8K number for that checkpoint. If the four rows were moved to the sibling repos, the post-trained MiMo, DeepSeek-V4-Pro and DeepSeek-V3 would have no GSM8K row at all, and the Llama 405B base repo would lose its rank 2.
The correct placement would not change the ordering of the numbers: 99.6 would still lead, from the right repo. It would change which model each number tells you about.
!Where the 28 GSM8K rows come from: open pull requests, main branch, verified
Where do the rows come from, and what does "verified" mean here?
I fetched the discussion for every row that carries a pullRequest number, and listed .eval_results on the main branch of every repo.
| Source | Rows |
|---|---|
| Open PR from an account flagged as Hugging Face (julien-c 9, nielsr 6, SaylorTwift 3, burtenshaw 1) | 19 |
| Open PR from a bot account (EvalEvalBot) | 1 |
| File on the repo's main branch | 8 |
| Verified | 0 |
All 20 pull requests are open, and for all 20 the YAML file is absent from main. For all 8 other rows it is present. Four of the 20 pull requests were opened on November 2, one day before I pulled the board, and one of them produced rank 3 (LLaDA2.0-flash, 96.06). The board moves when someone opens a pull request, with no action from the model's owner.
The Hugging Face documentation defines the badges. "Community" means a result submitted via an open PR that is not merged to main. "Verified" means a valid verify token, which requires the evaluation to have run in HF Jobs with inspect-ai. So 20 of 28 rows are community results by definition, and no row meets the verified bar. The documentation says community scores are visible while the PR is open. That is a reasonable design for bootstrapping a board. It also means a reader has to look at the badge to know which rows the model's owner stands behind, and a screenshot of the ranking does not show it.
The eight main-branch rows are not uniformly clean either. Two Mellum2 repos show the same 81.73. The YAML on the Base repo states that its numbers were measured on the pre-extension Base-Pretrain checkpoint, so it is one measurement displayed twice. This one is disclosed in the file, and it still occupies two ranks.
How much can 1,319 problems separate?
GSM8K is nearly saturated at the top, so the ordering is decided by a handful of problems.
| Quantity | Value |
|---|---|
| One problem | 0.076 points |
| Rank 1 to rank 2 gap (99.6 to 96.8) | 2.8 points, about 37 problems |
| Rank 2 to rank 3 gap (96.8 to 96.06) | 0.74 points, about 10 problems |
| Spread of ranks 3 to 6 (96.06 to 93.10) | 2.96 points |
| 95% half-width at 95% accuracy, n=1,319 | 1.18 points (illustrative) |
| 95% half-width at 99% accuracy, n=1,319 | 0.54 points (illustrative) |
The interval rows are a sampling calculation for a single run on a fixed test set. They ignore prompt variation, which is larger when rows use different shot counts. Even so, ranks 3 to 6 sit inside a band that a single run cannot separate, and the rows within it come from different protocols: 8-shot CoT for some, no note for others.
One other check is exact arithmetic. The two rows with a full setup report 84.9886 and 78.4685, and with n=1319 these are 1,121 and 1,035 correct answers. ProKope-7B's note says 1,075 of 1,319, which is 81.50. Those rows can be audited back to a count. Most rows cannot.
!How big is each effect: one problem, intervals, ceiling lost to relabeled keys, gaps between ranks
Can a 99.6 be right if some answer keys are wrong?
This part is arithmetic with one assumption, so I mark it as a question to check, not a finding.
The GSM8K-Platinum project (MIT, Gradient Science) reviewed the 1,319 test problems. Its dataset card reports 219 flagged by models, 110 rejected as poorly written, 10 relabeled, 99 verified, leaving 1,209 questions. The 10 relabeled items are answer keys that the authors found to be wrong.
Take a score of 99.6 on the original test set. 1,314 of 1,319 rounds to 99.6, so that is about 5 misses. Suppose the 10 relabels are correct and scoring is exact match against the original key. A model that solves an item correctly then loses it, because the key is wrong. A model that is right on all 10 would lose all 10 and top out at 99.24. To score 99.6 it must have matched the key on at least 5 of the 10, which means it gave the answer the key says, not the answer the authors say is right.
Three explanations fit: the relabels are wrong for some of those items, the scoring is more lenient than exact match, or the model has seen the original keys. I have not run the model, and a base model's 8-shot number with no note on answer extraction does not let me tell these apart. What I can say is that for a benchmark with known key errors, a score above the ceiling is a reason to read the setup, not a reason to rank it first. Platinum also reports that cleaning the set separates models that look tied on the original, which is the same point from the other side.
How do I pull the board and classify rows myself?
This script needs only the standard library. It counts rows by source and prints the sampling arithmetic used above.
import json, urllib.request, collections
def get(url):
req = urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0"})
return json.load(urllib.request.urlopen(req, timeout=60))
DATASET = "openai/gsm8k"
rows = get(f"https://huggingface.co/api/datasets/{DATASET}/leaderboard")
by_source = collections.Counter()
for r in rows:
pr = r.get("pullRequest")
if pr:
disc = get(f"https://huggingface.co/api/models/{r['modelId']}/discussions/{pr}")
by_source[("open PR" if disc["status"] == "open" else disc["status"])] += 1
else:
by_source["main branch"] += 1
print("rows:", len(rows), "| verified:", sum(bool(r["verified"]) for r in rows))
print(dict(by_source))
n = 1319
print("one problem = %.3f points" % (100 / n))
for acc in (0.95, 0.99):
se = (acc * (1 - acc) / n) ** 0.5 * 100
print("accuracy %.2f: 95%% half-width = %.2f points" % (acc, 1.96 * se))
print("ceiling if 10 keys are wrong: %.2f" % (100 * (n - 10) / n))
On November 3, 2026 it printed:
rows: 28 | verified: 0
{'open PR': 20, 'main branch': 8}
one problem = 0.076 points
accuracy 0.95: 95% half-width = 1.18 points
accuracy 0.99: 95% half-width = 0.54 points
ceiling if 10 keys are wrong: 99.24
The board endpoint is undocumented, so the field names may change. To check a row, open its pullRequest discussion, read the YAML, then open the source card and find the table the number came from.
Checklist for reading a saturated board
1. Open the badge before the rank. Community means an open PR, and verified means a run in HF Jobs. Neither is a statement from the model's owner.
2. Match the checkpoint, not the family. Find the card table the number came from and read its column header. Base, instruct and chat checkpoints sit in different tables.
3. Ask how many problems separate two rows. On GSM8K, 10 problems is 0.76 points.
4. Prefer rows that state their setup. Shot count, temperature, n and the answer-extraction rule decide the score more than rank does.
5. Check for the same measurement on two repos. A copied number is one result, not two.
6. Distrust a score above the key ceiling. If a benchmark has known label errors, read the evaluation setup before you trust the lead.
FAQ
How many GSM8K results on Hugging Face are verified?
None. On November 3, 2026 all 28 rows on the board show verified as false. Verified requires a valid verify token from an inspect-ai run in HF Jobs.
Is the 99.6 at rank 1 of the GSM8K board wrong?
Nothing here shows the number is wrong. It is the MiMo-V2.5-Pro Base score from the model card's Base Model table, 8-shot. The problem is that it is registered on the post-trained repo, which has its own separate row in the card's download table.
What does "community" mean on a Hugging Face leaderboard row?
It means the result was submitted through an open pull request that has not been merged to the repo's main branch. In the GSM8K board, 20 of 28 rows are in that state.
Why is the Llama 3.1 405B base repo ranked second on GSM8K?
Its row carries 96.8, which the shared model card gives for Llama 3.1 405B Instruct, 8-shot CoT. The card has no GSM8K row in its base table. The instruct repo is a separate repo.
How many problems does one point of GSM8K accuracy represent?
GSM8K has 1,319 test problems, so one point is about 13 problems and one problem is 0.076 points.
What is GSM8K-Platinum?
It is a revised version of the GSM8K test set from Gradient Science. Of 1,319 test problems it rejected 110, relabeled 10 and kept 1,209. It exists because the original set has ambiguous questions and wrong keys.
Further reading
- GPQA Diamond Has 198 Questions: What the Hugging Face Leaderboard's 115 Rows Can and Cannot Rank
- How Hugging Face Official Benchmark Leaderboards Actually Work: .eval_results YAML, the base_model Filter, and the 30% the Default View Hides
Sources
- GSM8K dataset and board: huggingface.co
- GSM8K evaluation definition (eval.yaml): huggingface.co
- Hugging Face documentation, Evaluation Results (badges, community PRs): huggingface.co
- GSM8K-Platinum dataset card: huggingface.co
- GSM8K-Platinum write-up: gradientscience.org
- Rank 1 model card (Base Model Evaluation table): huggingface.co
- Rank 1 registering pull request: huggingface.co
- Llama 3.1 model card (base and instruction tuned tables): github.com/meta-llama/llama-models/blob/main/mode…
- Rank 2 registering pull request: huggingface.co
- DeepSeek-V4-Pro model card: huggingface.co
- DeepSeek-V3 model card: huggingface.co
All counts in this article are my own measurements from the board JSON, pull request discussions, dataset cards and the cited model cards on November 3, 2026 (KST); I have no affiliation with any model, lab or benchmark named here. Which table a number came from is my reading of each card, and the board endpoint is undocumented and may change.
How long does it take to train an AI model on cloud GPUs?
Training an AI model takes anywhere from a few minutes to several months, depending on model size, how many tokens it trains on and how many GPUs run in parallel. On Nodus a small fine-tune can finish in under five minutes, while by our math a 7B language model trained from scratch on 140 billion tokens takes about 3.6 days on 64 NVIDIA H100s. The sections below show the formula, check it against published runs, and list what makes the real clock run longer than the math.
How do you estimate how long a training run will take?
Work out the total math the run needs, then divide by how fast your GPUs actually do math. Training a transformer takes about 6 floating-point operations per parameter per token, the rule from Kaplan et al.'s scaling-laws paper. An NVIDIA H100 SXM peaks at about 989 teraFLOPS of dense BF16 math, half the sparse figure on NVIDIA's H100 page.
No real run gets the full peak. The share you get is called model FLOPs utilization, and 30% is a fair planning number for code you have not tuned. That gives:
GPU-hours = (6 × parameters × tokens) / (989e12 × utilization × 3,600)
wall-clock = GPU-hours / number of GPUs (if scaling were perfect)
At 30% utilization, one H100-hour delivers about 1.07 × 10^18 useful operations. The second line is the optimistic part, and we come back to it below.
How long does it take to train a language model from scratch?
Days to weeks for small and mid-size models, if you have enough GPUs. Here is the formula at 30% utilization, using a compute-optimal budget of about 20 tokens per parameter from DeepMind's Chinchilla paper, plus one longer 7B run.
| Model and tokens | H100 GPU-hours | Fewer GPUs | More GPUs |
| --- | ---: | ---: | ---: |
| 1B on 20B tokens | 112 | 1 GPU: 4.7 days | 8 GPUs: 14 hours |
| 7B on 140B tokens | 5,505 | 8 GPUs: 28.7 days | 64 GPUs: 3.6 days |
| 7B on 2T tokens | 78,643 | 64 GPUs: 51.2 days | 512 GPUs: 6.4 days |
| 70B on 1.4T tokens | 550,500 | 512 GPUs: 44.8 days | 2,048 GPUs: 11.2 days |
Our arithmetic: 6 × parameters × tokens, 989 teraFLOPS dense BF16, 30% utilization, perfect scaling. Real runs land later; see the overheads section.
Token count moves the clock as much as model size. The same 7B model takes 14 times longer on 2 trillion tokens than on 140 billion. Many open models now train far past the Chinchilla ratio, so settle the token budget first. Our post on what it costs to train an LLM prices the same GPU-hours in dollars.
How long do real published training runs take?
Small reproductions take hours, and frontier models take months. Andrej Karpathy's llm.c reproduction of GPT-2 124M trained on 10 billion tokens in about 90 minutes on one 8 × NVIDIA A100 80GB node. His GPT-2 1.6B run took about 24 hours on one 8 × H100 node: 32,000 steps of roughly 1 million tokens each, at about 2.75 seconds per step.
That 1.6B run is a good check on the formula. It works out to 192 GPU-hours for about 33.6 billion tokens, which back-solves to roughly 46% utilization. Karpathy's own logs show 42% to 47% at the start, so well-tuned code beats our 30% planning figure by about half.
At the top end, Meta's Llama 3.1 model card lists 30.84 million H100 GPU-hours for the 405B model. The Llama 3 paper says it trained on up to 16,384 H100s. Divide one by the other and even that cluster needs about 78 days of pure compute, a floor on the real calendar time.
Do more GPUs make training proportionally faster?
Only roughly, and never for free. Adding GPUs splits the same GPU-hours across more machines, so the bill stays about the same while the calendar shrinks. In practice the bill creeps up, because GPUs spend more time talking to each other.
Meta's own numbers show the effect. The Llama 3 paper reports 43% utilization on 8,192 GPUs and 41% on 16,384 GPUs for the same model, a small but real loss from doubling the cluster.
!Days to train a 7B model on 2 trillion tokens: 409.6 days on 8 H100s, 51.2 days on 64, 6.4 days on 512, at 30% utilization and perfect scaling.
The same 78,643 GPU-hours: one 8-GPU node needs 409.6 days, a 512-GPU cluster about a week.
Our decision rule: pick the smallest GPU count that finishes before your deadline, not the largest you can get. Single nodes of 8 GPUs scale best because the GPUs share fast NVLink links. Multi-node runs add network cost and more machines that can fail.
How long does fine-tuning take?
Usually minutes to a few hours, because fine-tuning sees far fewer tokens than pretraining. Fine-tuning a 7B model on 50 million tokens is about 2 H100-hours of math at 30% utilization by the same formula, and LoRA does less than that because it skips most weight gradients.
For small jobs, the work around training often takes longer than training itself. In our published fine-tuning benchmark, Qwen2.5-0.5B on one NVIDIA L4 spent 112.8 seconds in the training loop on Nodus. From submission to a verified exported model took 4.8 minutes, or 288 seconds.
!Small fine-tune on Nodus: 112.8 seconds in the training loop versus 288 seconds from submission to exported model.
For a small fine-tune, startup, evaluation and export took longer than the training loop.
So for fine-tuning, the honest answer to "how long" is mostly startup, model download, evaluation and export. On Nodus, a TrainingJob imports a pinned Hugging Face model revision into a read-only Volume once, and later runs reuse it, so billed GPU time is not spent downloading weights.
Why does the real clock run longer than the math?
Because the formula counts only useful training steps. Four things add time on most projects:
- Waiting for capacity. A large multi-GPU request may queue until enough machines are free together.
- Startup. Pulling the image, loading the model and reading the first batch of data.
- Failures and restarts. Over a 54-day stretch of Llama 3 405B pretraining, Meta logged 466 job interruptions, 419 of them unexpected. It still reported more than 90% effective training time, meaning at least a tenth of the elapsed time did no useful training.
- Evaluation and export. Validation passes and writing out the final weights.
Our planning rule is to add at least 10% to the formula for multi-day runs. Go higher if your checkpoints are far apart, since every restart redoes the work since the last save.
How does Nodus show the time before you launch?
A dry run returns the estimate before anything is charged. For a managed TrainingJob, the training guide says the estimate is the number of steps times the measured seconds per step on the slowest GPU type you allowed, plus evaluation time. You can set spec.expectedDuration to use your own figure.
$ nodus apply -f sft.yaml --dry-run -o yaml # status.estimate: cost to completion, first hold, start time
For your own script, nodus run takes the same idea from flags. This prints the estimate only, using your expected duration:
$ nodus run --dry-run --gpu H100:8 --expected-duration 24h -- torchrun train.py
If the run must finish by a date, --complete-by (or placement.completeByTime) keeps it off any capacity whose p90 finish is later, per our placement docs. H100s are listed at $2.60/hr on the Nodus pricing page, checked October 31, 2026. If a machine is lost mid-run, Nodus starts a new attempt from the latest committed checkpoint, as described in attempts and recovery. Our post on how often spot GPUs are interrupted shows how much that matters on cheaper capacity.
FAQ
How long does it take to train a model like GPT-4?
Nobody outside the lab knows, because the GPU-hours were never published. The closest public figure is Llama 3.1 405B at 30.84 million H100 GPU-hours, which is months even on a cluster of more than 16,000 GPUs.
Can I train a useful model on a single GPU?
Yes, if it is small or a fine-tune. Karpathy notes his GPT-2 124M run would take roughly 4 to 24 hours on a single GPU, depending on the card.
Is a faster GPU always worth it for training time?
Not always. A GPU that costs twice as much per hour but runs the job 2.5 times faster is cheaper overall, and one that runs it 1.5 times faster is not. Compare per-step time on your own model. Nodus managed training runtimes publish measured step times for their presets, which is what their estimates use.
Does training time include data preparation?
No, none of the figures here do. Tokenizing and cleaning a large dataset is usually CPU work that you can finish before you pay for any GPU time.
Originally published on the Nodus blog.
@nikks_techie example: Llama 3.1 70B, bf16 KV, no prefix sharing:
320 KiB/token × 8,192 tokens × 100 active sequences = 250 GiB of cache alone.
70B isn't enough to size it. the model's KV heads and the cache dtype matter.
github.com/meta-llama/llama-models/blob/main/mode…
2 views
What Every AI Engineer (Builder) Needs to Know About the Data Behind AI
Table of Contents
Chapter 1 of 4, deep dive. Click any line to jump straight to it:
- [Introduction](#introduction)
- [Module 1.1: The Big Picture: What a Model Actually Learns](#module-11-the-big-picture-what-a-model-actually-learns)
- [1.1.1 The One Sentence Everything Builds On](#111-the-one-sentence-everything-builds-on)
- [1.1.2 The Pipeline Loop as Three Mathematical Operations](#112-the-pipeline-loop-as-three-mathematical-operations)
- [1.1.3 How Big Is "Big"?](#113-how-big-is-big)
- [1.1.4 The Quantity, Quality, Diversity Trade-off, Measured](#114-the-quantity-quality-diversity-trade-off-measured)
- [Module 1.2: The Mathematics of the Training Objective](#module-12-the-mathematics-of-the-training-objective)
- [1.2.1 Entropy, Cross-Entropy and KL Divergence](#121-entropy-cross-entropy-and-kl-divergence)
- [1.2.2 Training Is Maximum Likelihood](#122-training-is-maximum-likelihood)
- [1.2.3 The Cost of Spam, Derived](#123-the-cost-of-spam-derived)
- [1.2.4 Proof in Code: Clean vs Spam-Polluted Training](#124-proof-in-code-clean-vs-spam-polluted-training)
- [1.2.5 The ln(V) Sanity Check](#125-the-lnv-sanity-check)
- [Module 1.3: What the Model Is Truly Optimizing For](#module-13-what-the-model-is-truly-optimizing-for)
- [1.3.1 From Tokens to Loss to Perplexity](#131-from-tokens-to-loss-to-perplexity)
- [1.3.2 How the Data Pushes on the Model: the Gradient](#132-how-the-data-pushes-on-the-model-the-gradient)
- [1.3.3 Scaling Laws, With Real Numbers](#133-scaling-laws-with-real-numbers)
- [1.3.4 Compute-Optimal vs Over-Trained](#134-compute-optimal-vs-over-trained)
- [1.3.5 Where the Numbers From the Intro Come Back](#135-where-the-numbers-from-the-intro-come-back)
- [Chapter 1 Cheat Sheet](#chapter-1-cheat-sheet)
- [References](#references)
Chapters 2, 3 and 4 (cleaning the web, teaching filters to think, and the final mile into tokens and batches) publish as their own posts.
---
Introduction
I have been quiet for a while, not because I had nothing to say, but because I kept turning one question over in my head: how do people like us, without a fraction of the money the big labs spend, actually contribute to building AI that reflects who we are. The honest answer took time to arrive at, and the number that finally pushed me to write this down is too large to sit with quietly.
> Did you know? Meta's own published model card for Llama 3.1 states that training the 405-billion-parameter version alone used 30.84 million GPU-hours on H100 chips drawing 700 watts each. Run that math and it comes out to roughly 21.6 million kilowatt-hours of electricity, enough to power about 2,000 average households for a full year. Meta's own figures put the location-based emissions from that one training run at 8,930 tons of CO2, close to what 1,900 cars emit in a year. Spread across the 16,384 GPUs used, that workload still took about 78 days of continuous training, and at a modest $2 per GPU-hour cloud rate, the compute bill alone comes to roughly $62 million, before a single researcher, dataset, or building is paid for.
Sit with that $62 million for a second, then sit with this. In 2021, OpenAI contracted a company called Sama to have Kenyan workers label disturbing content so ChatGPT would not reproduce it, paying those workers roughly $1.50 to $2 an hour while internal documents showed OpenAI itself was billed $12.50 an hour for that same work. Separately, Scale AI, the company Alexandr Wang led as CEO until Meta bought 49 percent of it for $14.3 billion in 2025, has faced its own wave of US lawsuits after reports that its Remotasks platform paid some workers as little as a few cents per task. Two different companies, two separate scandals, the same pattern underneath: the humans doing the most psychologically demanding part of building this technology are paid the least, while the compute bill for the model they helped clean runs into the tens of millions.
!exploting Kenyan people to label data
Who this is for. This series is written for builders, students, self-taught engineers, researchers and curriculum designers anywhere who want to understand how large language models are actually built, not just how to call an API, especially those of us working with limited compute, limited funding and limited institutional backing. You do not need a GPU cluster or a research lab to read this. You need curiosity, a laptop, and the willingness to look closely at data most people scroll past.
And while that gap sits unresolved, the people steering this industry are moving at a different altitude entirely. In September 2025, Donald Trump told the United Nations General Assembly that the United States would start calling artificial intelligence "superintelligence," or SI, later signing an executive order requiring federal agencies to use the new term. That is the altitude at which this race is being run, nations rewriting policy and vocabulary around who leads next, while much of the data that trains these systems, our languages, our conversations, our culture, keeps flowing outward for free.
This is where the series begins, not with despair at the size of the gap, but with the one part of it we can actually close ourselves. Over the next four chapters we move from the big picture of why training data matters so completely, through where real data comes from and how it gets cleaned, into the filtering and quality techniques that separate a frontier lab's dataset from a pile of scraped pages, and finally into how that data becomes the tokens and batches a model actually trains on. None of it requires a data center. All of it requires showing up.
---
How to read this post. Every section follows the same rhythm: the idea in plain words, the math (each equation is followed by a "read it as" sentence), a worked example with real arithmetic you can check with a calculator, and why it matters for data. All code lives in one repository so this post stays readable:
!Source of truth
> 💻 Companion repo: iws3/before-we-train. Every script in chapter-01/ runs on a laptop CPU with plain Python and NumPy. No GPU needed.
Notation used throughout:
| Symbol | Meaning |
|---|---|
| x_t | the token at position t |
| x_{<t} | all tokens before position t (the context) |
| V | vocabulary size (number of distinct tokens) |
| p or p_data | the true distribution of your dataset |
| q or q_θ | the model's distribution, with parameters θ |
| n(·) | a count, e.g. n(the, cat) is how many times "cat" follows "the" |
| N | number of model parameters (used from 1.3.3 on) |
| D | number of training tokens (used from 1.3.3 on) |
| ln | natural logarithm. Losses in this post are in nats; divide by ln 2 ≈ 0.693 to get bits |
| Σ, Π | "add up" and "multiply together" over everything in the range shown |
---
Module 1.1: The Big Picture: What a Model Actually Learns
1.1.1 The One Sentence Everything Builds On
The idea. A language model becomes whatever you feed it. Not a feeling and not a simplification: it is what the training procedure is mathematically built to do. Whatever pattern sits in the text, grammar, facts, boilerplate, spam, a stereotype repeated often enough, gets absorbed. The model cannot tell your intentions apart from the data's contents. It only ever sees the data.
The math. A language model assigns a probability to any piece of text by splitting it into tokens and using the chain rule of probability:
P(x₁, x₂, …, x_T) = P(x₁) · P(x₂ | x₁) · P(x₃ | x₁, x₂) · … · P(x_T | x₁, …, x_T−1)
or, written compactly:
P(x₁, …, x_T) = ∏ P(xₜ | x₁, …, xₜ₋₁) (multiply over t = 1 … T)
Read it as: the probability of a whole text is the probability of the first token, times the probability of the second given the first, times the probability of the third given the first two, and so on.
For example, the sentence "the cat sat" is scored as:
P("the cat sat") = P(the) × P(cat | the) × P(sat | the cat)
A model with parameters θ tries to make its own probabilities match the real ones. Training picks the θ that makes the training text as probable as possible:
θ* = the θ that maximizes Σ log P_θ(documentᵢ) (add over i = 1 … n)
Read it as: "find the settings θ that give the highest total log-probability to all n training documents." (We add logs instead of multiplying probabilities; 1.3.1 explains why.)
Notice what is missing from that formula: any mention of what is good. The only input is the data. Change the data and θ* changes with it.
Worked example: a model you can compute by hand. The simplest language model is a bigram counter. It predicts the next word using only the previous word:
n(c, w)
P(w | c) = ─────────
n(c)
Read it as: the chance of word w after context word c is how often c was followed by w, divided by how often c appeared at all.
Train it on three sentences: "the cat sat", "the cat ran", "the dog sat". The word "the" starts all three, so n(the) = 3. After "the" comes "cat" twice and "dog" once:
- P(cat | the) = 2/3
- P(dog | the) = 1/3
Now an attacker, or just a careless crawler, adds three copies of the spam sentence "the free prize". The word "the" now starts six sentences, and "free" follows it three times:
| Next word after "the" | Clean corpus | After adding 3 spam lines |
|---|---|---|
| cat | 2/3 = 0.667 | 2/6 = 0.333 |
| dog | 1/3 = 0.333 | 1/6 = 0.167 |
| free | 0 | 3/6 = 0.500 |
Nobody told the model to like spam. Spam's share of the counts simply became spam's share of the probability. A transformer is far more powerful than a counter and generalizes beyond exact counts, but its training objective (Module 1.2) pulls it toward the same destination: the model's beliefs are the frequencies in your data.
Why it matters. This is why data preparation is not a chore before the real work of building AI. It is the real work. Most open models use nearly the same transformer design. What separates a strong model from a weak one is almost always what it was fed: how clean, how diverse, how deliberately chosen.
> 💻 Code: chapter-01/1_1_1_bigram_counter.py. Builds the bigram model above, prints the probability table before and after spam, and lets you change the number of spam copies to watch P(free | the) climb toward 1.
Expected Output:
1.1.2 The Pipeline Loop as Three Mathematical Operations
flowchart TD
A["Sources: web crawl, books, code, research, community data"] --> B["Extraction: raw pages to clean text"]
B --> C["Filtering: remove junk, duplicates, unsafe content"]
C --> D["Mixing: decide how much of each source to use"]
D --> E["Tokenizing and batching"]
E --> F["Training: the model learns to predict the next token"]
F -->|"evaluate, then adjust a filter or a weight"| B
Nobody gets the mixture right on the first try. Every strong dataset in use today is the record of a team running this loop dozens of times: train a small model, check what improved, adjust a filter or a weight, repeat. Anyone with a laptop can run that loop at small scale.
The idea. Every box in the diagram does one of three things to the data distribution. Once you see them as operations on a probability distribution, the whole pipeline becomes easy to reason about.
Operation 1: Mixing
If you draw documents from several sources, where source k has distribution p_k and weight w_k, the combined distribution is:
p_mix(x) = Σₖ wₖ · pₖ(x) where Σₖ wₖ = 1
Read it as: the mixed dataset is a weighted average of its sources, and the weights add up to 100%.
The Llama 3 paper reports a final mix of roughly 50% general knowledge, 25% mathematical and reasoning data, 17% code and 8% multilingual text. Those percentages are the w_k values (they add up to 100%). They are design decisions, not facts of nature.
Operation 2: Filtering
A filter assigns each document a keep-score f(x), which is 1 or 0 for a hard filter (keep or drop) and anywhere in between for a soft one. The distribution after filtering is:
p(x) · f(x)
p_filtered(x) = ─────────── where Z = Σₓ p(x) · f(x)
Z
Read it as: keep each document in proportion to its score, then divide by Z so the probabilities add up to 1 again. Z has a plain meaning: it is the keep rate, the fraction of the data that survives.
Operation 3: Deduplication
If a document appears m(x) times, its weight in the dataset is proportional to m(x). A boilerplate footer that appears in 1,000 of 1,000,000 pages carries 0.1% of the probability, while a unique page carries 0.0001%. The footer is 1,000 times over-represented. Deduplication sets every m(x) back to 1 and removes that distortion.
Worked example 1: filters multiply
Suppose a crawl passes through four stages with these keep rates (illustrative numbers):
| Stage | Keep rate |
|---|---|
| URL filter | 90% |
| Language filter | 60% |
| Quality filter | 50% |
| Deduplication | 50% |
The total keep rate is the product:
Z_total = 0.90 × 0.60 × 0.50 × 0.50 = 0.135
Only 13.5% survives. Planning backwards: if you want 1 billion clean tokens, you must start with
1 billion
───────── ≈ 7.4 billion raw tokens
0.135
Worked example 2: tokens seen versus tokens owned
A source with weight w_k in a run of D total tokens contributes w_k · D tokens. If your unique data for that source is U_k tokens, the model sees it for this many epochs (passes over the data):
wₖ · D
epochsₖ = ──────────
Uₖ
Take a 15 trillion token run and a language that gets 0.01% of the mix. Step by step:
1. Tokens drawn: 15 × 10^12 × 0.0001 = 1.5 billion
2. Unique tokens the community has collected: 200 million
3. Epochs: 1.5 billion ÷ 200 million = 7.5
Work on data-constrained scaling (Muennighoff et al., 2023) found that repeating data for up to roughly 4 epochs performs almost like fresh data, but returns fall off quickly beyond that. So for a small language, adding new unique text is worth more than upsampling what you have, and every genuinely new page counts.
Why it matters. You do not need to understand a thousand techniques. You need to ask of any pipeline step: is this mixing, filtering or deduplication, and how does it change the distribution the model will copy? The order in the diagram is also not sacred. Cheap, aggressive steps usually run first because they shrink the data before expensive steps touch it, and different teams arrange the same boxes differently. What never changes is the loop.
> 💻 Code: chapter-01/1_1_2_mixture_and_filters.py. Applies mixing weights, a chain of filters and deduplication to a toy corpus, then prints the distribution before and after each step so you can see every p_filtered change.
Expected code output:
!expected output
1.1.3 How Big Is "Big"?
The idea. A pretraining run is measured in tokens, and a token is, on average, about 0.75 of an English word. Llama 3 was trained on roughly 15 trillion tokens. What does that number mean for a human?
The math. The conversion is a chain of unit changes:
tokens × 0.75 words per token
years of reading = ─────────────────────────────────────────────────
250 words per minute × 525,600 minutes per year
Read it as: turn tokens into words, then words into minutes of reading, then minutes into years.
Worked example.
1. 15 trillion tokens × 0.75 words per token ≈ 11.25 trillion words
2. 11.25 trillion words ÷ 250 words per minute ≈ 45 billion minutes
3. 45 billion minutes ÷ 525,600 minutes per year ≈ 85,600 years of non-stop reading
What survives cleaning is small. FineWeb-Edu, the heavily filtered educational subset of the 15-trillion-token FineWeb dataset, keeps about 1.3 trillion tokens. That is the keep rate Z from the last section:
1.3 trillion
Z = ────────────── ≈ 0.087 (8.7%)
15 trillion
For every 100 pages in the full dataset, the educational filter judges about 91 unfit and keeps about 9.
A hidden cost: your language may need more tokens for the same words. The "0.75 words per token" figure is for English. The general quantity is called fertility, the average number of tokens a tokenizer needs per word:
tokens
φ = ──────── English: φ ≈ 1 / 0.75 ≈ 1.33
words
If a tokenizer was built mostly from English text, a language it sees rarely gets chopped into many small pieces, so its φ is higher. Suppose φ = 3 for some language (illustrative; you should measure your own). Three things follow directly:
| Effect | English (φ ≈ 1.33) | Your language (φ = 3) |
|---|---|---|
| Tokens needed per word | 1.33 | 3 |
| Relative cost of the same text | 1× | 3 ÷ 1.33 ≈ 2.25× |
| Words a fixed budget D teaches the model | D ÷ 1.33 | D ÷ 3 |
| Words that fit in an 8,192-token context window | 8192 × 0.75 = 6,144 | 8192 ÷ 3 ≈ 2,731 |
Tokenization (Chapter 4) is therefore not a neutral technical step. It decides how expensive your language is to learn.
> 💻 Code: chapter-01/1_1_3_scale_and_fertility.py. Reproduces the 85,600-year calculation, then measures fertility φ for any text file you point it at, so you can compare English with a local language on the tokenizer of your choice.
Expected Output:
!scale and fertility
1.1.4 The Quantity, Quality, Diversity Trade-off, Measured
The idea. Quantity, quality and diversity pull against each other. Aggressive filtering raises quality but shrinks quantity and diversity. Keeping everything preserves quantity but drowns the model in noise. Every technique in this series is a negotiation between the three. Let's make that negotiation measurable.
The math. Give every document a quality score q(x) from a classifier, and keep the documents at or above a threshold τ. Then:
Quantity: n(τ) = number of documents with q(x) ≥ τ
Quality: q̄(τ) = average of q(x) over the documents with q(x) ≥ τ
Diversity: H(τ) = − Σ_g p_g · ln(p_g)
Read it as:
- Quantity is how many documents survive the cut.
- Quality is the average score among the survivors.
- Diversity is the entropy of the share p_g that each group g (language, topic, region) holds in the kept set. It is highest when groups are balanced and zero when one group is everything.
n(τ) can only go down as τ rises, and the average quality can only go up. Diversity can do either, and raising τ can silently destroy it.
Worked example. Take 10 documents: 6 English web pages, 2 forum threads and 2 Yoruba pages. A quality classifier trained on English reference text gives these scores:
| Group | Scores | Sum |
|---|---|---|
| English web (6) | 0.90, 0.85, 0.80, 0.75, 0.70, 0.30 | 4.30 |
| Forum threads (2) | 0.60, 0.20 | 0.80 |
| Yoruba pages (2) | 0.20, 0.15 | 0.35 |
Set τ = 0.5. Only documents scoring 0.5 or higher survive: the five best English pages and the 0.60 forum thread. The 0.30 English page, the 0.20 forum thread and both Yoruba pages are removed. Here is the before and after:
| Measure | Before filter | After filter (τ = 0.5) |
|---|---|---|
| Quantity | 10 documents | 6 documents (60%) |
| Average quality | 5.45 ÷ 10 = 0.545 | 4.60 ÷ 6 = 0.767 |
| Group shares (English / forum / Yoruba) | 60% / 20% / 20% | 83% / 17% / 0% |
| Diversity H | 0.950 nats | 0.451 nats |
The two entropy values, step by step:
H_before = −( 0.6·ln 0.6 + 0.2·ln 0.2 + 0.2·ln 0.2 ) = 0.950
H_after = −( (5/6)·ln(5/6) + (1/6)·ln(1/6) ) = 0.451
Quality rose by 41% (0.767 ÷ 0.545 ≈ 1.41) and diversity fell by more than half. The Yoruba text was not low quality in any real sense. The classifier simply never learned what good Yoruba looks like.
Why it matters. FineWeb-Edu's 8.7% keep rate is quality bought at a steep, deliberate price in quantity. This example shows the second price: filters carry the biases of whatever they were trained on. Chapter 3 returns to this when we build filters that do not erase the languages we care about.
> 💻 Code: chapter-01/1_1_4_filter_tradeoff.py. Reproduces the table above, then sweeps τ from 0 to 1 and plots quantity, quality and diversity on one chart.
EXpected Output:
!filter tradeoff
[↑ Back to contents](#table-of-contents)
---
Module 1.2: The Mathematics of the Training Objective
1.2.1 Entropy, Cross-Entropy and KL Divergence
The idea. Training a language model means minimizing one quantity: the average surprise the model feels at the real next word, measured across the dataset. Information theory gives this quantity a name, cross-entropy, and splits it into two pieces with very different meanings.
The math. Let p be the true distribution of the data and q the model's distribution. Define three quantities.
Entropy, the irreducible randomness in the data:
H(p) = − Σₓ p(x) · ln p(x)
Cross-entropy, the average surprise of a model q on data that actually comes from p:
H(p, q) = − Σₓ p(x) · ln q(x)
KL divergence, how far the model is from the data:
p(x)
KL(p ‖ q) = Σₓ p(x) · ln ────
q(x)
Read them as: entropy is the surprise you'd feel even with perfect knowledge. Cross-entropy is the surprise you actually feel with an imperfect model. KL divergence is the extra surprise caused by the model being wrong.
These three are linked by one short identity. Expand the logarithm inside the KL divergence:
KL(p ‖ q) = Σₓ p(x) ln p(x) − Σₓ p(x) ln q(x)
= −H(p) + H(p, q)
Rearranging gives the decomposition everything in this series rests on:
╔═══════════════════════════════════╗
║ H(p, q) = H(p) + KL(p ‖ q) ║
╚═══════════════════════════════════╝
Read it as: total surprise = unavoidable surprise + surprise from being wrong.
Why the floor cannot move. KL divergence is never negative. The proof is short. It uses the fact that ln is concave (Jensen's inequality), which says Σ p · ln(r) ≤ ln(Σ p · r):
q(x)
−KL(p ‖ q) = Σₓ p(x) · ln ────
p(x)
≤ ln [ Σₓ p(x) · q(x)/p(x) ] (Jensen)
= ln [ Σₓ q(x) ] ≤ ln 1 = 0
So KL ≥ 0, with equality exactly when q = p. Therefore H(p, q) ≥ H(p): no model can score a cross-entropy below the entropy of the data.
Reading the two pieces:
| Piece | What it is | Can training change it? |
|---|---|---|
| H(p) | the genuine randomness of language (many words can follow "I went to the...") | No, it is a floor |
| KL(p ‖ q) | the gap between the model and the data | Yes, this is all training reduces |
Training pushes the model toward the data's distribution and nothing else. Read that with data quality in mind: the best possible outcome of training is a model that perfectly reproduces whatever is in your dataset.
> 💻 Code: chapter-01/1_2_1_entropy_kl_identity.py. Draws random distributions p and q, computes all three quantities, and checks H(p, q) = H(p) + KL to machine precision. Also checks that KL is never negative.
Expected Output:
!entropy_kl_identity
1.2.2 Training Is Maximum Likelihood
The idea. In practice we never know the true p. We only have a finite dataset. So what does training actually minimize?
The math. From a dataset of n tokens, build the empirical distribution p̂(x) = n(x) / n, the fraction of the dataset equal to each outcome. The training loss is the average negative log-probability the model assigns to the data:
L(θ) = −(1/n) · Σᵢ ln q_θ(xᵢ) = H(p̂, q_θ) = H(p̂) + KL(p̂ ‖ q_θ)
Read it as: the training loss is the cross-entropy between your dataset and the model. Minimizing it, maximizing likelihood and minimizing KL to the dataset are three names for the same thing. The target is p̂: not "language", not "truth", but your dataset.
Worked example: what is the best possible model? Let a model be a single categorical distribution with probabilities q_k over K outcomes, and let n_k be how often outcome k appears. We want to maximize the data's log-likelihood subject to the probabilities adding up to 1. Use a Lagrange multiplier λ.
Step 1. Write the objective with the constraint attached:
J = Σⱼ nⱼ · ln qⱼ − λ · ( Σⱼ qⱼ − 1 )
Step 2. Set the derivative with respect to each q_k to zero:
∂J nₖ
─── = ──── − λ = 0 ⟹ qₖ = nₖ / λ
∂qₖ qₖ
Step 3. The probabilities must add up to 1, so Σ n_k / λ = n / λ = 1, which forces λ = n. Therefore:
q*ₖ = nₖ / n
The optimal model is simply the frequency table. This is exactly the bigram counter from 1.1.1, now proven to be the best possible solution of its kind. A neural network is a vastly more flexible, smoother way of approximating the same target.
Why it matters. "The model learns from data" sounds soft. This is the hard version: the loss function has the dataset's frequencies baked in as its minimum. Remove a document and the target changes. Duplicate a document and its weight in the target grows.
> 💻 Code: chapter-01/1_2_2_mle_is_frequency.py. Runs gradient descent on a categorical model and shows the learned probabilities converge to n_k / n.
Expected Output:
!maximum likelihood frequency
1.2.3 The Cost of Spam, Derived
The idea. Suppose a fraction ε of your dataset is spam. How much does that hurt a model that trains perfectly? We can answer this exactly, with no experiment.
The math. The data is a mixture of clean text and spam:
p_data(x) = (1 − ε) · p_clean(x) + ε · p_spam(x)
A perfect training run reaches q = p_data (from 1.2.2). Now measure that perfect model on clean held-out text. Assume spam and clean text look different, so on clean text only the first term matters: p_data(x) = (1-ε) · p_clean(x). The loss is:
L_clean = − Σₓ p_clean(x) · ln[ (1 − ε) · p_clean(x) ]
= H(p_clean) + ln( 1 / (1 − ε) )
└──────────┘ └─────────────────┘
best a clean-trained the spam tax
model can do
Because perplexity is e^L, the spam tax becomes a clean multiplier:
1
perplexity inflation = e^(spam tax) = ─────
1 − ε
Read it as: if 10% of your data is spam, a perfectly trained model is 1 ÷ 0.9 = 1.11 times more confused on clean text than a model trained on clean data alone.
Worked example.
| Spam share ε | Tax in nats −ln(1−ε) | Perplexity multiplier 1/(1−ε) |
|---|---|---|
| 1% | 0.0101 | 1.010× |
| 5% | 0.0513 | 1.053× |
| 10% | 0.1054 | 1.111× |
| 25% | 0.2877 | 1.333× |
| 50% | 0.6931 | 2.000× |
The intuition is wasted probability. A model that has perfectly learned your data gives spam-shaped text a total probability of ε, so only 1 − ε is left for everything else. There is a second consequence worth more than the perplexity number: when you sample text from this model, a fraction ε of what it writes is spam. Pollution does not just make the model worse at measuring good text. It makes the model produce bad text at the rate it appeared in training.
A caveat for honesty. This is a best case. Real spam overlaps partly with ordinary language, and real models do not reach q = p_data exactly. The formula is a clean lower bound on the damage, not a full picture.
1.2.4 Proof in Code: Clean vs Spam-Polluted Training
The idea. Let's test the derivation with the smallest possible experiment. I built a character-level language model that predicts the next character using nothing but counts, and trained two copies on the same public-domain text: one clean, and one with spam lines inserted after roughly every quarter of the lines.
held-out loss, trained on clean text: 2.4838
held-out loss, trained on spam-polluted text: 2.6550
Both models were tested on the same clean held-out text. The math check: the loss gap is 2.6550 − 2.4838 = 0.1712 nats, so the perplexity ratio is
PPL_polluted
──────────── = e^(2.6550 − 2.4838) = e^0.1712 ≈ 1.187 (equivalently 14.2 / 12.0)
PPL_clean
If the bound from 1.2.3 were tight, a multiplier of 1.187 would mean 1/(1−ε) = 1.187. Solve for ε:
1
ε = 1 − ─────── ≈ 0.16
1.187
So the polluted model behaves as though about 16% of its probability mass is tied up in spam-shaped patterns. Treat that as a rough reading, since a toy counter does not satisfy the clean separation assumption exactly, but the order of magnitude is what the theory predicts.
Sample text generated from the polluted model shows fragments of the inserted spam, words like CLICK, BUY, FREE and www, sitting inside what should have been ordinary prose. A transformer with billions of parameters is a far better copier than this toy model. It will learn the spam more precisely, not less.
> 💻 Code: chapter-01/1_2_4_spam_pollution_experiment.py. Trains both character models on a public-domain text file, prints held-out loss and perplexity, measures the real spam share ε, and compares the observed perplexity ratio against 1/(1−ε). Change the spam rate to reproduce the table in 1.2.3.
1.2.5 The ln(V) Sanity Check
The idea. At the very start of training, before the model has learned anything, it should know nothing. What loss does knowing nothing give?
The math. A model that knows nothing assigns the same probability to every token: q(x) = 1/V. Plug that into the cross-entropy. This works for any data distribution p:
H(p, q) = − Σₓ p(x) · ln(1/V) = ln V · Σₓ p(x) = ln V · 1 = ln V
So the starting loss should sit almost exactly at ln V, and the starting perplexity at exactly V (the model is as unsure as if choosing among all V tokens).
Worked example.
| Tokenizer | Vocabulary V | Expected starting loss ln V |
|---|---|---|
| Llama 2 | 32,000 | 10.37 |
| GPT-2 | 50,257 | 10.82 |
| Llama 3 | 128,256 | 11.76 |
How to use it as a debugging tool. Compare your first logged loss against ln V:
- Far above ln V: the model starts overconfident, often because initial weights are too large.
- Far below ln V: the model somehow already knows the answers. Suspect label leakage, where the targets are accidentally visible in the inputs, or a data bug that makes text trivially predictable (heavy duplication, for instance).
- Right at ln V but never falling: the model is not receiving a usable signal. Check that inputs and targets are properly aligned and that the data is not noise.
This single check has caught more pipeline bugs than almost any other. It costs one print statement.
> 💻 Code: chapter-01/1_2_5_ln_vocab_check.py. Computes ln V for common tokenizers and runs a randomly initialized model on a batch to confirm the starting loss lands where theory says.
What this means practically: the moment you accept a document into your training set, you are voting for the model to reproduce it. Every filtering decision in the rest of this series exists because of this one mathematical fact.
[↑ Back to contents](#table-of-contents)
---
Module 1.3: What the Model Is Truly Optimizing For
1.3.1 From Tokens to Loss to Perplexity
The idea. A document is a sequence of tokens, and its probability is the product of each token's probability given what came before. Products of many small numbers collapse toward zero, so in practice everything runs in log space, where multiplication becomes addition.
The math. From the chain rule in 1.1.1, take the logarithm and average over the T tokens. The flipped, averaged log-probability is the loss:
L = −(1/T) · Σₜ ln p_θ(xₜ | x<ₜ) (add over t = 1 … T)
Perplexity is e raised to the loss:
PPL = e^L = ( ∏ₜ 1 / p_θ(xₜ | x<ₜ) ) ^ (1/T)
Read it as: perplexity is the geometric mean of the inverse probabilities. It has an intuitive reading: if the model were choosing uniformly among k options at every step, each probability would be 1/k and PPL = k. A perplexity of 15 means the model is, on average, about as unsure as if picking among 15 equally likely options.
Worked example. A four-token sentence where the model assigns probabilities 0.5, 0.25, 0.1 and 0.8 to the true tokens:
| Token | Probability | ln p |
|---|---|---|
| 1 | 0.50 | −0.6931 |
| 2 | 0.25 | −1.3863 |
| 3 | 0.10 | −2.3026 |
| 4 | 0.80 | −0.2231 |
| Total | product = 0.01 | −4.6051 |
Then:
4.6051
L = ──────── = 1.1513 PPL = e^1.1513 ≈ 3.16
4
Check it the other way, straight from the product: (1 ÷ 0.01)^(1/4) = 100^0.25 = 3.16. ✓
Why log space is not optional. A 1,000-token document where each token gets probability 0.1 has probability 10^-1000. The smallest number a 64-bit float can hold is about 10^-308. Multiplying directly gives zero, and zero's logarithm is minus infinity. Summing logs gives the stable answer:
Σ ln 0.1 (1,000 times) = 1000 × (−2.3026) = −2302.6
A trap: perplexity is not comparable across tokenizers. Per-token loss depends on how text was chopped into tokens. A tokenizer that splits text into many tiny, easy-to-predict tokens gets a lower per-token loss without being a better model. The fair, tokenizer-independent measure is bits per byte (BPB):
T × L
BPB = ────────
B × ln 2
where T is the number of tokens, L the loss per token in nats, and B the number of bytes in the text. The ln 2 converts nats to bits.
Worked example. A document of 1,000 tokens and 4,200 bytes at L = 2.3 nats per token:
1000 × 2.3 2300
BPB = ───────────── = ────── ≈ 0.790
4200 × 0.6931 2911
When you compare your model against someone else's, compare BPB, not perplexity. This matters especially for African languages, where tokenizers differ most.
> 💻 Code: chapter-01/1_3_1_perplexity_and_bpb.py. Computes the worked example both ways, demonstrates the underflow failure of naive multiplication, and converts loss to bits per byte.
1.3.2 How the Data Pushes on the Model: the Gradient
The idea. Now we can look inside a single training step and see exactly how data shapes the model. A model outputs a score (a logit) z_i for each of the V tokens, and a softmax turns scores into probabilities.
The math. The softmax and the loss for one example, where y is the true next token:
e^zᵢ
pᵢ = ──────── ℓ = −ln p_y = −z_y + ln Σⱼ e^zⱼ
Σⱼ e^zⱼ
Differentiate the loss with respect to a logit z_i. The first term contributes −1 only when i = y, and the second term differentiates to the softmax probability itself:
╔══════════════════════════════════════╗
║ ∂ℓ / ∂zᵢ = pᵢ − 𝟙[ i = y ] ║
╚══════════════════════════════════════╝
Read it as: the gradient is just what the model predicted minus what actually happened. (𝟙[i = y] is 1 for the true token and 0 for every other token.) Gradient descent moves each logit against this gradient: it raises the true token's score and lowers all the others in proportion to how much probability they stole.
Worked example. Three tokens with logits (2, 1, 0), and the true token is token 3. The softmax needs e² = 7.389, e¹ = 2.718, e⁰ = 1, which sum to 11.107:
| Token | Probability | Gradient p_i − 𝟙[i=y] | What the update does |
|---|---|---|---|
| 1 | 7.389 ÷ 11.107 = 0.665 | +0.665 | push score down |
| 2 | 2.718 ÷ 11.107 = 0.245 | +0.245 | push score down |
| 3 (true) | 1 ÷ 11.107 = 0.090 | 0.090 − 1 = −0.910 | push score up hard |
The loss is −ln 0.090 = 2.41. The true token was given only 9% probability, so the push is strong. The gradient entries sum to exactly zero (0.665 + 0.245 − 0.910 = 0), because probability moved is probability conserved.
Averaging over the dataset. For a fixed context c, the average gradient over all the training examples that share that context is:
E_data[ ∇_z ℓ ] = q_θ(· | c) − p_data(· | c)
Read it as: the average push equals "what the model believes" minus "what the data says." This is zero only when the model's distribution matches the data's distribution for that context. The update never stops pulling until the model has copied the data. If spam follows context c with frequency ε, then spam tokens receive a pull proportional to ε on every step, for as long as training runs. This is the one-line mechanism behind "the model becomes whatever you feed it."
> 💻 Code: chapter-01/1_3_2_softmax_gradient_check.py. Verifies the formula p − onehot against numerical finite-difference gradients, reproduces the three-token example, and shows the averaged gradient going to zero as the model matches the data.
1.3.3 Scaling Laws, With Real Numbers
The idea. Scaling laws describe how loss falls as a model and its training data both grow. The widely cited Chinchilla work (Hoffmann et al., 2022) fits a simple form with three terms: a floor, a term that shrinks as the model grows, and a term that shrinks as the data grows.
The math. With N parameters and D training tokens:
L(N, D) = E + A / N^α + B / D^β
───── ─────────── ───────────
floor model-size data-size
term term
The fitted constants reported in the paper's third estimation approach are approximately:
| Constant | Value | Meaning |
|---|---|---|
| E | 1.69 | irreducible loss, the entropy floor H(p) from 1.2.1 |
| A | 406.4 | model-size coefficient |
| B | 410.7 | data-size coefficient |
| α | 0.34 | how fast loss improves with more parameters |
| β | 0.28 | how fast loss improves with more data |
These constants belong to the paper's own dataset and tokenizer, so treat the absolute losses as illustrative rather than as predictions for your model. What transfers is the shape.
Worked example 1: what doubling the data buys. The data term scales as D^−β, so doubling D multiplies it by:
2^(−0.28) ≈ 0.824
Doubling your data shrinks the data-related part of the loss by about 18%. It never touches E.
Worked example 2: two real model sizes. Take Llama 3's 8B model at 15T tokens, and the 405B model at about 15.6T tokens. Each term, computed from the formula:
| | Model term A / N^α | Data term B / D^β | Total L = 1.69 + ... |
|---|---|---|---|
| 8B params, 15T tokens | 0.175 | 0.084 | 1.69 + 0.175 + 0.084 ≈ 1.95 |
| 405B params, 15.6T tokens | 0.046 | 0.083 | 1.69 + 0.046 + 0.083 ≈ 1.82 |
Look at the 405B row. The model-size term has shrunk to 0.046, while the data term (0.083) is nearly double it. For a model that large, data is the term still holding the loss up. This is the mathematical reason the field's appetite for clean, usable text keeps climbing.
Where data quality enters. The formula counts tokens, but not all tokens are equal. Better filtering behaves like a larger effective D, or equivalently a smaller B. This is the quantitative form of "a smaller, cleaner dataset can beat a larger, dirtier one."
> 💻 Code: chapter-01/1_3_3_chinchilla_calculator.py. Implements L(N, D) with the fitted constants, reproduces both rows of the table, and plots loss curves as you vary N and D.
1.3.4 Compute-Optimal vs Over-Trained
The idea. Given a fixed compute budget, how should you split it between a bigger model and more data?
The math. Training compute is approximately 6 floating-point operations per parameter per token:
C ≈ 6 · N · D
Substitute D = C / (6N) into L(N, D) and set the derivative with respect to N to zero. The result is that the best N and D both grow with compute, as power laws:
N_opt ∝ C^( β / (α+β) ) = C^( 0.28 / 0.62 ) ≈ C^0.45
D_opt ∝ C^( α / (α+β) ) = C^( 0.34 / 0.62 ) ≈ C^0.55
Read it as: when your budget grows, spend it on a bigger model and more data, at close to the same rate. The practical rule of thumb that came out of this is roughly 20 training tokens per parameter.
Worked example: how far modern models sit from that rule.
| Model | Parameters | Tokens | Tokens per parameter |
|---|---|---|---|
| Chinchilla | 70B | 1.4T | 20 |
| Llama 3 405B | 405B | ~15.6T | ~38 |
| Llama 3 8B | 8B | 15T | ~1,875 (about 94× the rule) |
Why anyone over-trains. The "optimal" above minimizes loss for a fixed training budget. But a model is trained once and used many times. Inference costs roughly 2 operations per parameter per token served, so with T tokens served over a model's lifetime:
inference cost 2 · N · T T
────────────── = ─────────── = ─────
training cost 6 · N · D 3 D
The model size N cancels out. If a model serves more than 3D tokens in its lifetime, inference dominates the bill. For D = 15T, that threshold is 3 × 15T = 45 trillion tokens served, which a widely deployed model can pass. So it pays to train a smaller model on far more data: slightly worse for the training bill, much cheaper for everyone who runs it afterward. That single design choice is why a smaller model's hunger for data keeps rising even as model sizes stay similar. It is also good news for builders with limited compute: a small model trained on excellent data is exactly the shape of model we can afford to run.
> 💻 Code: chapter-01/1_3_4_compute_optimal_and_inference.py. Finds the loss-minimizing N and D for a given compute budget, and finds the break-even point where inference cost passes training cost.
1.3.5 Where the Numbers From the Intro Come Back
This is where the opening numbers of the post connect back to the mathematics. Meta's published figures say training the 405-billion-parameter model consumed 30.84 million GPU-hours. Check that against C ≈ 6ND.
Step 1: total compute needed.
C ≈ 6 × (405 × 10⁹) × (15.6 × 10¹²) ≈ 3.8 × 10²⁵ FLOPs
Step 2: total GPU-seconds available.
30.84 × 10⁶ hours × 3600 s/hour ≈ 1.11 × 10¹¹ GPU-seconds
Step 3: compute delivered per GPU per second.
3.8 × 10²⁵
──────────── ≈ 3.4 × 10¹⁴ FLOP/s per GPU
1.11 × 10¹¹
An H100's peak throughput in the precision used for training is on the order of 10^15 FLOP/s, so each GPU ran at roughly a third of peak, which is in the normal range for large training runs. The introduction's numbers are not exaggerations; they follow from the same arithmetic as the equation above.
All of that compute exists to shrink one number, the loss, by pushing the model closer to the data it was shown. In the language of 1.2.1, it drives KL(p̂ ‖ q_θ) toward zero and leaves H(p) exactly where it is. None of it can make the model smarter than its data. A perfectly optimized model trained on a flawed dataset is still a flawed model, just a very confident one.
What you can do on a laptop. You cannot afford 30.84 million GPU-hours. You can afford to measure the data term B / D^β for your own data. Train a tiny model on 1%, 2%, 4%, 8% of your corpus, plot held-out loss against tokens, and fit a line on log-log axes. The slope is your β. Compare two versions of your pipeline (before and after a filter) and you have a number for whether the filter helped. That is the loop from 1.1.2, with math attached.
> 💻 Code: chapter-01/1_3_5_mini_scaling_experiment.py. Trains small models on growing slices of a text file, fits L ≈ E + B / D^β on the results, and compares two dataset versions side by side.
[↑ Back to contents](#table-of-contents)
---
Chapter 1 Cheat Sheet
| Idea | Equation | One-line meaning |
|---|---|---|
| Chain rule | P(x) = Π P(xₜ ∣ x<ₜ) | text probability is a product of next-token probabilities |
| Mixing | p_mix = Σ wₖ pₖ | source weights are design decisions |
| Filtering | p_f = p·f / Z | Z is the keep rate |
| Cross-entropy split | H(p,q) = H(p) + KL(p‖q) | floor plus the only part training can reduce |
| MLE optimum | q*ₖ = nₖ / n | the best model is the frequency table of your data |
| Spam tax | PPL × 1/(1−ε) | spam wastes exactly its share of probability |
| Starting loss | L₀ = ln V | the free bug detector |
| Perplexity | PPL = e^L | effective number of equally likely choices |
| Bits per byte | BPB = T·L / (B·ln 2) | tokenizer-independent comparison |
| Gradient | ∂ℓ/∂zᵢ = pᵢ − 1[i=y] | predicted minus actual |
| Scaling law | L = E + A/Nᵅ + B/Dᵝ | floor + model term + data term |
| Training compute | C ≈ 6ND | cost grows with both size and data |
| Inference vs training | T / 3D | model size cancels out |
Try it yourself. Four exercises, solutions in the repo under chapter-01/exercises/:
1. In 1.1.1, how many spam copies make P(free | the) exceed 0.9?
2. A dataset is 8% spam. By what factor does a perfect model's perplexity on clean text rise? What fraction of its generated text will be spam?
3. A tokenizer has V = 65,536. What starting loss should you expect?
4. In 1.3.3, how many tokens would the 8B model need to push the data term B / D^β below 0.05?
---
References
- Meta AI, Llama 3.1 Model Card: official source for the 30.84 million GPU-hour, 700W and CO2 figures, and used again in 1.3.5.
- TIME, "Exclusive: OpenAI Used Kenyan Workers on Less Than $2 Per Hour to Make ChatGPT Less Toxic", January 18, 2023: the original investigation, based on payslips and internal documents.
- CBS News, "Kenyan workers with AI jobs thought they had tickets to the future until the grim reality set in", 60 Minutes, 2025: confirms the $12.50-to-$2 pay gap and separately implicates Scale AI's Remotasks platform.
- Alexandr Wang, Wikipedia: background on Scale AI and Meta's 2025 investment.
- Fox Business, ["Trump rebrands AI, rejects 'globalist scheme'
27 reactions · 5 comments
I HAVE JUST FOUND SOMETHING INSANE
Meta's official Llama repo has a default sample image
it's a dog onboard
github.com/meta-llama/llama-models/blob/main/mode…
Llama 4's multimodal chat script loads this dog every time a dev tests vision. Meta's AI has a mascot and they never told anyone
Also pumpfun is spending millions on onboarding. In callout rewards, updates, KOLs and now referrals.
never been run wtf??
So genny
onboarddog - $onboard
407 views
I HAVE JUST FOUND SOMETHING INSANE
Meta's official Llama repo has a default sample image
it's a dog onboard
github.com/meta-llama/llama-models/blob/main/mode…
Llama 4's multimodal chat script loads this dog every time a dev tests vision. Meta's AI has a mascot and they never told anyone
Also pumpfun is spending millions on onboarding. In callout rewards, updates, KOLs and now referrals.
never been run wtf??
So genny
onboarddog
2 likes · 83 views
A Xiaomi treinou uma IA gigante ao vivo, com o custo na tela #shorts
O MiMo V2.6 foi treinado por reforço com painel público e custo aberto: US$ 2,6 milhões, ao vivo. DeepSeek, Meta, BLOOM, OLMo e SmolLM3 já tinham aberto partes do treino. O que é inédito aqui, e o furo que a transparência revelou.
Painel ao vivo do treino: mimo.xiaomi.com
Post oficial MiMo-V2.6: mimo.mi.com
Relatório técnico: arxiv.org
Framework de treino (verl): XiaomiMiMo/verl
Ambientes de RL: huggingface.co
Auditoria Vals AI: vals.ai
Comparação:
DeepSeek-V3 (custo): arxiv.org
DeepSeek-R1 na Nature: nature.com
Llama 3.1 (horas de GPU): github.com/meta-llama/llama-models/blob/main/mode…
BLOOM (logs ao vivo, 2022): huggingface.co
OLMo 2: allenai.org
SmolLM3 (logs e checkpoints): huggingface.co
Ideia: vídeo de bycloud.
#opensource #programacao #ia
32 views
Mentions
Can you train your own LLM, and what does it actually take? Yes, you can train your own LLM, and for most teams that means fine-tuning an open model, not building one from scratch. On Nodus, fine-tuning, continued pretraining and scratch pretraining all run on one managed training runtime. By our math they cost about $2, $1,200 and $19,000 in GPU time for an 8B model, but the deciding factor is how much data you have. What are the three ways to train your own LLM? There are three paths, and they differ by how much of the model you change. Each one starts from a different place and needs a different amount of text. | Path | Starts from | What it changes | Data you need | |---|---|---|---| | Fine-tuning (LoRA or full) | An open model's weights | Behavior: format, tone, task skill | Thousands of examples, millions of tokens | | Continued pretraining | An open model's weights | Knowledge of a domain or language | Billions of tokens of raw text | | Pretraining from scratch | Only an architecture and tokenizer | Everything | Hundreds of billions of tokens or more | Fine-tuning teaches a capable model to behave a certain way on your task. Continued pretraining keeps training the base model on raw domain text, such as legal filings or a low-resource language, before any fine-tuning. Scratch training begins with random weights, so the model learns language itself from your data. If what you really need is fresh facts at answer time, you may not need training at all. Our post on RAG vs fine-tuning covers that choice. How much does each path cost for an 8B model? Fine-tuning costs a few dollars, continued pretraining about a thousand, and scratch training about nineteen thousand. We priced all three for the same 8-billion-parameter model on NVIDIA H100s, listed at $2.60/hr on the Nodus pricing page. The method comes from our post on what it costs to train an LLM. It assumes about 6 operations per parameter per training token and an H100 at 989 teraFLOPS of dense BF16 math. At 30% utilization, one H100-hour delivers about 1.07 × 10^18 useful operations. | Path (8B model) | Training tokens | H100-hours | GPU cost | |---|---|---|---| | LoRA fine-tune | 20 million | 0.9 | $2.34 | | Continued pretraining (full) | 10 billion | 449 | $1,168 | | Scratch, compute-optimal | 160 billion | 7,190 | $18,695 | Prices checked November 5, 2026. These are planning numbers for GPU time only, not quotes. The 20 million tokens are, for example, 10,000 examples of 2,000 tokens each. The 10 billion tokens are our example size for a domain corpus. The 160 billion tokens follow the rule of about 20 tokens per parameter from DeepMind's Chinchilla paper. We use 6 operations per token for LoRA too: gradient checkpointing adds back about as much work as frozen weights save, the same assumption as our 70B fine-tuning cost post. !GPU cost for an 8B model at $2.60 per H100-hour and 30% utilization: LoRA fine-tune on 20 million tokens $2.34, continued pretraining on 10 billion tokens $1,168, scratch training on 160 billion tokens $18,695. Each step down the ladder costs roughly 16 to 500 times more GPU time than the one before. How long does each path take? A LoRA fine-tune of an 8B model takes about an hour on one GPU, while the other two take days. The H100-hours above divide across however many GPUs you run. Full training needs more memory than one GPU has. Adam in mixed precision keeps about 16 bytes per parameter, per the ZeRO paper, so an 8B model carries about 128 GB of training state before activations. That rules out one 80 GB H100 and points to an 8-GPU machine with the state sharded across it. On one 8 × H100 machine, continued pretraining on 10 billion tokens takes about 56 hours. Compute-optimal scratch training takes about 899 hours, or 37 days. !Wall-clock hours for an 8B model: LoRA fine-tune 0.9 hours on one H100, continued pretraining 56 hours on 8 H100s, scratch training 899 hours on 8 H100s. Scratch training on a single machine runs for over a month, so it usually means several machines at once. Nodus multi-node training (Beta) runs one job across 2 to 8 machines. Eight 8 × H100 machines bring the scratch run down to about 112 hours, or 4.7 days, for the same $18,695. Our post on how many GPUs you need shows how to size this for other models. Is training from scratch ever worth it? Rarely, because the open models you would compete with saw far more data than a budget scratch run can afford. Meta's Llama 3.1 model card says the family was pretrained on about 15 trillion tokens. That is about 94 times the 160 billion tokens of our compute-optimal 8B run. The card also lists 1.46 million H100 GPU-hours for the 8B model alone. So a scratch 8B model for $18,695 will usually be weaker at general language than a free open model you fine-tune for a few dollars. Scratch training still makes sense in a few cases: - You need a tokenizer or architecture that no open model has, such as a new script or a non-text modality. - Your domain looks nothing like web text, such as genomic or machine-log sequences. - You need full control over every token the model saw, for licensing or research. - The model is small. Our cost post puts a compute-optimal 1B model at about $292. Which path should you pick? Pick the cheapest path your data and goal justify, and move down only when the one above has failed on held-out tests. This is the rule we use. 1. Start with an open model, prompting and retrieval. Measure it on 50 to 100 real tasks you keep aside. 2. If it knows enough but behaves wrong (format, tone, tool calls, a narrow task), fine-tune with LoRA. Our LoRA vs QLoRA post covers the memory trade-off. 3. If it lacks the vocabulary and facts of your field, and you have billions of tokens of clean domain text, run continued pretraining, then fine-tune on top. 4. Train from scratch only for one of the reasons in the previous section. The data test settles most cases. If your corpus is measured in millions of tokens, a fine-tune or a retrieval index fits it, and continued pretraining has too little to learn from. What does it take to run each path on Nodus? One training runtime covers all three paths, so moving down the ladder changes a few lines, not your tooling. The managed nodus/sft runtime does supervised fine-tuning (full, LoRA or QLoRA), and with task: Pretrain it does continued or from-scratch pretraining. This example is from the Nodus training guide: spec: runtime: nodus/sft task: Pretrain model: {uri: hf://Qwen/Qwen3-0.6B@c1899de289a04d12100db370d81485cdf75e47ca} data: {volume: corpus, format: JSONL} parameters: {steps: 10000, initialization: Scratch, packing: true} resources: {gpu: {type: [h100-sxm-80g], count: 8}} initialization: Scratch keeps only the model's architecture and tokenizer. Continued, the default, keeps its weights, which turns the same file into continued pretraining. For a fine-tune you drop task: Pretrain and add a lora block, as the guide shows. Price the run before it starts. nodus apply -f sft.yaml --dry-run -o yaml returns an estimate of cost to completion, built from steps times measured seconds per step. Then set spec.maxCostUSD: when the run reaches the cap it is suspended, not failed, and raising the cap resumes it from its checkpoint. Training runs are checkpointed, and a lost machine resumes from the latest checkpoint, so a long pretraining run does not restart at zero. FAQ Can I train my own LLM on a single consumer GPU? You can fine-tune on one, but not pretrain a useful model. QLoRA stores the base in 4 bits, so an 8B model's weights take about 4 GB, and the QLoRA paper fine-tuned a 65B model on one 48 GB GPU. Nodus lists the 24 GB RTX 4090 at $0.29/hr on its pricing page. Does fine-tuning add new knowledge to a model? Not reliably. In a 2023 comparison of knowledge injection, retrieval beat unsupervised fine-tuning for both facts seen in training and new ones. Use fine-tuning for behavior and retrieval for facts that change. How do I know if my trained model is actually better? Compare it with the untouched base model on the same held-out tasks you set aside before training. On Nodus, set data.validationFraction to pick the checkpoint with the lowest validation loss, then judge quality on a separate test set, as the training guide advises. Is training on Nodus generally available? TrainingJobs are in Beta, and multi-node training is also in Beta, so fields may still change. Each change is announced in the changelog with a migration note. Originally published on the Nodus blog.
The Carmack Pipeline: How DOOM's 1993 Hacks Predict Modern LLM Architecture > "People who are really serious about software should make their own hardware." — Alan Kay John Carmack wrote DOOM in 1993 on hardware that makes a modern smart thermostat look like a supercomputer. No GPU. No FPU on mainstream target machines. 4 MB of RAM. And yet DOOM rendered a pseudo-3D world at 35 fps, sustained multiplayer over dial-up links, and kept every networked PC in bitwise lock-step synchrony. Thirty years later, the systems engineers scaling frontier large language models are solving structurally identical problems. The bottleneck is unchanged: your math is too slow for your silicon, so you change your math. This article traces five direct lines from DOOM's source code to the systems architecture of modern large language models. --- 1. The Root Problem: Your Floating-Point is Too Expensive DOOM's Situation (1993) The Intel 80386 and 80486SX processors — the dominant consumer CPUs when DOOM shipped — had no hardware Floating-Point Unit (FPU). Any floating-point operation invoked software emulation, running 50–300× slower than integer instructions. For a real-time rasterizer transforming wall vertices and actor projections every frame, floating-point math was a non-starter. Carmack's solution was to banish floating-point numbers from the engine entirely, standardizing on 16.16 fixed-point arithmetic: // DOOM's 16.16 Fixed-Point (m_fixed.h / Watcom C 1993) #define FRACBITS 16 #define FRACUNIT (1 << FRACBITS) // 1.0 = 65536 typedef int fixed_t; // In 1993, 32-bit compilers lacked 64-bit integer types. // Carmack used Watcom inline x86 assembly: // imul edx -> 64-bit product in EDX:EAX // shrd eax, edx, 16 -> shift 16 bits from EDX into EAX (fixed-point scaling) #pragma aux FixedMul = \ "imul edx" \ "shrd eax, edx, 16"\ parm [eax] [edx] // Modern portable C equivalent: // fixed_t FixedMul(fixed_t a, fixed_t b) { // return (fixed_t)(((int64_t)a * b) >> FRACBITS); // } Modern LLMs: Memory Walls and Low-Precision Quantization A 70-billion-parameter model stored in FP32 requires 280 GB of memory just for its weights. During inference, memory bandwidth — not compute throughput — is the primary wall. Accelerators like NVIDIA H100s can perform tensor operations far faster than High Bandwidth Memory (HBM) can deliver the weight matrices to the compute cores. The modern response mirrors Carmack's: replace wide floating-point types with compact integers. | Format | Bits per Weight | Memory vs. FP32 | Primary Benefit | | :--- | :--- | :--- | :--- | | FP32 | 32 | 1× | Baseline precision | | FP16 / BF16 | 16 | 2× | Half-precision training standard | | INT8 | 8 | 4× | Standard server-side inference quantization | | INT4 (AWQ/GPTQ) | 4 | 8× | Consumer GPU deployment (fits 70B in 48 GB) | | Ternary (BitNet 1.58b) | ~1.58 | ~20× | Replaces matrix multiplies with pure addition | In PyTorch, post-training quantization moves weights from FP32 down to 8-bit or 4-bit integer containers: import torch # Standard FP32 weight matrix: 4096 x 4096 = 64 MB weight_fp32 = torch.randn(4096, 4096) # INT8 per-channel quantization (16 MB — 4x memory bandwidth reduction) scales = weight_fp32.abs().max(dim=1).values / 127.0 zero_points = torch.zeros(4096, dtype=torch.int32) weight_int8 = torch.quantize_per_channel( weight_fp32, scales=scales, zero_points=zero_points, axis=0, dtype=torch.qint8 ) # In advanced INT4 kernels (e.g., bitsandbytes, AWQ, Marlin), pairs of 4-bit # weights pack into single uint8 bytes, cutting storage to ~8 MB (8x reduction). Carmack eliminated the FPU because integer ALUs were fast and coprocessors were absent. Modern ML engineers downcast to INT8/INT4/FP8 because memory buses cannot feed FP32 weights fast enough. The constraint shifted from compute cycles to bus width, but the architectural pattern is identical. --- 2. Precomputed Trigonometry: LUTs Then, RoPE Now DOOM's Binary Angular Measurement (BAM) Rather than storing angles in degrees (0–360) or radians (0–2π), DOOM maps a complete circle to the entire domain of an unsigned 32-bit integer (0 to 2^32 - 1): // tables.h typedef unsigned angle_t; #define ANG45 0x20000000 // 2^32 / 8 #define ANG90 0x40000000 // 2^32 / 4 #define ANG180 0x80000000 // 2^32 / 2 #define ANG270 0xc0000000 // 3 * 2^32 / 4 This representation is called Binary Angular Measurement (BAM). Its advantage: circular wrapping is free. When an actor turns past 360°, native 32-bit integer overflow automatically wraps the value modulo 2^32. No conditionals (if (ang >= 360)), no branching, and no division instructions. The Shared-Array Cosine Trick To evaluate trigonometric functions without runtime series expansion, Carmack discretized the circle into 8,192 fine angles (FINEANGLES = 8192). Converting an angle_t into a table index requires only a 19-bit right shift (32 - 13 = 19): #define FINEANGLES 8192 #define ANGLETOFINESHIFT 19 // Sized to 5/4 * 8192 = 10,240 entries (full 360° circle + 90° padding) extern fixed_t finesine[5 * FINEANGLES / 4]; // Cosine is a pointer offset 90° (2,048 entries) into the sine buffer fixed_t* finecosine = &finesine[FINEANGLES / 4]; Because cos(θ) = sin(θ + 90°), finecosine requires no dedicated array. It aliases the same contiguous buffer at a 2,048-element offset. // Zero runtime trigonometry in hot rendering paths: void MoveActor(fixed_t* x, fixed_t* y, angle_t angle, fixed_t speed) { int idx = angle >> ANGLETOFINESHIFT; // Single shift to LUT index *x += FixedMul(speed, finecosine[idx]); // Array read: no sin(), no cos() *y += FixedMul(speed, finesine[idx]); } RoPE (Rotary Position Embeddings): The Modern Equivalent Modern transformer architectures (LLaMA 3, Mistral, Gemma, DeepSeek) encode relative token order using Rotary Position Embeddings (RoPE). RoPE rotates token query and key vectors in 2D subspaces by an angle proportional to the token's position in the context window: θ_{pos, i} = pos · 10000^(-2i / d) Evaluating transcendental trigonometric functions dynamically for every token, across 32 attention heads and 128,000 sequence positions, would destroy inference throughput. Modern LLMs apply Carmack's exact strategy: precompute all frequencies into a static cache and index by position. # RoPE frequency precomputation (Meta LLaMA architecture) import torch def precompute_freqs_cis(dim: int, seq_len: int, theta: float = 10000.0): # Precompute inverse frequencies (the BAM fine-angle equivalent) inv_freq = 1.0 / (theta ** (torch.arange(0, dim, 2).float() / dim)) t = torch.arange(seq_len) freqs = torch.outer(t, inv_freq) # Shape: [seq_len, dim/2] # Pre-bake sin and cos into one contiguous complex tensor: cis(θ) = cos(θ) + i·sin(θ) freqs_cis = torch.polar(torch.ones_like(freqs), freqs) return freqs_cis # Cached once at model initialization def reshape_for_broadcast(freqs_cis: torch.Tensor, x: torch.Tensor): # Align [seq_len, head_dim/2] to [1, seq_len, 1, head_dim/2] for multi-head attention shape = [d if i in (1, x.ndim - 1) else 1 for i, d in enumerate(x.shape)] return freqs_cis.view(*shape) def apply_rotary_emb(xq, xk, freqs_cis): # Group real feature pairs into complex numbers xq_ = torch.view_as_complex(xq.float().reshape(*xq.shape[:-1], -1, 2)) xk_ = torch.view_as_complex(xk.float().reshape(*xk.shape[:-1], -1, 2)) # Slice frequency cache to current sequence length and broadcast freqs_cis = freqs_cis[:xq_.shape[1]] freqs_cis = reshape_for_broadcast(freqs_cis, xq_) # Complex multiplication performs 2D rotation with zero runtime trig calls xq_out = torch.view_as_real(xq_ * freqs_cis).flatten(3) xk_out = torch.view_as_real(xk_ * freqs_cis).flatten(3) return xq_out.type_as(xq), xk_out.type_as(xk) | DOOM (1993) | Modern LLM / RoPE (2024) | | :--- | :--- | | finesine[10240] table | freqs_cis complex frequency buffer | | angle >> ANGLETOFINESHIFT | Token sequence position slicing [:seq_len] | | finecosine = &finesine[2048] | torch.polar() packaging cos + i·sin in contiguous memory | | Zero runtime trig evaluations | Zero runtime trig evaluations in attention layers | --- 3. Memory Hierarchy: L1 Cache Lines in 1993, SRAM Tiling in 2024 DOOM's Data Layout Carmack's combined trigonometric table fits into 10,240 × 4 bytes = 40.96 KB. On an Intel 486 with an 8 KB to 16 KB L1 cache, maintaining spatial locality was vital. By packing sine and cosine into a single contiguous array, index calculations stayed cache-dense and avoided pipeline-stalling RAM roundtrips. FlashAttention: Managing Modern Memory Walls The breakthrough of FlashAttention (Dao et al., 2022) was recognizing that self-attention is memory-bound, not compute-bound. Standard attention materializes an intermediate N × N attention score matrix in high-latency GPU High Bandwidth Memory (HBM). For long contexts, transferring this matrix back and forth across the bus dominates execution time. Standard Attention: Read Q, K from HBM -> Compute QK^T -> Write S (N x N) to HBM Read S from HBM -> Compute Softmax -> Write P (N x N) to HBM Read P, V from HBM -> Compute Attention Output -> Write to HBM Intermediate HBM footprint: O(N²) FlashAttention (Tiled): Load blocks of Q, K, V into fast on-chip SRAM (~20 MB on H100) Compute attention, online softmax, and reduction entirely within SRAM Write only the final output back to HBM Intermediate HBM footprint: O(N) Carmack laid out tables to stay within CPU cache boundaries; Tri Dao restructured attention into SRAM tiles. The engineering rule remains constant: compute is nearly free; data transit across memory hierarchies is where performance dies. --- 4. Bitwise Determinism: Multiplayer DOOM to Distributed RL DOOM's Input-Only Lockstep Protocol In 1993, sending complete 3D player coordinates across 9600-baud modems or IPX local networks overwhelmed available bandwidth. Carmack designed DOOM's multiplayer engine around input-synchronous lockstep simulation. Instead of syncing world states, peer machines exchanged only tiny 4-byte ticcmd_t input structs containing controller deltas: // DOOM network input packet (d_ticcmd.h) typedef struct { char forwardmove; // Movement delta char sidemove; // Strafe delta short angleturn; // 16-bit angle delta (unpacks via: angleturn << 16) short consistancy; // Game state checksum to detect desynchronization byte chatchar; byte buttons; } ticcmd_t; For lockstep to work without desynchronization, every machine had to arrive at the exact same game state given identical input sequences. With floating-point arithmetic, disparate processor stepping levels and differing rounding modes produce divergent results within seconds. Fixed-point integer arithmetic made DOOM's simulation 100% bitwise deterministic across diverse hardware. Distributed RL and Robotics Rollouts Modern reinforcement learning systems (e.g., Isaac Gym, MuJoCo parallel rollouts, Brax) run thousands of simultaneous environment instances to train robotic policies and autonomous agents. Non-deterministic floating-point accumulation (e.g., arbitrary thread execution order in CUDA non-associative atomicAdd) leads to divergent trajectory states, making rollouts un-reproducible and gradient updates noisy: 1. Integer State Representation: Environments keep discrete counters, grid coordinates, and action indices in fixed integer domains. 2. Deterministic Reduction Kernels: Distributed frameworks enforce fixed reduction tree structures (torch.use_deterministic_algorithms(True)). 3. Reproducible Trajectory Caching: Bitwise exact simulation is the only way distributed workers can validate agent rollout integrity without logging gigabytes of intermediate physics states. The lockstep multiplayer network model of 1993 and distributed reinforcement learning rollouts in 2024 both depend on the same foundation: uncompromising numerical determinism. --- 5. Synthesis: Hardware Sympathy vs. The Bitter Lesson Richard Sutton's influential 2019 essay The Bitter Lesson observes: > "The biggest lesson that can be read from 70 years of AI research is that general methods that leverage computation are ultimately the most effective, and by a large margin." Sutton's thesis is that human researchers continuously attempt to build domain-specific shortcuts into algorithms, but general methods (learning and search) consistently win once compute scales up. Does that invalidate Carmack's philosophy? Not at all. There are two distinct abstraction layers in systems design: - Algorithm-to-Problem (Sutton's domain): Do not hardcode heuristics about human vision or language into model architectures. Let general transformers learn the representations. - Hardware-to-Algorithm (Carmack's domain): Always align the mathematical execution of that general model with the physical silicon running it. Scaling laws only remain economically viable when systems engineers practice aggressive hardware sympathy. The reason frontier models can scale across thousands of GPUs is not because someone abandoned the Bitter Lesson; it is because kernel engineers wrote custom Triton and CUDA kernels, tiled SRAM accesses, precomputed rotary frequencies, and compressed weights into 4-bit integers. Sutton tells us what to compute. Carmack shows us how to make it physically computable. --- Summary of Timeless Systems Principles 1. Quantize Early: When floating-point pipelines saturate memory bandwidth or lack native hardware acceleration, discrete integer representations win. 2. Precompute Transcendental Math: Never evaluate sine, cosine, or square roots inside tight inner loops; cache them in contiguous memory. 3. Exploit Symmetries: Reusing memory via pointer phase shifts (finecosine = &finesine[2048]) saves cache footprint with zero runtime computation. 4. Design for Determinism: Complex distributed systems stay in sync only when numerical transformations are bitwise reproducible. 5. Silicon Dictates Algorithms: Software architecture that ignores memory hierarchy and hardware instruction costs will always lose to hardware-sympathetic engineering. --- References - DOOM Source Code (id-Software/DOOM) — Original tables.h, m_fixed.h, r_main.c, d_ticcmd.h - RoFormer: Enhanced Transformer with Rotary Position Embedding — Su et al. (2021) - FlashAttention: Fast and Memory-Efficient Exact Attention — Dao et al. (2022) - BitNet b1.58: 1-bit LLMs for All — Ma et al. (2024) - The Bitter Lesson — Richard S. Sutton (2019) - LLaMA Architecture Source Code — Meta Platforms - Original DOOM Trig Note by @TrisH0x2A
GSM8K on Hugging Face: 4 of the Top 12 Rows Show a Sibling Checkpoint's Score, and 20 of 28 Rows Are Unmerged PRs TL;DR - On November 3, 2026 the Hugging Face GSM8K board lists 28 rows from 28 repos. 20 come from pull requests that are still open, 8 sit on the repo's main branch, and 0 are verified. - Four of the top 12 rows show a number that the model card reports for a different checkpoint of the same family. Rank 1 (99.6) is the MiMo-V2.5-Pro Base score, registered on the post-trained repo. Rank 2 (96.8) is the Llama 3.1 405B Instruct score, registered on the base repo. Ranks 7 and 12 are DeepSeek base scores on the post-trained repos. - The numbers are real self-reports. The labels are what is off, and the board has no column that would show it. - GSM8K has 1,319 test problems, so one problem is 0.076 points. The rank 2 to rank 3 gap (0.74) is 10 problems, and a 95% interval at 95% accuracy is about 1.2 points wide on each side. - A score of 99.6 means about 5 misses. The GSM8K-Platinum project relabeled 10 answers in the same test set. If those relabels are right, a score that high implies the model matched at least 5 of the 10 keys they call wrong. !Top 12 rows of the GSM8K board on Hugging Face, with the four rows whose number belongs to a sibling checkpoint in red In the chart, red marks rows where the model card gives this number for a sibling checkpoint, blue marks a row whose card table matches the repo (LLaDA2.0-flash), and grey marks rows I did not check against the card. What does the GSM8K board on Hugging Face contain? GSM8K is a set of grade-school math word problems with a train split of 7,473 and a test split of 1,319. Hugging Face lists openai/gsm8k as a benchmark dataset, so a model repo can publish a result in a .eval_results/*.yaml file and the dataset's board collects it. I pulled the board on November 3, 2026 (KST) from the JSON behind the leaderboard view. It has 28 rows. Each row is from a different repo, verified is false on every row, and the top of the page reads like a clean ranking: 99.6, 96.8, 96.06, 94.16, 93.93. The dataset's own eval.yaml describes a specific protocol: an inspect-ai task, a zero-shot prompt template that asks for a boxed final answer, four epochs reduced to pass@1, and a model-graded scorer. The rows do not follow one protocol. Of the 28 notes, 15 are empty. Where a note gives a shot count it ranges across 0, 4, 5 and 8. Two rows from one lab give a complete setup (8-shot, temperature 0, n=1319, lm-evaluation-harness), and one row reports a strict-match score over 1,075 of the 1,319 problems. So I treat each row as a claim with a source link, not as a measurement the board produced. Why does rank 1 belong to a different checkpoint? The top row is XiaomiMiMo/MiMo-V2.5-Pro at 99.6. Its model card has a section titled "Base Model Evaluation", and the GSM8K row there reads 8-shot with these values: 99.6 for MiMo-V2.5-Pro Base, 83.3 for MiMo-V2.5 Base, 92.6 for DeepSeek-V4-Pro Base. That is the only place GSM8K appears in the card. The card's own download table lists two repos: MiMo-V2.5-Pro and MiMo-V2.5-Pro-Base, and the second one exists on the Hub. So the board shows the base-model score on the post-trained repo. The pull request that registered it says the results "were extracted from the model card". It was opened on April 27, 2026 and is still open. Nothing in the YAML says "base", and the 8-shot setting is not in the note, which is empty. The card is not hiding anything. It labels its table as Base. The distinction is lost only in the step that turns a card into a board row. Is that one slip, or does it repeat? I checked the sources for the top rows. It repeats in four of the top 12, and in both directions. | Board rank | Repo on the board | Score | Where the card puts this number | Sibling repo | |---|---|---|---|---| | 1 | MiMo-V2.5-Pro | 99.6 | "Base Model Evaluation" table, Pro Base column, 8-shot | MiMo-V2.5-Pro-Base | | 2 | Llama-3.1-405B | 96.8 | "Instruction tuned models" table, 405B Instruct column, 8-shot CoT, em_maj1@1 | Llama-3.1-405B-Instruct | | 7 | DeepSeek-V4-Pro | 92.6 | Base table, DeepSeek-V4-Pro-Base column, 8-shot | DeepSeek-V4-Pro-Base | | 12 | DeepSeek-V3 | 89.3 | "Base Model" section, DeepSeek-V3 column, 8-shot | DeepSeek-V3-Base | Rank 2 is the mirror image. The Llama 3.1 model card covers every size and both variants. Its "Base pretrained models" table has no GSM8K row. Its "Instruction tuned models" table has one, and the 405B Instruct value is 96.8. The pull request is titled "Add GSM8K eval result (96.8, 8-shot CoT)" and attaches that number to the base repo meta-llama/Llama-3.1-405B. For DeepSeek-V3 the card has a Base Model section with GSM8K at 89.3 and a Chat Model section with no GSM8K row. For DeepSeek-V4-Pro the table header names the three columns DeepSeek-V3.2-Base, DeepSeek-V4-Flash-Base and DeepSeek-V4-Pro-Base, and 92.6 is the last one. Why does it matter if the number is the same? Because the board is read by repo name. Someone choosing a model for math word problems sees MiMo-V2.5-Pro at the top and downloads that repo. The card reports no GSM8K number for that checkpoint. If the four rows were moved to the sibling repos, the post-trained MiMo, DeepSeek-V4-Pro and DeepSeek-V3 would have no GSM8K row at all, and the Llama 405B base repo would lose its rank 2. The correct placement would not change the ordering of the numbers: 99.6 would still lead, from the right repo. It would change which model each number tells you about. !Where the 28 GSM8K rows come from: open pull requests, main branch, verified Where do the rows come from, and what does "verified" mean here? I fetched the discussion for every row that carries a pullRequest number, and listed .eval_results on the main branch of every repo. | Source | Rows | |---|---| | Open PR from an account flagged as Hugging Face (julien-c 9, nielsr 6, SaylorTwift 3, burtenshaw 1) | 19 | | Open PR from a bot account (EvalEvalBot) | 1 | | File on the repo's main branch | 8 | | Verified | 0 | All 20 pull requests are open, and for all 20 the YAML file is absent from main. For all 8 other rows it is present. Four of the 20 pull requests were opened on November 2, one day before I pulled the board, and one of them produced rank 3 (LLaDA2.0-flash, 96.06). The board moves when someone opens a pull request, with no action from the model's owner. The Hugging Face documentation defines the badges. "Community" means a result submitted via an open PR that is not merged to main. "Verified" means a valid verify token, which requires the evaluation to have run in HF Jobs with inspect-ai. So 20 of 28 rows are community results by definition, and no row meets the verified bar. The documentation says community scores are visible while the PR is open. That is a reasonable design for bootstrapping a board. It also means a reader has to look at the badge to know which rows the model's owner stands behind, and a screenshot of the ranking does not show it. The eight main-branch rows are not uniformly clean either. Two Mellum2 repos show the same 81.73. The YAML on the Base repo states that its numbers were measured on the pre-extension Base-Pretrain checkpoint, so it is one measurement displayed twice. This one is disclosed in the file, and it still occupies two ranks. How much can 1,319 problems separate? GSM8K is nearly saturated at the top, so the ordering is decided by a handful of problems. | Quantity | Value | |---|---| | One problem | 0.076 points | | Rank 1 to rank 2 gap (99.6 to 96.8) | 2.8 points, about 37 problems | | Rank 2 to rank 3 gap (96.8 to 96.06) | 0.74 points, about 10 problems | | Spread of ranks 3 to 6 (96.06 to 93.10) | 2.96 points | | 95% half-width at 95% accuracy, n=1,319 | 1.18 points (illustrative) | | 95% half-width at 99% accuracy, n=1,319 | 0.54 points (illustrative) | The interval rows are a sampling calculation for a single run on a fixed test set. They ignore prompt variation, which is larger when rows use different shot counts. Even so, ranks 3 to 6 sit inside a band that a single run cannot separate, and the rows within it come from different protocols: 8-shot CoT for some, no note for others. One other check is exact arithmetic. The two rows with a full setup report 84.9886 and 78.4685, and with n=1319 these are 1,121 and 1,035 correct answers. ProKope-7B's note says 1,075 of 1,319, which is 81.50. Those rows can be audited back to a count. Most rows cannot. !How big is each effect: one problem, intervals, ceiling lost to relabeled keys, gaps between ranks Can a 99.6 be right if some answer keys are wrong? This part is arithmetic with one assumption, so I mark it as a question to check, not a finding. The GSM8K-Platinum project (MIT, Gradient Science) reviewed the 1,319 test problems. Its dataset card reports 219 flagged by models, 110 rejected as poorly written, 10 relabeled, 99 verified, leaving 1,209 questions. The 10 relabeled items are answer keys that the authors found to be wrong. Take a score of 99.6 on the original test set. 1,314 of 1,319 rounds to 99.6, so that is about 5 misses. Suppose the 10 relabels are correct and scoring is exact match against the original key. A model that solves an item correctly then loses it, because the key is wrong. A model that is right on all 10 would lose all 10 and top out at 99.24. To score 99.6 it must have matched the key on at least 5 of the 10, which means it gave the answer the key says, not the answer the authors say is right. Three explanations fit: the relabels are wrong for some of those items, the scoring is more lenient than exact match, or the model has seen the original keys. I have not run the model, and a base model's 8-shot number with no note on answer extraction does not let me tell these apart. What I can say is that for a benchmark with known key errors, a score above the ceiling is a reason to read the setup, not a reason to rank it first. Platinum also reports that cleaning the set separates models that look tied on the original, which is the same point from the other side. How do I pull the board and classify rows myself? This script needs only the standard library. It counts rows by source and prints the sampling arithmetic used above. import json, urllib.request, collections def get(url): req = urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0"}) return json.load(urllib.request.urlopen(req, timeout=60)) DATASET = "openai/gsm8k" rows = get(f"https://huggingface.co/api/datasets/{DATASET}/leaderboard") by_source = collections.Counter() for r in rows: pr = r.get("pullRequest") if pr: disc = get(f"https://huggingface.co/api/models/{r['modelId']}/discussions/{pr}") by_source[("open PR" if disc["status"] == "open" else disc["status"])] += 1 else: by_source["main branch"] += 1 print("rows:", len(rows), "| verified:", sum(bool(r["verified"]) for r in rows)) print(dict(by_source)) n = 1319 print("one problem = %.3f points" % (100 / n)) for acc in (0.95, 0.99): se = (acc * (1 - acc) / n) ** 0.5 * 100 print("accuracy %.2f: 95%% half-width = %.2f points" % (acc, 1.96 * se)) print("ceiling if 10 keys are wrong: %.2f" % (100 * (n - 10) / n)) On November 3, 2026 it printed: rows: 28 | verified: 0 {'open PR': 20, 'main branch': 8} one problem = 0.076 points accuracy 0.95: 95% half-width = 1.18 points accuracy 0.99: 95% half-width = 0.54 points ceiling if 10 keys are wrong: 99.24 The board endpoint is undocumented, so the field names may change. To check a row, open its pullRequest discussion, read the YAML, then open the source card and find the table the number came from. Checklist for reading a saturated board 1. Open the badge before the rank. Community means an open PR, and verified means a run in HF Jobs. Neither is a statement from the model's owner. 2. Match the checkpoint, not the family. Find the card table the number came from and read its column header. Base, instruct and chat checkpoints sit in different tables. 3. Ask how many problems separate two rows. On GSM8K, 10 problems is 0.76 points. 4. Prefer rows that state their setup. Shot count, temperature, n and the answer-extraction rule decide the score more than rank does. 5. Check for the same measurement on two repos. A copied number is one result, not two. 6. Distrust a score above the key ceiling. If a benchmark has known label errors, read the evaluation setup before you trust the lead. FAQ How many GSM8K results on Hugging Face are verified? None. On November 3, 2026 all 28 rows on the board show verified as false. Verified requires a valid verify token from an inspect-ai run in HF Jobs. Is the 99.6 at rank 1 of the GSM8K board wrong? Nothing here shows the number is wrong. It is the MiMo-V2.5-Pro Base score from the model card's Base Model table, 8-shot. The problem is that it is registered on the post-trained repo, which has its own separate row in the card's download table. What does "community" mean on a Hugging Face leaderboard row? It means the result was submitted through an open pull request that has not been merged to the repo's main branch. In the GSM8K board, 20 of 28 rows are in that state. Why is the Llama 3.1 405B base repo ranked second on GSM8K? Its row carries 96.8, which the shared model card gives for Llama 3.1 405B Instruct, 8-shot CoT. The card has no GSM8K row in its base table. The instruct repo is a separate repo. How many problems does one point of GSM8K accuracy represent? GSM8K has 1,319 test problems, so one point is about 13 problems and one problem is 0.076 points. What is GSM8K-Platinum? It is a revised version of the GSM8K test set from Gradient Science. Of 1,319 test problems it rejected 110, relabeled 10 and kept 1,209. It exists because the original set has ambiguous questions and wrong keys. Further reading - GPQA Diamond Has 198 Questions: What the Hugging Face Leaderboard's 115 Rows Can and Cannot Rank - How Hugging Face Official Benchmark Leaderboards Actually Work: .eval_results YAML, the base_model Filter, and the 30% the Default View Hides Sources - GSM8K dataset and board: huggingface.co - GSM8K evaluation definition (eval.yaml): huggingface.co - Hugging Face documentation, Evaluation Results (badges, community PRs): huggingface.co - GSM8K-Platinum dataset card: huggingface.co - GSM8K-Platinum write-up: gradientscience.org - Rank 1 model card (Base Model Evaluation table): huggingface.co - Rank 1 registering pull request: huggingface.co - Llama 3.1 model card (base and instruction tuned tables): github.com/meta-llama/llama-models/blob/main/mode… - Rank 2 registering pull request: huggingface.co - DeepSeek-V4-Pro model card: huggingface.co - DeepSeek-V3 model card: huggingface.co All counts in this article are my own measurements from the board JSON, pull request discussions, dataset cards and the cited model cards on November 3, 2026 (KST); I have no affiliation with any model, lab or benchmark named here. Which table a number came from is my reading of each card, and the board endpoint is undocumented and may change.
How long does it take to train an AI model on cloud GPUs? Training an AI model takes anywhere from a few minutes to several months, depending on model size, how many tokens it trains on and how many GPUs run in parallel. On Nodus a small fine-tune can finish in under five minutes, while by our math a 7B language model trained from scratch on 140 billion tokens takes about 3.6 days on 64 NVIDIA H100s. The sections below show the formula, check it against published runs, and list what makes the real clock run longer than the math. How do you estimate how long a training run will take? Work out the total math the run needs, then divide by how fast your GPUs actually do math. Training a transformer takes about 6 floating-point operations per parameter per token, the rule from Kaplan et al.'s scaling-laws paper. An NVIDIA H100 SXM peaks at about 989 teraFLOPS of dense BF16 math, half the sparse figure on NVIDIA's H100 page. No real run gets the full peak. The share you get is called model FLOPs utilization, and 30% is a fair planning number for code you have not tuned. That gives: GPU-hours = (6 × parameters × tokens) / (989e12 × utilization × 3,600) wall-clock = GPU-hours / number of GPUs (if scaling were perfect) At 30% utilization, one H100-hour delivers about 1.07 × 10^18 useful operations. The second line is the optimistic part, and we come back to it below. How long does it take to train a language model from scratch? Days to weeks for small and mid-size models, if you have enough GPUs. Here is the formula at 30% utilization, using a compute-optimal budget of about 20 tokens per parameter from DeepMind's Chinchilla paper, plus one longer 7B run. | Model and tokens | H100 GPU-hours | Fewer GPUs | More GPUs | | --- | ---: | ---: | ---: | | 1B on 20B tokens | 112 | 1 GPU: 4.7 days | 8 GPUs: 14 hours | | 7B on 140B tokens | 5,505 | 8 GPUs: 28.7 days | 64 GPUs: 3.6 days | | 7B on 2T tokens | 78,643 | 64 GPUs: 51.2 days | 512 GPUs: 6.4 days | | 70B on 1.4T tokens | 550,500 | 512 GPUs: 44.8 days | 2,048 GPUs: 11.2 days | Our arithmetic: 6 × parameters × tokens, 989 teraFLOPS dense BF16, 30% utilization, perfect scaling. Real runs land later; see the overheads section. Token count moves the clock as much as model size. The same 7B model takes 14 times longer on 2 trillion tokens than on 140 billion. Many open models now train far past the Chinchilla ratio, so settle the token budget first. Our post on what it costs to train an LLM prices the same GPU-hours in dollars. How long do real published training runs take? Small reproductions take hours, and frontier models take months. Andrej Karpathy's llm.c reproduction of GPT-2 124M trained on 10 billion tokens in about 90 minutes on one 8 × NVIDIA A100 80GB node. His GPT-2 1.6B run took about 24 hours on one 8 × H100 node: 32,000 steps of roughly 1 million tokens each, at about 2.75 seconds per step. That 1.6B run is a good check on the formula. It works out to 192 GPU-hours for about 33.6 billion tokens, which back-solves to roughly 46% utilization. Karpathy's own logs show 42% to 47% at the start, so well-tuned code beats our 30% planning figure by about half. At the top end, Meta's Llama 3.1 model card lists 30.84 million H100 GPU-hours for the 405B model. The Llama 3 paper says it trained on up to 16,384 H100s. Divide one by the other and even that cluster needs about 78 days of pure compute, a floor on the real calendar time. Do more GPUs make training proportionally faster? Only roughly, and never for free. Adding GPUs splits the same GPU-hours across more machines, so the bill stays about the same while the calendar shrinks. In practice the bill creeps up, because GPUs spend more time talking to each other. Meta's own numbers show the effect. The Llama 3 paper reports 43% utilization on 8,192 GPUs and 41% on 16,384 GPUs for the same model, a small but real loss from doubling the cluster. !Days to train a 7B model on 2 trillion tokens: 409.6 days on 8 H100s, 51.2 days on 64, 6.4 days on 512, at 30% utilization and perfect scaling. The same 78,643 GPU-hours: one 8-GPU node needs 409.6 days, a 512-GPU cluster about a week. Our decision rule: pick the smallest GPU count that finishes before your deadline, not the largest you can get. Single nodes of 8 GPUs scale best because the GPUs share fast NVLink links. Multi-node runs add network cost and more machines that can fail. How long does fine-tuning take? Usually minutes to a few hours, because fine-tuning sees far fewer tokens than pretraining. Fine-tuning a 7B model on 50 million tokens is about 2 H100-hours of math at 30% utilization by the same formula, and LoRA does less than that because it skips most weight gradients. For small jobs, the work around training often takes longer than training itself. In our published fine-tuning benchmark, Qwen2.5-0.5B on one NVIDIA L4 spent 112.8 seconds in the training loop on Nodus. From submission to a verified exported model took 4.8 minutes, or 288 seconds. !Small fine-tune on Nodus: 112.8 seconds in the training loop versus 288 seconds from submission to exported model. For a small fine-tune, startup, evaluation and export took longer than the training loop. So for fine-tuning, the honest answer to "how long" is mostly startup, model download, evaluation and export. On Nodus, a TrainingJob imports a pinned Hugging Face model revision into a read-only Volume once, and later runs reuse it, so billed GPU time is not spent downloading weights. Why does the real clock run longer than the math? Because the formula counts only useful training steps. Four things add time on most projects: - Waiting for capacity. A large multi-GPU request may queue until enough machines are free together. - Startup. Pulling the image, loading the model and reading the first batch of data. - Failures and restarts. Over a 54-day stretch of Llama 3 405B pretraining, Meta logged 466 job interruptions, 419 of them unexpected. It still reported more than 90% effective training time, meaning at least a tenth of the elapsed time did no useful training. - Evaluation and export. Validation passes and writing out the final weights. Our planning rule is to add at least 10% to the formula for multi-day runs. Go higher if your checkpoints are far apart, since every restart redoes the work since the last save. How does Nodus show the time before you launch? A dry run returns the estimate before anything is charged. For a managed TrainingJob, the training guide says the estimate is the number of steps times the measured seconds per step on the slowest GPU type you allowed, plus evaluation time. You can set spec.expectedDuration to use your own figure. $ nodus apply -f sft.yaml --dry-run -o yaml # status.estimate: cost to completion, first hold, start time For your own script, nodus run takes the same idea from flags. This prints the estimate only, using your expected duration: $ nodus run --dry-run --gpu H100:8 --expected-duration 24h -- torchrun train.py If the run must finish by a date, --complete-by (or placement.completeByTime) keeps it off any capacity whose p90 finish is later, per our placement docs. H100s are listed at $2.60/hr on the Nodus pricing page, checked October 31, 2026. If a machine is lost mid-run, Nodus starts a new attempt from the latest committed checkpoint, as described in attempts and recovery. Our post on how often spot GPUs are interrupted shows how much that matters on cheaper capacity. FAQ How long does it take to train a model like GPT-4? Nobody outside the lab knows, because the GPU-hours were never published. The closest public figure is Llama 3.1 405B at 30.84 million H100 GPU-hours, which is months even on a cluster of more than 16,000 GPUs. Can I train a useful model on a single GPU? Yes, if it is small or a fine-tune. Karpathy notes his GPT-2 124M run would take roughly 4 to 24 hours on a single GPU, depending on the card. Is a faster GPU always worth it for training time? Not always. A GPU that costs twice as much per hour but runs the job 2.5 times faster is cheaper overall, and one that runs it 1.5 times faster is not. Compare per-step time on your own model. Nodus managed training runtimes publish measured step times for their presets, which is what their estimates use. Does training time include data preparation? No, none of the figures here do. Tokenizing and cleaning a large dataset is usually CPU work that you can finish before you pay for any GPU time. Originally published on the Nodus blog.
@nikks_techie example: Llama 3.1 70B, bf16 KV, no prefix sharing: 320 KiB/token × 8,192 tokens × 100 active sequences = 250 GiB of cache alone. 70B isn't enough to size it. the model's KV heads and the cache dtype matter. github.com/meta-llama/llama-models/blob/main/mode…
What Every AI Engineer (Builder) Needs to Know About the Data Behind AI Table of Contents Chapter 1 of 4, deep dive. Click any line to jump straight to it: - [Introduction](#introduction) - [Module 1.1: The Big Picture: What a Model Actually Learns](#module-11-the-big-picture-what-a-model-actually-learns) - [1.1.1 The One Sentence Everything Builds On](#111-the-one-sentence-everything-builds-on) - [1.1.2 The Pipeline Loop as Three Mathematical Operations](#112-the-pipeline-loop-as-three-mathematical-operations) - [1.1.3 How Big Is "Big"?](#113-how-big-is-big) - [1.1.4 The Quantity, Quality, Diversity Trade-off, Measured](#114-the-quantity-quality-diversity-trade-off-measured) - [Module 1.2: The Mathematics of the Training Objective](#module-12-the-mathematics-of-the-training-objective) - [1.2.1 Entropy, Cross-Entropy and KL Divergence](#121-entropy-cross-entropy-and-kl-divergence) - [1.2.2 Training Is Maximum Likelihood](#122-training-is-maximum-likelihood) - [1.2.3 The Cost of Spam, Derived](#123-the-cost-of-spam-derived) - [1.2.4 Proof in Code: Clean vs Spam-Polluted Training](#124-proof-in-code-clean-vs-spam-polluted-training) - [1.2.5 The ln(V) Sanity Check](#125-the-lnv-sanity-check) - [Module 1.3: What the Model Is Truly Optimizing For](#module-13-what-the-model-is-truly-optimizing-for) - [1.3.1 From Tokens to Loss to Perplexity](#131-from-tokens-to-loss-to-perplexity) - [1.3.2 How the Data Pushes on the Model: the Gradient](#132-how-the-data-pushes-on-the-model-the-gradient) - [1.3.3 Scaling Laws, With Real Numbers](#133-scaling-laws-with-real-numbers) - [1.3.4 Compute-Optimal vs Over-Trained](#134-compute-optimal-vs-over-trained) - [1.3.5 Where the Numbers From the Intro Come Back](#135-where-the-numbers-from-the-intro-come-back) - [Chapter 1 Cheat Sheet](#chapter-1-cheat-sheet) - [References](#references) Chapters 2, 3 and 4 (cleaning the web, teaching filters to think, and the final mile into tokens and batches) publish as their own posts. --- Introduction I have been quiet for a while, not because I had nothing to say, but because I kept turning one question over in my head: how do people like us, without a fraction of the money the big labs spend, actually contribute to building AI that reflects who we are. The honest answer took time to arrive at, and the number that finally pushed me to write this down is too large to sit with quietly. > Did you know? Meta's own published model card for Llama 3.1 states that training the 405-billion-parameter version alone used 30.84 million GPU-hours on H100 chips drawing 700 watts each. Run that math and it comes out to roughly 21.6 million kilowatt-hours of electricity, enough to power about 2,000 average households for a full year. Meta's own figures put the location-based emissions from that one training run at 8,930 tons of CO2, close to what 1,900 cars emit in a year. Spread across the 16,384 GPUs used, that workload still took about 78 days of continuous training, and at a modest $2 per GPU-hour cloud rate, the compute bill alone comes to roughly $62 million, before a single researcher, dataset, or building is paid for. Sit with that $62 million for a second, then sit with this. In 2021, OpenAI contracted a company called Sama to have Kenyan workers label disturbing content so ChatGPT would not reproduce it, paying those workers roughly $1.50 to $2 an hour while internal documents showed OpenAI itself was billed $12.50 an hour for that same work. Separately, Scale AI, the company Alexandr Wang led as CEO until Meta bought 49 percent of it for $14.3 billion in 2025, has faced its own wave of US lawsuits after reports that its Remotasks platform paid some workers as little as a few cents per task. Two different companies, two separate scandals, the same pattern underneath: the humans doing the most psychologically demanding part of building this technology are paid the least, while the compute bill for the model they helped clean runs into the tens of millions. !exploting Kenyan people to label data Who this is for. This series is written for builders, students, self-taught engineers, researchers and curriculum designers anywhere who want to understand how large language models are actually built, not just how to call an API, especially those of us working with limited compute, limited funding and limited institutional backing. You do not need a GPU cluster or a research lab to read this. You need curiosity, a laptop, and the willingness to look closely at data most people scroll past. And while that gap sits unresolved, the people steering this industry are moving at a different altitude entirely. In September 2025, Donald Trump told the United Nations General Assembly that the United States would start calling artificial intelligence "superintelligence," or SI, later signing an executive order requiring federal agencies to use the new term. That is the altitude at which this race is being run, nations rewriting policy and vocabulary around who leads next, while much of the data that trains these systems, our languages, our conversations, our culture, keeps flowing outward for free. This is where the series begins, not with despair at the size of the gap, but with the one part of it we can actually close ourselves. Over the next four chapters we move from the big picture of why training data matters so completely, through where real data comes from and how it gets cleaned, into the filtering and quality techniques that separate a frontier lab's dataset from a pile of scraped pages, and finally into how that data becomes the tokens and batches a model actually trains on. None of it requires a data center. All of it requires showing up. --- How to read this post. Every section follows the same rhythm: the idea in plain words, the math (each equation is followed by a "read it as" sentence), a worked example with real arithmetic you can check with a calculator, and why it matters for data. All code lives in one repository so this post stays readable: !Source of truth > 💻 Companion repo: iws3/before-we-train. Every script in chapter-01/ runs on a laptop CPU with plain Python and NumPy. No GPU needed. Notation used throughout: | Symbol | Meaning | |---|---| | x_t | the token at position t | | x_{<t} | all tokens before position t (the context) | | V | vocabulary size (number of distinct tokens) | | p or p_data | the true distribution of your dataset | | q or q_θ | the model's distribution, with parameters θ | | n(·) | a count, e.g. n(the, cat) is how many times "cat" follows "the" | | N | number of model parameters (used from 1.3.3 on) | | D | number of training tokens (used from 1.3.3 on) | | ln | natural logarithm. Losses in this post are in nats; divide by ln 2 ≈ 0.693 to get bits | | Σ, Π | "add up" and "multiply together" over everything in the range shown | --- Module 1.1: The Big Picture: What a Model Actually Learns 1.1.1 The One Sentence Everything Builds On The idea. A language model becomes whatever you feed it. Not a feeling and not a simplification: it is what the training procedure is mathematically built to do. Whatever pattern sits in the text, grammar, facts, boilerplate, spam, a stereotype repeated often enough, gets absorbed. The model cannot tell your intentions apart from the data's contents. It only ever sees the data. The math. A language model assigns a probability to any piece of text by splitting it into tokens and using the chain rule of probability: P(x₁, x₂, …, x_T) = P(x₁) · P(x₂ | x₁) · P(x₃ | x₁, x₂) · … · P(x_T | x₁, …, x_T−1) or, written compactly: P(x₁, …, x_T) = ∏ P(xₜ | x₁, …, xₜ₋₁) (multiply over t = 1 … T) Read it as: the probability of a whole text is the probability of the first token, times the probability of the second given the first, times the probability of the third given the first two, and so on. For example, the sentence "the cat sat" is scored as: P("the cat sat") = P(the) × P(cat | the) × P(sat | the cat) A model with parameters θ tries to make its own probabilities match the real ones. Training picks the θ that makes the training text as probable as possible: θ* = the θ that maximizes Σ log P_θ(documentᵢ) (add over i = 1 … n) Read it as: "find the settings θ that give the highest total log-probability to all n training documents." (We add logs instead of multiplying probabilities; 1.3.1 explains why.) Notice what is missing from that formula: any mention of what is good. The only input is the data. Change the data and θ* changes with it. Worked example: a model you can compute by hand. The simplest language model is a bigram counter. It predicts the next word using only the previous word: n(c, w) P(w | c) = ───────── n(c) Read it as: the chance of word w after context word c is how often c was followed by w, divided by how often c appeared at all. Train it on three sentences: "the cat sat", "the cat ran", "the dog sat". The word "the" starts all three, so n(the) = 3. After "the" comes "cat" twice and "dog" once: - P(cat | the) = 2/3 - P(dog | the) = 1/3 Now an attacker, or just a careless crawler, adds three copies of the spam sentence "the free prize". The word "the" now starts six sentences, and "free" follows it three times: | Next word after "the" | Clean corpus | After adding 3 spam lines | |---|---|---| | cat | 2/3 = 0.667 | 2/6 = 0.333 | | dog | 1/3 = 0.333 | 1/6 = 0.167 | | free | 0 | 3/6 = 0.500 | Nobody told the model to like spam. Spam's share of the counts simply became spam's share of the probability. A transformer is far more powerful than a counter and generalizes beyond exact counts, but its training objective (Module 1.2) pulls it toward the same destination: the model's beliefs are the frequencies in your data. Why it matters. This is why data preparation is not a chore before the real work of building AI. It is the real work. Most open models use nearly the same transformer design. What separates a strong model from a weak one is almost always what it was fed: how clean, how diverse, how deliberately chosen. > 💻 Code: chapter-01/1_1_1_bigram_counter.py. Builds the bigram model above, prints the probability table before and after spam, and lets you change the number of spam copies to watch P(free | the) climb toward 1. Expected Output: 1.1.2 The Pipeline Loop as Three Mathematical Operations flowchart TD A["Sources: web crawl, books, code, research, community data"] --> B["Extraction: raw pages to clean text"] B --> C["Filtering: remove junk, duplicates, unsafe content"] C --> D["Mixing: decide how much of each source to use"] D --> E["Tokenizing and batching"] E --> F["Training: the model learns to predict the next token"] F -->|"evaluate, then adjust a filter or a weight"| B Nobody gets the mixture right on the first try. Every strong dataset in use today is the record of a team running this loop dozens of times: train a small model, check what improved, adjust a filter or a weight, repeat. Anyone with a laptop can run that loop at small scale. The idea. Every box in the diagram does one of three things to the data distribution. Once you see them as operations on a probability distribution, the whole pipeline becomes easy to reason about. Operation 1: Mixing If you draw documents from several sources, where source k has distribution p_k and weight w_k, the combined distribution is: p_mix(x) = Σₖ wₖ · pₖ(x) where Σₖ wₖ = 1 Read it as: the mixed dataset is a weighted average of its sources, and the weights add up to 100%. The Llama 3 paper reports a final mix of roughly 50% general knowledge, 25% mathematical and reasoning data, 17% code and 8% multilingual text. Those percentages are the w_k values (they add up to 100%). They are design decisions, not facts of nature. Operation 2: Filtering A filter assigns each document a keep-score f(x), which is 1 or 0 for a hard filter (keep or drop) and anywhere in between for a soft one. The distribution after filtering is: p(x) · f(x) p_filtered(x) = ─────────── where Z = Σₓ p(x) · f(x) Z Read it as: keep each document in proportion to its score, then divide by Z so the probabilities add up to 1 again. Z has a plain meaning: it is the keep rate, the fraction of the data that survives. Operation 3: Deduplication If a document appears m(x) times, its weight in the dataset is proportional to m(x). A boilerplate footer that appears in 1,000 of 1,000,000 pages carries 0.1% of the probability, while a unique page carries 0.0001%. The footer is 1,000 times over-represented. Deduplication sets every m(x) back to 1 and removes that distortion. Worked example 1: filters multiply Suppose a crawl passes through four stages with these keep rates (illustrative numbers): | Stage | Keep rate | |---|---| | URL filter | 90% | | Language filter | 60% | | Quality filter | 50% | | Deduplication | 50% | The total keep rate is the product: Z_total = 0.90 × 0.60 × 0.50 × 0.50 = 0.135 Only 13.5% survives. Planning backwards: if you want 1 billion clean tokens, you must start with 1 billion ───────── ≈ 7.4 billion raw tokens 0.135 Worked example 2: tokens seen versus tokens owned A source with weight w_k in a run of D total tokens contributes w_k · D tokens. If your unique data for that source is U_k tokens, the model sees it for this many epochs (passes over the data): wₖ · D epochsₖ = ────────── Uₖ Take a 15 trillion token run and a language that gets 0.01% of the mix. Step by step: 1. Tokens drawn: 15 × 10^12 × 0.0001 = 1.5 billion 2. Unique tokens the community has collected: 200 million 3. Epochs: 1.5 billion ÷ 200 million = 7.5 Work on data-constrained scaling (Muennighoff et al., 2023) found that repeating data for up to roughly 4 epochs performs almost like fresh data, but returns fall off quickly beyond that. So for a small language, adding new unique text is worth more than upsampling what you have, and every genuinely new page counts. Why it matters. You do not need to understand a thousand techniques. You need to ask of any pipeline step: is this mixing, filtering or deduplication, and how does it change the distribution the model will copy? The order in the diagram is also not sacred. Cheap, aggressive steps usually run first because they shrink the data before expensive steps touch it, and different teams arrange the same boxes differently. What never changes is the loop. > 💻 Code: chapter-01/1_1_2_mixture_and_filters.py. Applies mixing weights, a chain of filters and deduplication to a toy corpus, then prints the distribution before and after each step so you can see every p_filtered change. Expected code output: !expected output 1.1.3 How Big Is "Big"? The idea. A pretraining run is measured in tokens, and a token is, on average, about 0.75 of an English word. Llama 3 was trained on roughly 15 trillion tokens. What does that number mean for a human? The math. The conversion is a chain of unit changes: tokens × 0.75 words per token years of reading = ───────────────────────────────────────────────── 250 words per minute × 525,600 minutes per year Read it as: turn tokens into words, then words into minutes of reading, then minutes into years. Worked example. 1. 15 trillion tokens × 0.75 words per token ≈ 11.25 trillion words 2. 11.25 trillion words ÷ 250 words per minute ≈ 45 billion minutes 3. 45 billion minutes ÷ 525,600 minutes per year ≈ 85,600 years of non-stop reading What survives cleaning is small. FineWeb-Edu, the heavily filtered educational subset of the 15-trillion-token FineWeb dataset, keeps about 1.3 trillion tokens. That is the keep rate Z from the last section: 1.3 trillion Z = ────────────── ≈ 0.087 (8.7%) 15 trillion For every 100 pages in the full dataset, the educational filter judges about 91 unfit and keeps about 9. A hidden cost: your language may need more tokens for the same words. The "0.75 words per token" figure is for English. The general quantity is called fertility, the average number of tokens a tokenizer needs per word: tokens φ = ──────── English: φ ≈ 1 / 0.75 ≈ 1.33 words If a tokenizer was built mostly from English text, a language it sees rarely gets chopped into many small pieces, so its φ is higher. Suppose φ = 3 for some language (illustrative; you should measure your own). Three things follow directly: | Effect | English (φ ≈ 1.33) | Your language (φ = 3) | |---|---|---| | Tokens needed per word | 1.33 | 3 | | Relative cost of the same text | 1× | 3 ÷ 1.33 ≈ 2.25× | | Words a fixed budget D teaches the model | D ÷ 1.33 | D ÷ 3 | | Words that fit in an 8,192-token context window | 8192 × 0.75 = 6,144 | 8192 ÷ 3 ≈ 2,731 | Tokenization (Chapter 4) is therefore not a neutral technical step. It decides how expensive your language is to learn. > 💻 Code: chapter-01/1_1_3_scale_and_fertility.py. Reproduces the 85,600-year calculation, then measures fertility φ for any text file you point it at, so you can compare English with a local language on the tokenizer of your choice. Expected Output: !scale and fertility 1.1.4 The Quantity, Quality, Diversity Trade-off, Measured The idea. Quantity, quality and diversity pull against each other. Aggressive filtering raises quality but shrinks quantity and diversity. Keeping everything preserves quantity but drowns the model in noise. Every technique in this series is a negotiation between the three. Let's make that negotiation measurable. The math. Give every document a quality score q(x) from a classifier, and keep the documents at or above a threshold τ. Then: Quantity: n(τ) = number of documents with q(x) ≥ τ Quality: q̄(τ) = average of q(x) over the documents with q(x) ≥ τ Diversity: H(τ) = − Σ_g p_g · ln(p_g) Read it as: - Quantity is how many documents survive the cut. - Quality is the average score among the survivors. - Diversity is the entropy of the share p_g that each group g (language, topic, region) holds in the kept set. It is highest when groups are balanced and zero when one group is everything. n(τ) can only go down as τ rises, and the average quality can only go up. Diversity can do either, and raising τ can silently destroy it. Worked example. Take 10 documents: 6 English web pages, 2 forum threads and 2 Yoruba pages. A quality classifier trained on English reference text gives these scores: | Group | Scores | Sum | |---|---|---| | English web (6) | 0.90, 0.85, 0.80, 0.75, 0.70, 0.30 | 4.30 | | Forum threads (2) | 0.60, 0.20 | 0.80 | | Yoruba pages (2) | 0.20, 0.15 | 0.35 | Set τ = 0.5. Only documents scoring 0.5 or higher survive: the five best English pages and the 0.60 forum thread. The 0.30 English page, the 0.20 forum thread and both Yoruba pages are removed. Here is the before and after: | Measure | Before filter | After filter (τ = 0.5) | |---|---|---| | Quantity | 10 documents | 6 documents (60%) | | Average quality | 5.45 ÷ 10 = 0.545 | 4.60 ÷ 6 = 0.767 | | Group shares (English / forum / Yoruba) | 60% / 20% / 20% | 83% / 17% / 0% | | Diversity H | 0.950 nats | 0.451 nats | The two entropy values, step by step: H_before = −( 0.6·ln 0.6 + 0.2·ln 0.2 + 0.2·ln 0.2 ) = 0.950 H_after = −( (5/6)·ln(5/6) + (1/6)·ln(1/6) ) = 0.451 Quality rose by 41% (0.767 ÷ 0.545 ≈ 1.41) and diversity fell by more than half. The Yoruba text was not low quality in any real sense. The classifier simply never learned what good Yoruba looks like. Why it matters. FineWeb-Edu's 8.7% keep rate is quality bought at a steep, deliberate price in quantity. This example shows the second price: filters carry the biases of whatever they were trained on. Chapter 3 returns to this when we build filters that do not erase the languages we care about. > 💻 Code: chapter-01/1_1_4_filter_tradeoff.py. Reproduces the table above, then sweeps τ from 0 to 1 and plots quantity, quality and diversity on one chart. EXpected Output: !filter tradeoff [↑ Back to contents](#table-of-contents) --- Module 1.2: The Mathematics of the Training Objective 1.2.1 Entropy, Cross-Entropy and KL Divergence The idea. Training a language model means minimizing one quantity: the average surprise the model feels at the real next word, measured across the dataset. Information theory gives this quantity a name, cross-entropy, and splits it into two pieces with very different meanings. The math. Let p be the true distribution of the data and q the model's distribution. Define three quantities. Entropy, the irreducible randomness in the data: H(p) = − Σₓ p(x) · ln p(x) Cross-entropy, the average surprise of a model q on data that actually comes from p: H(p, q) = − Σₓ p(x) · ln q(x) KL divergence, how far the model is from the data: p(x) KL(p ‖ q) = Σₓ p(x) · ln ──── q(x) Read them as: entropy is the surprise you'd feel even with perfect knowledge. Cross-entropy is the surprise you actually feel with an imperfect model. KL divergence is the extra surprise caused by the model being wrong. These three are linked by one short identity. Expand the logarithm inside the KL divergence: KL(p ‖ q) = Σₓ p(x) ln p(x) − Σₓ p(x) ln q(x) = −H(p) + H(p, q) Rearranging gives the decomposition everything in this series rests on: ╔═══════════════════════════════════╗ ║ H(p, q) = H(p) + KL(p ‖ q) ║ ╚═══════════════════════════════════╝ Read it as: total surprise = unavoidable surprise + surprise from being wrong. Why the floor cannot move. KL divergence is never negative. The proof is short. It uses the fact that ln is concave (Jensen's inequality), which says Σ p · ln(r) ≤ ln(Σ p · r): q(x) −KL(p ‖ q) = Σₓ p(x) · ln ──── p(x) ≤ ln [ Σₓ p(x) · q(x)/p(x) ] (Jensen) = ln [ Σₓ q(x) ] ≤ ln 1 = 0 So KL ≥ 0, with equality exactly when q = p. Therefore H(p, q) ≥ H(p): no model can score a cross-entropy below the entropy of the data. Reading the two pieces: | Piece | What it is | Can training change it? | |---|---|---| | H(p) | the genuine randomness of language (many words can follow "I went to the...") | No, it is a floor | | KL(p ‖ q) | the gap between the model and the data | Yes, this is all training reduces | Training pushes the model toward the data's distribution and nothing else. Read that with data quality in mind: the best possible outcome of training is a model that perfectly reproduces whatever is in your dataset. > 💻 Code: chapter-01/1_2_1_entropy_kl_identity.py. Draws random distributions p and q, computes all three quantities, and checks H(p, q) = H(p) + KL to machine precision. Also checks that KL is never negative. Expected Output: !entropy_kl_identity 1.2.2 Training Is Maximum Likelihood The idea. In practice we never know the true p. We only have a finite dataset. So what does training actually minimize? The math. From a dataset of n tokens, build the empirical distribution p̂(x) = n(x) / n, the fraction of the dataset equal to each outcome. The training loss is the average negative log-probability the model assigns to the data: L(θ) = −(1/n) · Σᵢ ln q_θ(xᵢ) = H(p̂, q_θ) = H(p̂) + KL(p̂ ‖ q_θ) Read it as: the training loss is the cross-entropy between your dataset and the model. Minimizing it, maximizing likelihood and minimizing KL to the dataset are three names for the same thing. The target is p̂: not "language", not "truth", but your dataset. Worked example: what is the best possible model? Let a model be a single categorical distribution with probabilities q_k over K outcomes, and let n_k be how often outcome k appears. We want to maximize the data's log-likelihood subject to the probabilities adding up to 1. Use a Lagrange multiplier λ. Step 1. Write the objective with the constraint attached: J = Σⱼ nⱼ · ln qⱼ − λ · ( Σⱼ qⱼ − 1 ) Step 2. Set the derivative with respect to each q_k to zero: ∂J nₖ ─── = ──── − λ = 0 ⟹ qₖ = nₖ / λ ∂qₖ qₖ Step 3. The probabilities must add up to 1, so Σ n_k / λ = n / λ = 1, which forces λ = n. Therefore: q*ₖ = nₖ / n The optimal model is simply the frequency table. This is exactly the bigram counter from 1.1.1, now proven to be the best possible solution of its kind. A neural network is a vastly more flexible, smoother way of approximating the same target. Why it matters. "The model learns from data" sounds soft. This is the hard version: the loss function has the dataset's frequencies baked in as its minimum. Remove a document and the target changes. Duplicate a document and its weight in the target grows. > 💻 Code: chapter-01/1_2_2_mle_is_frequency.py. Runs gradient descent on a categorical model and shows the learned probabilities converge to n_k / n. Expected Output: !maximum likelihood frequency 1.2.3 The Cost of Spam, Derived The idea. Suppose a fraction ε of your dataset is spam. How much does that hurt a model that trains perfectly? We can answer this exactly, with no experiment. The math. The data is a mixture of clean text and spam: p_data(x) = (1 − ε) · p_clean(x) + ε · p_spam(x) A perfect training run reaches q = p_data (from 1.2.2). Now measure that perfect model on clean held-out text. Assume spam and clean text look different, so on clean text only the first term matters: p_data(x) = (1-ε) · p_clean(x). The loss is: L_clean = − Σₓ p_clean(x) · ln[ (1 − ε) · p_clean(x) ] = H(p_clean) + ln( 1 / (1 − ε) ) └──────────┘ └─────────────────┘ best a clean-trained the spam tax model can do Because perplexity is e^L, the spam tax becomes a clean multiplier: 1 perplexity inflation = e^(spam tax) = ───── 1 − ε Read it as: if 10% of your data is spam, a perfectly trained model is 1 ÷ 0.9 = 1.11 times more confused on clean text than a model trained on clean data alone. Worked example. | Spam share ε | Tax in nats −ln(1−ε) | Perplexity multiplier 1/(1−ε) | |---|---|---| | 1% | 0.0101 | 1.010× | | 5% | 0.0513 | 1.053× | | 10% | 0.1054 | 1.111× | | 25% | 0.2877 | 1.333× | | 50% | 0.6931 | 2.000× | The intuition is wasted probability. A model that has perfectly learned your data gives spam-shaped text a total probability of ε, so only 1 − ε is left for everything else. There is a second consequence worth more than the perplexity number: when you sample text from this model, a fraction ε of what it writes is spam. Pollution does not just make the model worse at measuring good text. It makes the model produce bad text at the rate it appeared in training. A caveat for honesty. This is a best case. Real spam overlaps partly with ordinary language, and real models do not reach q = p_data exactly. The formula is a clean lower bound on the damage, not a full picture. 1.2.4 Proof in Code: Clean vs Spam-Polluted Training The idea. Let's test the derivation with the smallest possible experiment. I built a character-level language model that predicts the next character using nothing but counts, and trained two copies on the same public-domain text: one clean, and one with spam lines inserted after roughly every quarter of the lines. held-out loss, trained on clean text: 2.4838 held-out loss, trained on spam-polluted text: 2.6550 Both models were tested on the same clean held-out text. The math check: the loss gap is 2.6550 − 2.4838 = 0.1712 nats, so the perplexity ratio is PPL_polluted ──────────── = e^(2.6550 − 2.4838) = e^0.1712 ≈ 1.187 (equivalently 14.2 / 12.0) PPL_clean If the bound from 1.2.3 were tight, a multiplier of 1.187 would mean 1/(1−ε) = 1.187. Solve for ε: 1 ε = 1 − ─────── ≈ 0.16 1.187 So the polluted model behaves as though about 16% of its probability mass is tied up in spam-shaped patterns. Treat that as a rough reading, since a toy counter does not satisfy the clean separation assumption exactly, but the order of magnitude is what the theory predicts. Sample text generated from the polluted model shows fragments of the inserted spam, words like CLICK, BUY, FREE and www, sitting inside what should have been ordinary prose. A transformer with billions of parameters is a far better copier than this toy model. It will learn the spam more precisely, not less. > 💻 Code: chapter-01/1_2_4_spam_pollution_experiment.py. Trains both character models on a public-domain text file, prints held-out loss and perplexity, measures the real spam share ε, and compares the observed perplexity ratio against 1/(1−ε). Change the spam rate to reproduce the table in 1.2.3. 1.2.5 The ln(V) Sanity Check The idea. At the very start of training, before the model has learned anything, it should know nothing. What loss does knowing nothing give? The math. A model that knows nothing assigns the same probability to every token: q(x) = 1/V. Plug that into the cross-entropy. This works for any data distribution p: H(p, q) = − Σₓ p(x) · ln(1/V) = ln V · Σₓ p(x) = ln V · 1 = ln V So the starting loss should sit almost exactly at ln V, and the starting perplexity at exactly V (the model is as unsure as if choosing among all V tokens). Worked example. | Tokenizer | Vocabulary V | Expected starting loss ln V | |---|---|---| | Llama 2 | 32,000 | 10.37 | | GPT-2 | 50,257 | 10.82 | | Llama 3 | 128,256 | 11.76 | How to use it as a debugging tool. Compare your first logged loss against ln V: - Far above ln V: the model starts overconfident, often because initial weights are too large. - Far below ln V: the model somehow already knows the answers. Suspect label leakage, where the targets are accidentally visible in the inputs, or a data bug that makes text trivially predictable (heavy duplication, for instance). - Right at ln V but never falling: the model is not receiving a usable signal. Check that inputs and targets are properly aligned and that the data is not noise. This single check has caught more pipeline bugs than almost any other. It costs one print statement. > 💻 Code: chapter-01/1_2_5_ln_vocab_check.py. Computes ln V for common tokenizers and runs a randomly initialized model on a batch to confirm the starting loss lands where theory says. What this means practically: the moment you accept a document into your training set, you are voting for the model to reproduce it. Every filtering decision in the rest of this series exists because of this one mathematical fact. [↑ Back to contents](#table-of-contents) --- Module 1.3: What the Model Is Truly Optimizing For 1.3.1 From Tokens to Loss to Perplexity The idea. A document is a sequence of tokens, and its probability is the product of each token's probability given what came before. Products of many small numbers collapse toward zero, so in practice everything runs in log space, where multiplication becomes addition. The math. From the chain rule in 1.1.1, take the logarithm and average over the T tokens. The flipped, averaged log-probability is the loss: L = −(1/T) · Σₜ ln p_θ(xₜ | x<ₜ) (add over t = 1 … T) Perplexity is e raised to the loss: PPL = e^L = ( ∏ₜ 1 / p_θ(xₜ | x<ₜ) ) ^ (1/T) Read it as: perplexity is the geometric mean of the inverse probabilities. It has an intuitive reading: if the model were choosing uniformly among k options at every step, each probability would be 1/k and PPL = k. A perplexity of 15 means the model is, on average, about as unsure as if picking among 15 equally likely options. Worked example. A four-token sentence where the model assigns probabilities 0.5, 0.25, 0.1 and 0.8 to the true tokens: | Token | Probability | ln p | |---|---|---| | 1 | 0.50 | −0.6931 | | 2 | 0.25 | −1.3863 | | 3 | 0.10 | −2.3026 | | 4 | 0.80 | −0.2231 | | Total | product = 0.01 | −4.6051 | Then: 4.6051 L = ──────── = 1.1513 PPL = e^1.1513 ≈ 3.16 4 Check it the other way, straight from the product: (1 ÷ 0.01)^(1/4) = 100^0.25 = 3.16. ✓ Why log space is not optional. A 1,000-token document where each token gets probability 0.1 has probability 10^-1000. The smallest number a 64-bit float can hold is about 10^-308. Multiplying directly gives zero, and zero's logarithm is minus infinity. Summing logs gives the stable answer: Σ ln 0.1 (1,000 times) = 1000 × (−2.3026) = −2302.6 A trap: perplexity is not comparable across tokenizers. Per-token loss depends on how text was chopped into tokens. A tokenizer that splits text into many tiny, easy-to-predict tokens gets a lower per-token loss without being a better model. The fair, tokenizer-independent measure is bits per byte (BPB): T × L BPB = ──────── B × ln 2 where T is the number of tokens, L the loss per token in nats, and B the number of bytes in the text. The ln 2 converts nats to bits. Worked example. A document of 1,000 tokens and 4,200 bytes at L = 2.3 nats per token: 1000 × 2.3 2300 BPB = ───────────── = ────── ≈ 0.790 4200 × 0.6931 2911 When you compare your model against someone else's, compare BPB, not perplexity. This matters especially for African languages, where tokenizers differ most. > 💻 Code: chapter-01/1_3_1_perplexity_and_bpb.py. Computes the worked example both ways, demonstrates the underflow failure of naive multiplication, and converts loss to bits per byte. 1.3.2 How the Data Pushes on the Model: the Gradient The idea. Now we can look inside a single training step and see exactly how data shapes the model. A model outputs a score (a logit) z_i for each of the V tokens, and a softmax turns scores into probabilities. The math. The softmax and the loss for one example, where y is the true next token: e^zᵢ pᵢ = ──────── ℓ = −ln p_y = −z_y + ln Σⱼ e^zⱼ Σⱼ e^zⱼ Differentiate the loss with respect to a logit z_i. The first term contributes −1 only when i = y, and the second term differentiates to the softmax probability itself: ╔══════════════════════════════════════╗ ║ ∂ℓ / ∂zᵢ = pᵢ − 𝟙[ i = y ] ║ ╚══════════════════════════════════════╝ Read it as: the gradient is just what the model predicted minus what actually happened. (𝟙[i = y] is 1 for the true token and 0 for every other token.) Gradient descent moves each logit against this gradient: it raises the true token's score and lowers all the others in proportion to how much probability they stole. Worked example. Three tokens with logits (2, 1, 0), and the true token is token 3. The softmax needs e² = 7.389, e¹ = 2.718, e⁰ = 1, which sum to 11.107: | Token | Probability | Gradient p_i − 𝟙[i=y] | What the update does | |---|---|---|---| | 1 | 7.389 ÷ 11.107 = 0.665 | +0.665 | push score down | | 2 | 2.718 ÷ 11.107 = 0.245 | +0.245 | push score down | | 3 (true) | 1 ÷ 11.107 = 0.090 | 0.090 − 1 = −0.910 | push score up hard | The loss is −ln 0.090 = 2.41. The true token was given only 9% probability, so the push is strong. The gradient entries sum to exactly zero (0.665 + 0.245 − 0.910 = 0), because probability moved is probability conserved. Averaging over the dataset. For a fixed context c, the average gradient over all the training examples that share that context is: E_data[ ∇_z ℓ ] = q_θ(· | c) − p_data(· | c) Read it as: the average push equals "what the model believes" minus "what the data says." This is zero only when the model's distribution matches the data's distribution for that context. The update never stops pulling until the model has copied the data. If spam follows context c with frequency ε, then spam tokens receive a pull proportional to ε on every step, for as long as training runs. This is the one-line mechanism behind "the model becomes whatever you feed it." > 💻 Code: chapter-01/1_3_2_softmax_gradient_check.py. Verifies the formula p − onehot against numerical finite-difference gradients, reproduces the three-token example, and shows the averaged gradient going to zero as the model matches the data. 1.3.3 Scaling Laws, With Real Numbers The idea. Scaling laws describe how loss falls as a model and its training data both grow. The widely cited Chinchilla work (Hoffmann et al., 2022) fits a simple form with three terms: a floor, a term that shrinks as the model grows, and a term that shrinks as the data grows. The math. With N parameters and D training tokens: L(N, D) = E + A / N^α + B / D^β ───── ─────────── ─────────── floor model-size data-size term term The fitted constants reported in the paper's third estimation approach are approximately: | Constant | Value | Meaning | |---|---|---| | E | 1.69 | irreducible loss, the entropy floor H(p) from 1.2.1 | | A | 406.4 | model-size coefficient | | B | 410.7 | data-size coefficient | | α | 0.34 | how fast loss improves with more parameters | | β | 0.28 | how fast loss improves with more data | These constants belong to the paper's own dataset and tokenizer, so treat the absolute losses as illustrative rather than as predictions for your model. What transfers is the shape. Worked example 1: what doubling the data buys. The data term scales as D^−β, so doubling D multiplies it by: 2^(−0.28) ≈ 0.824 Doubling your data shrinks the data-related part of the loss by about 18%. It never touches E. Worked example 2: two real model sizes. Take Llama 3's 8B model at 15T tokens, and the 405B model at about 15.6T tokens. Each term, computed from the formula: | | Model term A / N^α | Data term B / D^β | Total L = 1.69 + ... | |---|---|---|---| | 8B params, 15T tokens | 0.175 | 0.084 | 1.69 + 0.175 + 0.084 ≈ 1.95 | | 405B params, 15.6T tokens | 0.046 | 0.083 | 1.69 + 0.046 + 0.083 ≈ 1.82 | Look at the 405B row. The model-size term has shrunk to 0.046, while the data term (0.083) is nearly double it. For a model that large, data is the term still holding the loss up. This is the mathematical reason the field's appetite for clean, usable text keeps climbing. Where data quality enters. The formula counts tokens, but not all tokens are equal. Better filtering behaves like a larger effective D, or equivalently a smaller B. This is the quantitative form of "a smaller, cleaner dataset can beat a larger, dirtier one." > 💻 Code: chapter-01/1_3_3_chinchilla_calculator.py. Implements L(N, D) with the fitted constants, reproduces both rows of the table, and plots loss curves as you vary N and D. 1.3.4 Compute-Optimal vs Over-Trained The idea. Given a fixed compute budget, how should you split it between a bigger model and more data? The math. Training compute is approximately 6 floating-point operations per parameter per token: C ≈ 6 · N · D Substitute D = C / (6N) into L(N, D) and set the derivative with respect to N to zero. The result is that the best N and D both grow with compute, as power laws: N_opt ∝ C^( β / (α+β) ) = C^( 0.28 / 0.62 ) ≈ C^0.45 D_opt ∝ C^( α / (α+β) ) = C^( 0.34 / 0.62 ) ≈ C^0.55 Read it as: when your budget grows, spend it on a bigger model and more data, at close to the same rate. The practical rule of thumb that came out of this is roughly 20 training tokens per parameter. Worked example: how far modern models sit from that rule. | Model | Parameters | Tokens | Tokens per parameter | |---|---|---|---| | Chinchilla | 70B | 1.4T | 20 | | Llama 3 405B | 405B | ~15.6T | ~38 | | Llama 3 8B | 8B | 15T | ~1,875 (about 94× the rule) | Why anyone over-trains. The "optimal" above minimizes loss for a fixed training budget. But a model is trained once and used many times. Inference costs roughly 2 operations per parameter per token served, so with T tokens served over a model's lifetime: inference cost 2 · N · T T ────────────── = ─────────── = ───── training cost 6 · N · D 3 D The model size N cancels out. If a model serves more than 3D tokens in its lifetime, inference dominates the bill. For D = 15T, that threshold is 3 × 15T = 45 trillion tokens served, which a widely deployed model can pass. So it pays to train a smaller model on far more data: slightly worse for the training bill, much cheaper for everyone who runs it afterward. That single design choice is why a smaller model's hunger for data keeps rising even as model sizes stay similar. It is also good news for builders with limited compute: a small model trained on excellent data is exactly the shape of model we can afford to run. > 💻 Code: chapter-01/1_3_4_compute_optimal_and_inference.py. Finds the loss-minimizing N and D for a given compute budget, and finds the break-even point where inference cost passes training cost. 1.3.5 Where the Numbers From the Intro Come Back This is where the opening numbers of the post connect back to the mathematics. Meta's published figures say training the 405-billion-parameter model consumed 30.84 million GPU-hours. Check that against C ≈ 6ND. Step 1: total compute needed. C ≈ 6 × (405 × 10⁹) × (15.6 × 10¹²) ≈ 3.8 × 10²⁵ FLOPs Step 2: total GPU-seconds available. 30.84 × 10⁶ hours × 3600 s/hour ≈ 1.11 × 10¹¹ GPU-seconds Step 3: compute delivered per GPU per second. 3.8 × 10²⁵ ──────────── ≈ 3.4 × 10¹⁴ FLOP/s per GPU 1.11 × 10¹¹ An H100's peak throughput in the precision used for training is on the order of 10^15 FLOP/s, so each GPU ran at roughly a third of peak, which is in the normal range for large training runs. The introduction's numbers are not exaggerations; they follow from the same arithmetic as the equation above. All of that compute exists to shrink one number, the loss, by pushing the model closer to the data it was shown. In the language of 1.2.1, it drives KL(p̂ ‖ q_θ) toward zero and leaves H(p) exactly where it is. None of it can make the model smarter than its data. A perfectly optimized model trained on a flawed dataset is still a flawed model, just a very confident one. What you can do on a laptop. You cannot afford 30.84 million GPU-hours. You can afford to measure the data term B / D^β for your own data. Train a tiny model on 1%, 2%, 4%, 8% of your corpus, plot held-out loss against tokens, and fit a line on log-log axes. The slope is your β. Compare two versions of your pipeline (before and after a filter) and you have a number for whether the filter helped. That is the loop from 1.1.2, with math attached. > 💻 Code: chapter-01/1_3_5_mini_scaling_experiment.py. Trains small models on growing slices of a text file, fits L ≈ E + B / D^β on the results, and compares two dataset versions side by side. [↑ Back to contents](#table-of-contents) --- Chapter 1 Cheat Sheet | Idea | Equation | One-line meaning | |---|---|---| | Chain rule | P(x) = Π P(xₜ ∣ x<ₜ) | text probability is a product of next-token probabilities | | Mixing | p_mix = Σ wₖ pₖ | source weights are design decisions | | Filtering | p_f = p·f / Z | Z is the keep rate | | Cross-entropy split | H(p,q) = H(p) + KL(p‖q) | floor plus the only part training can reduce | | MLE optimum | q*ₖ = nₖ / n | the best model is the frequency table of your data | | Spam tax | PPL × 1/(1−ε) | spam wastes exactly its share of probability | | Starting loss | L₀ = ln V | the free bug detector | | Perplexity | PPL = e^L | effective number of equally likely choices | | Bits per byte | BPB = T·L / (B·ln 2) | tokenizer-independent comparison | | Gradient | ∂ℓ/∂zᵢ = pᵢ − 1[i=y] | predicted minus actual | | Scaling law | L = E + A/Nᵅ + B/Dᵝ | floor + model term + data term | | Training compute | C ≈ 6ND | cost grows with both size and data | | Inference vs training | T / 3D | model size cancels out | Try it yourself. Four exercises, solutions in the repo under chapter-01/exercises/: 1. In 1.1.1, how many spam copies make P(free | the) exceed 0.9? 2. A dataset is 8% spam. By what factor does a perfect model's perplexity on clean text rise? What fraction of its generated text will be spam? 3. A tokenizer has V = 65,536. What starting loss should you expect? 4. In 1.3.3, how many tokens would the 8B model need to push the data term B / D^β below 0.05? --- References - Meta AI, Llama 3.1 Model Card: official source for the 30.84 million GPU-hour, 700W and CO2 figures, and used again in 1.3.5. - TIME, "Exclusive: OpenAI Used Kenyan Workers on Less Than $2 Per Hour to Make ChatGPT Less Toxic", January 18, 2023: the original investigation, based on payslips and internal documents. - CBS News, "Kenyan workers with AI jobs thought they had tickets to the future until the grim reality set in", 60 Minutes, 2025: confirms the $12.50-to-$2 pay gap and separately implicates Scale AI's Remotasks platform. - Alexandr Wang, Wikipedia: background on Scale AI and Meta's 2025 investment. - Fox Business, ["Trump rebrands AI, rejects 'globalist scheme'
I HAVE JUST FOUND SOMETHING INSANE Meta's official Llama repo has a default sample image it's a dog onboard github.com/meta-llama/llama-models/blob/main/mode… Llama 4's multimodal chat script loads this dog every time a dev tests vision. Meta's AI has a mascot and they never told anyone Also pumpfun is spending millions on onboarding. In callout rewards, updates, KOLs and now referrals. never been run wtf?? So genny onboarddog - $onboard
I HAVE JUST FOUND SOMETHING INSANE Meta's official Llama repo has a default sample image it's a dog onboard github.com/meta-llama/llama-models/blob/main/mode… Llama 4's multimodal chat script loads this dog every time a dev tests vision. Meta's AI has a mascot and they never told anyone Also pumpfun is spending millions on onboarding. In callout rewards, updates, KOLs and now referrals. never been run wtf?? So genny onboarddog
A Xiaomi treinou uma IA gigante ao vivo, com o custo na tela #shorts O MiMo V2.6 foi treinado por reforço com painel público e custo aberto: US$ 2,6 milhões, ao vivo. DeepSeek, Meta, BLOOM, OLMo e SmolLM3 já tinham aberto partes do treino. O que é inédito aqui, e o furo que a transparência revelou. Painel ao vivo do treino: mimo.xiaomi.com Post oficial MiMo-V2.6: mimo.mi.com Relatório técnico: arxiv.org Framework de treino (verl): XiaomiMiMo/verl Ambientes de RL: huggingface.co Auditoria Vals AI: vals.ai Comparação: DeepSeek-V3 (custo): arxiv.org DeepSeek-R1 na Nature: nature.com Llama 3.1 (horas de GPU): github.com/meta-llama/llama-models/blob/main/mode… BLOOM (logs ao vivo, 2022): huggingface.co OLMo 2: allenai.org SmolLM3 (logs e checkpoints): huggingface.co Ideia: vídeo de bycloud. #opensource #programacao #ia