GSM8K on Hugging Face: 4 of the Top 12 Rows Show a Sibling Checkpoint's Score, and 20 of 28 Rows Are Unmerged PRs
TL;DR
- On November 3, 2026 the Hugging Face GSM8K board lists 28 rows from 28 repos. 20 come from pull requests that are still open, 8 sit on the repo's main branch, and 0 are verified.
- Four of the top 12 rows show a number that the model card reports for a different checkpoint of the same family. Rank 1 (99.6) is the MiMo-V2.5-Pro Base score, registered on the post-trained repo. Rank 2 (96.8) is the Llama 3.1 405B Instruct score, registered on the base repo. Ranks 7 and 12 are DeepSeek base scores on the post-trained repos.
- The numbers are real self-reports. The labels are what is off, and the board has no column that would show it.
- GSM8K has 1,319 test problems, so one problem is 0.076 points. The rank 2 to rank 3 gap (0.74) is 10 problems, and a 95% interval at 95% accuracy is about 1.2 points wide on each side.
- A score of 99.6 means about 5 misses. The GSM8K-Platinum project relabeled 10 answers in the same test set. If those relabels are right, a score that high implies the model matched at least 5 of the 10 keys they call wrong.
!Top 12 rows of the GSM8K board on Hugging Face, with the four rows whose number belongs to a sibling checkpoint in red
In the chart, red marks rows where the model card gives this number for a sibling checkpoint, blue marks a row whose card table matches the repo (LLaDA2.0-flash), and grey marks rows I did not check against the card.
What does the GSM8K board on Hugging Face contain?
GSM8K is a set of grade-school math word problems with a train split of 7,473 and a test split of 1,319. Hugging Face lists openai/gsm8k as a benchmark dataset, so a model repo can publish a result in a .eval_results/*.yaml file and the dataset's board collects it.
I pulled the board on November 3, 2026 (KST) from the JSON behind the leaderboard view. It has 28 rows. Each row is from a different repo, verified is false on every row, and the top of the page reads like a clean ranking: 99.6, 96.8, 96.06, 94.16, 93.93.
The dataset's own eval.yaml describes a specific protocol: an inspect-ai task, a zero-shot prompt template that asks for a boxed final answer, four epochs reduced to pass@1, and a model-graded scorer. The rows do not follow one protocol. Of the 28 notes, 15 are empty. Where a note gives a shot count it ranges across 0, 4, 5 and 8. Two rows from one lab give a complete setup (8-shot, temperature 0, n=1319, lm-evaluation-harness), and one row reports a strict-match score over 1,075 of the 1,319 problems. So I treat each row as a claim with a source link, not as a measurement the board produced.
Why does rank 1 belong to a different checkpoint?
The top row is XiaomiMiMo/MiMo-V2.5-Pro at 99.6. Its model card has a section titled "Base Model Evaluation", and the GSM8K row there reads 8-shot with these values: 99.6 for MiMo-V2.5-Pro Base, 83.3 for MiMo-V2.5 Base, 92.6 for DeepSeek-V4-Pro Base. That is the only place GSM8K appears in the card. The card's own download table lists two repos: MiMo-V2.5-Pro and MiMo-V2.5-Pro-Base, and the second one exists on the Hub.
So the board shows the base-model score on the post-trained repo. The pull request that registered it says the results "were extracted from the model card". It was opened on April 27, 2026 and is still open. Nothing in the YAML says "base", and the 8-shot setting is not in the note, which is empty.
The card is not hiding anything. It labels its table as Base. The distinction is lost only in the step that turns a card into a board row.
Is that one slip, or does it repeat?
I checked the sources for the top rows. It repeats in four of the top 12, and in both directions.
| Board rank | Repo on the board | Score | Where the card puts this number | Sibling repo |
|---|---|---|---|---|
| 1 | MiMo-V2.5-Pro | 99.6 | "Base Model Evaluation" table, Pro Base column, 8-shot | MiMo-V2.5-Pro-Base |
| 2 | Llama-3.1-405B | 96.8 | "Instruction tuned models" table, 405B Instruct column, 8-shot CoT, em_maj1@1 | Llama-3.1-405B-Instruct |
| 7 | DeepSeek-V4-Pro | 92.6 | Base table, DeepSeek-V4-Pro-Base column, 8-shot | DeepSeek-V4-Pro-Base |
| 12 | DeepSeek-V3 | 89.3 | "Base Model" section, DeepSeek-V3 column, 8-shot | DeepSeek-V3-Base |
Rank 2 is the mirror image. The Llama 3.1 model card covers every size and both variants. Its "Base pretrained models" table has no GSM8K row. Its "Instruction tuned models" table has one, and the 405B Instruct value is 96.8. The pull request is titled "Add GSM8K eval result (96.8, 8-shot CoT)" and attaches that number to the base repo meta-llama/Llama-3.1-405B.
For DeepSeek-V3 the card has a Base Model section with GSM8K at 89.3 and a Chat Model section with no GSM8K row. For DeepSeek-V4-Pro the table header names the three columns DeepSeek-V3.2-Base, DeepSeek-V4-Flash-Base and DeepSeek-V4-Pro-Base, and 92.6 is the last one.
Why does it matter if the number is the same? Because the board is read by repo name. Someone choosing a model for math word problems sees MiMo-V2.5-Pro at the top and downloads that repo. The card reports no GSM8K number for that checkpoint. If the four rows were moved to the sibling repos, the post-trained MiMo, DeepSeek-V4-Pro and DeepSeek-V3 would have no GSM8K row at all, and the Llama 405B base repo would lose its rank 2.
The correct placement would not change the ordering of the numbers: 99.6 would still lead, from the right repo. It would change which model each number tells you about.
!Where the 28 GSM8K rows come from: open pull requests, main branch, verified
Where do the rows come from, and what does "verified" mean here?
I fetched the discussion for every row that carries a pullRequest number, and listed .eval_results on the main branch of every repo.
| Source | Rows |
|---|---|
| Open PR from an account flagged as Hugging Face (julien-c 9, nielsr 6, SaylorTwift 3, burtenshaw 1) | 19 |
| Open PR from a bot account (EvalEvalBot) | 1 |
| File on the repo's main branch | 8 |
| Verified | 0 |
All 20 pull requests are open, and for all 20 the YAML file is absent from main. For all 8 other rows it is present. Four of the 20 pull requests were opened on November 2, one day before I pulled the board, and one of them produced rank 3 (LLaDA2.0-flash, 96.06). The board moves when someone opens a pull request, with no action from the model's owner.
The Hugging Face documentation defines the badges. "Community" means a result submitted via an open PR that is not merged to main. "Verified" means a valid verify token, which requires the evaluation to have run in HF Jobs with inspect-ai. So 20 of 28 rows are community results by definition, and no row meets the verified bar. The documentation says community scores are visible while the PR is open. That is a reasonable design for bootstrapping a board. It also means a reader has to look at the badge to know which rows the model's owner stands behind, and a screenshot of the ranking does not show it.
The eight main-branch rows are not uniformly clean either. Two Mellum2 repos show the same 81.73. The YAML on the Base repo states that its numbers were measured on the pre-extension Base-Pretrain checkpoint, so it is one measurement displayed twice. This one is disclosed in the file, and it still occupies two ranks.
How much can 1,319 problems separate?
GSM8K is nearly saturated at the top, so the ordering is decided by a handful of problems.
| Quantity | Value |
|---|---|
| One problem | 0.076 points |
| Rank 1 to rank 2 gap (99.6 to 96.8) | 2.8 points, about 37 problems |
| Rank 2 to rank 3 gap (96.8 to 96.06) | 0.74 points, about 10 problems |
| Spread of ranks 3 to 6 (96.06 to 93.10) | 2.96 points |
| 95% half-width at 95% accuracy, n=1,319 | 1.18 points (illustrative) |
| 95% half-width at 99% accuracy, n=1,319 | 0.54 points (illustrative) |
The interval rows are a sampling calculation for a single run on a fixed test set. They ignore prompt variation, which is larger when rows use different shot counts. Even so, ranks 3 to 6 sit inside a band that a single run cannot separate, and the rows within it come from different protocols: 8-shot CoT for some, no note for others.
One other check is exact arithmetic. The two rows with a full setup report 84.9886 and 78.4685, and with n=1319 these are 1,121 and 1,035 correct answers. ProKope-7B's note says 1,075 of 1,319, which is 81.50. Those rows can be audited back to a count. Most rows cannot.
!How big is each effect: one problem, intervals, ceiling lost to relabeled keys, gaps between ranks
Can a 99.6 be right if some answer keys are wrong?
This part is arithmetic with one assumption, so I mark it as a question to check, not a finding.
The GSM8K-Platinum project (MIT, Gradient Science) reviewed the 1,319 test problems. Its dataset card reports 219 flagged by models, 110 rejected as poorly written, 10 relabeled, 99 verified, leaving 1,209 questions. The 10 relabeled items are answer keys that the authors found to be wrong.
Take a score of 99.6 on the original test set. 1,314 of 1,319 rounds to 99.6, so that is about 5 misses. Suppose the 10 relabels are correct and scoring is exact match against the original key. A model that solves an item correctly then loses it, because the key is wrong. A model that is right on all 10 would lose all 10 and top out at 99.24. To score 99.6 it must have matched the key on at least 5 of the 10, which means it gave the answer the key says, not the answer the authors say is right.
Three explanations fit: the relabels are wrong for some of those items, the scoring is more lenient than exact match, or the model has seen the original keys. I have not run the model, and a base model's 8-shot number with no note on answer extraction does not let me tell these apart. What I can say is that for a benchmark with known key errors, a score above the ceiling is a reason to read the setup, not a reason to rank it first. Platinum also reports that cleaning the set separates models that look tied on the original, which is the same point from the other side.
How do I pull the board and classify rows myself?
This script needs only the standard library. It counts rows by source and prints the sampling arithmetic used above.
import json, urllib.request, collections
def get(url):
req = urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0"})
return json.load(urllib.request.urlopen(req, timeout=60))
DATASET = "openai/gsm8k"
rows = get(f"https://huggingface.co/api/datasets/{DATASET}/leaderboard")
by_source = collections.Counter()
for r in rows:
pr = r.get("pullRequest")
if pr:
disc = get(f"https://huggingface.co/api/models/{r['modelId']}/discussions/{pr}")
by_source[("open PR" if disc["status"] == "open" else disc["status"])] += 1
else:
by_source["main branch"] += 1
print("rows:", len(rows), "| verified:", sum(bool(r["verified"]) for r in rows))
print(dict(by_source))
n = 1319
print("one problem = %.3f points" % (100 / n))
for acc in (0.95, 0.99):
se = (acc * (1 - acc) / n) ** 0.5 * 100
print("accuracy %.2f: 95%% half-width = %.2f points" % (acc, 1.96 * se))
print("ceiling if 10 keys are wrong: %.2f" % (100 * (n - 10) / n))
On November 3, 2026 it printed:
rows: 28 | verified: 0
{'open PR': 20, 'main branch': 8}
one problem = 0.076 points
accuracy 0.95: 95% half-width = 1.18 points
accuracy 0.99: 95% half-width = 0.54 points
ceiling if 10 keys are wrong: 99.24
The board endpoint is undocumented, so the field names may change. To check a row, open its pullRequest discussion, read the YAML, then open the source card and find the table the number came from.
Checklist for reading a saturated board
1. Open the badge before the rank. Community means an open PR, and verified means a run in HF Jobs. Neither is a statement from the model's owner.
2. Match the checkpoint, not the family. Find the card table the number came from and read its column header. Base, instruct and chat checkpoints sit in different tables.
3. Ask how many problems separate two rows. On GSM8K, 10 problems is 0.76 points.
4. Prefer rows that state their setup. Shot count, temperature, n and the answer-extraction rule decide the score more than rank does.
5. Check for the same measurement on two repos. A copied number is one result, not two.
6. Distrust a score above the key ceiling. If a benchmark has known label errors, read the evaluation setup before you trust the lead.
FAQ
How many GSM8K results on Hugging Face are verified?
None. On November 3, 2026 all 28 rows on the board show verified as false. Verified requires a valid verify token from an inspect-ai run in HF Jobs.
Is the 99.6 at rank 1 of the GSM8K board wrong?
Nothing here shows the number is wrong. It is the MiMo-V2.5-Pro Base score from the model card's Base Model table, 8-shot. The problem is that it is registered on the post-trained repo, which has its own separate row in the card's download table.
What does "community" mean on a Hugging Face leaderboard row?
It means the result was submitted through an open pull request that has not been merged to the repo's main branch. In the GSM8K board, 20 of 28 rows are in that state.
Why is the Llama 3.1 405B base repo ranked second on GSM8K?
Its row carries 96.8, which the shared model card gives for Llama 3.1 405B Instruct, 8-shot CoT. The card has no GSM8K row in its base table. The instruct repo is a separate repo.
How many problems does one point of GSM8K accuracy represent?
GSM8K has 1,319 test problems, so one point is about 13 problems and one problem is 0.076 points.
What is GSM8K-Platinum?
It is a revised version of the GSM8K test set from Gradient Science. Of 1,319 test problems it rejected 110, relabeled 10 and kept 1,209. It exists because the original set has ambiguous questions and wrong keys.
Further reading
- GPQA Diamond Has 198 Questions: What the Hugging Face Leaderboard's 115 Rows Can and Cannot Rank
- How Hugging Face Official Benchmark Leaderboards Actually Work: .eval_results YAML, the base_model Filter, and the 30% the Default View Hides
Sources
- GSM8K dataset and board: huggingface.co
- GSM8K evaluation definition (eval.yaml): huggingface.co
- Hugging Face documentation, Evaluation Results (badges, community PRs): huggingface.co
- GSM8K-Platinum dataset card: huggingface.co
- GSM8K-Platinum write-up: gradientscience.org
- Rank 1 model card (Base Model Evaluation table): huggingface.co
- Rank 1 registering pull request: huggingface.co
- Llama 3.1 model card (base and instruction tuned tables): github.com/meta-llama/llama-models/blob/main/mode…
- Rank 2 registering pull request: huggingface.co
- DeepSeek-V4-Pro model card: huggingface.co
- DeepSeek-V3 model card: huggingface.co
All counts in this article are my own measurements from the board JSON, pull request discussions, dataset cards and the cited model cards on November 3, 2026 (KST); I have no affiliation with any model, lab or benchmark named here. Which table a number came from is my reading of each card, and the board endpoint is undocumented and may change.
How long does it take to train an AI model on cloud GPUs?
Training an AI model takes anywhere from a few minutes to several months, depending on model size, how many tokens it trains on and how many GPUs run in parallel. On Nodus a small fine-tune can finish in under five minutes, while by our math a 7B language model trained from scratch on 140 billion tokens takes about 3.6 days on 64 NVIDIA H100s. The sections below show the formula, check it against published runs, and list what makes the real clock run longer than the math.
How do you estimate how long a training run will take?
Work out the total math the run needs, then divide by how fast your GPUs actually do math. Training a transformer takes about 6 floating-point operations per parameter per token, the rule from Kaplan et al.'s scaling-laws paper. An NVIDIA H100 SXM peaks at about 989 teraFLOPS of dense BF16 math, half the sparse figure on NVIDIA's H100 page.
No real run gets the full peak. The share you get is called model FLOPs utilization, and 30% is a fair planning number for code you have not tuned. That gives:
GPU-hours = (6 × parameters × tokens) / (989e12 × utilization × 3,600)
wall-clock = GPU-hours / number of GPUs (if scaling were perfect)
At 30% utilization, one H100-hour delivers about 1.07 × 10^18 useful operations. The second line is the optimistic part, and we come back to it below.
How long does it take to train a language model from scratch?
Days to weeks for small and mid-size models, if you have enough GPUs. Here is the formula at 30% utilization, using a compute-optimal budget of about 20 tokens per parameter from DeepMind's Chinchilla paper, plus one longer 7B run.
| Model and tokens | H100 GPU-hours | Fewer GPUs | More GPUs |
| --- | ---: | ---: | ---: |
| 1B on 20B tokens | 112 | 1 GPU: 4.7 days | 8 GPUs: 14 hours |
| 7B on 140B tokens | 5,505 | 8 GPUs: 28.7 days | 64 GPUs: 3.6 days |
| 7B on 2T tokens | 78,643 | 64 GPUs: 51.2 days | 512 GPUs: 6.4 days |
| 70B on 1.4T tokens | 550,500 | 512 GPUs: 44.8 days | 2,048 GPUs: 11.2 days |
Our arithmetic: 6 × parameters × tokens, 989 teraFLOPS dense BF16, 30% utilization, perfect scaling. Real runs land later; see the overheads section.
Token count moves the clock as much as model size. The same 7B model takes 14 times longer on 2 trillion tokens than on 140 billion. Many open models now train far past the Chinchilla ratio, so settle the token budget first. Our post on what it costs to train an LLM prices the same GPU-hours in dollars.
How long do real published training runs take?
Small reproductions take hours, and frontier models take months. Andrej Karpathy's llm.c reproduction of GPT-2 124M trained on 10 billion tokens in about 90 minutes on one 8 × NVIDIA A100 80GB node. His GPT-2 1.6B run took about 24 hours on one 8 × H100 node: 32,000 steps of roughly 1 million tokens each, at about 2.75 seconds per step.
That 1.6B run is a good check on the formula. It works out to 192 GPU-hours for about 33.6 billion tokens, which back-solves to roughly 46% utilization. Karpathy's own logs show 42% to 47% at the start, so well-tuned code beats our 30% planning figure by about half.
At the top end, Meta's Llama 3.1 model card lists 30.84 million H100 GPU-hours for the 405B model. The Llama 3 paper says it trained on up to 16,384 H100s. Divide one by the other and even that cluster needs about 78 days of pure compute, a floor on the real calendar time.
Do more GPUs make training proportionally faster?
Only roughly, and never for free. Adding GPUs splits the same GPU-hours across more machines, so the bill stays about the same while the calendar shrinks. In practice the bill creeps up, because GPUs spend more time talking to each other.
Meta's own numbers show the effect. The Llama 3 paper reports 43% utilization on 8,192 GPUs and 41% on 16,384 GPUs for the same model, a small but real loss from doubling the cluster.
!Days to train a 7B model on 2 trillion tokens: 409.6 days on 8 H100s, 51.2 days on 64, 6.4 days on 512, at 30% utilization and perfect scaling.
The same 78,643 GPU-hours: one 8-GPU node needs 409.6 days, a 512-GPU cluster about a week.
Our decision rule: pick the smallest GPU count that finishes before your deadline, not the largest you can get. Single nodes of 8 GPUs scale best because the GPUs share fast NVLink links. Multi-node runs add network cost and more machines that can fail.
How long does fine-tuning take?
Usually minutes to a few hours, because fine-tuning sees far fewer tokens than pretraining. Fine-tuning a 7B model on 50 million tokens is about 2 H100-hours of math at 30% utilization by the same formula, and LoRA does less than that because it skips most weight gradients.
For small jobs, the work around training often takes longer than training itself. In our published fine-tuning benchmark, Qwen2.5-0.5B on one NVIDIA L4 spent 112.8 seconds in the training loop on Nodus. From submission to a verified exported model took 4.8 minutes, or 288 seconds.
!Small fine-tune on Nodus: 112.8 seconds in the training loop versus 288 seconds from submission to exported model.
For a small fine-tune, startup, evaluation and export took longer than the training loop.
So for fine-tuning, the honest answer to "how long" is mostly startup, model download, evaluation and export. On Nodus, a TrainingJob imports a pinned Hugging Face model revision into a read-only Volume once, and later runs reuse it, so billed GPU time is not spent downloading weights.
Why does the real clock run longer than the math?
Because the formula counts only useful training steps. Four things add time on most projects:
- Waiting for capacity. A large multi-GPU request may queue until enough machines are free together.
- Startup. Pulling the image, loading the model and reading the first batch of data.
- Failures and restarts. Over a 54-day stretch of Llama 3 405B pretraining, Meta logged 466 job interruptions, 419 of them unexpected. It still reported more than 90% effective training time, meaning at least a tenth of the elapsed time did no useful training.
- Evaluation and export. Validation passes and writing out the final weights.
Our planning rule is to add at least 10% to the formula for multi-day runs. Go higher if your checkpoints are far apart, since every restart redoes the work since the last save.
How does Nodus show the time before you launch?
A dry run returns the estimate before anything is charged. For a managed TrainingJob, the training guide says the estimate is the number of steps times the measured seconds per step on the slowest GPU type you allowed, plus evaluation time. You can set spec.expectedDuration to use your own figure.
$ nodus apply -f sft.yaml --dry-run -o yaml # status.estimate: cost to completion, first hold, start time
For your own script, nodus run takes the same idea from flags. This prints the estimate only, using your expected duration:
$ nodus run --dry-run --gpu H100:8 --expected-duration 24h -- torchrun train.py
If the run must finish by a date, --complete-by (or placement.completeByTime) keeps it off any capacity whose p90 finish is later, per our placement docs. H100s are listed at $2.60/hr on the Nodus pricing page, checked October 31, 2026. If a machine is lost mid-run, Nodus starts a new attempt from the latest committed checkpoint, as described in attempts and recovery. Our post on how often spot GPUs are interrupted shows how much that matters on cheaper capacity.
FAQ
How long does it take to train a model like GPT-4?
Nobody outside the lab knows, because the GPU-hours were never published. The closest public figure is Llama 3.1 405B at 30.84 million H100 GPU-hours, which is months even on a cluster of more than 16,000 GPUs.
Can I train a useful model on a single GPU?
Yes, if it is small or a fine-tune. Karpathy notes his GPT-2 124M run would take roughly 4 to 24 hours on a single GPU, depending on the card.
Is a faster GPU always worth it for training time?
Not always. A GPU that costs twice as much per hour but runs the job 2.5 times faster is cheaper overall, and one that runs it 1.5 times faster is not. Compare per-step time on your own model. Nodus managed training runtimes publish measured step times for their presets, which is what their estimates use.
Does training time include data preparation?
No, none of the figures here do. Tokenizing and cleaning a large dataset is usually CPU work that you can finish before you pay for any GPU time.
Originally published on the Nodus blog.
How much does it cost to train an LLM from scratch?
Training an LLM from scratch costs from a few hundred dollars for a small model to tens of millions for a frontier model, and almost all of it is GPU-hours times the hourly rate. At the $2.60/hr Nodus lists for an NVIDIA H100 on its pricing page, a compute-optimal 1B model costs about $292 and a 7B model about $14,300 in compute. The rest of this post shows the formula, checks it against what Meta and DeepSeek actually published, and lists the costs the headline numbers leave out.
How do you estimate what an LLM training run will cost?
You need three numbers: model size, training tokens and how much of the GPU's peak speed you will really get. Training a transformer takes about 6 floating-point operations per parameter per token, the rule from Kaplan et al.'s scaling-laws paper. So a run needs roughly 6 × parameters × tokens operations in total.
An NVIDIA H100 SXM peaks at about 989 teraFLOPS of dense BF16 math. NVIDIA's H100 page lists 1,979 teraFLOPS, but that figure assumes sparsity, so halve it for normal training. No real run gets the full peak. The share you do get is called utilization, and it is the number that moves your budget most.
Put together, the estimate is:
GPU-hours = (6 × parameters × tokens) / (989e12 × utilization × 3,600)
cost = GPU-hours × hourly rate
At 30% utilization, one H100-hour delivers about 1.07 × 10^18 useful operations. Every cost in this post comes from that line and a rate.
How many tokens should a model train on?
A common starting point is about 20 tokens per parameter. DeepMind's Chinchilla paper found that model size and training tokens should grow together for the best model at a fixed compute budget, which works out to roughly that ratio: Chinchilla itself paired 70 billion parameters with 1.4 trillion tokens. A 1B model gets 20 billion tokens and a 7B model gets 140 billion.
Most open models now train far past that point. Meta trained every Llama 3.1 model on about 15 trillion tokens, which is over 1,800 tokens per parameter for the 8B model. Extra tokens cost more to train but give a better small model that is cheaper to serve. That choice can multiply your bill by ten or more, so decide it before you price anything.
What did real LLM training runs actually cost?
Published GPU-hour counts are the best reality check, and they run from about 1.5 million to over 30 million for recent open models. Meta's Llama 3.1 model card lists the H100-80GB hours for each size. DeepSeek's V3 technical report lists 2.788 million H800 GPU-hours for its full training, which it prices at $5.576 million assuming $2 per GPU-hour.
| Model | Published GPU-hours | At $2.60/GPU-hour |
| --- | ---: | ---: |
| Llama 3.1 8B | 1.46M H100 | $3.80M |
| Llama 3.1 70B | 7M H100 | $18.2M |
| Llama 3.1 405B | 30.84M H100 | $80.2M |
| DeepSeek-V3 (37B active) | 2.788M H800 | $7.25M |
GPU-hours from the Llama 3.1 model card and the DeepSeek-V3 report; Nodus H100 list rate checked October 30, 2026. The last column is for scale only: runs this size need thousands of GPUs held for weeks, which is a reserved-capacity contract, not an hourly bill.
!Published GPU-hours: Llama 3.1 8B 1.46 million, DeepSeek-V3 2.788 million, Llama 3.1 70B 7.0 million, Llama 3.1 405B 30.84 million.
DeepSeek-V3 used fewer GPU-hours than Llama 3.1 70B because only 37B of its parameters are active per token.
DeepSeek-V3 is a mixture-of-experts model, so each token only touches 37 billion of its parameters. That is why its compute bill sits between the 8B and 70B dense Llama models even though it is much larger in total.
What utilization should you plan for?
Plan for 30% unless you have measured better on your own code. We back-solved the effective utilization of each published run from its parameters, tokens and GPU-hours, using the formula above. The results were about 14% for Llama 3.1 8B, 25% for Llama 3.1 70B and 33% for Llama 3.1 405B.
The best small-scale result we found is higher. Andrej Karpathy's llm.c reproduction of the 1.6B GPT-2 trained on 33.6 billion tokens in 24 hours on one 8 × H100 node. That is 192 GPU-hours, which back-solves to about 46% utilization, close to the roughly 50% he reports.
!Effective utilization back-solved from published runs: Llama 3.1 8B 14%, Llama 3.1 70B 25%, Llama 3.1 405B 33%, llm.c GPT-2 1.6B 46%.
The same formula on four real runs: plan for 30% and treat 45% as a well-tuned best case.
The gap between running speed and the full bill is the useful part. The Llama 3 paper reports 38% to 43% utilization while the 405B model was training, yet the whole GPU-hour count back-solves to 33%. Over a 54-day stretch of that run, Meta logged 466 job interruptions, 419 of them unexpected. Restarts, redone work and idle time all land on the bill, and that is why we plan at 30%, not at peak.
How much does it cost to train a small LLM from scratch?
A small model is cheap enough to train for learning or for a narrow domain. Here is the formula at 30% utilization and the Nodus H100 list rate of $2.60/hr.
| Model and tokens | Total operations | H100 GPU-hours | Compute cost |
| --- | ---: | ---: | ---: |
| 1B on 20B tokens | 1.2 × 10^20 | 112 | $292 |
| 7B on 140B tokens | 5.9 × 10^21 | 5,505 | $14,313 |
| 7B on 2T tokens | 8.4 × 10^22 | 78,643 | $204,471 |
| 70B on 1.4T tokens | 5.9 × 10^23 | 550,500 | $1.43M |
Our arithmetic from 6 × parameters × tokens, 989 teraFLOPS dense BF16, 30% utilization and $2.60 per H100-hour (checked October 30, 2026).
To check the method against a real bill, take the llm.c GPT-2 run. Karpathy priced its 192 H100-hours at $672, which is $3.50 per GPU-hour. At $2.60, the same 192 GPU-hours cost $499.20. If your code reaches 45% instead of 30%, every row above falls by a third.
What costs do these numbers leave out?
The formula prices one clean run, and real projects are never one clean run. DeepSeek says plainly that its $5.576 million covers only the official training run and excludes prior research and ablation experiments on architecture, algorithms and data. Expect to spend real money on small test runs before the big one.
Four other costs show up on most projects:
- Lost work after an interruption. If an 8-node, 64-GPU run loses 6 hours of progress, that is 384 GPU-hours, or $998.40 at $2.60. With a checkpoint no more than 30 minutes old, the same failure costs 32 GPU-hours, or $83.20.
- Idle GPUs. A node that sits waiting for data, a fix or a person bills the same as one that trains.
- Data preparation and evaluation. Cleaning and tokenizing a large corpus and running benchmarks takes its own compute.
- Storage for checkpoints and datasets, which grows with model size.
The first two are the ones you control with tooling. That is where most of the gap between the formula and the final invoice comes from.
Should you train from scratch at all?
Usually not. Train from scratch only when you need your own architecture or tokenizer, a model with no inherited weights, or research results about pretraining itself. For most teams, continuing to pretrain an open model on your own text, or fine-tuning it, gets a better model for a small fraction of the cost.
Our own continued-pretraining benchmark shows the scale difference. Continuing to train Qwen2.5-0.5B on raw text on one NVIDIA L4 cut held-out perplexity by 21.8% for a recorded charge of $0.121164. That was a small pilot on a tiny model. Still, it shows the order of magnitude: well under a dollar to adapt a small model, against hundreds to train even a 1B model from nothing. If you are fine-tuning instead, the cheapest way to fine-tune Llama on an H100 walks through that cost.
How do you run a from-scratch pretraining job on Nodus?
Nodus runs pretraining as a TrainingJob with task: Pretrain and initialization: Scratch, which keeps only the model's architecture and tokenizer. This manifest follows the training guide, with a spending cap and two 8-GPU nodes added:
apiVersion: nodus.dev/v1beta1
kind: TrainingJob
metadata:
name: scratch-pretrain
spec:
runtime: nodus/sft
task: Pretrain
model: {uri: hf://Qwen/Qwen3-0.6B@c1899de289a04d12100db370d81485cdf75e47ca}
data: {volume: corpus, format: JSONL}
parameters: {steps: 10000, initialization: Scratch, packing: true}
resources: {gpu: {type: [h100-sxm-80g], count: 8}, nodes: 2}
maxCostUSD: "500.00"
Price it before anything runs:
$ nodus apply -f scratch-pretrain.yaml --dry-run -o yaml
$ nodus apply -f scratch-pretrain.yaml
The dry run's estimate is the number of steps times the measured seconds per step on the slowest GPU type you allowed. maxCostUSD caps the whole run, including retries; when it is reached the run is suspended, not failed, and raising the cap resumes it from its checkpoint. If a node is lost, the gang restarts from the latest checkpoint every rank committed, up to three times. Multi-node TrainingJobs are in Beta and go up to 8 nodes, as the multi-node guide explains.
FAQ
How much did it cost to train GPT-4?
OpenAI has not published GPT-4's training compute or cost, so any figure you see is an outside estimate. The best public anchors are open models that publish GPU-hours, such as Llama 3.1 405B at 30.84 million H100-hours.
Can you train an LLM on one GPU?
Yes, for small models. Karpathy notes the 1.6B GPT-2 run would take about 8 days on one GPU instead of 1 day on eight, for the same GPU-hours. Nodus lists prices per GPU-hour, so the GPU-hours, not the GPU count, set the compute cost.
Is a B200 cheaper than an H100 for pretraining?
Only if it finishes the job more than proportionally faster. Nodus lists the NVIDIA B200 at $5.50/hr against $2.60/hr for the H100, so the B200 needs over 2.1 times the H100's throughput on your model to cost less per run. Measure seconds per step on both before you commit.
Does a mixture-of-experts model cost less to train?
Per token, yes, because compute scales with active parameters, not total ones. DeepSeek-V3 has 37B active parameters and needed 2.788 million GPU-hours, fewer than the dense Llama 3.1 70B at 7.0 million. You still need enough GPU memory to hold all the experts.
Originally published on the Nodus blog.
Exploring Other Large Language Models for Improving Kinyarwanda Text Generation
Choosing the Best Open-Source Model for Text Generation
1. Introduction
Text generation is an important task in Natural Language Processing (NLP). It allows computers to generate text from a prompt. For example, when we write “The people of Kicukiro say”, a language model can predict what words should come next.
In our project, we studied GPT-2 and compared it with a tiny GPT model that our group built from scratch. Our tiny GPT was trained on Kinyarwanda news from the KINNEWS dataset.
Our experiments showed that GPT-2 can generate fluent English, but it struggles to continue a sentence correctly in Kinyarwanda. Our tiny GPT generates text that looks more like Kinyarwanda, but its sentences do not always make sense.
GPT-2 has 124 million parameters and was trained mainly on English text. Our tiny GPT has about 0.82 million parameters and was trained on approximately 24 million characters of Kinyarwanda news. This showed us that both model size and training data are important for text generation.
The next step is to find a better language model that can generate useful text and follow instructions. In this blog, I compare four open-weight models and choose the one that is most suitable for improving our project.
2. Models We Considered
Model 1: Qwen3-4B-Instruct-2507
Qwen3-4B-Instruct-2507 was developed by the Qwen team at Alibaba. It has 4 billion parameters and supports a context length of up to 262,144 tokens. It uses the Apache 2.0 licence.
This model is designed to follow instructions, understand text, and generate helpful responses. It also has multilingual capabilities and can handle many different tasks.
One advantage is that it does not require a separate thinking mode to generate its answers. This makes its output easier to use when we want a direct response.
However, it is larger than our original GPT-2 model, so it needs more memory and computing power. We also need to test its Kinyarwanda output instead of assuming that it will always generate correct Kinyarwanda.
Model 2: Qwen3-1.7B
Qwen3-1.7B is another model from Alibaba's Qwen team. It has approximately 1.7 billion parameters and uses the Apache 2.0 licence. Its standard context length is around 32,768 tokens.
Its main advantage is its smaller size compared with Qwen3-4B. This can make it easier to run and faster to test on a computer with limited memory.
It can follow instructions and generate text, making it useful for small NLP applications. However, its smaller size may affect the quality of some responses.
This model would be a good choice if our main concern was reducing memory use. We would still need to compare its Kinyarwanda generation with the output from the larger model.
Model 3: Llama 3.2-3B-Instruct
Llama 3.2-3B-Instruct was developed by Meta. It has approximately 3 billion parameters and a context length of up to 128,000 tokens. It uses the Llama 3.2 Community Licence.
This model is designed for tasks such as text generation, summarisation, and following instructions. It is relatively small compared with many larger language models.
One advantage is that it can generate conversational text and follow user instructions. However, its licence has specific terms, and access to its model files may require accepting Meta's conditions. Its officially supported languages also do not include Kinyarwanda.
This means it is worth considering, but its licence and language limitations make it less convenient for our project.
Model 4: SmolLM3-3B
SmolLM3-3B was developed by Hugging Face and its contributors. It has approximately 3 billion parameters and supports a standard context length of 65,536 tokens, with extensions available for longer contexts. It uses the Apache 2.0 licence.
The model is designed for instruction following, text generation, and reasoning. It is also released with information about its training process, which makes it interesting for learning and research.
Its size makes it a possible option for running on a GPU with limited memory. However, its documented multilingual support focuses on six languages: English, French, Spanish, German, Italian, and Portuguese. Kinyarwanda is not among them.
For our project, we would need to test whether it produces useful Kinyarwanda text before choosing it.
3. Comparison Table
| Model | Developer | Parameters | Context length | Licence | Main strength |
|---|---|---:|---:|---|---|
| Qwen3-4B-Instruct-2507 | Alibaba Qwen team | 4B | 262,144 tokens | Apache 2.0 | Instruction following and multilingual text generation |
| Qwen3-1.7B | Alibaba Qwen team | 1.7B | 32,768 tokens | Apache 2.0 | Smaller size and lower memory needs |
| Llama 3.2-3B-Instruct | Meta | 3B | 128,000 tokens | Llama Community Licence | Conversational text and instruction following |
| SmolLM3-3B | Hugging Face and contributors | 3B | 65,536 tokens by default | Apache 2.0 | Efficient text generation and transparent training details |
Context lengths are the models' documented limits. A longer context does not automatically mean better text generation.
4. The Model Chosen
I would choose Qwen3-4B-Instruct-2507 to improve our text-generation project.
The first reason is its instruction-following ability. We want a model that can receive a prompt and generate a response in the requested format. For example, we could ask it to continue a Kinyarwanda news sentence, produce a short news paragraph, or rewrite text more clearly.
The second reason is its multilingual capability. Our notebook showed that GPT-2 struggles with Kinyarwanda partly because its tokenizer breaks many Kinyarwanda words into smaller pieces. The Kinyarwanda example required about 1.7 times as many tokens as the English example. Qwen3 may handle this task better, but we must test it because multilingual support does not guarantee accurate Kinyarwanda.
The third reason is that it uses the Apache 2.0 licence and its Hugging Face model page provides access to the model files without the gated-access process required by some other models. We can load it directly in a notebook rather than depending on a paid text-generation API.
Can it run on free Google Colab?
Qwen3-4B-Instruct-2507 is a realistic model to test on a free Google Colab T4 GPU with 16 GB of GPU memory. Its 4-billion-parameter size makes local inference more practical than using a much larger model. The exact memory use depends on the precision, context length, and generation settings.
For our first test, we should use a moderate prompt length and generate a limited number of new tokens. If memory becomes a problem, we can try a quantized version.
The model itself does not require a paid API key. However, free Colab GPU access is not guaranteed at all times, and Colab still requires a Google account. We should also check the model's licence terms before using it in a published application.
How will it improve our project?
We can compare Qwen3 with GPT-2 and our tiny GPT using the same Kinyarwanda prompts. For example, we can use the prompt:
"Abaturage bo mu Kicukiro bavuga ko..."
We can then compare the generated text for language quality, sentence meaning, grammar, repetition, and whether it follows the prompt.
We should keep the prompts and generation settings as similar as possible. We can also test English prompts to compare performance across languages.
Our tiny GPT is still valuable because it helps us understand how a transformer works and how training data affects the results. Qwen3 would be an additional model for comparison, not a replacement for the lessons we learned from building our own model.
5. Conclusion
Our experiments showed that generating text is not only about having a large model. The model's training data, tokenizer, and ability to follow instructions also affect the result.
Qwen3-4B-Instruct-2507 is my preferred option because it combines a manageable model size, instruction following, multilingual capabilities, and an Apache 2.0 licence. It is also practical to test on a free Colab T4 GPU, although actual memory use must be checked.
Most importantly, we should not assume that it will automatically produce perfect Kinyarwanda. We need to test it with real Kinyarwanda prompts and compare the results with GPT-2 and our tiny GPT. These tests will help us decide whether it is a useful improvement for our project.
6. References
1. OpenAI. Language Models are Unsupervised Multitask Learners (2019). openai.com
2. Qwen Team. Qwen3-4B-Instruct-2507 model card. Hugging Face. huggingface.co
3. Qwen Team. Qwen3-1.7B model card. Hugging Face. huggingface.co
4. Meta. Llama 3.2 model card and licence information. github.com/meta-llama/llama-models/blob/main/mode…
5. Hugging Face. SmolLM3-3B model card. huggingface.co
6. Hugging Face. Transformers documentation. huggingface.co
7. Vaswani, A. et al. Attention Is All You Need (2017). arxiv.org
Beyond GPT-2: Choosing a Modern Model for Our Poem Generator
Blog assignment: Introduction to Large Language Models (LLMs)
Introduction
In our Natural Language Processing (NLP) course, we developed a Poem Generator using Transformer models. Our project explores three stages: building a small GPT-style model (Tiny GPT) from scratch, experimenting with the pretrained GPT-2 model, and comparing their generated poems.
Although these models helped us understand how language generation works, we wanted to investigate whether modern Large Language Models (LLMs) could produce better poems. In this blog, we compare six models, examine their strengths and limitations, and select one model that fits our project and available computing resources.
1. Limitations of Our Current Project
Tiny GPT learns from a small poetry dataset. Consequently, it may learn common poetic patterns without fully understanding the relationships between ideas. GPT-2 was pretrained on a much larger text collection, making it more capable of generating fluent text. However, its original context window is 1,024 tokens, and its base version is not specifically trained to follow detailed instructions.
For example, asking GPT-2 to write four rhyming lines about rain at night does not guarantee that it will produce exactly four lines or maintain a consistent rhyme scheme.
We therefore need a model that follows instructions, generates coherent text, respects requested poem structures, and runs on hardware available to our team.
2. What We Considered When Selecting a Model
We used five criteria to compare candidate models:
1. Instruction following: Can the model follow requests about topic, length, tone, and rhyme?
2. Computing requirements: Can it run on a free Google Colab T4 GPU with 16 GB of GPU memory?
3. Licence and access: Can we download and run it without paying for an API or supplying an API key? What licence conditions apply?
4. Generation quality: Can it produce relevant, creative, coherent, and grammatically correct poems?
5. Compatibility: Does it work with Hugging Face Transformers, allowing us to reuse our existing Python workflow?
3. Comparison of Open-Weight LLMs
We researched six models from different developers. Their parameter counts, context windows, licences, and intended strengths are summarized below.
| Model | Developer | Parameters | Context Window | Licence |
|---|---|---|---|---|
| GPT-2 Medium / Large | OpenAI | 355M / 774M | 1,024 tokens | MIT |
| Gemma 3 4B Instruct | Google | 4B | 128K tokens | Gemma Terms of Use |
| Llama 3.2 3B Instruct | Meta | 3B | 128K tokens | Llama Community Licence |
| Phi-4 Mini Instruct | Microsoft | 3.8B | 128K tokens | MIT |
| SmolLM3-3B | Hugging Face | 3B | Up to 128K tokens | Apache 2.0 |
| Qwen3-4B-Instruct-2507 | Qwen team, Alibaba | 4B | 262,144 tokens | Apache 2.0 |
Note: Context limits can depend on the model variant and software configuration. Open-weight does not mean every model has an unrestricted open-source licence.
GPT-2
GPT-2, developed by OpenAI, is a useful baseline for our project because it demonstrates how a Transformer predicts the next token to generate text. Its relatively short context window and lack of instruction tuning in the original base model make it less convenient for prompts with several constraints. It remains valuable for comparing modern models against an established text-generation approach.
Gemma 3 4B Instruct
Gemma 3 is developed by Google. Its 4-billion-parameter instruction-tuned version supports a 128K-token context window and is designed for tasks including text generation, reasoning, and summarization. It is a strong candidate for creative writing, but its Gemma-specific terms differ from standard permissive licences.
Llama 3.2 3B Instruct
Llama 3.2 is developed by Meta. Its 3-billion-parameter text model is designed for dialogue and other language tasks. Its relatively small size makes it attractive for resource-constrained environments. However, its Community Licence and acceptable-use requirements must be considered, and its poetry performance would need to be tested.
Phi-4 Mini Instruct
Phi-4 Mini Instruct is developed by Microsoft and contains approximately 3.8 billion parameters. It supports a 128K-token context window and is particularly focused on reasoning, mathematics, coding, and instruction following. Although these capabilities are useful, we would need experimental evidence to establish whether it offers an advantage for poetry.
SmolLM3-3B
SmolLM3-3B is developed by Hugging Face. It is a compact instruction-tuned model with support for long-context tasks and multiple languages. Its Apache 2.0 licence and publicly documented training approach make it attractive for experimentation. Nevertheless, we cannot assume that it generates better poems without testing it on our prompts.
Qwen3-4B-Instruct-2507
Qwen3-4B-Instruct-2507 is developed by the Qwen team at Alibaba. It contains approximately four billion parameters and supports a native context window of 262,144 tokens. Its model card reports capabilities in instruction following, text comprehension, reasoning, and open-ended text generation. These features make it a promising candidate for our project.
4. Our Chosen Model: Qwen3-4B-Instruct-2507
We selected Qwen3-4B-Instruct-2507 because it offers a useful balance between instruction following, generation capabilities, computing requirements, and licence flexibility.
First, it follows detailed instructions. We can prompt it to write a poem with a specific topic, number of lines, tone, and rhyme scheme. Its instruction-tuned design makes it a better fit for this task than an unmodified GPT-2 base model, although compliance must still be measured.
Second, it is practical to experiment with. The model has approximately four billion parameters. Its published BF16 weights occupy about 8 GB, so loading it on a 16 GB Colab T4 may be feasible. However, model weights are not the only memory requirement: the attention cache, input length, generated tokens, and software overhead also consume GPU memory. We will begin with short prompts and modest generation lengths, monitor GPU memory, and reduce the workload or use quantization if necessary.
Third, its licence supports our research. Qwen3-4B-Instruct-2507 uses Apache 2.0, which permits use and modification subject to its conditions. We can download the model weights and run local inference through supported software without needing a paid inference API or API key. We must still follow the licence and any applicable platform requirements.
Fourth, it fits our existing workflow. The model supports Hugging Face Transformers and uses a chat template for structured prompts. This allows us to build on our existing Python knowledge and keep the experiment manageable.
Finally, its long context window provides room for larger prompts and additional instructions. Our current poems do not require 262K tokens, but the capability is useful if we later experiment with longer creative-writing tasks.
5. How We Will Evaluate the Models
To make our comparison fairer, we will use the same set of poem prompts for Tiny GPT, GPT-2, and Qwen3. We will keep the requested topics, poem lengths, and evaluation criteria consistent. Where possible, we will use comparable generation settings, while respecting each model's recommended configuration.
We will evaluate each output using six criteria:
- Relevance: Does the poem stay on the requested topic?
- Structure: Does it follow the requested number of lines or stanzas?
- Rhyme: Does it follow the requested rhyme scheme?
- Coherence: Do the lines connect logically?
- Grammar: Is the language understandable and grammatically correct?
- Creativity: Does the poem use interesting descriptions and original ideas?
We can use a simple scoring rubric, such as 1–5 for each criterion, and ask human reviewers to assess the results. We will report the actual scores after running the experiment rather than inventing results in advance. We will also record generation time and peak GPU memory usage to understand the practical cost of each model.
6. Limitations and Future Improvements
Our comparison has several limitations. First, Tiny GPT, GPT-2, and Qwen3 differ in size, training data, and training methods. Therefore, differences in their output cannot be attributed to model architecture alone.
Second, even an instruction-tuned model can ignore constraints, repeat phrases, or generate awkward rhymes. Fluency does not guarantee creativity or quality. Third, the models may reproduce biases or familiar patterns learned during training.
Finally, Qwen3 requires more computing resources than Tiny GPT. If we want it to imitate the specific style of our poetry dataset, we could explore parameter-efficient fine-tuning, such as LoRA, if sufficient training data and hardware are available.
Conclusion
Our experiments with Tiny GPT and GPT-2 introduced us to Transformer-based text generation. Researching modern LLMs showed us that instruction tuning, training methods, model size, and context length can make a difference to how well a model responds to creative-writing requests.
We chose Qwen3-4B-Instruct-2507 because it combines instruction-following capabilities, a permissive licence, compatibility with our Python workflow, and a model size that may fit our Colab environment with careful memory management.
Our next step is to test it against Tiny GPT and GPT-2 using identical prompts and transparent evaluation criteria. This will help us determine whether the modern model actually improves our Poem Generator rather than relying on assumptions.
References
1. OpenAI. GPT-2. openai/gpt-2
2. Google. Gemma 3 Model Card. ai.google.dev
3. Meta. Llama 3.2 Model Card. github.com/meta-llama/llama-models/blob/main/mode…
4. Microsoft. Phi-4 Mini Instruct. huggingface.co
5. Hugging Face. SmolLM3-3B. huggingface.co
6. Qwen Team. Qwen3-4B-Instruct-2507 Model Card. huggingface.co
7. Vaswani, A., et al. (2017). Attention Is All You Need. arxiv.org
@nikks_techie example: Llama 3.1 70B, bf16 KV, no prefix sharing:
320 KiB/token × 8,192 tokens × 100 active sequences = 250 GiB of cache alone.
70B isn't enough to size it. the model's KV heads and the cache dtype matter.
github.com/meta-llama/llama-models/blob/main/mode…
2 views
I HAVE JUST FOUND SOMETHING INSANE
Meta's official Llama repo has a default sample image
it's a dog onboard
github.com/meta-llama/llama-models/blob/main/mode…
Llama 4's multimodal chat script loads this dog every time a dev tests vision. Meta's AI has a mascot and they never told anyone
Also pumpfun is spending millions on onboarding. In callout rewards, updates, KOLs and now referrals.
never been run wtf??
So genny
onboarddog - $onboard
407 views
I HAVE JUST FOUND SOMETHING INSANE
Meta's official Llama repo has a default sample image
it's a dog onboard
github.com/meta-llama/llama-models/blob/main/mode…
Llama 4's multimodal chat script loads this dog every time a dev tests vision. Meta's AI has a mascot and they never told anyone
Also pumpfun is spending millions on onboarding. In callout rewards, updates, KOLs and now referrals.
never been run wtf??
So genny
onboarddog
2 likes · 83 views
A Xiaomi treinou uma IA gigante ao vivo, com o custo na tela #shorts
O MiMo V2.6 foi treinado por reforço com painel público e custo aberto: US$ 2,6 milhões, ao vivo. DeepSeek, Meta, BLOOM, OLMo e SmolLM3 já tinham aberto partes do treino. O que é inédito aqui, e o furo que a transparência revelou.
Painel ao vivo do treino: mimo.xiaomi.com
Post oficial MiMo-V2.6: mimo.mi.com
Relatório técnico: arxiv.org
Framework de treino (verl): XiaomiMiMo/verl
Ambientes de RL: huggingface.co
Auditoria Vals AI: vals.ai
Comparação:
DeepSeek-V3 (custo): arxiv.org
DeepSeek-R1 na Nature: nature.com
Llama 3.1 (horas de GPU): github.com/meta-llama/llama-models/blob/main/mode…
BLOOM (logs ao vivo, 2022): huggingface.co
OLMo 2: allenai.org
SmolLM3 (logs e checkpoints): huggingface.co
Ideia: vídeo de bycloud.
#opensource #programacao #ia
30 views
Mentions
GSM8K on Hugging Face: 4 of the Top 12 Rows Show a Sibling Checkpoint's Score, and 20 of 28 Rows Are Unmerged PRs TL;DR - On November 3, 2026 the Hugging Face GSM8K board lists 28 rows from 28 repos. 20 come from pull requests that are still open, 8 sit on the repo's main branch, and 0 are verified. - Four of the top 12 rows show a number that the model card reports for a different checkpoint of the same family. Rank 1 (99.6) is the MiMo-V2.5-Pro Base score, registered on the post-trained repo. Rank 2 (96.8) is the Llama 3.1 405B Instruct score, registered on the base repo. Ranks 7 and 12 are DeepSeek base scores on the post-trained repos. - The numbers are real self-reports. The labels are what is off, and the board has no column that would show it. - GSM8K has 1,319 test problems, so one problem is 0.076 points. The rank 2 to rank 3 gap (0.74) is 10 problems, and a 95% interval at 95% accuracy is about 1.2 points wide on each side. - A score of 99.6 means about 5 misses. The GSM8K-Platinum project relabeled 10 answers in the same test set. If those relabels are right, a score that high implies the model matched at least 5 of the 10 keys they call wrong. !Top 12 rows of the GSM8K board on Hugging Face, with the four rows whose number belongs to a sibling checkpoint in red In the chart, red marks rows where the model card gives this number for a sibling checkpoint, blue marks a row whose card table matches the repo (LLaDA2.0-flash), and grey marks rows I did not check against the card. What does the GSM8K board on Hugging Face contain? GSM8K is a set of grade-school math word problems with a train split of 7,473 and a test split of 1,319. Hugging Face lists openai/gsm8k as a benchmark dataset, so a model repo can publish a result in a .eval_results/*.yaml file and the dataset's board collects it. I pulled the board on November 3, 2026 (KST) from the JSON behind the leaderboard view. It has 28 rows. Each row is from a different repo, verified is false on every row, and the top of the page reads like a clean ranking: 99.6, 96.8, 96.06, 94.16, 93.93. The dataset's own eval.yaml describes a specific protocol: an inspect-ai task, a zero-shot prompt template that asks for a boxed final answer, four epochs reduced to pass@1, and a model-graded scorer. The rows do not follow one protocol. Of the 28 notes, 15 are empty. Where a note gives a shot count it ranges across 0, 4, 5 and 8. Two rows from one lab give a complete setup (8-shot, temperature 0, n=1319, lm-evaluation-harness), and one row reports a strict-match score over 1,075 of the 1,319 problems. So I treat each row as a claim with a source link, not as a measurement the board produced. Why does rank 1 belong to a different checkpoint? The top row is XiaomiMiMo/MiMo-V2.5-Pro at 99.6. Its model card has a section titled "Base Model Evaluation", and the GSM8K row there reads 8-shot with these values: 99.6 for MiMo-V2.5-Pro Base, 83.3 for MiMo-V2.5 Base, 92.6 for DeepSeek-V4-Pro Base. That is the only place GSM8K appears in the card. The card's own download table lists two repos: MiMo-V2.5-Pro and MiMo-V2.5-Pro-Base, and the second one exists on the Hub. So the board shows the base-model score on the post-trained repo. The pull request that registered it says the results "were extracted from the model card". It was opened on April 27, 2026 and is still open. Nothing in the YAML says "base", and the 8-shot setting is not in the note, which is empty. The card is not hiding anything. It labels its table as Base. The distinction is lost only in the step that turns a card into a board row. Is that one slip, or does it repeat? I checked the sources for the top rows. It repeats in four of the top 12, and in both directions. | Board rank | Repo on the board | Score | Where the card puts this number | Sibling repo | |---|---|---|---|---| | 1 | MiMo-V2.5-Pro | 99.6 | "Base Model Evaluation" table, Pro Base column, 8-shot | MiMo-V2.5-Pro-Base | | 2 | Llama-3.1-405B | 96.8 | "Instruction tuned models" table, 405B Instruct column, 8-shot CoT, em_maj1@1 | Llama-3.1-405B-Instruct | | 7 | DeepSeek-V4-Pro | 92.6 | Base table, DeepSeek-V4-Pro-Base column, 8-shot | DeepSeek-V4-Pro-Base | | 12 | DeepSeek-V3 | 89.3 | "Base Model" section, DeepSeek-V3 column, 8-shot | DeepSeek-V3-Base | Rank 2 is the mirror image. The Llama 3.1 model card covers every size and both variants. Its "Base pretrained models" table has no GSM8K row. Its "Instruction tuned models" table has one, and the 405B Instruct value is 96.8. The pull request is titled "Add GSM8K eval result (96.8, 8-shot CoT)" and attaches that number to the base repo meta-llama/Llama-3.1-405B. For DeepSeek-V3 the card has a Base Model section with GSM8K at 89.3 and a Chat Model section with no GSM8K row. For DeepSeek-V4-Pro the table header names the three columns DeepSeek-V3.2-Base, DeepSeek-V4-Flash-Base and DeepSeek-V4-Pro-Base, and 92.6 is the last one. Why does it matter if the number is the same? Because the board is read by repo name. Someone choosing a model for math word problems sees MiMo-V2.5-Pro at the top and downloads that repo. The card reports no GSM8K number for that checkpoint. If the four rows were moved to the sibling repos, the post-trained MiMo, DeepSeek-V4-Pro and DeepSeek-V3 would have no GSM8K row at all, and the Llama 405B base repo would lose its rank 2. The correct placement would not change the ordering of the numbers: 99.6 would still lead, from the right repo. It would change which model each number tells you about. !Where the 28 GSM8K rows come from: open pull requests, main branch, verified Where do the rows come from, and what does "verified" mean here? I fetched the discussion for every row that carries a pullRequest number, and listed .eval_results on the main branch of every repo. | Source | Rows | |---|---| | Open PR from an account flagged as Hugging Face (julien-c 9, nielsr 6, SaylorTwift 3, burtenshaw 1) | 19 | | Open PR from a bot account (EvalEvalBot) | 1 | | File on the repo's main branch | 8 | | Verified | 0 | All 20 pull requests are open, and for all 20 the YAML file is absent from main. For all 8 other rows it is present. Four of the 20 pull requests were opened on November 2, one day before I pulled the board, and one of them produced rank 3 (LLaDA2.0-flash, 96.06). The board moves when someone opens a pull request, with no action from the model's owner. The Hugging Face documentation defines the badges. "Community" means a result submitted via an open PR that is not merged to main. "Verified" means a valid verify token, which requires the evaluation to have run in HF Jobs with inspect-ai. So 20 of 28 rows are community results by definition, and no row meets the verified bar. The documentation says community scores are visible while the PR is open. That is a reasonable design for bootstrapping a board. It also means a reader has to look at the badge to know which rows the model's owner stands behind, and a screenshot of the ranking does not show it. The eight main-branch rows are not uniformly clean either. Two Mellum2 repos show the same 81.73. The YAML on the Base repo states that its numbers were measured on the pre-extension Base-Pretrain checkpoint, so it is one measurement displayed twice. This one is disclosed in the file, and it still occupies two ranks. How much can 1,319 problems separate? GSM8K is nearly saturated at the top, so the ordering is decided by a handful of problems. | Quantity | Value | |---|---| | One problem | 0.076 points | | Rank 1 to rank 2 gap (99.6 to 96.8) | 2.8 points, about 37 problems | | Rank 2 to rank 3 gap (96.8 to 96.06) | 0.74 points, about 10 problems | | Spread of ranks 3 to 6 (96.06 to 93.10) | 2.96 points | | 95% half-width at 95% accuracy, n=1,319 | 1.18 points (illustrative) | | 95% half-width at 99% accuracy, n=1,319 | 0.54 points (illustrative) | The interval rows are a sampling calculation for a single run on a fixed test set. They ignore prompt variation, which is larger when rows use different shot counts. Even so, ranks 3 to 6 sit inside a band that a single run cannot separate, and the rows within it come from different protocols: 8-shot CoT for some, no note for others. One other check is exact arithmetic. The two rows with a full setup report 84.9886 and 78.4685, and with n=1319 these are 1,121 and 1,035 correct answers. ProKope-7B's note says 1,075 of 1,319, which is 81.50. Those rows can be audited back to a count. Most rows cannot. !How big is each effect: one problem, intervals, ceiling lost to relabeled keys, gaps between ranks Can a 99.6 be right if some answer keys are wrong? This part is arithmetic with one assumption, so I mark it as a question to check, not a finding. The GSM8K-Platinum project (MIT, Gradient Science) reviewed the 1,319 test problems. Its dataset card reports 219 flagged by models, 110 rejected as poorly written, 10 relabeled, 99 verified, leaving 1,209 questions. The 10 relabeled items are answer keys that the authors found to be wrong. Take a score of 99.6 on the original test set. 1,314 of 1,319 rounds to 99.6, so that is about 5 misses. Suppose the 10 relabels are correct and scoring is exact match against the original key. A model that solves an item correctly then loses it, because the key is wrong. A model that is right on all 10 would lose all 10 and top out at 99.24. To score 99.6 it must have matched the key on at least 5 of the 10, which means it gave the answer the key says, not the answer the authors say is right. Three explanations fit: the relabels are wrong for some of those items, the scoring is more lenient than exact match, or the model has seen the original keys. I have not run the model, and a base model's 8-shot number with no note on answer extraction does not let me tell these apart. What I can say is that for a benchmark with known key errors, a score above the ceiling is a reason to read the setup, not a reason to rank it first. Platinum also reports that cleaning the set separates models that look tied on the original, which is the same point from the other side. How do I pull the board and classify rows myself? This script needs only the standard library. It counts rows by source and prints the sampling arithmetic used above. import json, urllib.request, collections def get(url): req = urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0"}) return json.load(urllib.request.urlopen(req, timeout=60)) DATASET = "openai/gsm8k" rows = get(f"https://huggingface.co/api/datasets/{DATASET}/leaderboard") by_source = collections.Counter() for r in rows: pr = r.get("pullRequest") if pr: disc = get(f"https://huggingface.co/api/models/{r['modelId']}/discussions/{pr}") by_source[("open PR" if disc["status"] == "open" else disc["status"])] += 1 else: by_source["main branch"] += 1 print("rows:", len(rows), "| verified:", sum(bool(r["verified"]) for r in rows)) print(dict(by_source)) n = 1319 print("one problem = %.3f points" % (100 / n)) for acc in (0.95, 0.99): se = (acc * (1 - acc) / n) ** 0.5 * 100 print("accuracy %.2f: 95%% half-width = %.2f points" % (acc, 1.96 * se)) print("ceiling if 10 keys are wrong: %.2f" % (100 * (n - 10) / n)) On November 3, 2026 it printed: rows: 28 | verified: 0 {'open PR': 20, 'main branch': 8} one problem = 0.076 points accuracy 0.95: 95% half-width = 1.18 points accuracy 0.99: 95% half-width = 0.54 points ceiling if 10 keys are wrong: 99.24 The board endpoint is undocumented, so the field names may change. To check a row, open its pullRequest discussion, read the YAML, then open the source card and find the table the number came from. Checklist for reading a saturated board 1. Open the badge before the rank. Community means an open PR, and verified means a run in HF Jobs. Neither is a statement from the model's owner. 2. Match the checkpoint, not the family. Find the card table the number came from and read its column header. Base, instruct and chat checkpoints sit in different tables. 3. Ask how many problems separate two rows. On GSM8K, 10 problems is 0.76 points. 4. Prefer rows that state their setup. Shot count, temperature, n and the answer-extraction rule decide the score more than rank does. 5. Check for the same measurement on two repos. A copied number is one result, not two. 6. Distrust a score above the key ceiling. If a benchmark has known label errors, read the evaluation setup before you trust the lead. FAQ How many GSM8K results on Hugging Face are verified? None. On November 3, 2026 all 28 rows on the board show verified as false. Verified requires a valid verify token from an inspect-ai run in HF Jobs. Is the 99.6 at rank 1 of the GSM8K board wrong? Nothing here shows the number is wrong. It is the MiMo-V2.5-Pro Base score from the model card's Base Model table, 8-shot. The problem is that it is registered on the post-trained repo, which has its own separate row in the card's download table. What does "community" mean on a Hugging Face leaderboard row? It means the result was submitted through an open pull request that has not been merged to the repo's main branch. In the GSM8K board, 20 of 28 rows are in that state. Why is the Llama 3.1 405B base repo ranked second on GSM8K? Its row carries 96.8, which the shared model card gives for Llama 3.1 405B Instruct, 8-shot CoT. The card has no GSM8K row in its base table. The instruct repo is a separate repo. How many problems does one point of GSM8K accuracy represent? GSM8K has 1,319 test problems, so one point is about 13 problems and one problem is 0.076 points. What is GSM8K-Platinum? It is a revised version of the GSM8K test set from Gradient Science. Of 1,319 test problems it rejected 110, relabeled 10 and kept 1,209. It exists because the original set has ambiguous questions and wrong keys. Further reading - GPQA Diamond Has 198 Questions: What the Hugging Face Leaderboard's 115 Rows Can and Cannot Rank - How Hugging Face Official Benchmark Leaderboards Actually Work: .eval_results YAML, the base_model Filter, and the 30% the Default View Hides Sources - GSM8K dataset and board: huggingface.co - GSM8K evaluation definition (eval.yaml): huggingface.co - Hugging Face documentation, Evaluation Results (badges, community PRs): huggingface.co - GSM8K-Platinum dataset card: huggingface.co - GSM8K-Platinum write-up: gradientscience.org - Rank 1 model card (Base Model Evaluation table): huggingface.co - Rank 1 registering pull request: huggingface.co - Llama 3.1 model card (base and instruction tuned tables): github.com/meta-llama/llama-models/blob/main/mode… - Rank 2 registering pull request: huggingface.co - DeepSeek-V4-Pro model card: huggingface.co - DeepSeek-V3 model card: huggingface.co All counts in this article are my own measurements from the board JSON, pull request discussions, dataset cards and the cited model cards on November 3, 2026 (KST); I have no affiliation with any model, lab or benchmark named here. Which table a number came from is my reading of each card, and the board endpoint is undocumented and may change.
How long does it take to train an AI model on cloud GPUs? Training an AI model takes anywhere from a few minutes to several months, depending on model size, how many tokens it trains on and how many GPUs run in parallel. On Nodus a small fine-tune can finish in under five minutes, while by our math a 7B language model trained from scratch on 140 billion tokens takes about 3.6 days on 64 NVIDIA H100s. The sections below show the formula, check it against published runs, and list what makes the real clock run longer than the math. How do you estimate how long a training run will take? Work out the total math the run needs, then divide by how fast your GPUs actually do math. Training a transformer takes about 6 floating-point operations per parameter per token, the rule from Kaplan et al.'s scaling-laws paper. An NVIDIA H100 SXM peaks at about 989 teraFLOPS of dense BF16 math, half the sparse figure on NVIDIA's H100 page. No real run gets the full peak. The share you get is called model FLOPs utilization, and 30% is a fair planning number for code you have not tuned. That gives: GPU-hours = (6 × parameters × tokens) / (989e12 × utilization × 3,600) wall-clock = GPU-hours / number of GPUs (if scaling were perfect) At 30% utilization, one H100-hour delivers about 1.07 × 10^18 useful operations. The second line is the optimistic part, and we come back to it below. How long does it take to train a language model from scratch? Days to weeks for small and mid-size models, if you have enough GPUs. Here is the formula at 30% utilization, using a compute-optimal budget of about 20 tokens per parameter from DeepMind's Chinchilla paper, plus one longer 7B run. | Model and tokens | H100 GPU-hours | Fewer GPUs | More GPUs | | --- | ---: | ---: | ---: | | 1B on 20B tokens | 112 | 1 GPU: 4.7 days | 8 GPUs: 14 hours | | 7B on 140B tokens | 5,505 | 8 GPUs: 28.7 days | 64 GPUs: 3.6 days | | 7B on 2T tokens | 78,643 | 64 GPUs: 51.2 days | 512 GPUs: 6.4 days | | 70B on 1.4T tokens | 550,500 | 512 GPUs: 44.8 days | 2,048 GPUs: 11.2 days | Our arithmetic: 6 × parameters × tokens, 989 teraFLOPS dense BF16, 30% utilization, perfect scaling. Real runs land later; see the overheads section. Token count moves the clock as much as model size. The same 7B model takes 14 times longer on 2 trillion tokens than on 140 billion. Many open models now train far past the Chinchilla ratio, so settle the token budget first. Our post on what it costs to train an LLM prices the same GPU-hours in dollars. How long do real published training runs take? Small reproductions take hours, and frontier models take months. Andrej Karpathy's llm.c reproduction of GPT-2 124M trained on 10 billion tokens in about 90 minutes on one 8 × NVIDIA A100 80GB node. His GPT-2 1.6B run took about 24 hours on one 8 × H100 node: 32,000 steps of roughly 1 million tokens each, at about 2.75 seconds per step. That 1.6B run is a good check on the formula. It works out to 192 GPU-hours for about 33.6 billion tokens, which back-solves to roughly 46% utilization. Karpathy's own logs show 42% to 47% at the start, so well-tuned code beats our 30% planning figure by about half. At the top end, Meta's Llama 3.1 model card lists 30.84 million H100 GPU-hours for the 405B model. The Llama 3 paper says it trained on up to 16,384 H100s. Divide one by the other and even that cluster needs about 78 days of pure compute, a floor on the real calendar time. Do more GPUs make training proportionally faster? Only roughly, and never for free. Adding GPUs splits the same GPU-hours across more machines, so the bill stays about the same while the calendar shrinks. In practice the bill creeps up, because GPUs spend more time talking to each other. Meta's own numbers show the effect. The Llama 3 paper reports 43% utilization on 8,192 GPUs and 41% on 16,384 GPUs for the same model, a small but real loss from doubling the cluster. !Days to train a 7B model on 2 trillion tokens: 409.6 days on 8 H100s, 51.2 days on 64, 6.4 days on 512, at 30% utilization and perfect scaling. The same 78,643 GPU-hours: one 8-GPU node needs 409.6 days, a 512-GPU cluster about a week. Our decision rule: pick the smallest GPU count that finishes before your deadline, not the largest you can get. Single nodes of 8 GPUs scale best because the GPUs share fast NVLink links. Multi-node runs add network cost and more machines that can fail. How long does fine-tuning take? Usually minutes to a few hours, because fine-tuning sees far fewer tokens than pretraining. Fine-tuning a 7B model on 50 million tokens is about 2 H100-hours of math at 30% utilization by the same formula, and LoRA does less than that because it skips most weight gradients. For small jobs, the work around training often takes longer than training itself. In our published fine-tuning benchmark, Qwen2.5-0.5B on one NVIDIA L4 spent 112.8 seconds in the training loop on Nodus. From submission to a verified exported model took 4.8 minutes, or 288 seconds. !Small fine-tune on Nodus: 112.8 seconds in the training loop versus 288 seconds from submission to exported model. For a small fine-tune, startup, evaluation and export took longer than the training loop. So for fine-tuning, the honest answer to "how long" is mostly startup, model download, evaluation and export. On Nodus, a TrainingJob imports a pinned Hugging Face model revision into a read-only Volume once, and later runs reuse it, so billed GPU time is not spent downloading weights. Why does the real clock run longer than the math? Because the formula counts only useful training steps. Four things add time on most projects: - Waiting for capacity. A large multi-GPU request may queue until enough machines are free together. - Startup. Pulling the image, loading the model and reading the first batch of data. - Failures and restarts. Over a 54-day stretch of Llama 3 405B pretraining, Meta logged 466 job interruptions, 419 of them unexpected. It still reported more than 90% effective training time, meaning at least a tenth of the elapsed time did no useful training. - Evaluation and export. Validation passes and writing out the final weights. Our planning rule is to add at least 10% to the formula for multi-day runs. Go higher if your checkpoints are far apart, since every restart redoes the work since the last save. How does Nodus show the time before you launch? A dry run returns the estimate before anything is charged. For a managed TrainingJob, the training guide says the estimate is the number of steps times the measured seconds per step on the slowest GPU type you allowed, plus evaluation time. You can set spec.expectedDuration to use your own figure. $ nodus apply -f sft.yaml --dry-run -o yaml # status.estimate: cost to completion, first hold, start time For your own script, nodus run takes the same idea from flags. This prints the estimate only, using your expected duration: $ nodus run --dry-run --gpu H100:8 --expected-duration 24h -- torchrun train.py If the run must finish by a date, --complete-by (or placement.completeByTime) keeps it off any capacity whose p90 finish is later, per our placement docs. H100s are listed at $2.60/hr on the Nodus pricing page, checked October 31, 2026. If a machine is lost mid-run, Nodus starts a new attempt from the latest committed checkpoint, as described in attempts and recovery. Our post on how often spot GPUs are interrupted shows how much that matters on cheaper capacity. FAQ How long does it take to train a model like GPT-4? Nobody outside the lab knows, because the GPU-hours were never published. The closest public figure is Llama 3.1 405B at 30.84 million H100 GPU-hours, which is months even on a cluster of more than 16,000 GPUs. Can I train a useful model on a single GPU? Yes, if it is small or a fine-tune. Karpathy notes his GPT-2 124M run would take roughly 4 to 24 hours on a single GPU, depending on the card. Is a faster GPU always worth it for training time? Not always. A GPU that costs twice as much per hour but runs the job 2.5 times faster is cheaper overall, and one that runs it 1.5 times faster is not. Compare per-step time on your own model. Nodus managed training runtimes publish measured step times for their presets, which is what their estimates use. Does training time include data preparation? No, none of the figures here do. Tokenizing and cleaning a large dataset is usually CPU work that you can finish before you pay for any GPU time. Originally published on the Nodus blog.
How much does it cost to train an LLM from scratch? Training an LLM from scratch costs from a few hundred dollars for a small model to tens of millions for a frontier model, and almost all of it is GPU-hours times the hourly rate. At the $2.60/hr Nodus lists for an NVIDIA H100 on its pricing page, a compute-optimal 1B model costs about $292 and a 7B model about $14,300 in compute. The rest of this post shows the formula, checks it against what Meta and DeepSeek actually published, and lists the costs the headline numbers leave out. How do you estimate what an LLM training run will cost? You need three numbers: model size, training tokens and how much of the GPU's peak speed you will really get. Training a transformer takes about 6 floating-point operations per parameter per token, the rule from Kaplan et al.'s scaling-laws paper. So a run needs roughly 6 × parameters × tokens operations in total. An NVIDIA H100 SXM peaks at about 989 teraFLOPS of dense BF16 math. NVIDIA's H100 page lists 1,979 teraFLOPS, but that figure assumes sparsity, so halve it for normal training. No real run gets the full peak. The share you do get is called utilization, and it is the number that moves your budget most. Put together, the estimate is: GPU-hours = (6 × parameters × tokens) / (989e12 × utilization × 3,600) cost = GPU-hours × hourly rate At 30% utilization, one H100-hour delivers about 1.07 × 10^18 useful operations. Every cost in this post comes from that line and a rate. How many tokens should a model train on? A common starting point is about 20 tokens per parameter. DeepMind's Chinchilla paper found that model size and training tokens should grow together for the best model at a fixed compute budget, which works out to roughly that ratio: Chinchilla itself paired 70 billion parameters with 1.4 trillion tokens. A 1B model gets 20 billion tokens and a 7B model gets 140 billion. Most open models now train far past that point. Meta trained every Llama 3.1 model on about 15 trillion tokens, which is over 1,800 tokens per parameter for the 8B model. Extra tokens cost more to train but give a better small model that is cheaper to serve. That choice can multiply your bill by ten or more, so decide it before you price anything. What did real LLM training runs actually cost? Published GPU-hour counts are the best reality check, and they run from about 1.5 million to over 30 million for recent open models. Meta's Llama 3.1 model card lists the H100-80GB hours for each size. DeepSeek's V3 technical report lists 2.788 million H800 GPU-hours for its full training, which it prices at $5.576 million assuming $2 per GPU-hour. | Model | Published GPU-hours | At $2.60/GPU-hour | | --- | ---: | ---: | | Llama 3.1 8B | 1.46M H100 | $3.80M | | Llama 3.1 70B | 7M H100 | $18.2M | | Llama 3.1 405B | 30.84M H100 | $80.2M | | DeepSeek-V3 (37B active) | 2.788M H800 | $7.25M | GPU-hours from the Llama 3.1 model card and the DeepSeek-V3 report; Nodus H100 list rate checked October 30, 2026. The last column is for scale only: runs this size need thousands of GPUs held for weeks, which is a reserved-capacity contract, not an hourly bill. !Published GPU-hours: Llama 3.1 8B 1.46 million, DeepSeek-V3 2.788 million, Llama 3.1 70B 7.0 million, Llama 3.1 405B 30.84 million. DeepSeek-V3 used fewer GPU-hours than Llama 3.1 70B because only 37B of its parameters are active per token. DeepSeek-V3 is a mixture-of-experts model, so each token only touches 37 billion of its parameters. That is why its compute bill sits between the 8B and 70B dense Llama models even though it is much larger in total. What utilization should you plan for? Plan for 30% unless you have measured better on your own code. We back-solved the effective utilization of each published run from its parameters, tokens and GPU-hours, using the formula above. The results were about 14% for Llama 3.1 8B, 25% for Llama 3.1 70B and 33% for Llama 3.1 405B. The best small-scale result we found is higher. Andrej Karpathy's llm.c reproduction of the 1.6B GPT-2 trained on 33.6 billion tokens in 24 hours on one 8 × H100 node. That is 192 GPU-hours, which back-solves to about 46% utilization, close to the roughly 50% he reports. !Effective utilization back-solved from published runs: Llama 3.1 8B 14%, Llama 3.1 70B 25%, Llama 3.1 405B 33%, llm.c GPT-2 1.6B 46%. The same formula on four real runs: plan for 30% and treat 45% as a well-tuned best case. The gap between running speed and the full bill is the useful part. The Llama 3 paper reports 38% to 43% utilization while the 405B model was training, yet the whole GPU-hour count back-solves to 33%. Over a 54-day stretch of that run, Meta logged 466 job interruptions, 419 of them unexpected. Restarts, redone work and idle time all land on the bill, and that is why we plan at 30%, not at peak. How much does it cost to train a small LLM from scratch? A small model is cheap enough to train for learning or for a narrow domain. Here is the formula at 30% utilization and the Nodus H100 list rate of $2.60/hr. | Model and tokens | Total operations | H100 GPU-hours | Compute cost | | --- | ---: | ---: | ---: | | 1B on 20B tokens | 1.2 × 10^20 | 112 | $292 | | 7B on 140B tokens | 5.9 × 10^21 | 5,505 | $14,313 | | 7B on 2T tokens | 8.4 × 10^22 | 78,643 | $204,471 | | 70B on 1.4T tokens | 5.9 × 10^23 | 550,500 | $1.43M | Our arithmetic from 6 × parameters × tokens, 989 teraFLOPS dense BF16, 30% utilization and $2.60 per H100-hour (checked October 30, 2026). To check the method against a real bill, take the llm.c GPT-2 run. Karpathy priced its 192 H100-hours at $672, which is $3.50 per GPU-hour. At $2.60, the same 192 GPU-hours cost $499.20. If your code reaches 45% instead of 30%, every row above falls by a third. What costs do these numbers leave out? The formula prices one clean run, and real projects are never one clean run. DeepSeek says plainly that its $5.576 million covers only the official training run and excludes prior research and ablation experiments on architecture, algorithms and data. Expect to spend real money on small test runs before the big one. Four other costs show up on most projects: - Lost work after an interruption. If an 8-node, 64-GPU run loses 6 hours of progress, that is 384 GPU-hours, or $998.40 at $2.60. With a checkpoint no more than 30 minutes old, the same failure costs 32 GPU-hours, or $83.20. - Idle GPUs. A node that sits waiting for data, a fix or a person bills the same as one that trains. - Data preparation and evaluation. Cleaning and tokenizing a large corpus and running benchmarks takes its own compute. - Storage for checkpoints and datasets, which grows with model size. The first two are the ones you control with tooling. That is where most of the gap between the formula and the final invoice comes from. Should you train from scratch at all? Usually not. Train from scratch only when you need your own architecture or tokenizer, a model with no inherited weights, or research results about pretraining itself. For most teams, continuing to pretrain an open model on your own text, or fine-tuning it, gets a better model for a small fraction of the cost. Our own continued-pretraining benchmark shows the scale difference. Continuing to train Qwen2.5-0.5B on raw text on one NVIDIA L4 cut held-out perplexity by 21.8% for a recorded charge of $0.121164. That was a small pilot on a tiny model. Still, it shows the order of magnitude: well under a dollar to adapt a small model, against hundreds to train even a 1B model from nothing. If you are fine-tuning instead, the cheapest way to fine-tune Llama on an H100 walks through that cost. How do you run a from-scratch pretraining job on Nodus? Nodus runs pretraining as a TrainingJob with task: Pretrain and initialization: Scratch, which keeps only the model's architecture and tokenizer. This manifest follows the training guide, with a spending cap and two 8-GPU nodes added: apiVersion: nodus.dev/v1beta1 kind: TrainingJob metadata: name: scratch-pretrain spec: runtime: nodus/sft task: Pretrain model: {uri: hf://Qwen/Qwen3-0.6B@c1899de289a04d12100db370d81485cdf75e47ca} data: {volume: corpus, format: JSONL} parameters: {steps: 10000, initialization: Scratch, packing: true} resources: {gpu: {type: [h100-sxm-80g], count: 8}, nodes: 2} maxCostUSD: "500.00" Price it before anything runs: $ nodus apply -f scratch-pretrain.yaml --dry-run -o yaml $ nodus apply -f scratch-pretrain.yaml The dry run's estimate is the number of steps times the measured seconds per step on the slowest GPU type you allowed. maxCostUSD caps the whole run, including retries; when it is reached the run is suspended, not failed, and raising the cap resumes it from its checkpoint. If a node is lost, the gang restarts from the latest checkpoint every rank committed, up to three times. Multi-node TrainingJobs are in Beta and go up to 8 nodes, as the multi-node guide explains. FAQ How much did it cost to train GPT-4? OpenAI has not published GPT-4's training compute or cost, so any figure you see is an outside estimate. The best public anchors are open models that publish GPU-hours, such as Llama 3.1 405B at 30.84 million H100-hours. Can you train an LLM on one GPU? Yes, for small models. Karpathy notes the 1.6B GPT-2 run would take about 8 days on one GPU instead of 1 day on eight, for the same GPU-hours. Nodus lists prices per GPU-hour, so the GPU-hours, not the GPU count, set the compute cost. Is a B200 cheaper than an H100 for pretraining? Only if it finishes the job more than proportionally faster. Nodus lists the NVIDIA B200 at $5.50/hr against $2.60/hr for the H100, so the B200 needs over 2.1 times the H100's throughput on your model to cost less per run. Measure seconds per step on both before you commit. Does a mixture-of-experts model cost less to train? Per token, yes, because compute scales with active parameters, not total ones. DeepSeek-V3 has 37B active parameters and needed 2.788 million GPU-hours, fewer than the dense Llama 3.1 70B at 7.0 million. You still need enough GPU memory to hold all the experts. Originally published on the Nodus blog.
Exploring Other Large Language Models for Improving Kinyarwanda Text Generation Choosing the Best Open-Source Model for Text Generation 1. Introduction Text generation is an important task in Natural Language Processing (NLP). It allows computers to generate text from a prompt. For example, when we write “The people of Kicukiro say”, a language model can predict what words should come next. In our project, we studied GPT-2 and compared it with a tiny GPT model that our group built from scratch. Our tiny GPT was trained on Kinyarwanda news from the KINNEWS dataset. Our experiments showed that GPT-2 can generate fluent English, but it struggles to continue a sentence correctly in Kinyarwanda. Our tiny GPT generates text that looks more like Kinyarwanda, but its sentences do not always make sense. GPT-2 has 124 million parameters and was trained mainly on English text. Our tiny GPT has about 0.82 million parameters and was trained on approximately 24 million characters of Kinyarwanda news. This showed us that both model size and training data are important for text generation. The next step is to find a better language model that can generate useful text and follow instructions. In this blog, I compare four open-weight models and choose the one that is most suitable for improving our project. 2. Models We Considered Model 1: Qwen3-4B-Instruct-2507 Qwen3-4B-Instruct-2507 was developed by the Qwen team at Alibaba. It has 4 billion parameters and supports a context length of up to 262,144 tokens. It uses the Apache 2.0 licence. This model is designed to follow instructions, understand text, and generate helpful responses. It also has multilingual capabilities and can handle many different tasks. One advantage is that it does not require a separate thinking mode to generate its answers. This makes its output easier to use when we want a direct response. However, it is larger than our original GPT-2 model, so it needs more memory and computing power. We also need to test its Kinyarwanda output instead of assuming that it will always generate correct Kinyarwanda. Model 2: Qwen3-1.7B Qwen3-1.7B is another model from Alibaba's Qwen team. It has approximately 1.7 billion parameters and uses the Apache 2.0 licence. Its standard context length is around 32,768 tokens. Its main advantage is its smaller size compared with Qwen3-4B. This can make it easier to run and faster to test on a computer with limited memory. It can follow instructions and generate text, making it useful for small NLP applications. However, its smaller size may affect the quality of some responses. This model would be a good choice if our main concern was reducing memory use. We would still need to compare its Kinyarwanda generation with the output from the larger model. Model 3: Llama 3.2-3B-Instruct Llama 3.2-3B-Instruct was developed by Meta. It has approximately 3 billion parameters and a context length of up to 128,000 tokens. It uses the Llama 3.2 Community Licence. This model is designed for tasks such as text generation, summarisation, and following instructions. It is relatively small compared with many larger language models. One advantage is that it can generate conversational text and follow user instructions. However, its licence has specific terms, and access to its model files may require accepting Meta's conditions. Its officially supported languages also do not include Kinyarwanda. This means it is worth considering, but its licence and language limitations make it less convenient for our project. Model 4: SmolLM3-3B SmolLM3-3B was developed by Hugging Face and its contributors. It has approximately 3 billion parameters and supports a standard context length of 65,536 tokens, with extensions available for longer contexts. It uses the Apache 2.0 licence. The model is designed for instruction following, text generation, and reasoning. It is also released with information about its training process, which makes it interesting for learning and research. Its size makes it a possible option for running on a GPU with limited memory. However, its documented multilingual support focuses on six languages: English, French, Spanish, German, Italian, and Portuguese. Kinyarwanda is not among them. For our project, we would need to test whether it produces useful Kinyarwanda text before choosing it. 3. Comparison Table | Model | Developer | Parameters | Context length | Licence | Main strength | |---|---|---:|---:|---|---| | Qwen3-4B-Instruct-2507 | Alibaba Qwen team | 4B | 262,144 tokens | Apache 2.0 | Instruction following and multilingual text generation | | Qwen3-1.7B | Alibaba Qwen team | 1.7B | 32,768 tokens | Apache 2.0 | Smaller size and lower memory needs | | Llama 3.2-3B-Instruct | Meta | 3B | 128,000 tokens | Llama Community Licence | Conversational text and instruction following | | SmolLM3-3B | Hugging Face and contributors | 3B | 65,536 tokens by default | Apache 2.0 | Efficient text generation and transparent training details | Context lengths are the models' documented limits. A longer context does not automatically mean better text generation. 4. The Model Chosen I would choose Qwen3-4B-Instruct-2507 to improve our text-generation project. The first reason is its instruction-following ability. We want a model that can receive a prompt and generate a response in the requested format. For example, we could ask it to continue a Kinyarwanda news sentence, produce a short news paragraph, or rewrite text more clearly. The second reason is its multilingual capability. Our notebook showed that GPT-2 struggles with Kinyarwanda partly because its tokenizer breaks many Kinyarwanda words into smaller pieces. The Kinyarwanda example required about 1.7 times as many tokens as the English example. Qwen3 may handle this task better, but we must test it because multilingual support does not guarantee accurate Kinyarwanda. The third reason is that it uses the Apache 2.0 licence and its Hugging Face model page provides access to the model files without the gated-access process required by some other models. We can load it directly in a notebook rather than depending on a paid text-generation API. Can it run on free Google Colab? Qwen3-4B-Instruct-2507 is a realistic model to test on a free Google Colab T4 GPU with 16 GB of GPU memory. Its 4-billion-parameter size makes local inference more practical than using a much larger model. The exact memory use depends on the precision, context length, and generation settings. For our first test, we should use a moderate prompt length and generate a limited number of new tokens. If memory becomes a problem, we can try a quantized version. The model itself does not require a paid API key. However, free Colab GPU access is not guaranteed at all times, and Colab still requires a Google account. We should also check the model's licence terms before using it in a published application. How will it improve our project? We can compare Qwen3 with GPT-2 and our tiny GPT using the same Kinyarwanda prompts. For example, we can use the prompt: "Abaturage bo mu Kicukiro bavuga ko..." We can then compare the generated text for language quality, sentence meaning, grammar, repetition, and whether it follows the prompt. We should keep the prompts and generation settings as similar as possible. We can also test English prompts to compare performance across languages. Our tiny GPT is still valuable because it helps us understand how a transformer works and how training data affects the results. Qwen3 would be an additional model for comparison, not a replacement for the lessons we learned from building our own model. 5. Conclusion Our experiments showed that generating text is not only about having a large model. The model's training data, tokenizer, and ability to follow instructions also affect the result. Qwen3-4B-Instruct-2507 is my preferred option because it combines a manageable model size, instruction following, multilingual capabilities, and an Apache 2.0 licence. It is also practical to test on a free Colab T4 GPU, although actual memory use must be checked. Most importantly, we should not assume that it will automatically produce perfect Kinyarwanda. We need to test it with real Kinyarwanda prompts and compare the results with GPT-2 and our tiny GPT. These tests will help us decide whether it is a useful improvement for our project. 6. References 1. OpenAI. Language Models are Unsupervised Multitask Learners (2019). openai.com 2. Qwen Team. Qwen3-4B-Instruct-2507 model card. Hugging Face. huggingface.co 3. Qwen Team. Qwen3-1.7B model card. Hugging Face. huggingface.co 4. Meta. Llama 3.2 model card and licence information. github.com/meta-llama/llama-models/blob/main/mode… 5. Hugging Face. SmolLM3-3B model card. huggingface.co 6. Hugging Face. Transformers documentation. huggingface.co 7. Vaswani, A. et al. Attention Is All You Need (2017). arxiv.org
Beyond GPT-2: Choosing a Modern Model for Our Poem Generator Blog assignment: Introduction to Large Language Models (LLMs) Introduction In our Natural Language Processing (NLP) course, we developed a Poem Generator using Transformer models. Our project explores three stages: building a small GPT-style model (Tiny GPT) from scratch, experimenting with the pretrained GPT-2 model, and comparing their generated poems. Although these models helped us understand how language generation works, we wanted to investigate whether modern Large Language Models (LLMs) could produce better poems. In this blog, we compare six models, examine their strengths and limitations, and select one model that fits our project and available computing resources. 1. Limitations of Our Current Project Tiny GPT learns from a small poetry dataset. Consequently, it may learn common poetic patterns without fully understanding the relationships between ideas. GPT-2 was pretrained on a much larger text collection, making it more capable of generating fluent text. However, its original context window is 1,024 tokens, and its base version is not specifically trained to follow detailed instructions. For example, asking GPT-2 to write four rhyming lines about rain at night does not guarantee that it will produce exactly four lines or maintain a consistent rhyme scheme. We therefore need a model that follows instructions, generates coherent text, respects requested poem structures, and runs on hardware available to our team. 2. What We Considered When Selecting a Model We used five criteria to compare candidate models: 1. Instruction following: Can the model follow requests about topic, length, tone, and rhyme? 2. Computing requirements: Can it run on a free Google Colab T4 GPU with 16 GB of GPU memory? 3. Licence and access: Can we download and run it without paying for an API or supplying an API key? What licence conditions apply? 4. Generation quality: Can it produce relevant, creative, coherent, and grammatically correct poems? 5. Compatibility: Does it work with Hugging Face Transformers, allowing us to reuse our existing Python workflow? 3. Comparison of Open-Weight LLMs We researched six models from different developers. Their parameter counts, context windows, licences, and intended strengths are summarized below. | Model | Developer | Parameters | Context Window | Licence | |---|---|---|---|---| | GPT-2 Medium / Large | OpenAI | 355M / 774M | 1,024 tokens | MIT | | Gemma 3 4B Instruct | Google | 4B | 128K tokens | Gemma Terms of Use | | Llama 3.2 3B Instruct | Meta | 3B | 128K tokens | Llama Community Licence | | Phi-4 Mini Instruct | Microsoft | 3.8B | 128K tokens | MIT | | SmolLM3-3B | Hugging Face | 3B | Up to 128K tokens | Apache 2.0 | | Qwen3-4B-Instruct-2507 | Qwen team, Alibaba | 4B | 262,144 tokens | Apache 2.0 | Note: Context limits can depend on the model variant and software configuration. Open-weight does not mean every model has an unrestricted open-source licence. GPT-2 GPT-2, developed by OpenAI, is a useful baseline for our project because it demonstrates how a Transformer predicts the next token to generate text. Its relatively short context window and lack of instruction tuning in the original base model make it less convenient for prompts with several constraints. It remains valuable for comparing modern models against an established text-generation approach. Gemma 3 4B Instruct Gemma 3 is developed by Google. Its 4-billion-parameter instruction-tuned version supports a 128K-token context window and is designed for tasks including text generation, reasoning, and summarization. It is a strong candidate for creative writing, but its Gemma-specific terms differ from standard permissive licences. Llama 3.2 3B Instruct Llama 3.2 is developed by Meta. Its 3-billion-parameter text model is designed for dialogue and other language tasks. Its relatively small size makes it attractive for resource-constrained environments. However, its Community Licence and acceptable-use requirements must be considered, and its poetry performance would need to be tested. Phi-4 Mini Instruct Phi-4 Mini Instruct is developed by Microsoft and contains approximately 3.8 billion parameters. It supports a 128K-token context window and is particularly focused on reasoning, mathematics, coding, and instruction following. Although these capabilities are useful, we would need experimental evidence to establish whether it offers an advantage for poetry. SmolLM3-3B SmolLM3-3B is developed by Hugging Face. It is a compact instruction-tuned model with support for long-context tasks and multiple languages. Its Apache 2.0 licence and publicly documented training approach make it attractive for experimentation. Nevertheless, we cannot assume that it generates better poems without testing it on our prompts. Qwen3-4B-Instruct-2507 Qwen3-4B-Instruct-2507 is developed by the Qwen team at Alibaba. It contains approximately four billion parameters and supports a native context window of 262,144 tokens. Its model card reports capabilities in instruction following, text comprehension, reasoning, and open-ended text generation. These features make it a promising candidate for our project. 4. Our Chosen Model: Qwen3-4B-Instruct-2507 We selected Qwen3-4B-Instruct-2507 because it offers a useful balance between instruction following, generation capabilities, computing requirements, and licence flexibility. First, it follows detailed instructions. We can prompt it to write a poem with a specific topic, number of lines, tone, and rhyme scheme. Its instruction-tuned design makes it a better fit for this task than an unmodified GPT-2 base model, although compliance must still be measured. Second, it is practical to experiment with. The model has approximately four billion parameters. Its published BF16 weights occupy about 8 GB, so loading it on a 16 GB Colab T4 may be feasible. However, model weights are not the only memory requirement: the attention cache, input length, generated tokens, and software overhead also consume GPU memory. We will begin with short prompts and modest generation lengths, monitor GPU memory, and reduce the workload or use quantization if necessary. Third, its licence supports our research. Qwen3-4B-Instruct-2507 uses Apache 2.0, which permits use and modification subject to its conditions. We can download the model weights and run local inference through supported software without needing a paid inference API or API key. We must still follow the licence and any applicable platform requirements. Fourth, it fits our existing workflow. The model supports Hugging Face Transformers and uses a chat template for structured prompts. This allows us to build on our existing Python knowledge and keep the experiment manageable. Finally, its long context window provides room for larger prompts and additional instructions. Our current poems do not require 262K tokens, but the capability is useful if we later experiment with longer creative-writing tasks. 5. How We Will Evaluate the Models To make our comparison fairer, we will use the same set of poem prompts for Tiny GPT, GPT-2, and Qwen3. We will keep the requested topics, poem lengths, and evaluation criteria consistent. Where possible, we will use comparable generation settings, while respecting each model's recommended configuration. We will evaluate each output using six criteria: - Relevance: Does the poem stay on the requested topic? - Structure: Does it follow the requested number of lines or stanzas? - Rhyme: Does it follow the requested rhyme scheme? - Coherence: Do the lines connect logically? - Grammar: Is the language understandable and grammatically correct? - Creativity: Does the poem use interesting descriptions and original ideas? We can use a simple scoring rubric, such as 1–5 for each criterion, and ask human reviewers to assess the results. We will report the actual scores after running the experiment rather than inventing results in advance. We will also record generation time and peak GPU memory usage to understand the practical cost of each model. 6. Limitations and Future Improvements Our comparison has several limitations. First, Tiny GPT, GPT-2, and Qwen3 differ in size, training data, and training methods. Therefore, differences in their output cannot be attributed to model architecture alone. Second, even an instruction-tuned model can ignore constraints, repeat phrases, or generate awkward rhymes. Fluency does not guarantee creativity or quality. Third, the models may reproduce biases or familiar patterns learned during training. Finally, Qwen3 requires more computing resources than Tiny GPT. If we want it to imitate the specific style of our poetry dataset, we could explore parameter-efficient fine-tuning, such as LoRA, if sufficient training data and hardware are available. Conclusion Our experiments with Tiny GPT and GPT-2 introduced us to Transformer-based text generation. Researching modern LLMs showed us that instruction tuning, training methods, model size, and context length can make a difference to how well a model responds to creative-writing requests. We chose Qwen3-4B-Instruct-2507 because it combines instruction-following capabilities, a permissive licence, compatibility with our Python workflow, and a model size that may fit our Colab environment with careful memory management. Our next step is to test it against Tiny GPT and GPT-2 using identical prompts and transparent evaluation criteria. This will help us determine whether the modern model actually improves our Poem Generator rather than relying on assumptions. References 1. OpenAI. GPT-2. openai/gpt-2 2. Google. Gemma 3 Model Card. ai.google.dev 3. Meta. Llama 3.2 Model Card. github.com/meta-llama/llama-models/blob/main/mode… 4. Microsoft. Phi-4 Mini Instruct. huggingface.co 5. Hugging Face. SmolLM3-3B. huggingface.co 6. Qwen Team. Qwen3-4B-Instruct-2507 Model Card. huggingface.co 7. Vaswani, A., et al. (2017). Attention Is All You Need. arxiv.org
@nikks_techie example: Llama 3.1 70B, bf16 KV, no prefix sharing: 320 KiB/token × 8,192 tokens × 100 active sequences = 250 GiB of cache alone. 70B isn't enough to size it. the model's KV heads and the cache dtype matter. github.com/meta-llama/llama-models/blob/main/mode…
I HAVE JUST FOUND SOMETHING INSANE Meta's official Llama repo has a default sample image it's a dog onboard github.com/meta-llama/llama-models/blob/main/mode… Llama 4's multimodal chat script loads this dog every time a dev tests vision. Meta's AI has a mascot and they never told anyone Also pumpfun is spending millions on onboarding. In callout rewards, updates, KOLs and now referrals. never been run wtf?? So genny onboarddog - $onboard
I HAVE JUST FOUND SOMETHING INSANE Meta's official Llama repo has a default sample image it's a dog onboard github.com/meta-llama/llama-models/blob/main/mode… Llama 4's multimodal chat script loads this dog every time a dev tests vision. Meta's AI has a mascot and they never told anyone Also pumpfun is spending millions on onboarding. In callout rewards, updates, KOLs and now referrals. never been run wtf?? So genny onboarddog
A Xiaomi treinou uma IA gigante ao vivo, com o custo na tela #shorts O MiMo V2.6 foi treinado por reforço com painel público e custo aberto: US$ 2,6 milhões, ao vivo. DeepSeek, Meta, BLOOM, OLMo e SmolLM3 já tinham aberto partes do treino. O que é inédito aqui, e o furo que a transparência revelou. Painel ao vivo do treino: mimo.xiaomi.com Post oficial MiMo-V2.6: mimo.mi.com Relatório técnico: arxiv.org Framework de treino (verl): XiaomiMiMo/verl Ambientes de RL: huggingface.co Auditoria Vals AI: vals.ai Comparação: DeepSeek-V3 (custo): arxiv.org DeepSeek-R1 na Nature: nature.com Llama 3.1 (horas de GPU): github.com/meta-llama/llama-models/blob/main/mode… BLOOM (logs ao vivo, 2022): huggingface.co OLMo 2: allenai.org SmolLM3 (logs e checkpoints): huggingface.co Ideia: vídeo de bycloud. #opensource #programacao #ia