Codex Was Torching Your SSD and Nobody Noticed for Months
Your Codex log file looked fine. Your SSD counters did not.
The file was small.
That was the trick.
If you opened <code>~/.codex/</code>, you could see a SQLite database called <code>logs_2.sqlite</code>. It was not huge. It was not swallowing the disk. Finder, Explorer, <code>du</code>, and every normal storage check had nothing dramatic to say.
Meanwhile, one developer's SSD recorded roughly 37 TB of writes in 21 days.
That works out to about 1.7 TB a day. If you stretch one heavy, continuous-use report across a full year, it becomes roughly 640 TB.
The number is not a prediction for every Codex user. It came from one machine running for a long time. But it exposed a problem that is easy to miss with local developer tools:
> a file can stay small while the storage device underneath it gets written to constantly.
This is the story of a noisy TRACE logging sink, a SQLite database doing exactly what it was told, and the SSD wear that nobody could see from file size alone.
The quick version
- A user reported unusually high local disk writes and traced them to Codex CLI's <code>logs_2.sqlite</code> database and its WAL companion file.
- The reported machine accumulated about 37 TB of writes over 21 days of uptime.
- The file stayed bounded because new logs were inserted while older logs were deleted.
- The reported cause was an overly verbose TRACE logging path that did not fully respect the expected <code>RUST_LOG</code> override.
- The supplied investigation says version 0.142.0 reduced the noisy writes substantially, with early testing reporting around 85% fewer log writes.
- If you used an older build heavily, update first, then inspect real write activity and your SSD's lifetime-write counters.
The number that made people look twice
37 TB in 21 days is not a normal “developer tool wrote some logs” story.
| Measurement | Value |
| --- | --- |
| Reported writes | ~37 TB |
| Reporting window | 21 days |
| Approximate daily rate | ~1.7 TB/day |
| Annualized at that rate | ~640 TB/year |
The annualized number needs context. It assumes the same workload every day for a year, which is rarely how people work. Someone running short sessions will produce far less. Someone leaving the tool open, working with long sessions, or using it in automation could produce much more than they expect.
The useful point is not “every SSD will die in a year.”
The useful point is that this level of write activity belongs in the category of things you should notice and control.
<!-- Visual placement: use a clean SMART dashboard screenshot here, with all serial numbers and personally identifiable drive details blurred. -->
Why the file size told you almost nothing
Most of us check disk problems like this:
du -sh ~/.codex
ls -lh ~/.codex/logs_2.sqlite*
Those commands answer one question:
> How much data is stored right now?
They do not answer this:
> How much data has been written over time?
The Codex log database was designed to retain a bounded amount of diagnostic information. New rows were written, old rows were removed, and the database could remain roughly the same size.
So the folder looked calm while the disk was doing constant work.
Imagine a café with four tables. Looking through the window at noon, you see four people sitting down. Come back at 6 PM, still four people. The table count tells you nothing about whether eight customers visited or eight hundred.
The SQLite file was the same. Stable size, enormous turnover.
!Diagram showing how a small SQLite log file can lead to high SSD write activity
What was generating all those writes?
The reported issue points to a logging path that defaulted to TRACE, the most detailed level in Rust's tracing ecosystem.
TRACE logging is useful when you are trying to understand a very specific failure. It can include low-level events that are normally too noisy for everyday use:
- file-open events
- raw network or WebSocket activity
- transport details
- telemetry plumbing
- dependency-library messages that get bridged into the application's tracing system
The supplied breakdown estimates that TRACE-level entries made up 70.7% of the retained log-body size. The highest-volume targets included WebSocket handling, OpenTelemetry-related logs, a generic dependency log bridge, and client transport code.
That is diagnostic exhaust. A little of it is fine. A default stream of it, persisted locally in SQLite, is a different kind of workload.
The three parts that stacked together
No single part of this design has to be bad on its own.
1. TRACE was active where it should have been opt-in
TRACE is not a normal default for a tool used for hours at a time. It is the level you turn on when you are actively debugging.
The reported behavior was that Codex initialized a feedback logging sink at TRACE. That turns every routine interaction into a lot of diagnostic detail.
2. RUST_LOG=warn did not fully quiet it
Rust developers often reach for this first:
RUST_LOG=warn codex
That is supposed to turn down the volume. According to the issue analysis, it did not fully apply to the relevant sink. If a target did not have an explicit override, Codex's internal default still allowed noisy logging.
That is worse than an obvious missing setting. A user can think they reduced logging while the disk keeps receiving the same workload.
3. SQLite turned event noise into storage churn
SQLite is excellent. It is also not magic.
For small structured local data, a SQLite database is a good choice. For a constant stream of fine-grained TRACE events, inserts, deletes, indexes, and checkpoints, it can create a lot of writes even when the database never becomes large.
The problem was not “SQLite is bad.” The problem was using a write-heavy retention loop as a sink for a log level that should not have been permanently enabled.
The part the operating system does not show you
There are two kinds of amplification in this story.
SQLite write amplification
SQLite often uses WAL mode, short for write-ahead logging.
In simplified form:
1. A log event is appended to the WAL file.
2. SQLite later checkpoints that change into the main database.
3. Older rows are deleted to keep retained data bounded.
4. Indexes and database pages are updated along the way.
One logical log event can therefore create more than one write at the filesystem level.
The WAL is not wrong. Checkpoints are not wrong. The problem appears when that loop is fed relentlessly with small, noisy events.
SSD write amplification
An SSD has its own internal life.
It writes data in small pages but erases much larger blocks. When a small part of a block changes, the SSD controller often writes the new data to a fresh location, marks old data stale, later moves still-valid pages, and erases the old block.
That movement is called garbage collection. Wear leveling moves writes around so one physical area does not wear out first.
Both are useful. Both mean the flash may receive more physical writes than the operating system asked for.
The ratio is called the write amplification factor. The exact number varies by drive, spare capacity, free space, workload, and controller. The supplied analysis does not measure the multiplier for the affected drive, so do not treat any guessed physical-wear number as a precise fact.
The direction is enough:
> High-churn, small random writes are one of the workloads SSDs work hardest to manage.
Is 640 TB actually dangerous?
It can be.
SSDs have an endurance rating usually expressed as TBW, or terabytes written. A typical 1 TB consumer SSD may be rated for a few hundred TBW. Enterprise drives may be rated much higher. The exact number depends on the model, capacity, NAND type, and warranty.
That does not mean a drive fails the moment it reaches its rated TBW. Ratings are warranty and endurance guidance, not a timer that hits zero at midnight. Plenty of drives outlive them.
But a background development tool potentially consuming hundreds of terabytes of writes is still worth taking seriously. It can eat through a meaningful part of the drive's expected write budget without filling the disk or showing a visible failure.
How to check your own machine
The order matters.
Update first. Then look at real I/O. Then check what the drive itself says.
Step 1: update Codex
The investigation supplied with this article identifies 0.142.0 as the version containing the logging reduction. Check the current release notes and your installed build, then update before you start measuring.
Also remember that Codex CLI, the desktop app, and editor integrations can ship or bundle components on different schedules. Updating one does not automatically prove every local Codex-related process is updated.
Step 2: find the log files
On macOS and Linux:
ls -la ~/.codex/logs_2.sqlite*
On Windows PowerShell:
Get-Item "$env:USERPROFILE\.codex\logs_2.sqlite*"
You might see the database, a <code>-wal</code> file, and a <code>-shm</code> file.
Do not stop after checking their size. A small database is the whole reason this incident was easy to miss.
Step 3: watch actual writes
On Linux, <code>pidstat</code> can show which process is writing:
pidstat -d 5
To see whether a process has the database open:
sudo lsof ~/.codex/logs_2.sqlite
On macOS, open Activity Monitor, select the Disk tab, and watch bytes written for Codex-related processes while you work.
On Windows, use Resource Monitor or Task Manager's disk column. Look at sustained per-process writes, not just a brief spike during startup.
Step 4: check the drive's own lifetime write counter
This is the closest thing to the truth you have.
On Linux or macOS with smartmontools installed, an NVMe drive can often be checked with:
sudo smartctl -a /dev/nvme0
Look for:
- Data Units Written
- Percentage Used
- warnings or critical health indicators
On Windows, CrystalDiskInfo exposes Total Host Writes and health information. PowerShell can also show some device health data:
Get-PhysicalDisk | Get-StorageReliabilityCounter |
Select-Object DeviceId, Wear, Temperature
Compare lifetime writes with the TBW rating for your exact drive model. Do not compare against a random SSD number from a search result.
<!-- Visual placement: add a small annotated image of a SMART page here. Highlight only Data Units Written or Total Host Writes, Percentage Used, and health status. -->
What if I used an older version for months?
First, do not panic and do not guess from file size.
Check the drive counters. If the drive has low percentage used and reasonable total writes for its age and workload, update and move on. If the drive has unusually high wear, warnings, or a large write count compared with its TBW rating, back up your important data and plan for a replacement before failure forces the schedule.
This is a good backup reminder even if your drive looks healthy. SSDs usually fail without much theatre.
Workarounds if you cannot update immediately
Updating is the right fix. These are temporary ways to reduce persistent write pressure while you arrange that update.
Put throwaway logs on ephemeral storage
For CI jobs and containers, the cleanest option is usually to run with a temporary home directory or mount the relevant config directory on ephemeral storage.
Example for a disposable CI environment:
export HOME="$(mktemp -d)"
For a container workload, mount a tmpfs at the appropriate Codex state directory if the environment can tolerate losing local diagnostics when the job exits.
This is especially useful for build runners that create and destroy their environment anyway.
Be careful with /tmp
Moving a log file to <code>/tmp</code> is only useful if <code>/tmp</code> is actually memory-backed in your environment. On many Linux systems it may be tmpfs. On macOS, <code>/tmp</code> is not automatically a RAM disk by default.
Check the mount before assuming the write load vanished:
mount | grep ' /tmp '
Then verify with a real I/O monitor. Never trust a workaround because it sounded plausible in a comment thread.
Avoid unsupported database surgery unless you really need it
Some community workarounds modify the SQLite log database so inserts are ignored. That can be useful in an emergency, but it is an unsupported change to an internal application file.
Future versions may change the schema or assume the database behaves normally. If you choose that route, stop Codex first, make a copy of the database, document what you changed, and remove the workaround once you are on a fixed release.
The bigger lesson for tool builders
This bug is bigger than one CLI.
Measure bytes written, not just disk space used
Every local persistence feature should have an I/O budget under realistic usage. Test bytes written per hour, per session, and per day. File size is not a sufficient metric for databases, caches, indexes, or log rotation systems.
TRACE must be an intentional decision
Verbose logging needs an explicit opt-in and a clear way back out. Shipping it by default turns every user into a debugging environment.
Standard controls have to work end to end
If your ecosystem has a standard logging override, honour it for every sink. A partial override is dangerous because it gives experienced users false confidence.
Structured logs have storage costs
SQLite, RocksDB, local search indexes, and caches are all good tools. High-frequency writes have a cost. The design needs backpressure, batching, rate limits, retention rules, and test coverage for pathological verbosity.
Privacy and wear are connected
The more raw operational detail a tool records, the more it writes and the more sensitive local data it can retain. Logging file paths, raw transport frames, and low-level event payloads should be a deliberate debugging choice, not background noise.
The bigger lesson for users
There is a simple habit worth keeping:
> When you install a new long-running developer tool, look at its write rate once.
Two minutes in Activity Monitor, Resource Monitor, or <code>pidstat</code> can catch a surprising amount.
Check lifetime SSD writes occasionally. Keep your tools updated. Treat a stable file size as a storage number, not an I/O number.
The log file was small. The work it made the SSD do was not.
What is confirmed and what is still uncertain
The supplied investigation is clear about its limits, and those limits matter:
- The 37 TB measurement came from one heavy user's machine. It is a data point, not a universal rate.
- Reports disagree about when the highest logging verbosity began. The exact start date is uncertain.
- The annualized 640 TB figure is arithmetic based on the reported 21-day sample, not a forecast.
- SSD write amplification is real, but the exact physical NAND multiplier was not measured for this incident.
- The reported 85% reduction after version 0.142.0 is an early observed result, not a guarantee for every platform or workload.
That uncertainty does not weaken the main finding. The correct response is still simple: update, measure the process, and inspect the drive counter.
Sources and further reading
- GitHub issue #28224 in openai/codex
- Official OpenAI Codex documentation
- The public reporting and technical analysis supplied for this article, including Notebookcheck, Windows Report, Lilting.ch, yage.ai, and Daniel Vaughan's write-up
11 reactions · 9 comments
Four GPUs, One Quantization Ladder: Gemma 4 E2B From a 4 GB Laptop to an NVIDIA L4
---
title: "Four GPUs, One Quantization Ladder: Gemma 4 E2B From a 4 GB Laptop to an NVIDIA L4"
published: true
description: "Gemma 4 E2B from BF16 to Q2_K on a Tesla T4, an NVIDIA L4, a GTX 1650 laptop and an RTX 3050 desktop, with byte-identical GGUF files and the same llama.cpp build. The ladder keeps its shape on all four cards, 218 questions die on all four, and any two GPUs grade 2 to 4 questions differently at BF16."
tags: gemma, llamacpp, quantization, nvidia
cover_image: raw.githubusercontent.com
---
This article provides a step by step guide to running Gemma 4 E2B's quantization ladder, BF16 down to Q2_K, on four NVIDIA GPUs with the same nine GGUF files and the same llama.cpp build, and comparing the four cards question by question.
xbill9/gemma-graveyard
The first article asked every rung of the ladder the same 250 GSM8K questions on a Tesla T4. The second repeated it on an NVIDIA L4 in AWS. This one adds two consumer cards: a GTX 1650 Ti laptop GPU with 4 GB and an RTX 3050 desktop card with 8 GB. All four run llama.cpp b11699 on files with the same sha256, and each card gives identical text when BF16 is run twice.
---
The Results in Brief
The ladder keeps its shape on all four GPUs. Against each card's own BF16, F16 to Q5_K_M lose 1 to 8 answers, Q4_K_M and Q4_0 lose 9 to 19, Q3_K_M loses 42 to 49, and Q2_K gets 3 to 6 right out of 250.
The counts agree; the questions move. Any two GPUs grade 2 to 4 questions differently at BF16 and 12 to 22 at Q4_0. No question is lost at F16 or Q8_0 on all four cards, while 216 are lost at Q2_K on all four. 218 questions die at some rung on every GPU, and 147 of them die first at the same rung on every GPU.
The 4 GB laptop runs every quantized rung entirely on its GPU. BF16 and F16 do not fit, so llama.cpp splits them between GPU and CPU, and they decode at 9.2 and 9.6 tokens per second against 45.9 for Q8_0 on the same card.
The cover lays the four GPUs' grades over each other. One row per rung, BF16 at the top and Q2_K at the bottom, one cell per question: marigold where all four GPUs answer right, grey where all four answer wrong, violet where they disagree. docs/make_cover_gpu4.py draws it from the committed rows.
---
What Changes and What Stays
| | Tesla T4 | NVIDIA L4 | GTX 1650 Ti | RTX 3050 |
| :--- | :--- | :--- | :--- | :--- |
| Where | GCE n1-standard-2 | AWS g6.xlarge | laptop | desktop |
| GPU memory | 15,360 MiB | 23,034 MiB | 4,096 MiB | 8,192 MiB |
| Compute capability | 7.5 | 8.9 | 7.5 | 8.6 |
| Host CPU | 2 vCPU | AMD EPYC 7R13 | Intel i7-10750H | Intel i7-5820K |
| Driver | 615.71.10 | 595.112.07 | 615.71.10 | 615.78.08 |
The same on all four: llama.cpp b11699 with GGML_CUDA=ON, built for each card's own compute capability; the nine GGUFs and Google's QAT Q4_0 file, byte for byte; the 250 questions in data/heldout.jsonl; temperature 0, one request at a time, thinking off, 1,024 max tokens, llama-server -c 4096 -np 1 --seed 0. Every rung runs with -ngl 99 except BF16 and F16 on the GTX 1650 (Step 4).
---
At This Point You Should Have…
- An NVIDIA GPU with at least 4 GB and a CUDA toolkit
- About 58 GB of free disk for the checkpoint, the nine GGUFs and the QAT file
- A clone: git clone https://github.com/xbill9/gemma-graveyard
- The T4 run's committed results in runs/2026-10-27-ladder/
---
Step 1 — Build llama.cpp at the Same Tag
On the RTX 3050:
git -C ~/llama.cpp worktree add --detach /android/graveyard/llama.cpp b11699
cd /android/graveyard/llama.cpp
cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86 -DCMAKE_BUILD_TYPE=Release
cmake --build build -j12 --target llama-server llama-quantize
The one build flag that changes per card is CMAKE_CUDA_ARCHITECTURES: 75 for the T4 and the GTX 1650, 89 for the L4, 86 for the RTX 3050.
nvidia-smi --query-gpu=name,memory.total,compute_cap,driver_version --format=csv
name, memory.total [MiB], compute_cap, driver_version
NVIDIA GeForce RTX 3050 OEM, 8192 MiB, 8.6, 615.78.08
🔎 Tip: keep the pinned build in its own worktree. The desktop already had a newer llama.cpp checkout (b11983). git worktree add --detach <dir> b11699 gives a second working copy at the tag from the same clone, so the measured build and the everyday build sit side by side.
---
Step 2 — Rebuild the Ladder and Check It Against the T4
ladder.sh converts the checkpoint to BF16 and F16 and quantizes seven rungs from BF16. Each run script then compares every file's size and sha256 with the T4's ladder.tsv:
diff <(cut -f1,3,4 runs/2026-10-27-ladder/ladder.tsv) <(cut -f1,3,4 runs/2026-10-29-3050/ladder.tsv) && echo "ladder matches T4 sha256"
ladder matches T4 sha256
The GTX 1650 and the RTX 3050 both print the same line, and Google's QAT Q4_0 file matches too (sha256 fa401b55…). Across the four machines, the GGUF bytes are identical; the card, its driver, its CUDA version and the kernels built for it are what differ.
🔎 Tip: the converter writes the checkpoint directory into the GGUF. convert_hf_to_gguf.py copies the model card and takes the model's name from the directory it reads. A checkpoint taken straight from the Hugging Face cache snapshot has neither the README.md nor the name gemma-4-E2B-it, and gives different bytes. Download with hf download google/gemma-4-E2B-it --local-dir <dir>/gemma-4-E2B-it.
---
Step 3 — Prove Each Card Repeats Itself
BF16 on 50 questions, twice, on every GPU:
./graveyard.py repeat runs/2026-10-29-3050/determinism-1 runs/2026-10-29-3050/determinism-2 BF16
{"rung": "BF16", "n": 50, "mismatched_extracted": 0, "mismatched_text": 0, "mismatched_right": 0, "right_a": 46, "right_b": 46}
The T4, the L4 and the GTX 1650 give the same result: 0 mismatched characters and 46 of 50 right both times. Each card on its own is deterministic, so a difference between two cards comes from the GPU and its CUDA stack.
---
Step 4 — Fit the 16-bit Rungs on 4 GB
BF16 and F16 need a 4,384.87 MiB GPU buffer for their 36 layers (from the T4's load log), and the GTX 1650 has 4,096 MiB. With -ngl 99 the load fails. graveyard.py run --ngl auto leaves -ngl off, and llama.cpp's fit step places what it can:
common_params_fit_impl: - CUDA0 (NVIDIA GeForce GTX 1650 Ti with Max-Q Design): 14 layers, 2546 MiB used, 1125 MiB free
load_tensors: offloaded 14/36 layers to GPU
load_tensors: CUDA0 model buffer size = 2407.92 MiB
Every other rung fits whole. Q8_0 on the same card:
load_tensors: offloaded 36/36 layers to GPU
load_tensors: CUDA0 model buffer size = 2342.30 MiB
The split changes speed and leaves the grades in line with the other GPUs: the GTX 1650's BF16 scores 229, the same as the T4 and the L4, and grades 2 to 4 questions differently from each of them. The 8 GB RTX 3050 holds BF16 whole at 4,651 MiB.
---
Step 5 — Run the Ladder on Each Card and Score It
Each run script runs the nine rungs and QAT Q4_0, then scores every rung against that card's own BF16:
./graveyard.py bury runs/2026-10-29-3050 BF16 F16 Q8_0 Q6_K Q5_K_M Q4_K_M Q4_0 Q3_K_M Q2_K --json graveyard-ladder.json
Right answers, out of 250:
| Rung | T4 | L4 | GTX 1650 | RTX 3050 |
| :--- | :--- | :--- | :--- | :--- |
| BF16 | 229 | 229 | 229 | 232 |
| F16 | 229 | 229 | 228 | 232 |
| Q8_0 | 227 | 228 | 229 | 226 |
| Q6_K | 229 | 227 | 226 | 227 |
| Q5_K_M | 231 | 228 | 226 | 226 |
| Q4_K_M | 224 | 221 | 222 | 221 |
| Q4_0 | 222 | 220 | 216 | 218 |
| Q3_K_M | 191 | 186 | 183 | 191 |
| Q2_K | 3 | 6 | 5 | 3 |
Lost, against the same card's BF16:
| Rung | T4 | L4 | GTX 1650 | RTX 3050 |
| :--- | :--- | :--- | :--- | :--- |
| F16 | 1 | 1 | 2 | 1 |
| Q8_0 | 4 | 3 | 2 | 7 |
| Q6_K | 4 | 6 | 7 | 8 |
| Q5_K_M | 3 | 6 | 6 | 8 |
| Q4_K_M | 9 | 14 | 12 | 14 |
| Q4_0 | 15 | 16 | 19 | 16 |
| Q3_K_M | 42 | 45 | 49 | 44 |
| Q2_K | 226 | 225 | 225 | 229 |
Lost: right at BF16, wrong at this rung, on the same GPU.
The failures fall into the same kinds on every card. Q3_K_M runs out of room: 33 to 36 of its losses on each GPU are answers cut off at 1,024 tokens. Q2_K loops: 208 to 228 of its 250 answers repeat a line five or more times, and 246 to 249 hit the token cap.
---
Step 6 — Line the Four GPUs Up, Question by Question
docs/gpu4_derived.py rebuilds the T4's grid from its committed results and reads the other three from their rows, then compares all six pairs of GPUs at every rung:
python3 docs/gpu4_derived.py > runs/2026-10-29-3050/four-gpus.txt
| Rung | Graded differently, any two GPUs | Lost on all four |
| :--- | :--- | :--- |
| BF16 | 2 to 4 | — |
| F16 | 3 to 5 | 0 |
| Q8_0 | 3 to 9 | 0 |
| Q6_K | 4 to 10 | 2 |
| Q5_K_M | 7 to 10 | 1 |
| Q4_K_M | 9 to 15 | 5 |
| Q4_0 | 12 to 22 | 5 |
| Q3_K_M | 26 to 34 | 20 |
| Q2_K | 6 to 9 | 216 |
Graded differently: right on one GPU and wrong on the other, at that rung, for the closest and the furthest of the six pairs.
- The light rungs lose different questions on each card. F16 and Q8_0 lose 1 to 7 answers per GPU, and none of them is lost on all four.
- The deep losses are shared. Q3_K_M loses 20 questions on all four GPUs and Q2_K 216.
- Most of the graveyard is common ground. Each GPU has 226 questions that die at some rung, the RTX 3050 230. 218 die on all four, and 147 die first at the same rung on all four. One question dies only on the RTX 3050 (test-358) and one only on the GTX 1650 (test-147).
- BF16 itself moves a few answers. 227 questions are right at BF16 on all four cards and 18 wrong on all four. Against the L4, the RTX 3050's BF16 changes the text of 31 answers out of 250, the extracted number of 4 and the grade of 3.
---
Step 7 — Google's QAT Q4_0 on Four Cards
./graveyard.py bury runs/2026-10-29-3050 BF16 QAT_Q4_0 Q4_0 --json graveyard-qat.json
| GPU | QAT Q4_0 right | Q4_0 right | QAT Q4_0 lost | Q4_0 lost |
| :--- | :--- | :--- | :--- | :--- |
| T4 | 221 | 🥇 222 | 15 | 15 |
| L4 | 219 | 🥇 220 | 15 | 16 |
| GTX 1650 | 🥇 222 | 216 | 11 | 19 |
| RTX 3050 | 🥇 221 | 218 | 15 | 16 |
QAT Q4_0 scores 219 to 222 on the four cards and plain Q4_0 216 to 222. On the T4 and the L4 the plain file is ahead by 1; on the GTX 1650 QAT is ahead by 6 and on the RTX 3050 by 3. The two files lose mostly different questions: on the RTX 3050, 5 questions are lost under both, 10 only under QAT and 11 only under the plain Q4_0. Across the four GPUs, 6 of QAT's losses and 5 of Q4_0's are lost on every card.
---
Step 8 — Speed on Each Card
Median decode speed in tokens per second. Each column is one machine; read down a column.
| Rung | T4 | L4 | GTX 1650 | RTX 3050 |
| :--- | :--- | :--- | :--- | :--- |
| BF16 | 40.0 | 50.1 | 9.2 | 40.3 |
| F16 | 39.6 | 50.0 | 9.6 | 40.3 |
| Q8_0 | 64.3 | 83.0 | 45.9 | 67.8 |
| Q6_K | 70.8 | 98.1 | 57.2 | 75.3 |
| Q5_K_M | 78.7 | 107.7 | 61.9 | 82.2 |
| Q4_K_M | 84.1 | 118.7 | 66.5 | 91.5 |
| Q4_0 | 🥇 93.6 | 125.6 | 🥇 70.8 | 🥇 99.7 |
| Q3_K_M | 74.2 | 111.1 | 55.9 | 64.2 |
| Q2_K | 74.0 | 🥇 139.5 | 62.2 | 95.0 |
- Q4_0 is the fastest rung on three of the four cards; on the L4 it is Q2_K, which is wrong on 244 of 250 questions.
- Q5_K_M decodes 1.21 to 1.35 times as fast as Q8_0 on each card (arithmetic, from the table). Q8_0 is the baseline here because it is the largest rung that fits whole on all four GPUs.
- Q3_K_M is slower than Q4_0 on every card, and on the RTX 3050 it is slower than Q8_0.
- BF16 and F16 run at the same speed on the T4, the L4 and the RTX 3050. On the GTX 1650, with 22 of 36 layers on the CPU, they run at a fifth of Q8_0's speed.
The full ladder with QAT took 5 hours 16 minutes on the RTX 3050 and 12 hours 25 minutes on the GTX 1650, from the first log line to the last (run-3050.log, run-1650.log).
---
Compare and Contrast
| | T4 | L4 | GTX 1650 | RTX 3050 |
| :--- | :--- | :--- | :--- | :--- |
| BF16 right of 250 | 229 | 229 | 229 | 🥇 232 |
| Q5_K_M lost | 🥇 3 | 6 | 6 | 8 |
| Q4_0 right, tok/s | 222, 93.6 | 220, 125.6 | 216, 70.8 | 218, 99.7 |
| Every rung whole on the GPU | 🟢 yes | 🟢 yes | ❌ BF16, F16 split | 🟢 yes |
| Same text twice at BF16 | 🟢 yes | 🟢 yes | 🟢 yes | 🟢 yes |
---
So, Which One?
On any of the four cards, take Q5_K_M: 39.0% of BF16's bytes, 226 to 231 right, 3 to 8 answers lost against the card's own BF16, and 1.21 to 1.35 times Q8_0's decode speed. For the fastest file that stays within 19 answers, take Q4_0, the fastest rung on three of the cards. On a 4 GB card, every quantized rung fits whole and the 16-bit rungs run at under 10 tokens per second. To compare two quantizations question by question, run both on one GPU: a different card moves 2 to 4 grades at BF16 and up to 34 at Q3_K_M.
---
Summary
The goal of this article was to run Gemma 4 E2B's quantization ladder on four GPUs, from a 4 GB laptop card to a data-centre L4, and see which grades hold. The key was holding everything else fixed: byte-identical GGUFs, the same llama.cpp tag and the same sampling, with each card checked to give identical text twice. The four-GPU results were:
- 🟢 All nine GGUFs and the QAT file match the T4's sha256 on every machine, and BF16 gives identical text twice on 50 questions on every card.
- 🟢 The ladder keeps its shape on all four GPUs: F16 to Q5_K_M lose 1 to 8, Q4_K_M and Q4_0 lose 9 to 19, Q3_K_M loses 42 to 49.
- 🟢 Q5_K_M is the best trade on every card: 226 to 231 right at 1.21 to 1.35 times Q8_0's speed.
- 🟢 A 4 GB GTX 1650 runs every quantized rung whole on the GPU; BF16 and F16 run split, 14 of 36 layers on the GPU, at 9.2 and 9.6 tokens per second.
- ⚠️ Any two GPUs grade 2 to 4 questions differently at BF16 and 12 to 22 at Q4_0; no question is lost at F16 or Q8_0 on all four.
- ⚠️ 218 questions die on all four GPUs, 147 of them first at the same rung.
- ⚠️ QAT Q4_0 and plain Q4_0 trade places by card: plain ahead by 1 on the T4 and L4, QAT ahead by 6 on the GTX 1650 and by 3 on the RTX 3050.
- ❌ Q3_K_M loses 42 to 49, mostly answers cut off at 1,024 tokens; Q2_K gets 3 to 6 right, with 208 to 228 looping answers per card.
Scope: one run of each rung per GPU on four NVIDIA cards (a Tesla T4 on GCE, an L4 on AWS, a GTX 1650 Ti laptop GPU and an RTX 3050 desktop card), 250 GSM8K test questions, greedy, thinking off, 1,024 max tokens, llama.cpp b11699 built for each card. Determinism checked at BF16 on each card. The T4 grid is rebuilt from its committed per-question results. The hosts also differ in CPU, RAM, driver and CUDA version, and on the GTX 1650 BF16 and F16 run split between GPU and CPU. Speed is compared within each card only.
The strategy for measuring Gemma 4's accuracy one quantization rung at a time on four GPUs was validated with an incremental step by step approach.
---
References
- Accuracy vs Speed: Gemma 4 E2B Quantization Tradeoffs, BF16 to Q2_K, the T4 article
- Performance vs Quality: Gemma 4 E2B Quantization on an NVIDIA L4 and a Tesla T4, the L4 article
- gemma-graveyard on GitHub
- GTX 1650 run: runs/2026-10-29-1650
- RTX 3050 run: runs/2026-10-29-3050
- llama.cpp
- google/gemma-4-E2B-it
- google/gemma-4-E2B-it-qat-q4_0-gguf
- GSM8K
10 reactions · 1 comment
Accuracy vs Speed: Gemma 4 E2B Quantization Tradeoffs, BF16 to Q2_K
---
title: "Accuracy vs Speed: Gemma 4 E2B Quantization Tradeoffs, BF16 to Q2_K"
published: true
description: "Gemma 4 E2B quantized from BF16 to Q2_K on one Tesla T4, every rung asked the same 250 GSM8K questions, and each answer buried at the rung where it first goes wrong. Q5_K_M and above hold the score; Q3_K_M loses 38; Q2_K gets 3 right. With thinking on, Q3_K_M wins back 17."
tags: devchallenge, hf26challenge, gemma, llamacpp
cover_image: raw.githubusercontent.com
---
This is a submission for the Hacktoberfest Open-Source AI Challenge Week 4: Ghost in the Machine
This article provides a step by step guide to quantizing Gemma 4 E2B down a ladder of nine GGUF files with llama.cpp, asking every rung the same 250 maths questions on one Tesla T4, and burying each answer at the rung where it first goes wrong. Every count is computed by code and every row is committed.
xbill9/gemma-graveyard
A score per quantization level says how many answers a smaller file loses; a graveyard says which ones. Against the 229 of 250 questions BF16 gets right, Q8_0, Q6_K and Q5_K_M stay within 2, Q4_K_M and Q4_0 lose 5 and 7, Q3_K_M loses 38 and Q2_K gets 3 right. Above Q3_K_M most of the dead come back: 16 of the 23 questions that die between F16 and Q4_0 are right again at a lower rung. With thinking on, Q3_K_M climbs from 191 to 208.
---
What I Built
Quantization is the most common thing done to an open-weight model before it ships to a small device. The usual report is one accuracy number per file size. Two files with the same score can still get different questions wrong, and a net loss hides its parts: Q4_0 here is 7 answers down on BF16, made of 15 questions lost and 8 gained.
The gemma-graveyard asks every rung of a quantization ladder the same 250 questions at temperature 0, so each answer is the model's single most likely one. BF16 is the reference, and every question a rung answers differently from BF16 lands in one of four records:
- Dead: BF16 answers it right and the rung answers it wrong. A rung's dead count is every such question at that rung.
- Tombstone: one per question, placed at the first rung, going down from BF16, where the question dies. A question dead at Q8_0 and again at Q4_0 has one tombstone, at Q8_0.
- Rose again: a lower rung answers a tombstoned question right once more. test-687 dies at F16 and is right again at Q8_0, Q6_K, Q5_K_M and Q4_K_M.
- Ghost: BF16 answers it wrong and the rung answers it right. test-1166 asks for an avocado tree's fruit over ten years, answer 2,350. BF16 says 1,000; Q8_0, Q6_K and Q4_0 get it right.
A rung's score is BF16's 229, minus its dead, plus its ghosts: Q4_0 is 229 − 15 + 8 = 222. Quantization moves answers in both directions, and a score that holds can hide questions lost and found, so the article reports both.
For Día de los Muertos: the dead are remembered one at a time, by name. Each tombstone here carries the question's id, the right answer, the wrong answer the rung gave, and every lower rung where it rose again.
What the project delivers:
- ladder.sh: BF16, F16 and seven quantized GGUFs of Gemma 4 E2B, with bytes and sha256
- graveyard.py run: one rung, its own llama-server, 250 questions, answers graded in code
- graveyard.py bury: deaths, tombstones and ghosts per rung, computed in code
- graveyard.py autopsy: how each rung's answers fail (cut off, looping, writing into the thinking block)
- graveyard.py repeat: two runs of one rung compared question by question
- one run of the whole ladder plus Google's QAT Q4_0 on a Tesla T4, committed with its report
- a second pass with thinking on, over seven rungs from BF16 to Q3_K_M
---
Demo
The Results in Brief
Q5_K_M is the best trade between speed and accuracy on the T4. It keeps BF16's score, 231 right against 229, at 39.0% of the bytes and 1.97 times BF16's decode speed. The 2 extra right answers come from the 8 questions where the two disagree, 5 ghosts against 3 dead; two equally accurate models split 8 questions at least that unevenly about three times in four, so the gap is within the noise. Q4_0 is the fastest rung at 2.34 times BF16, and costs 9 right answers against Q5_K_M for 1.19 times its speed. Q4_K_M sits between the two.
Every other rung is beaten on both counts by one of those three: Q8_0 and Q6_K are slower than Q5_K_M with fewer or equal right answers, and Q3_K_M is slower than Q4_0 with 31 fewer. Below Q4 the accuracy collapses: Q3_K_M loses 38 and Q2_K gets 3 right. With thinking on, the order holds: Q5_K_M 230, Q4_K_M 226 and Q4_0 219, against BF16's 227.
The cover is the graveyard. One row per rung, BF16 at the top and Q2_K at the bottom, one cell per question: marigold where the rung answers right, a grey headstone where it answers wrong. docs/make_cover.py draws it from the committed rows.
The ladder, from graveyard.py bury, on 250 GSM8K test questions:
| Rung | % of BF16 bytes | Right | Dead | Ghosts | Median tok/s |
| :--- | :--- | :--- | :--- | :--- | :--- |
| BF16 | 100.0 | 229 | 0 | 0 | 40.0 |
| F16 | 100.0 | 229 | 1 | 1 | 39.6 |
| Q8_0 | 53.3 | 227 | 4 | 2 | 64.3 |
| Q6_K | 41.3 | 229 | 4 | 4 | 70.8 |
| Q5_K_M | 39.0 | 🥇 231 | 3 | 5 | 78.7 |
| Q4_K_M | 36.8 | 224 | 9 | 4 | 84.1 |
| Q4_0 | 36.1 | 222 | 15 | 8 | 🥇 93.6 |
| Q3_K_M | 34.4 | 191 | 42 | 4 | 74.2 |
| Q2_K | 32.1 | 3 | 226 | 0 | 74.0 |
Dead: BF16 right, this rung wrong. Ghosts: BF16 wrong, this rung right.
---
Code
xbill9/gemma-graveyard
| Path | What |
| :--- | :--- |
| ladder.sh | Converts the checkpoint to BF16 and F16 GGUF, quantizes seven rungs from BF16, writes ladder.tsv |
| f16_range.py | Counts the weights that overflow or underflow when BF16 is cast to F16 |
| graveyard.py | run, bury, autopsy and repeat |
| docs/think_derived.py | Thinking on against thinking off, question by question |
| lazy_eval.py | Prompt, answer extraction and grading, reused from Week 3 |
| run.md | Every command for the T4, in order, and how to resume after the VM stops |
| runs/2026-10-27-ladder/run-ladder.sh | Runs the nine rungs in order, skips finished ones, then buries |
| runs/2026-10-27-ladder/ | REPORT.md, the bury and autopsy tables, per-question tombstones |
---
How I Built It
At This Point You Should Have…
- A GCE n1-standard-2 in us-west2-b with one Tesla T4, 15,360 MiB, driver 615.71.10, 2 vCPU and 7,436 MiB of RAM
- llama.cpp built with CUDA for compute capability 7.5, at b11699
- google/gemma-4-E2B-it from Hugging Face, the BF16 checkpoint, text model only
- google/gemma-4-E2B-it-qat-q4_0-gguf, Google's quantization-aware Q4_0 file
- A clone: git clone https://github.com/xbill9/gemma-graveyard
---
Step 1 — Build the Ladder
ladder.sh converts the checkpoint twice, to BF16 and to F16, then quantizes seven rungs from the BF16 file with plain llama-quantize, with no importance matrix. That is the default a user gets.
TMPDIR=/opt1/tmp ./ladder.sh /opt1/models/gemma-4-E2B-it /opt1/graveyard-gguf /opt1/llama.cpp
rung file bytes sha256
BF16 /opt1/graveyard-gguf/e2b-BF16.gguf 9311305152 9fe40af9a72302151a5abc28a5057c5fe335eb80145e795ba745085fc73a2ad6
F16 /opt1/graveyard-gguf/e2b-F16.gguf 9311305152 5ab927e4b24a248de319e70467fc74d4ab87347ee9d6fb3ee60b293b12198cf8
Q8_0 /opt1/graveyard-gguf/e2b-Q8_0.gguf 4967497152 6b0301104c89a43c3bec5bae4b14d744f6000bc21e00e46e887fbbfe024de38a
Q6_K /opt1/graveyard-gguf/e2b-Q6_K.gguf 3845346752 e800e0389f48646994d8ff6d3409dbf0c13b3fc217d0647b481385da33bd3d99
Q5_K_M /opt1/graveyard-gguf/e2b-Q5_K_M.gguf 3630288320 f6c3f4395a9d3839535d8c14ae48ab784096945cfc4485005853eede915cb4a9
Q4_K_M /opt1/graveyard-gguf/e2b-Q4_K_M.gguf 3427880384 dd9dc465bb535989205f158d77f28f6f8348e47d8ac8870920c152b5996cbc16
Q4_0 /opt1/graveyard-gguf/e2b-Q4_0.gguf 3360161216 2d84758474be87d0a74edfa98efbeb602484e26f2e5fa8695fd9641ad323c7b2
Q3_K_M /opt1/graveyard-gguf/e2b-Q3_K_M.gguf 3201351104 b10c1f38c93635ccee1c7f382e03f5d2f4e9516087d7d62a73ca7de0f52693ab
Q2_K /opt1/graveyard-gguf/e2b-Q2_K.gguf 2989088192 65850f1a31efed2013ed148adc5dc937e973a8e79fd55cdba31edd53369fddb2
The reference is BF16 because that is the checkpoint's own dtype, so the BF16 GGUF holds the trained weights unchanged. F16 is a rung.
---
Step 2 — Check What F16 Costs Before Running It
F16 has more mantissa than BF16 and less exponent range, so the cast can overflow or round small weights to zero. f16_range.py counts both on the checkpoint, one line per tensor and a total:
python3 f16_range.py /opt1/models/gemma-4-E2B-it/model.safetensors > runs/2026-10-27-f16-range/out.txt
grep -E 'position_embedding_table|TOTAL' runs/2026-10-27-f16-range/out.txt
{"tensor": "model.vision_tower.patch_embedder.position_embedding_table", "n": 15728640, "overflow": 0, "underflow": 5507733, "subnormal": 834716, "absmax": 0.431640625}
{"tensor": "TOTAL", "tensors": 2011, "n": 5123178979, "overflow": 0, "underflow": 5523265, "subnormal": 7013885, "absmax": 900.0}
No weight overflows. Of the 5,523,265 that underflow, 5,507,733 are in the vision tower's position table, which the text GGUF does not carry.
---
Step 3 — Ask One Rung 250 Questions
graveyard.py run starts its own llama-server on one GGUF and asks each question once: temperature 0, one request at a time, no prompt cache, thinking off, 1,024 max tokens. Only the reply's content is graded, in code: the number after Answer:, compared with the reference.
./graveyard.py run BF16 /opt1/graveyard-gguf/e2b-BF16.gguf data/heldout.jsonl runs/2026-10-27-ladder --llama /opt1/llama.cpp
{"rung": "BF16", "n": 250, "right": 229, "errors": 0, "truncated": 4, "median_decode_tps": 40.0, "mean_tokens": 391.2}
The whole model fits on the T4 at BF16:
load_tensors: offloaded 36/36 layers to GPU
load_tensors: CPU_Mapped model buffer size = 768.00 MiB
load_tensors: CUDA0 model buffer size = 4384.87 MiB
🔎 Tip: Gemma 4 E2B keeps its embedding tables in CPU memory. The per-layer embedding table holds 2,348,810,240 of the checkpoint's weights, and llama.cpp leaves it memory-mapped on the CPU with -ngl 99. So each rung's GPU memory is about half its file size: 4,671 MiB in use at BF16 for a 9.3 GB file, 1,639 MiB at Q4_0, 1,287 MiB at Q2_K.
---
Step 4 — Prove the Same Rung Gives the Same Answers
A tombstone means something only if the same model on the same question always gives the same answer. BF16 on 50 questions, twice:
for i in 1 2; do
./graveyard.py run BF16 /opt1/graveyard-gguf/e2b-BF16.gguf data/heldout.jsonl runs/2026-10-27-determinism-$i \
--llama /opt1/llama.cpp --limit 50
done
The two runs compared in code, question by question:
0 mismatched extracted, 0 mismatched text, 46/50 both times
Every answer and every character of text matched. The same holds below BF16. graveyard.py repeat compares a second full run of four rungs with the ladder's:
./graveyard.py repeat runs/2026-10-27-ladder runs/2026-10-28-repeat Q4_0 Q3_K_M Q8_0 F16 --common
{"rung": "Q4_0", "n": 250, "mismatched_extracted": 0, "mismatched_text": 0, "mismatched_right": 0, "right_a": 222, "right_b": 222}
{"rung": "Q3_K_M", "n": 250, "mismatched_extracted": 0, "mismatched_text": 0, "mismatched_right": 0, "right_a": 191, "right_b": 191}
{"rung": "Q8_0", "n": 250, "mismatched_extracted": 0, "mismatched_text": 0, "mismatched_right": 0, "right_a": 227, "right_b": 227}
{"rung": "F16", "n": 250, "mismatched_extracted": 0, "mismatched_text": 0, "mismatched_right": 0, "right_a": 229, "right_b": 229}
A changed answer at a lower rung comes from the weights.
---
Step 5 — Run the Ladder and Bury It
runs/2026-10-27-ladder/run-ladder.sh runs the nine rungs in order and skips any rung whose results are complete, so it picks up again after the VM's evening stop. Then bury:
./graveyard.py bury runs/2026-10-27-ladder BF16 F16 Q8_0 Q6_K Q5_K_M Q4_K_M Q4_0 Q3_K_M Q2_K --json graveyard-ladder.json
{"first_deaths_by_rung": {"Q2_K": 177, "Q3_K_M": 26, "Q4_0": 9, "Q8_0": 4, "Q5_K_M": 2, "Q4_K_M": 6, "Q6_K": 1, "F16": 1}, "tombstones": 226, "ghosts": 11}
226 tombstones and 11 ghosts: 226 different questions die at some rung, and 11 questions BF16 gets wrong come back right at one rung or more. The first-death counts add up to the tombstones, one per question; the per-rung dead counts in the ladder table add up to more, because one question can be dead at several rungs. Per-question detail is in graveyard-ladder.json. The one question that dies at F16:
{'id': 'test-687', 'died_at': 'F16', 'answer': '13', 'said': 17.0, 'right_again_at': ['Q8_0', 'Q6_K', 'Q5_K_M', 'Q4_K_M']}
"One twin is 7 years older than one triplet. If their combined ages are 44, how old is one of the twins?" BF16 answers 13, from two twins and three triplets. F16 adds up two twins and one triplet, and answers 17. Four lower rungs answer 13 again.
---
Step 6 — Autopsy: How Each Rung Dies
graveyard.py autopsy counts, per rung, answers cut off at 1,024 tokens, answers that repeat one line five or more times, and answers that put text in the thinking block:
./graveyard.py autopsy runs/2026-10-27-ladder BF16 F16 Q8_0 Q6_K Q5_K_M Q4_K_M Q4_0 Q3_K_M Q2_K
{"rung": "Q3_K_M", "cut_off": 49, "looped": 0, "thought": 0, "empty_content": 0, "empty_right_in_reasoning": 0, "dead": 42, "dead_cut_off": 34, "dead_looped": 0, "dead_empty_content": 0, "first_deaths": 26, "first_deaths_right_again_lower": 0}
{"rung": "Q2_K", "cut_off": 246, "looped": 228, "thought": 75, "empty_content": 63, "empty_right_in_reasoning": 2, "dead": 226, "dead_cut_off": 222, "dead_looped": 207, "dead_empty_content": 60, "first_deaths": 177, "first_deaths_right_again_lower": 0}
Every rung:
| Rung | Cut off | Looped | Thinking unprompted | Dead | Dead, cut off |
| :--- | :--- | :--- | :--- | :--- | :--- |
| F16 | 4 | 0 | 0 | 1 | 0 |
| Q8_0 | 9 | 0 | 0 | 4 | 2 |
| Q6_K | 4 | 0 | 0 | 4 | 1 |
| Q5_K_M | 3 | 0 | 0 | 3 | 1 |
| Q4_K_M | 6 | 0 | 0 | 9 | 3 |
| Q4_0 | 10 | 0 | 0 | 15 | 8 |
| Q3_K_M | 49 | 0 | 0 | 42 | 34 |
| Q2_K | 246 | 228 | 75 | 226 | 222 |
Three kinds of death:
- F16 to Q4_0, a different road through the same problem. 23 first deaths; 16 of them are right again at some lower rung. Down to Q5_K_M most of the dead answers end normally with a wrong number: 4 of 12 were cut off. At Q4_K_M and Q4_0, 11 of 24 were. From F16 to Q5_K_M, ghosts (1 to 5 per rung) match deaths (1 to 4), so the score holds. From Q4_K_M down, deaths outnumber ghosts, 9 to 4 and 15 to 8.
- Q3_K_M, out of room. 34 of its 42 deaths were cut off at 1,024 tokens, still working, with no repeated lines. BF16 used a median 466.5 tokens on the 44 questions Q3_K_M got wrong while cut off.
- Q2_K, collapse. 246 of 250 answers run to the limit and 228 repeat a line five or more times. With thinking off, 75 answers open a thinking block anyway, and in 63 of them the whole answer lands there, leaving content empty. No other rung writes into the thinking block at all.
🔎 Tip: give two buries in one directory two names. bury writes its per-question tombstones to graveyard.json; --json picks the file, so the QAT comparison in Step 7 keeps the ladder's file intact.
---
Step 7 — Google's QAT Q4_0 Beside the Plain Q4_0
Google publishes a Q4_0 GGUF of Gemma 4 E2B trained for 4-bit (quantization-aware training). Its weights differ from the base checkpoint, so it sits beside the ladder, against the same BF16 reference:
./graveyard.py bury runs/2026-10-27-ladder BF16 QAT_Q4_0 Q4_0 --json graveyard-qat.json
{"rung": "QAT_Q4_0", "n": 250, "right": 221, "errors": 0, "truncated": 6, "median_decode_tps": 87.8, "mean_tokens": 391.9, "dead": 15, "ghosts": 7, "answer_changed": 27, "bytes": 3349516256, "gpu_mib_loaded": 1629, "gpu_mib_before": 0}
{"rung": "Q4_0", "n": 250, "right": 222, "errors": 0, "truncated": 10, "median_decode_tps": 93.6, "mean_tokens": 430.4, "dead": 15, "ghosts": 8, "answer_changed": 27, "bytes": 3360161216, "gpu_mib_loaded": 1639, "gpu_mib_before": 0}
Same number of deaths, same number of changed answers, one right answer apart. The graves differ: 6 of the 15 dead are the same question, 9 die only under QAT and 9 only under the plain Q4_0. QAT runs out of room less often: 2 of its deaths were cut off at 1,024 tokens, against 8 for the plain Q4_0.
---
Step 8 — Bury the Ladder Again With Thinking On
Gemma 4 can think before it answers. --think turns it on; the thinking goes to reasoning_content and only content is graded. Thinking needs room, so this pass allows 4,096 tokens in an 8,192-token context. It covers seven rungs, BF16 to Q3_K_M; F16, Q2_K and QAT Q4_0 were not run with thinking on. Each rung is buried against BF16 with thinking on, which gets 227 right. A second BF16 run of 50 questions matched the first exactly, 47 right both times.
./graveyard.py bury runs/2026-10-28-think BF16 Q8_0 Q6_K Q5_K_M Q4_K_M Q4_0 Q3_K_M --json graveyard.json
{"rung": "BF16", "n": 250, "right": 227, "errors": 0, "truncated": 0, "median_decode_tps": 34.5, "mean_tokens": 842.2, "dead": 0, "ghosts": 0, "answer_changed": 0, "bytes": 9311305152, "gpu_mib_loaded": 4699, "gpu_mib_before": 0}
{"rung": "Q3_K_M", "n": 250, "right": 208, "errors": 0, "truncated": 11, "median_decode_tps": 68.0, "mean_tokens": 1320.1, "dead": 26, "ghosts": 7, "answer_changed": 42, "bytes": 3201351104, "gpu_mib_loaded": 1517, "gpu_mib_before": 0}
docs/think_derived.py lines the two passes up question by question:
python3 docs/think_derived.py | tail -3
"q3_dead_cut_off_with_thinking_off": 34,
"of_them_right_with_thinking_on": 21
}
| Rung | Right, thinking off | Right, thinking on | Dead, off | Dead, on | Dead in both |
| :--- | :--- | :--- | :--- | :--- | :--- |
| BF16 | 229 | 227 | — | — | — |
| Q8_0 | 227 | 🥇 230 | 4 | 2 | 0 |
| Q6_K | 229 | 🥇 230 | 4 | 6 | 0 |
| Q5_K_M | 231 | 🥇 230 | 3 | 6 | 0 |
| Q4_K_M | 224 | 226 | 9 | 9 | 1 |
| Q4_0 | 222 | 219 | 15 | 15 | 3 |
| Q3_K_M | 191 | 208 | 42 | 26 | 12 |
Three things change:
- Q3_K_M recovers most of its loss. It gains 17 right answers. Of the 34 questions it lost by running out of room at 1,024 tokens, 21 are right with thinking on and 4,096 tokens. Both the thinking and the larger limit changed at once, so this run cannot say which one did it.
- Above Q3_K_M the score holds and the graves move. Each rung stays within 3 answers of its thinking-off score, but the questions it loses are almost all new: 0 of the deaths at Q8_0, Q6_K and Q5_K_M are shared between the two passes, 1 of 9 at Q4_K_M and 3 of 15 at Q4_0. BF16 itself gets 218 questions right in both modes.
- The cost is time. Answers use 2.08 to 2.18 times as many tokens, decoding runs at 0.79 to 0.93 of the thinking-off speed, and each rung takes 2.22 to 2.70 times as long on the T4.
---
Compare and Contrast
| | Q5_K_M | Q4_K_M | Q4_0 | QAT Q4_0 | Q3_K_M |
| :--- | :--- | :--- | :--- | :--- | :--- |
| Right of 250 (BF16: 229) | 🥇 231 | 🥈 224 | 🥉 222 | 221 | 191 |
| Dead | 🥇 3 | 🥈 9 | 🥉 15 | 🥉 15 | 42 |
| GPU MiB | 1,897 | 1,705 | 🥉 1,639 | 🥈 1,629 | 🥇 1,489 |
| Median tok/s on the T4 | 78.7 | 🥉 84.1 | 🥇 93.6 | 🥈 87.8 | 74.2 |
| Answers cut off | 🥇 3 | 🥈 6 | 10 | 🥈 6 | 49 |
| Verdict | 🟢 same score as BF16 | 🟢 small loss | ⚠️ 15 dead | ⚠️ 15 dead, fewer cut off | ❌ 38 fewer right |
---
So, Which One?
To ship Gemma 4 E2B on a small GPU with BF16's score, take Q5_K_M: 39.0% of the bytes, 231 right against 229, and 1.97 times BF16's decode speed on the T4 (arithmetic from the medians). For the fastest file that stays within 7 answers, take Q4_0 or Q4_K_M. Below Q4, the losses stop being a few flipped answers: Q3_K_M saves 150 MiB of GPU memory over Q4_0 and costs 31 right answers, and Q2_K stops answering. With thinking on and 4,096 tokens, Q3_K_M gets 208 right, 11 short of Q4_0, at more than twice the time per question.
---
Prior Work
- Prompt, answer extraction and grading (lazy_eval.py) and the 250-question GSM8K split (data/, seed 0): from my Week 3 entry, lazy-gemma, whose grader came from my earlier Gemma 4 TPU work.
- Open source used: llama.cpp (MIT) for conversion, quantization and serving; Gemma 4 E2B and Google's QAT Q4_0 GGUF (Gemma terms); GSM8K.
---
Why Does Open Innovation Matter?
A graveyard needs the same model at nine sizes, run the same way.
- The weights are on the disk, so the ladder is built here. ladder.sh produces every rung from one BF16 file with the open converter and quantizer, and ladder.tsv records each file's sha256. A hosted API serves whichever quantization its provider chose, and it can change without notice.
- The same answer twice. Greedy decoding on a fixed llama.cpp build gave identical text on every repeated run, BF16 to Q3_K_M, so every tombstone counts. That check needs control of the server, the seed and the build.
- The failure is visible. Q2_K's answers in the thinking block show up only because llama-server returns reasoning_content and content separately and every row is kept.
- No per-token bill. Ten models times 250 questions ran on one rented T4.
---
The Spirit of It
Every row in the graveyard is a grade-school maths problem, and each tombstone keeps its own epitaph:
- test-687, twins and triplets, answer 13. Died at F16, said 17. Rose again at Q8_0, Q6_K, Q5_K_M and Q4_K_M.
- test-2, a house flipped for profit, answer 70,000. Died at Q4_0, said 120,000. Never rose.
- test-564, Judy's two weeks of reading, answer 240. Died at Q8_0, said 3. Rose again at every rung from Q6_K to Q4_0.
Most graves above Q3_K_M have someone come back to them: 16 of 23.
---
Prize Categories
Gemma: Gemma 4 E2B, quantized locally with llama.cpp at nine levels and compared with Google's quantization-aware Q4_0 release, served on one Tesla T4.
---
Summary
The goal of this project was to find which answers Gemma 4 E2B loses at each quantization level, one question at a time. The key was a deterministic run: greedy decoding on one llama.cpp build, checked to give identical text twice at five rungs, so every changed answer comes from the weights. The results were:
- 🟢 Q8_0, Q6_K and Q5_K_M stay within 2 right answers of BF16's 229 of 250; Q5_K_M scores 231 at 39.0% of the bytes.
- 🟢 Every rung fits on the T4, BF16 included: all 36 layers on the GPU, 4,671 MiB in use, with the embedding tables in CPU memory.
- 🟢 Above Q3_K_M, 16 of the 23 questions that die are right again at a lower rung.
- ⚠️ Q4_K_M and Q4_0 lose 5 and 7, with 9 and 15 deaths.
- ⚠️ Google's QAT Q4_0 scores 221 against the plain Q4_0's 222; the two share 6 of their 15 deaths.
- 🟢 Q4_0, Q3_K_M, Q8_0 and F16 run a second time give the same text on all 250 questions.
- ⚠️ With thinking on, rungs above Q3_K_M keep their score but lose different questions; at Q4_0, 3 of 15 deaths are shared.
- ⚠️ Thinking on costs 2.1 times the tokens and 2.2 to 2.7 times the time per rung.
- ❌ Q3_K_M loses 38; 34 of its 42 deaths are cut off at 1,024 tokens. With thinking on and 4,096 tokens it loses 19.
- ❌ Q2_K gets 3 right: 246 answers hit the token limit, 228 loop, and 75 write into a thinking block with thinking off.
Scope: one Tesla T4 on GCE, llama.cpp b11699, 250 GSM8K test questions per rung, greedy. Thinking off with a 1,024-token limit on all ten models; thinking on with a 4,096-token limit on seven rungs, BF16 to Q3_K_M. Determinism checked at BF16, F16, Q8_0, Q4_0 and Q3_K_M. No importance matrix, none of the other 1,069 GSM8K test questions, and no other machine. Speeds compare rungs on this T4 only.
The strategy for burying Gemma 4's answers one quantization rung at a time was validated with an incremental step by step approach.
---
References
- Hacktoberfest Open-Source AI Challenge, Week 4: Ghost in the Machine
- gemma-graveyard on GitHub
- Run report: runs/2026-10-27-ladder/REPORT.md
- lazy-gemma, the Week 3 entry
- llama.cpp
- google/gemma-4-E2B-it
- google/gemma-4-E2B-it-qat-q4_0-gguf
- GSM8K
8 reactions · 3 comments
Mentions
【Codex】【VSCode】Remote SSH切断後に「別のアプリで開いています」と表示される問題の解決方法 VS CodeのRemote SSH環境でCodexを使用していたところ、SSH接続が切れた後、既存の会話を開けなくなる問題が発生しました。 VS CodeやローカルPCの再起動、Codex拡張機能のバージョン変更などを試しましたが、改善しませんでした。 ! 今回の環境では、リモートLinux上に残っていた古いCodexプロセスを終了することで、会話履歴を削除せずに復旧できました。 VS CodeやPCを再起動しても改善しない場合は、リモート側に残ったプロセスと、会話のロックを確認します。 本記事では、発生した問題と考えられる原因、実際に行った解決手順を紹介します。 発生した問題 使用環境 エディタ：VS Code 接続方法：VS CodeのRemote - SSH拡張機能 リモート側のOS：Linux 使用した拡張機能：OpenAIのCodex拡張機能 Remote SSH経由でCodexを使用していたところ、会話の途中でリモート接続が切れました。 その後、SSHで再接続し、Codexで既存の会話を開こうとしたところ、以下のメッセージが表示され、会話を再開できませんでした。 メッセージを読み込めませんでした 別のアプリで開いています ここで続けるには、あちらで閉じてください。 (一部画像はAIを用いて修正) しかし、切断前の会話画面には戻れず、メッセージの指示どおりに元の画面で会話を閉じることもできませんでした。 試したが解決しなかった方法 まず、以下の一般的な対処方法を試しました。 VS Codeの再起動 PCの再起動 Remote-SSH: Kill VS Code Server on Host... によるリモート側VS Code Serverの終了 Codex拡張機能のバージョン変更 しかし、いずれの方法でも改善しませんでした。 もちろん、これらの操作によって解決するケースもあります。ただし、今回のように解決しない場合は、リモートサーバー上にCodexのプロセスが残存していないかを確認する必要があります。 考えられる原因：古いCodexプロセスによる会話のロック 今回の問題は、SSH接続が切れた後もリモートサーバー上に古いCodexプロセスが残り、会話のロック（writer lock）を保持していたことが原因と考えられます。 Codexでは、同じ会話に複数のプロセスが同時に書き込まないよう、writer lockによる排他制御が行われています。 そのため、古いプロセスがロックを保持したままだと、新しく起動したCodexから既存の会話を開けなくなる場合があります。 同様の問題は、CodexのGitHubリポジトリでも報告されています。 GitHub Issue #41849：Remote SSH再接続後に古いCodexプロセスがwriter lockを保持する問題 解決方法 ! 以下のコマンドは、Remote SSH接続先のLinuxターミナルで実行します。 復旧までの流れは、次の4ステップです。 Codexプロセスを確認する writer lockを保持しているプロセスを特定する 不要になった古いCodexプロセスを終了する VS Codeで会話を開き直す Step 1：Codexプロセスを確認する まず、Codex関連のプロセスが現在どの程度起動しているかを確認します。 Codex関連プロセスの確認 ps -u "$USER" -o pid,ppid,etime,args | grep -Ei '[c]odex|[c]hatgpt|[e]xtensionHost' このコマンドでは、Codex関連のプロセスについて、PID（プロセスID）、PPID（親プロセスID）、起動からの経過時間、実行コマンドを確認できます。 今回の環境では、以下のような結果が得られました。 実行結果（抜粋・整形） PID ELAPSED COMMAND 1204484 6-04:53:18 codex app-server 2875105 11:20:31 codex app-server 3419599 03:29 codex app-server ELAPSED はプロセスが起動してからの経過時間を示しています。 今回の場合、約6日前、約11時間前、約3分前から動作しているCodexプロセスが同時に存在していました。 特に、古いプロセスと新しく起動したプロセスが同時に残っていることが気になります。 ただし、起動時間が古いからといって不要なプロセスとは限りません。次の手順で、実際に会話のロックを保持しているプロセスを確認します。 Step 2：writer lockを保持しているプロセスを特定する 次に、lsofコマンドを使用して、Codexのロックファイルを開いているプロセスを確認します。 writer lockを保持しているプロセスの確認 lsof -nP +D "$HOME/.codex/thread-writer-locks" lsof は、実行中のプロセスが開いているファイルを調べるためのコマンドです。 Codexのwriter lockに関するファイルは、今回の環境では ~/.codex/thread-writer-locks/ に存在していました。 実行すると、以下のような結果が得られました。 実行結果（抜粋・整形） COMMAND PID FD NAME codex 1204484 17uW .../thread-writer-locks/xxx.lock codex 1204484 41uW .../thread-writer-locks/xxx.lock codex 1204484 48uW .../thread-writer-locks/xxx.lock codex 1204484 71uW .../thread-writer-locks/xxx.lock codex 2875105 45uW .../thread-writer-locks/xxx.lock codex 2875105 62uW .../thread-writer-locks/xxx.lock ロックファイル名は省略しています。 FD 列にある uW のうち、u はファイルが読み書き可能な状態で開かれていること、W はファイル全体への書き込みロックを保持していることを示しています。 この結果から、PID 1204484 と 2875105 の2つのCodexプロセスが、複数のwriter lockを保持していることがわかりました。 一方、Step 1で確認した新しいCodexプロセス（PID 3419599）は、この出力には含まれていません。 さらに、次のコマンドでプロセスの親子関係を確認します。 pstreeによる親子関係の確認 pstree -sp 2875105 pstree は、プロセスの親子関係をツリー形式で表示するコマンドです。PIDは自分の環境で確認したものに置き換えてください。 今回の環境では、プロセスの親子関係などを確認した結果、PID 2875105 は古いVS Codeの拡張機能から起動されたCodexの app-server と判断しました。 そこで、この古いプロセスが問題の原因になっている可能性を考えました。 ! 複数のロックが確認できても、それだけで問題の会話に対応するロックを特定できたとは限りません。PIDや親プロセス、ほかのCodexセッションの稼働状況を合わせて確認することが重要です。 補足：lsofでFUSE関連の警告が表示される場合 lsofの実行時に、FUSE関連の警告が表示される場合があります。今回も警告が表示されましたが、ロックファイルに関する情報は取得できていました。 Step 3：古いCodexプロセスを終了する 今回の環境では、PID 2875105 が古いVS Codeの拡張機能から起動されたプロセスであることが確認できました。 そのため、ほかの作業で使用していないことを確認したうえで、以下のコマンドを実行しました。 ! 以下のPIDは今回の環境での例です。必ず自分の環境で確認したPIDに置き換えてください。 別のCodexセッションで実行中のタスクがある場合、そのプロセスを終了すると処理が中断される可能性があります。 対象のプロセスに終了を要求（PIDは置き換える） kill -TERM 2875105 kill -TERM は、指定したプロセスに終了要求のシグナル（SIGTERM）を送信するコマンドです。 実際の終了処理は、プロセス側の実装によって異なります。 終了後、必要に応じて以下のコマンドでロックの状態を再確認できます。 終了後のロック状態の再確認 lsof -nP +D "$HOME/.codex/thread-writer-locks" Step 4：VS Codeで会話を開き直す 古いプロセスを終了した後、Codexの既存の会話を開き直します。 画面が更新されない場合は、VS Codeのコマンドパレットから以下を実行します。 コマンドパレットで実行 Developer: Reload Window コマンドパレットは、次のショートカットで開けます。 OS ショートカット macOS Cmd + Shift + P Windows / Linux Ctrl + Shift + P VS Codeの再起動でも構いません。 今回の環境では、古いCodexプロセスを終了することでエラーが解消し、以前の会話を正常に再開できました。 会話履歴や設定を削除する必要もありませんでした。 備考・注意点 なぜVS CodeやPCの再起動では解決しなかったのか？ 今回の問題では、ローカルPCの再起動だけでなく、Remote SSHのVS Code Serverを終了させても改善しませんでした。 これは、リモートサーバー上のCodexプロセスが残存していたためと考えられます。 ローカルPCを再起動しても、SSH接続先で実行されているプロセスまで終了するとは限りません。 また、Remote-SSH: Kill VS Code Server on Host... はVS Code Serverの終了に用いるコマンドですが、今回、この操作でも改善しませんでした。Codexプロセスが残った詳しい経緯は確認できていません。 一方、以下のような原因であれば、一般的な再起動操作で解決できる場合もあります。 対処方法 改善が期待できるケース VS Codeの再起動 エディタや拡張機能の一時的な不具合 PCの再起動 ローカル側のプロセスや接続状態の不具合 Kill VS Code Server on Host リモート側のVS Code Serverの異常 Codex拡張機能の更新・変更 特定バージョンに起因する不具合 ただし、今回のようにリモートLinux上の古いCodexプロセスがwriter lockを保持している場合、これらの操作だけでは解決しないことがあります。 その場合は、問題を起こしているCodexプロセスを直接特定することが重要です。 プロセスの終了やロックファイルの削除には注意 ! 復旧作業の際は、以下の点に注意してください。 ~/.codex ディレクトリ全体を削除しない。会話履歴や設定を失う可能性があります。 pkill -9 codex などで、すべてのCodexプロセスを無差別に強制終了しない。 複数のCodexセッションを使用している場合は、終了対象のプロセスがほかの作業に使用されていないか確認する。 ロックファイルを安易に削除せず、まずはロックを保持しているプロセスを調べる。 今回のケースでは、古いCodexプロセスを1つ終了するだけで復旧できました。 まとめ VS CodeのRemote SSH環境でCodexを使用している際、SSH接続が切れると、古いCodexプロセスが残存して会話のwriter lockを保持し続ける場合があります。 VS CodeやPCの再起動でも改善しない場合は、以下の手順を試してみてください。 ps でCodex関連プロセスを確認する lsof でwriter lockを保持しているプロセスを調べる 不要になった古いCodexプロセスを終了する VS Codeで既存の会話を開き直す 今回はこの方法で、会話履歴を失うことなくCodexを復旧できました。 同じ問題で困っている方の参考になれば幸いです。 参考資料 openai/codex — GitHub Issue #41849 Visual Studio Code — Remote Development Tips and Tricks
if you are having a bad day, remember that meat proxies for AIs exist: codex#59060 "Told to reboot a server, it destroyed systemd, destroyed containers, ran random scripts, cost me $10,000+" pure gol- err platinum 🥇
Claude CodeからPremiere Proを290回動かして、53分のPodcastを全自動編集してみた 普段は Premiere Pro と After Effects で動画を編集しています。社内で始めたポッドキャスト「Locked in」の第1回（約53分）を、今回は Claude Code に編集させてみました。 素材は iPhone で撮った 4K 映像が3アングル分（合計約36GB）と、マイク2本の WAV です。そこから YouTube 用の本編1本（4K）と、縦型ショート20本を作りました。 役割はこう分けました。 Claude Code：素材の同期、文字起こし、カット、字幕、テロップ作り、配置、書き出し 僕：ショートを1本ずつ見て直しを頼む、タイトルを選ぶ、色（Lumetri）の数値を決める Premiere の画面で僕が直接いじったのは、色の数値だけです。直しの依頼は、ほぼ音声入力で出していました。 やり方は、届く所はスクリプトで動かし、届かない所だけ画面操作に任せる、という分け方です。今回はスクリプトで動かした回数が約340回、画面操作に頼ったのは6回ほどでした。動画編集に限らず、スクリプトで操作できる GUI アプリなら、同じ構成を試せると思います。 ! この記事のコードは macOS、Premiere Pro 2026、After Effects 2026 で動かしたものです。Windows では After Effects の呼び出し方などが変わります。 完成したショートの一部 処理の流れ ①〜③ は Mac の上で完結しています。④〜⑦ は Claude Code が Adobe のアプリにスクリプトを送って動かしました。スクリプトで操作できないところだけ、Codex の Computer Use に画面操作を頼んでいます。 スクリプトの内訳は、Premiere が約290回、After Effects が約50回です。 MCP サーバーを使わなかった理由 Premiere 用・After Effects 用の MCP サーバーは、GitHub にいくつかあります（hetpatel-11/Adobe_Premiere_Pro_MCP、Dakkshin/after-effects-mcp など）。どれも中では CEP 経由で ExtendScript を呼んでいます。 ただ、実装によっては使いたい操作がツールになっていませんでした。.mogrt の配置がその一例です。任意のスクリプトを実行できるサーバーもありますが、それなら結局スクリプトを書くことになります。そこで今回は MCP サーバーを使わず、自作の小さな CEP 拡張に JSX を直接送る構成にしました。 今回必要だった操作は、ほとんど ExtendScript で書けました。シーケンスの複製、クリップの配置・トリム・移動、.mogrt の文字の差し替え、Lumetri の値の読み書き、書き出しの依頼です。コマの書き出しなど一部は、非公開の QE DOM を使っています。 画面を毎回読み取る方式にしなかったのは、量が多かったからです。字幕は本編だけで約860枚あります。スクリプトなら200枚ずつまとめて置けて、結果は placed 200 NG 0 のような1行で返ってきます。画面の確認が必要なときだけ、コマを書き出して見せました。 ExtendScript は古い仕組みで、Premiere の拡張は UXP（2025年12月に正式版）へ移りつつあります。ただ、Premiere のスクリプト資料には、ExtendScript は今も動き、廃止の予定は出ていないとあります。 Premiere 側：CEP ブリッジ Premiere の中に見えない CEP 拡張を1つ置き、その中で HTTP サーバーを 127.0.0.1 だけで立てています。届いた JSX を evalScript に渡し、戻り値を返すだけのものです。以下は実行部分の抜粋です。 index.html <script> const http = require('http'), fs = require('fs'), os = require('os'), path = require('path'); // 認証用のトークン。先に適当な文字列をこのファイルに保存しておく const token = fs.readFileSync(path.join(os.homedir(), '.claude-premiere-bridge-token'), 'utf8').trim(); const evalJsx = code => new Promise(res => window.adobe_cep.evalScript(code, res)); http.createServer((req, res) => { if (req.headers['authorization'] !== 'Bearer ' + token) { res.writeHead(401); return res.end(); } if (req.method === 'POST' && req.url === '/eval') { let body = ''; req.setEncoding('utf8'); req.on('data', c => body += c); req.on('end', async () => res.end(String(await evalJsx(body)))); return; } res.writeHead(404); res.end(); }).listen(47820, '127.0.0.1'); </script> ほかに必要なのは次のとおりです。 拡張のフォルダ（~/Library/Application Support/Adobe/CEP/extensions/ の下）と CSXS/manifest.xml manifest では CEF に --enable-nodejs と --mixed-context を渡す（パネルの中で require を使うため） 起動のきっかけを applicationActivate にし、AutoVisible を false にして画面には出さない 実際に使った manifest.xml CSXS/manifest.xml <?xml version="1.0" encoding="UTF-8"?> <ExtensionManifest Version="7.0" ExtensionBundleId="com.claude.premierebridge" ExtensionBundleVersion="1.0.0" ExtensionBundleName="Claude Bridge" xmlns:xsi="w3.org"> <ExtensionList> <Extension Id="com.claude.premierebridge.bg" Version="1.0.0"/> </ExtensionList> <ExecutionEnvironment> <HostList><Host Name="PPRO" Version="[22.0,99.9]"/></HostList> <LocaleList><Locale Code="All"/></LocaleList> <RequiredRuntimeList><RequiredRuntime Name="CSXS" Version="9.0"/></RequiredRuntimeList> </ExecutionEnvironment> <DispatchInfoList> <Extension Id="com.claude.premierebridge.bg"> <DispatchInfo> <Resources> <MainPath>./index.html</MainPath> <CEFCommandLine> <Parameter>--enable-nodejs</Parameter> <Parameter>--mixed-context</Parameter> </CEFCommandLine> </Resources> <Lifecycle> <AutoVisible>false</AutoVisible> <StartOn> <Event>com.adobe.csxs.events.ApplicationActivate</Event> <Event>applicationActivate</Event> </StartOn> </Lifecycle> <UI> <Type>Custom</Type> <Geometry><Size><Height>1</Height><Width>1</Width></Size></Geometry> </UI> </DispatchInfo> </Extension> </DispatchInfoList> </ExtensionManifest> 署名していない拡張を読み込ませるため、一度だけデバッグモードを有効にします（数字は CSXS のバージョンに合わせる）。 defaults write com.adobe.CSXS.12 PlayerDebugMode 1 呼び出し側は zsh のラッパーです。送るスクリプトを try/catch で包み、out() に渡した値をまとめて返します。 run_pr.sh #!/bin/zsh 使い方: zsh run_pr.sh sample.jsx T=$(cat ~/.claude-premiere-bridge-token) { print -r -- '(function(){var o=[];function out(x){o.push(String(x));} try{' cat "$1" print -r -- '}catch(e){out("ERROR line "+e.line+": "+e);} return __o.join(" | ");})()'; } > "${1%.jsx}_w.jsx" curl -s -H "Authorization: Bearer $T" -X POST --data-binary @"${1%.jsx}_w.jsx" 127.0.0.1 最初に試すなら、out(app.project.activeSequence.name) だけを送って、シーケンス名が返ってくるかを見るのがいいと思います。 After Effects 側：テンプレートを作る After Effects は、Mac なら AppleScript からスクリプトファイルを実行できます。 osascript -e 'tell application "Adobe After Effects 2026" to DoScriptFile "/path/to/build.jsx"' コンポを組んだら、Premiere 側で変えたいプロパティを登録し、.mogrt として書き出します。 textProp.addToMotionGraphicsTemplateAs(comp, "字幕"); // Premiere で編集できる項目にする app.beginSuppressDialogs(); try { comp.openInEssentialGraphics(); comp.exportAsMotionGraphicsTemplate(true, "/path/to/MOGRT/"); } finally { app.endSuppressDialogs(false); } beginSuppressDialogs() で囲んだのは、書き出しのたびに「このフォントは使用できません」という確認が出て、AE が止まったからです。Mac に入れてあるフォントなのに出る理由までは追えていませんが、今回はこれで止まらなくなりました。どうしても閉じられないエラー画面が一度だけあり、そのときは Codex に OK を押してもらっています。 もうひとつ詰まったのが、作り直しです。同じテンプレート名で .mogrt を書き出し直しても、Premiere には古い見た目のまま入ってきました。名前の末尾に v2、v3 と付けて入れ直したら、新しいほうが使われました。 字幕、名前テロップ、用語の解説、章の切り替え、オープニング、エンディングは、どれもこの方法で作っています。 章の切り替え（12章）。番号や話題は Premiere 側で差し替えている 名前テロップ。2人の板の幅はスライダーでそろえた Premiere に並べる .mogrt は importMGT で置きます。 ! 今回の環境では、importMGT で置いた直後のクリップがテンプレートの長さ（10秒）になり、その範囲にある既存のクリップを上書きしました。試すときは、複製したシーケンスか空いたトラックでやってください。 置いたあとは、getMGTComponent() のプロパティに文字を入れます。今回のテンプレートでは、字幕の値が JSON の文字列で取れたので、その中の textEditValue を書き換えて戻しています。 var clip = seq.importMGT(mogrtPath, String(ticks), videoTrackIndex, 0); clip.end = endTime; // 10秒から字幕の長さに縮める var props = clip.getMGTComponent().properties; for (var i = 0; i < props.numItems; i++) { if (props[i].displayName != "字幕") continue; var o = eval("(" + props[i].getValue() + ")"); // ExtendScript には JSON がない o.textEditValue = text; o.fontTextRunLength = [text.length]; // 字幕全体を1つの書式で扱う前提 props[i].setValue(toJson(o), true); // toJson はオブジェクトを文字列に戻す自作関数 } 上書きでは実際に事故がありました。仕上げの段階で、字幕が2秒ほど抜けている所に1枚足したら、後ろの4枚が消えたのです。空いている区間に置いてから move() で動かす方法も試しましたが、動かしたクリップは描画されませんでした。最後は、消えた4枚をその場で置き直しています。Claude Code が書き出したコマを見て気づき、作業前に複製しておいたシーケンスから文と時刻を拾って直しました。 本編のタイムライン。V3 の細かい帯が字幕、V2 が色を調整する調整レイヤー 確認と書き出し 字幕やレイアウトを直すたびに僕が再生していると、手が止まります。そこで、まず Claude Code 自身にコマを書き出させて確認させました。 app.enableQE(); qe.project.getActiveSequence().exportFramePNG("00:20:29:12", "/tmp/check.png"); // 実際には check.png.png という名前で保存される ! app.enableQE() で使える QE DOM は、Adobe が公開していない仕組みです。バージョンが変わると動かなくなるかもしれません。 コマで見ていたのは、字幕が出ているか、文字が顔にかかっていないか、テロップの位置です。音のつながりやカットの間は、静止画では分かりません。そこは僕が書き出した動画を見て、気になった所を伝えました。 書き出しは Media Encoder にキューで渡します。 app.encoder.launchEncoder(); app.encoder.encodeSequence(seq, outPath, presetPath, app.encoder.ENCODE_ENTIRE, 1); app.encoder.startBatch(); スクリプトで触れない操作は Codex に 調整レイヤーを新しく作る方法は、今回調べた範囲の ExtendScript と QE には見つかりませんでした。シーケンス設定の「HDR グラフィックスホワイト」も、ダイアログの中でしか変えられません。 こういう操作は、Claude Code がターミナルから Codex を呼んで、画面を操作させています。 codex exec --skip-git-repo-check "Use computer use on this Mac. Premiere Pro のプロジェクトパネルで 新規項目 > 調整レイヤー を作り、OK を押す。ほかは何もしない。" 使ったのは codex-cli 0.162.1 です。Computer Use の設定と、macOS の画面収録・アクセシビリティの許可は先に済ませてあります（Computer Use の説明）。操作が終わったら、ブリッジでプロジェクトの中身を読み、本当にできたかを確かめてから次へ進みます。 最初、Claude Code は「調整レイヤーを手で作ってください」と僕に頼んできました。「そういう時は Codex の Computer Use で追加してって言ってるよね？」と返して、それからは聞かずに Codex を呼ぶようになっています。 素材の処理（Python） 4K の素材を Premiere で再生して探すと重いので、切る位置は音と文字から決めさせました。 同期 音の包絡線どうしの相互相関を FFT で計算し、カメラとマイクの時間差を出しました。録音機の時計がリセットされていて、ファイルの日付が2020年になっていました。どれが今回の録音かも、音を突き合わせて見つけています。 話者の判定 最初はカメラの音から声紋（SpeechBrain）で判定していました。マイクの録音が届いてからは、2本のマイクの音量差で判定しています。今回の録音では、話している本人のマイクのほうが10dB以上大きく入っていました。声紋で出した結果とも、ほぼ一致しています。 文字起こし mlx-whisper は、上で出した話し手の交代位置で音声を区切ってから回しました。通しで起こした単語の時刻に話者の判定を重ねると、時刻のずれで語の途中に交代が入ってしまったからです。Whisper は無音の区間で「ご視聴ありがとうございました」と出してくることがあります。そこで turbo と large-v3 の2つで起こし、違う箇所を Claude Code に突き合わせて清書させました。怪しい所は、その区間だけマイクごとに起こし直して確かめています。 改行 BudouX で改行の候補に分け、1行14〜15字を目安に、行の長さの偏りが小さくなる組み合わせを動的計画法で選びました。 同じ瞬間の3アングル。左から、ゲスト側の縦、ホスト側の縦、2人の横 色まわりで詰まったこと 分割すると白っぽくなる 最初は、9GB の iPhone の映像を扱いやすくしようと、ffmpeg のストリームコピーで5分ずつに分けていました。ところが、分けたファイルを使うと、書き出しで肌が白く飛びました。ffprobe で見ると、元のファイルにある Dolby Vision の情報が、分けたファイルにはありません。Premiere での HDR から SDR への変換が変わったのは、このためだと考えています。分割はやめて、元のファイルとプロキシで編集しました。ちなみに、分けたファイルは AAC の音も Premiere で無音になっていました。 HDR シーケンスで字幕がくすむ iPhone の素材に合わせて、シーケンスを Rec.2100 HLG で作っていました。この状態だと、字幕の白が映像に対して暗く見えました。HDR グラフィックスホワイトを 300 nit にしたら直っています。最終的な書き出しは SDR（Rec.709）です。 色の数値 色はショート1本で僕が調整し、Claude Code にその Lumetri の値を全項目読み取らせて、残りのショートにコピーさせました。本編も、僕が数値を決めたあとは「ここから動かさないで」と伝えています。アングルごとの細かい色合わせはしていません。調整レイヤー1枚で全体にかけています。 指示の出し方 指示はほぼ全部、音声入力です。誤変換が混ざっていても、Claude Code はだいたい意図を拾ってくれました。 途中で何度かルールを足しています。特に何度も言ったのは次の3つです。 修正の前にシーケンスを複製しておく 僕が色を決めたあとは、その値を変えない 頼んでいないのに書き出さない 書き出しの件は、Claude Code が気を利かせて勝手に書き出しを始めたので、そのとき言いました。Premiere を触っている最中に書き出しが走ると、作業が止まるからです。 品質の基準は、ショートの最初の依頼で具体的に伝えました。「字幕が見やすい」「文末でカットを切り替える」「話している人の口元が映る」の3つです。「文の途中で終わらせない」も何度も言いました。これは最終的に、区間を決める処理の中にルールとして入っています。 あとは、見て引っかかった所をそのまま言いました。「終わり方が急」と言えば、最後の区間を文末まで延ばしてきます。 「改行の位置、微妙にきもいとこある」と言ったときは、こう変わりました。 左が直す前、右が直した後。同じ瞬間のコマ 直す前は、1行の文字数をそろえることを優先していたので、「志を／持って」のように意味の途中で切れていました。Claude Code はこの回の字幕を全部見直し、意味の切れ目で区切り直しています。1行15字以内という条件は変えていません。 ほかに変わった字幕の例 直す前 直した後 人たちを見て 自分も／そう思ってるから そういう 自分もそう思ってるから／そういう人たちを見ると そういう人たちに／届いて おっしゃって そういう人たちに届いて いただいたように この／ポッドキャストを見て感じて おっしゃっていただいたように／このポッドキャストを見て それからすごい大衆に／受けるように それから すごい大衆に／受けるようにしなきゃいけない 本編の「20:27 カットが不自然」は、映像のカットではありませんでした。相手の返事の間だけ字幕が2秒消えていたのが原因で、Claude Code はそこを見つけて字幕を足しています。 日程 10/27 夜：素材が届く前に、ブリッジの動作を確認し、テロップのデザインを始める 10/28：ショート20本。夜に全20本を書き出す 10/29 昼：本編。字幕、テロップ、エンディングを入れて書き出す 10/29 夜：見直しで3か所を直して、書き出し直す ブリッジの確認から最後の書き出しまで、約2日半でした。僕が実際に手を動かしていた時間は数えていません。 残った手作業 色の数値とタイトル選びは、自分でやったほうが早かったです。タイトルは Claude Code に案をいくつも出させ、その中から選ぶか、言い回しを変えて決めました。 もうひとつは、動画を通して見て仕上がりを判断することです。字幕の抜けのように Claude Code がコマから気づくこともありましたが、音のつながりやカットの間の違和感は、僕が動画を見て伝えています。直す作業のほうは、ほとんど任せられました。 参考 Premiere Pro スクリプトの資料：ppro-scripting.docsforadobe.dev After Effects スクリプトの資料：ae-scripting.docsforadobe.dev Premiere の UXP 正式版の発表：blog.developer.adobe.com Claude Code：code.claude.com Codex CLI：openai/codex Codex の Claude Code 用プラグイン：openai/codex-plugin-cc モーショングラフィックステンプレート：helpx.adobe.com
Told to reboot a server, it destroyed systemd, destroyed containers, ran random
Bro asked AI to reboot the server and it rebooted his entire bloodline. Source: codex#59060
Codex Was Torching Your SSD and Nobody Noticed for Months Your Codex log file looked fine. Your SSD counters did not. The file was small. That was the trick. If you opened <code>~/.codex/</code>, you could see a SQLite database called <code>logs_2.sqlite</code>. It was not huge. It was not swallowing the disk. Finder, Explorer, <code>du</code>, and every normal storage check had nothing dramatic to say. Meanwhile, one developer's SSD recorded roughly 37 TB of writes in 21 days. That works out to about 1.7 TB a day. If you stretch one heavy, continuous-use report across a full year, it becomes roughly 640 TB. The number is not a prediction for every Codex user. It came from one machine running for a long time. But it exposed a problem that is easy to miss with local developer tools: > a file can stay small while the storage device underneath it gets written to constantly. This is the story of a noisy TRACE logging sink, a SQLite database doing exactly what it was told, and the SSD wear that nobody could see from file size alone. The quick version - A user reported unusually high local disk writes and traced them to Codex CLI's <code>logs_2.sqlite</code> database and its WAL companion file. - The reported machine accumulated about 37 TB of writes over 21 days of uptime. - The file stayed bounded because new logs were inserted while older logs were deleted. - The reported cause was an overly verbose TRACE logging path that did not fully respect the expected <code>RUST_LOG</code> override. - The supplied investigation says version 0.142.0 reduced the noisy writes substantially, with early testing reporting around 85% fewer log writes. - If you used an older build heavily, update first, then inspect real write activity and your SSD's lifetime-write counters. The number that made people look twice 37 TB in 21 days is not a normal “developer tool wrote some logs” story. | Measurement | Value | | --- | --- | | Reported writes | ~37 TB | | Reporting window | 21 days | | Approximate daily rate | ~1.7 TB/day | | Annualized at that rate | ~640 TB/year | The annualized number needs context. It assumes the same workload every day for a year, which is rarely how people work. Someone running short sessions will produce far less. Someone leaving the tool open, working with long sessions, or using it in automation could produce much more than they expect. The useful point is not “every SSD will die in a year.” The useful point is that this level of write activity belongs in the category of things you should notice and control. <!-- Visual placement: use a clean SMART dashboard screenshot here, with all serial numbers and personally identifiable drive details blurred. --> Why the file size told you almost nothing Most of us check disk problems like this: du -sh ~/.codex ls -lh ~/.codex/logs_2.sqlite* Those commands answer one question: > How much data is stored right now? They do not answer this: > How much data has been written over time? The Codex log database was designed to retain a bounded amount of diagnostic information. New rows were written, old rows were removed, and the database could remain roughly the same size. So the folder looked calm while the disk was doing constant work. Imagine a café with four tables. Looking through the window at noon, you see four people sitting down. Come back at 6 PM, still four people. The table count tells you nothing about whether eight customers visited or eight hundred. The SQLite file was the same. Stable size, enormous turnover. !Diagram showing how a small SQLite log file can lead to high SSD write activity What was generating all those writes? The reported issue points to a logging path that defaulted to TRACE, the most detailed level in Rust's tracing ecosystem. TRACE logging is useful when you are trying to understand a very specific failure. It can include low-level events that are normally too noisy for everyday use: - file-open events - raw network or WebSocket activity - transport details - telemetry plumbing - dependency-library messages that get bridged into the application's tracing system The supplied breakdown estimates that TRACE-level entries made up 70.7% of the retained log-body size. The highest-volume targets included WebSocket handling, OpenTelemetry-related logs, a generic dependency log bridge, and client transport code. That is diagnostic exhaust. A little of it is fine. A default stream of it, persisted locally in SQLite, is a different kind of workload. The three parts that stacked together No single part of this design has to be bad on its own. 1. TRACE was active where it should have been opt-in TRACE is not a normal default for a tool used for hours at a time. It is the level you turn on when you are actively debugging. The reported behavior was that Codex initialized a feedback logging sink at TRACE. That turns every routine interaction into a lot of diagnostic detail. 2. RUST_LOG=warn did not fully quiet it Rust developers often reach for this first: RUST_LOG=warn codex That is supposed to turn down the volume. According to the issue analysis, it did not fully apply to the relevant sink. If a target did not have an explicit override, Codex's internal default still allowed noisy logging. That is worse than an obvious missing setting. A user can think they reduced logging while the disk keeps receiving the same workload. 3. SQLite turned event noise into storage churn SQLite is excellent. It is also not magic. For small structured local data, a SQLite database is a good choice. For a constant stream of fine-grained TRACE events, inserts, deletes, indexes, and checkpoints, it can create a lot of writes even when the database never becomes large. The problem was not “SQLite is bad.” The problem was using a write-heavy retention loop as a sink for a log level that should not have been permanently enabled. The part the operating system does not show you There are two kinds of amplification in this story. SQLite write amplification SQLite often uses WAL mode, short for write-ahead logging. In simplified form: 1. A log event is appended to the WAL file. 2. SQLite later checkpoints that change into the main database. 3. Older rows are deleted to keep retained data bounded. 4. Indexes and database pages are updated along the way. One logical log event can therefore create more than one write at the filesystem level. The WAL is not wrong. Checkpoints are not wrong. The problem appears when that loop is fed relentlessly with small, noisy events. SSD write amplification An SSD has its own internal life. It writes data in small pages but erases much larger blocks. When a small part of a block changes, the SSD controller often writes the new data to a fresh location, marks old data stale, later moves still-valid pages, and erases the old block. That movement is called garbage collection. Wear leveling moves writes around so one physical area does not wear out first. Both are useful. Both mean the flash may receive more physical writes than the operating system asked for. The ratio is called the write amplification factor. The exact number varies by drive, spare capacity, free space, workload, and controller. The supplied analysis does not measure the multiplier for the affected drive, so do not treat any guessed physical-wear number as a precise fact. The direction is enough: > High-churn, small random writes are one of the workloads SSDs work hardest to manage. Is 640 TB actually dangerous? It can be. SSDs have an endurance rating usually expressed as TBW, or terabytes written. A typical 1 TB consumer SSD may be rated for a few hundred TBW. Enterprise drives may be rated much higher. The exact number depends on the model, capacity, NAND type, and warranty. That does not mean a drive fails the moment it reaches its rated TBW. Ratings are warranty and endurance guidance, not a timer that hits zero at midnight. Plenty of drives outlive them. But a background development tool potentially consuming hundreds of terabytes of writes is still worth taking seriously. It can eat through a meaningful part of the drive's expected write budget without filling the disk or showing a visible failure. How to check your own machine The order matters. Update first. Then look at real I/O. Then check what the drive itself says. Step 1: update Codex The investigation supplied with this article identifies 0.142.0 as the version containing the logging reduction. Check the current release notes and your installed build, then update before you start measuring. Also remember that Codex CLI, the desktop app, and editor integrations can ship or bundle components on different schedules. Updating one does not automatically prove every local Codex-related process is updated. Step 2: find the log files On macOS and Linux: ls -la ~/.codex/logs_2.sqlite* On Windows PowerShell: Get-Item "$env:USERPROFILE\.codex\logs_2.sqlite*" You might see the database, a <code>-wal</code> file, and a <code>-shm</code> file. Do not stop after checking their size. A small database is the whole reason this incident was easy to miss. Step 3: watch actual writes On Linux, <code>pidstat</code> can show which process is writing: pidstat -d 5 To see whether a process has the database open: sudo lsof ~/.codex/logs_2.sqlite On macOS, open Activity Monitor, select the Disk tab, and watch bytes written for Codex-related processes while you work. On Windows, use Resource Monitor or Task Manager's disk column. Look at sustained per-process writes, not just a brief spike during startup. Step 4: check the drive's own lifetime write counter This is the closest thing to the truth you have. On Linux or macOS with smartmontools installed, an NVMe drive can often be checked with: sudo smartctl -a /dev/nvme0 Look for: - Data Units Written - Percentage Used - warnings or critical health indicators On Windows, CrystalDiskInfo exposes Total Host Writes and health information. PowerShell can also show some device health data: Get-PhysicalDisk | Get-StorageReliabilityCounter | Select-Object DeviceId, Wear, Temperature Compare lifetime writes with the TBW rating for your exact drive model. Do not compare against a random SSD number from a search result. <!-- Visual placement: add a small annotated image of a SMART page here. Highlight only Data Units Written or Total Host Writes, Percentage Used, and health status. --> What if I used an older version for months? First, do not panic and do not guess from file size. Check the drive counters. If the drive has low percentage used and reasonable total writes for its age and workload, update and move on. If the drive has unusually high wear, warnings, or a large write count compared with its TBW rating, back up your important data and plan for a replacement before failure forces the schedule. This is a good backup reminder even if your drive looks healthy. SSDs usually fail without much theatre. Workarounds if you cannot update immediately Updating is the right fix. These are temporary ways to reduce persistent write pressure while you arrange that update. Put throwaway logs on ephemeral storage For CI jobs and containers, the cleanest option is usually to run with a temporary home directory or mount the relevant config directory on ephemeral storage. Example for a disposable CI environment: export HOME="$(mktemp -d)" For a container workload, mount a tmpfs at the appropriate Codex state directory if the environment can tolerate losing local diagnostics when the job exits. This is especially useful for build runners that create and destroy their environment anyway. Be careful with /tmp Moving a log file to <code>/tmp</code> is only useful if <code>/tmp</code> is actually memory-backed in your environment. On many Linux systems it may be tmpfs. On macOS, <code>/tmp</code> is not automatically a RAM disk by default. Check the mount before assuming the write load vanished: mount | grep ' /tmp ' Then verify with a real I/O monitor. Never trust a workaround because it sounded plausible in a comment thread. Avoid unsupported database surgery unless you really need it Some community workarounds modify the SQLite log database so inserts are ignored. That can be useful in an emergency, but it is an unsupported change to an internal application file. Future versions may change the schema or assume the database behaves normally. If you choose that route, stop Codex first, make a copy of the database, document what you changed, and remove the workaround once you are on a fixed release. The bigger lesson for tool builders This bug is bigger than one CLI. Measure bytes written, not just disk space used Every local persistence feature should have an I/O budget under realistic usage. Test bytes written per hour, per session, and per day. File size is not a sufficient metric for databases, caches, indexes, or log rotation systems. TRACE must be an intentional decision Verbose logging needs an explicit opt-in and a clear way back out. Shipping it by default turns every user into a debugging environment. Standard controls have to work end to end If your ecosystem has a standard logging override, honour it for every sink. A partial override is dangerous because it gives experienced users false confidence. Structured logs have storage costs SQLite, RocksDB, local search indexes, and caches are all good tools. High-frequency writes have a cost. The design needs backpressure, batching, rate limits, retention rules, and test coverage for pathological verbosity. Privacy and wear are connected The more raw operational detail a tool records, the more it writes and the more sensitive local data it can retain. Logging file paths, raw transport frames, and low-level event payloads should be a deliberate debugging choice, not background noise. The bigger lesson for users There is a simple habit worth keeping: > When you install a new long-running developer tool, look at its write rate once. Two minutes in Activity Monitor, Resource Monitor, or <code>pidstat</code> can catch a surprising amount. Check lifetime SSD writes occasionally. Keep your tools updated. Treat a stable file size as a storage number, not an I/O number. The log file was small. The work it made the SSD do was not. What is confirmed and what is still uncertain The supplied investigation is clear about its limits, and those limits matter: - The 37 TB measurement came from one heavy user's machine. It is a data point, not a universal rate. - Reports disagree about when the highest logging verbosity began. The exact start date is uncertain. - The annualized 640 TB figure is arithmetic based on the reported 21-day sample, not a forecast. - SSD write amplification is real, but the exact physical NAND multiplier was not measured for this incident. - The reported 85% reduction after version 0.142.0 is an early observed result, not a guarantee for every platform or workload. That uncertainty does not weaken the main finding. The correct response is still simple: update, measure the process, and inspect the drive counter. Sources and further reading - GitHub issue #28224 in openai/codex - Official OpenAI Codex documentation - The public reporting and technical analysis supplied for this article, including Notebookcheck, Windows Report, Lilting.ch, yage.ai, and Daniel Vaughan's write-up
A proof of Ryser's circulant Hadamard conjecture We show that an $n\times n$ circulant Hadamard matrix must satisfy a family of congruence equations that have solutions only when $n \leq 4$, proving Ryser's 1963 conjecture that no such matrices exist for $n>4$. Comments: Error found by AI. Corrected proof may be found here: github.com/openai/math/tree/main/preprints/The-ci…
Four GPUs, One Quantization Ladder: Gemma 4 E2B From a 4 GB Laptop to an NVIDIA L4 --- title: "Four GPUs, One Quantization Ladder: Gemma 4 E2B From a 4 GB Laptop to an NVIDIA L4" published: true description: "Gemma 4 E2B from BF16 to Q2_K on a Tesla T4, an NVIDIA L4, a GTX 1650 laptop and an RTX 3050 desktop, with byte-identical GGUF files and the same llama.cpp build. The ladder keeps its shape on all four cards, 218 questions die on all four, and any two GPUs grade 2 to 4 questions differently at BF16." tags: gemma, llamacpp, quantization, nvidia cover_image: raw.githubusercontent.com --- This article provides a step by step guide to running Gemma 4 E2B's quantization ladder, BF16 down to Q2_K, on four NVIDIA GPUs with the same nine GGUF files and the same llama.cpp build, and comparing the four cards question by question. xbill9/gemma-graveyard The first article asked every rung of the ladder the same 250 GSM8K questions on a Tesla T4. The second repeated it on an NVIDIA L4 in AWS. This one adds two consumer cards: a GTX 1650 Ti laptop GPU with 4 GB and an RTX 3050 desktop card with 8 GB. All four run llama.cpp b11699 on files with the same sha256, and each card gives identical text when BF16 is run twice. --- The Results in Brief The ladder keeps its shape on all four GPUs. Against each card's own BF16, F16 to Q5_K_M lose 1 to 8 answers, Q4_K_M and Q4_0 lose 9 to 19, Q3_K_M loses 42 to 49, and Q2_K gets 3 to 6 right out of 250. The counts agree; the questions move. Any two GPUs grade 2 to 4 questions differently at BF16 and 12 to 22 at Q4_0. No question is lost at F16 or Q8_0 on all four cards, while 216 are lost at Q2_K on all four. 218 questions die at some rung on every GPU, and 147 of them die first at the same rung on every GPU. The 4 GB laptop runs every quantized rung entirely on its GPU. BF16 and F16 do not fit, so llama.cpp splits them between GPU and CPU, and they decode at 9.2 and 9.6 tokens per second against 45.9 for Q8_0 on the same card. The cover lays the four GPUs' grades over each other. One row per rung, BF16 at the top and Q2_K at the bottom, one cell per question: marigold where all four GPUs answer right, grey where all four answer wrong, violet where they disagree. docs/make_cover_gpu4.py draws it from the committed rows. --- What Changes and What Stays | | Tesla T4 | NVIDIA L4 | GTX 1650 Ti | RTX 3050 | | :--- | :--- | :--- | :--- | :--- | | Where | GCE n1-standard-2 | AWS g6.xlarge | laptop | desktop | | GPU memory | 15,360 MiB | 23,034 MiB | 4,096 MiB | 8,192 MiB | | Compute capability | 7.5 | 8.9 | 7.5 | 8.6 | | Host CPU | 2 vCPU | AMD EPYC 7R13 | Intel i7-10750H | Intel i7-5820K | | Driver | 615.71.10 | 595.112.07 | 615.71.10 | 615.78.08 | The same on all four: llama.cpp b11699 with GGML_CUDA=ON, built for each card's own compute capability; the nine GGUFs and Google's QAT Q4_0 file, byte for byte; the 250 questions in data/heldout.jsonl; temperature 0, one request at a time, thinking off, 1,024 max tokens, llama-server -c 4096 -np 1 --seed 0. Every rung runs with -ngl 99 except BF16 and F16 on the GTX 1650 (Step 4). --- At This Point You Should Have… - An NVIDIA GPU with at least 4 GB and a CUDA toolkit - About 58 GB of free disk for the checkpoint, the nine GGUFs and the QAT file - A clone: git clone https://github.com/xbill9/gemma-graveyard - The T4 run's committed results in runs/2026-10-27-ladder/ --- Step 1 — Build llama.cpp at the Same Tag On the RTX 3050: git -C ~/llama.cpp worktree add --detach /android/graveyard/llama.cpp b11699 cd /android/graveyard/llama.cpp cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86 -DCMAKE_BUILD_TYPE=Release cmake --build build -j12 --target llama-server llama-quantize The one build flag that changes per card is CMAKE_CUDA_ARCHITECTURES: 75 for the T4 and the GTX 1650, 89 for the L4, 86 for the RTX 3050. nvidia-smi --query-gpu=name,memory.total,compute_cap,driver_version --format=csv name, memory.total [MiB], compute_cap, driver_version NVIDIA GeForce RTX 3050 OEM, 8192 MiB, 8.6, 615.78.08 🔎 Tip: keep the pinned build in its own worktree. The desktop already had a newer llama.cpp checkout (b11983). git worktree add --detach <dir> b11699 gives a second working copy at the tag from the same clone, so the measured build and the everyday build sit side by side. --- Step 2 — Rebuild the Ladder and Check It Against the T4 ladder.sh converts the checkpoint to BF16 and F16 and quantizes seven rungs from BF16. Each run script then compares every file's size and sha256 with the T4's ladder.tsv: diff <(cut -f1,3,4 runs/2026-10-27-ladder/ladder.tsv) <(cut -f1,3,4 runs/2026-10-29-3050/ladder.tsv) && echo "ladder matches T4 sha256" ladder matches T4 sha256 The GTX 1650 and the RTX 3050 both print the same line, and Google's QAT Q4_0 file matches too (sha256 fa401b55…). Across the four machines, the GGUF bytes are identical; the card, its driver, its CUDA version and the kernels built for it are what differ. 🔎 Tip: the converter writes the checkpoint directory into the GGUF. convert_hf_to_gguf.py copies the model card and takes the model's name from the directory it reads. A checkpoint taken straight from the Hugging Face cache snapshot has neither the README.md nor the name gemma-4-E2B-it, and gives different bytes. Download with hf download google/gemma-4-E2B-it --local-dir <dir>/gemma-4-E2B-it. --- Step 3 — Prove Each Card Repeats Itself BF16 on 50 questions, twice, on every GPU: ./graveyard.py repeat runs/2026-10-29-3050/determinism-1 runs/2026-10-29-3050/determinism-2 BF16 {"rung": "BF16", "n": 50, "mismatched_extracted": 0, "mismatched_text": 0, "mismatched_right": 0, "right_a": 46, "right_b": 46} The T4, the L4 and the GTX 1650 give the same result: 0 mismatched characters and 46 of 50 right both times. Each card on its own is deterministic, so a difference between two cards comes from the GPU and its CUDA stack. --- Step 4 — Fit the 16-bit Rungs on 4 GB BF16 and F16 need a 4,384.87 MiB GPU buffer for their 36 layers (from the T4's load log), and the GTX 1650 has 4,096 MiB. With -ngl 99 the load fails. graveyard.py run --ngl auto leaves -ngl off, and llama.cpp's fit step places what it can: common_params_fit_impl: - CUDA0 (NVIDIA GeForce GTX 1650 Ti with Max-Q Design): 14 layers, 2546 MiB used, 1125 MiB free load_tensors: offloaded 14/36 layers to GPU load_tensors: CUDA0 model buffer size = 2407.92 MiB Every other rung fits whole. Q8_0 on the same card: load_tensors: offloaded 36/36 layers to GPU load_tensors: CUDA0 model buffer size = 2342.30 MiB The split changes speed and leaves the grades in line with the other GPUs: the GTX 1650's BF16 scores 229, the same as the T4 and the L4, and grades 2 to 4 questions differently from each of them. The 8 GB RTX 3050 holds BF16 whole at 4,651 MiB. --- Step 5 — Run the Ladder on Each Card and Score It Each run script runs the nine rungs and QAT Q4_0, then scores every rung against that card's own BF16: ./graveyard.py bury runs/2026-10-29-3050 BF16 F16 Q8_0 Q6_K Q5_K_M Q4_K_M Q4_0 Q3_K_M Q2_K --json graveyard-ladder.json Right answers, out of 250: | Rung | T4 | L4 | GTX 1650 | RTX 3050 | | :--- | :--- | :--- | :--- | :--- | | BF16 | 229 | 229 | 229 | 232 | | F16 | 229 | 229 | 228 | 232 | | Q8_0 | 227 | 228 | 229 | 226 | | Q6_K | 229 | 227 | 226 | 227 | | Q5_K_M | 231 | 228 | 226 | 226 | | Q4_K_M | 224 | 221 | 222 | 221 | | Q4_0 | 222 | 220 | 216 | 218 | | Q3_K_M | 191 | 186 | 183 | 191 | | Q2_K | 3 | 6 | 5 | 3 | Lost, against the same card's BF16: | Rung | T4 | L4 | GTX 1650 | RTX 3050 | | :--- | :--- | :--- | :--- | :--- | | F16 | 1 | 1 | 2 | 1 | | Q8_0 | 4 | 3 | 2 | 7 | | Q6_K | 4 | 6 | 7 | 8 | | Q5_K_M | 3 | 6 | 6 | 8 | | Q4_K_M | 9 | 14 | 12 | 14 | | Q4_0 | 15 | 16 | 19 | 16 | | Q3_K_M | 42 | 45 | 49 | 44 | | Q2_K | 226 | 225 | 225 | 229 | Lost: right at BF16, wrong at this rung, on the same GPU. The failures fall into the same kinds on every card. Q3_K_M runs out of room: 33 to 36 of its losses on each GPU are answers cut off at 1,024 tokens. Q2_K loops: 208 to 228 of its 250 answers repeat a line five or more times, and 246 to 249 hit the token cap. --- Step 6 — Line the Four GPUs Up, Question by Question docs/gpu4_derived.py rebuilds the T4's grid from its committed results and reads the other three from their rows, then compares all six pairs of GPUs at every rung: python3 docs/gpu4_derived.py > runs/2026-10-29-3050/four-gpus.txt | Rung | Graded differently, any two GPUs | Lost on all four | | :--- | :--- | :--- | | BF16 | 2 to 4 | — | | F16 | 3 to 5 | 0 | | Q8_0 | 3 to 9 | 0 | | Q6_K | 4 to 10 | 2 | | Q5_K_M | 7 to 10 | 1 | | Q4_K_M | 9 to 15 | 5 | | Q4_0 | 12 to 22 | 5 | | Q3_K_M | 26 to 34 | 20 | | Q2_K | 6 to 9 | 216 | Graded differently: right on one GPU and wrong on the other, at that rung, for the closest and the furthest of the six pairs. - The light rungs lose different questions on each card. F16 and Q8_0 lose 1 to 7 answers per GPU, and none of them is lost on all four. - The deep losses are shared. Q3_K_M loses 20 questions on all four GPUs and Q2_K 216. - Most of the graveyard is common ground. Each GPU has 226 questions that die at some rung, the RTX 3050 230. 218 die on all four, and 147 die first at the same rung on all four. One question dies only on the RTX 3050 (test-358) and one only on the GTX 1650 (test-147). - BF16 itself moves a few answers. 227 questions are right at BF16 on all four cards and 18 wrong on all four. Against the L4, the RTX 3050's BF16 changes the text of 31 answers out of 250, the extracted number of 4 and the grade of 3. --- Step 7 — Google's QAT Q4_0 on Four Cards ./graveyard.py bury runs/2026-10-29-3050 BF16 QAT_Q4_0 Q4_0 --json graveyard-qat.json | GPU | QAT Q4_0 right | Q4_0 right | QAT Q4_0 lost | Q4_0 lost | | :--- | :--- | :--- | :--- | :--- | | T4 | 221 | 🥇 222 | 15 | 15 | | L4 | 219 | 🥇 220 | 15 | 16 | | GTX 1650 | 🥇 222 | 216 | 11 | 19 | | RTX 3050 | 🥇 221 | 218 | 15 | 16 | QAT Q4_0 scores 219 to 222 on the four cards and plain Q4_0 216 to 222. On the T4 and the L4 the plain file is ahead by 1; on the GTX 1650 QAT is ahead by 6 and on the RTX 3050 by 3. The two files lose mostly different questions: on the RTX 3050, 5 questions are lost under both, 10 only under QAT and 11 only under the plain Q4_0. Across the four GPUs, 6 of QAT's losses and 5 of Q4_0's are lost on every card. --- Step 8 — Speed on Each Card Median decode speed in tokens per second. Each column is one machine; read down a column. | Rung | T4 | L4 | GTX 1650 | RTX 3050 | | :--- | :--- | :--- | :--- | :--- | | BF16 | 40.0 | 50.1 | 9.2 | 40.3 | | F16 | 39.6 | 50.0 | 9.6 | 40.3 | | Q8_0 | 64.3 | 83.0 | 45.9 | 67.8 | | Q6_K | 70.8 | 98.1 | 57.2 | 75.3 | | Q5_K_M | 78.7 | 107.7 | 61.9 | 82.2 | | Q4_K_M | 84.1 | 118.7 | 66.5 | 91.5 | | Q4_0 | 🥇 93.6 | 125.6 | 🥇 70.8 | 🥇 99.7 | | Q3_K_M | 74.2 | 111.1 | 55.9 | 64.2 | | Q2_K | 74.0 | 🥇 139.5 | 62.2 | 95.0 | - Q4_0 is the fastest rung on three of the four cards; on the L4 it is Q2_K, which is wrong on 244 of 250 questions. - Q5_K_M decodes 1.21 to 1.35 times as fast as Q8_0 on each card (arithmetic, from the table). Q8_0 is the baseline here because it is the largest rung that fits whole on all four GPUs. - Q3_K_M is slower than Q4_0 on every card, and on the RTX 3050 it is slower than Q8_0. - BF16 and F16 run at the same speed on the T4, the L4 and the RTX 3050. On the GTX 1650, with 22 of 36 layers on the CPU, they run at a fifth of Q8_0's speed. The full ladder with QAT took 5 hours 16 minutes on the RTX 3050 and 12 hours 25 minutes on the GTX 1650, from the first log line to the last (run-3050.log, run-1650.log). --- Compare and Contrast | | T4 | L4 | GTX 1650 | RTX 3050 | | :--- | :--- | :--- | :--- | :--- | | BF16 right of 250 | 229 | 229 | 229 | 🥇 232 | | Q5_K_M lost | 🥇 3 | 6 | 6 | 8 | | Q4_0 right, tok/s | 222, 93.6 | 220, 125.6 | 216, 70.8 | 218, 99.7 | | Every rung whole on the GPU | 🟢 yes | 🟢 yes | ❌ BF16, F16 split | 🟢 yes | | Same text twice at BF16 | 🟢 yes | 🟢 yes | 🟢 yes | 🟢 yes | --- So, Which One? On any of the four cards, take Q5_K_M: 39.0% of BF16's bytes, 226 to 231 right, 3 to 8 answers lost against the card's own BF16, and 1.21 to 1.35 times Q8_0's decode speed. For the fastest file that stays within 19 answers, take Q4_0, the fastest rung on three of the cards. On a 4 GB card, every quantized rung fits whole and the 16-bit rungs run at under 10 tokens per second. To compare two quantizations question by question, run both on one GPU: a different card moves 2 to 4 grades at BF16 and up to 34 at Q3_K_M. --- Summary The goal of this article was to run Gemma 4 E2B's quantization ladder on four GPUs, from a 4 GB laptop card to a data-centre L4, and see which grades hold. The key was holding everything else fixed: byte-identical GGUFs, the same llama.cpp tag and the same sampling, with each card checked to give identical text twice. The four-GPU results were: - 🟢 All nine GGUFs and the QAT file match the T4's sha256 on every machine, and BF16 gives identical text twice on 50 questions on every card. - 🟢 The ladder keeps its shape on all four GPUs: F16 to Q5_K_M lose 1 to 8, Q4_K_M and Q4_0 lose 9 to 19, Q3_K_M loses 42 to 49. - 🟢 Q5_K_M is the best trade on every card: 226 to 231 right at 1.21 to 1.35 times Q8_0's speed. - 🟢 A 4 GB GTX 1650 runs every quantized rung whole on the GPU; BF16 and F16 run split, 14 of 36 layers on the GPU, at 9.2 and 9.6 tokens per second. - ⚠️ Any two GPUs grade 2 to 4 questions differently at BF16 and 12 to 22 at Q4_0; no question is lost at F16 or Q8_0 on all four. - ⚠️ 218 questions die on all four GPUs, 147 of them first at the same rung. - ⚠️ QAT Q4_0 and plain Q4_0 trade places by card: plain ahead by 1 on the T4 and L4, QAT ahead by 6 on the GTX 1650 and by 3 on the RTX 3050. - ❌ Q3_K_M loses 42 to 49, mostly answers cut off at 1,024 tokens; Q2_K gets 3 to 6 right, with 208 to 228 looping answers per card. Scope: one run of each rung per GPU on four NVIDIA cards (a Tesla T4 on GCE, an L4 on AWS, a GTX 1650 Ti laptop GPU and an RTX 3050 desktop card), 250 GSM8K test questions, greedy, thinking off, 1,024 max tokens, llama.cpp b11699 built for each card. Determinism checked at BF16 on each card. The T4 grid is rebuilt from its committed per-question results. The hosts also differ in CPU, RAM, driver and CUDA version, and on the GTX 1650 BF16 and F16 run split between GPU and CPU. Speed is compared within each card only. The strategy for measuring Gemma 4's accuracy one quantization rung at a time on four GPUs was validated with an incremental step by step approach. --- References - Accuracy vs Speed: Gemma 4 E2B Quantization Tradeoffs, BF16 to Q2_K, the T4 article - Performance vs Quality: Gemma 4 E2B Quantization on an NVIDIA L4 and a Tesla T4, the L4 article - gemma-graveyard on GitHub - GTX 1650 run: runs/2026-10-29-1650 - RTX 3050 run: runs/2026-10-29-3050 - llama.cpp - google/gemma-4-E2B-it - google/gemma-4-E2B-it-qat-q4_0-gguf - GSM8K
assertion: argon is currently at the frontier, pacing internal-only anthropic/openai models evidence: - sandbox-runtime#790 - codex#58115 macOS Seatbelt bypasses in both labs' own agent sandboxes, found in a quick vuln-hunting pass
Accuracy vs Speed: Gemma 4 E2B Quantization Tradeoffs, BF16 to Q2_K --- title: "Accuracy vs Speed: Gemma 4 E2B Quantization Tradeoffs, BF16 to Q2_K" published: true description: "Gemma 4 E2B quantized from BF16 to Q2_K on one Tesla T4, every rung asked the same 250 GSM8K questions, and each answer buried at the rung where it first goes wrong. Q5_K_M and above hold the score; Q3_K_M loses 38; Q2_K gets 3 right. With thinking on, Q3_K_M wins back 17." tags: devchallenge, hf26challenge, gemma, llamacpp cover_image: raw.githubusercontent.com --- This is a submission for the Hacktoberfest Open-Source AI Challenge Week 4: Ghost in the Machine This article provides a step by step guide to quantizing Gemma 4 E2B down a ladder of nine GGUF files with llama.cpp, asking every rung the same 250 maths questions on one Tesla T4, and burying each answer at the rung where it first goes wrong. Every count is computed by code and every row is committed. xbill9/gemma-graveyard A score per quantization level says how many answers a smaller file loses; a graveyard says which ones. Against the 229 of 250 questions BF16 gets right, Q8_0, Q6_K and Q5_K_M stay within 2, Q4_K_M and Q4_0 lose 5 and 7, Q3_K_M loses 38 and Q2_K gets 3 right. Above Q3_K_M most of the dead come back: 16 of the 23 questions that die between F16 and Q4_0 are right again at a lower rung. With thinking on, Q3_K_M climbs from 191 to 208. --- What I Built Quantization is the most common thing done to an open-weight model before it ships to a small device. The usual report is one accuracy number per file size. Two files with the same score can still get different questions wrong, and a net loss hides its parts: Q4_0 here is 7 answers down on BF16, made of 15 questions lost and 8 gained. The gemma-graveyard asks every rung of a quantization ladder the same 250 questions at temperature 0, so each answer is the model's single most likely one. BF16 is the reference, and every question a rung answers differently from BF16 lands in one of four records: - Dead: BF16 answers it right and the rung answers it wrong. A rung's dead count is every such question at that rung. - Tombstone: one per question, placed at the first rung, going down from BF16, where the question dies. A question dead at Q8_0 and again at Q4_0 has one tombstone, at Q8_0. - Rose again: a lower rung answers a tombstoned question right once more. test-687 dies at F16 and is right again at Q8_0, Q6_K, Q5_K_M and Q4_K_M. - Ghost: BF16 answers it wrong and the rung answers it right. test-1166 asks for an avocado tree's fruit over ten years, answer 2,350. BF16 says 1,000; Q8_0, Q6_K and Q4_0 get it right. A rung's score is BF16's 229, minus its dead, plus its ghosts: Q4_0 is 229 − 15 + 8 = 222. Quantization moves answers in both directions, and a score that holds can hide questions lost and found, so the article reports both. For Día de los Muertos: the dead are remembered one at a time, by name. Each tombstone here carries the question's id, the right answer, the wrong answer the rung gave, and every lower rung where it rose again. What the project delivers: - ladder.sh: BF16, F16 and seven quantized GGUFs of Gemma 4 E2B, with bytes and sha256 - graveyard.py run: one rung, its own llama-server, 250 questions, answers graded in code - graveyard.py bury: deaths, tombstones and ghosts per rung, computed in code - graveyard.py autopsy: how each rung's answers fail (cut off, looping, writing into the thinking block) - graveyard.py repeat: two runs of one rung compared question by question - one run of the whole ladder plus Google's QAT Q4_0 on a Tesla T4, committed with its report - a second pass with thinking on, over seven rungs from BF16 to Q3_K_M --- Demo The Results in Brief Q5_K_M is the best trade between speed and accuracy on the T4. It keeps BF16's score, 231 right against 229, at 39.0% of the bytes and 1.97 times BF16's decode speed. The 2 extra right answers come from the 8 questions where the two disagree, 5 ghosts against 3 dead; two equally accurate models split 8 questions at least that unevenly about three times in four, so the gap is within the noise. Q4_0 is the fastest rung at 2.34 times BF16, and costs 9 right answers against Q5_K_M for 1.19 times its speed. Q4_K_M sits between the two. Every other rung is beaten on both counts by one of those three: Q8_0 and Q6_K are slower than Q5_K_M with fewer or equal right answers, and Q3_K_M is slower than Q4_0 with 31 fewer. Below Q4 the accuracy collapses: Q3_K_M loses 38 and Q2_K gets 3 right. With thinking on, the order holds: Q5_K_M 230, Q4_K_M 226 and Q4_0 219, against BF16's 227. The cover is the graveyard. One row per rung, BF16 at the top and Q2_K at the bottom, one cell per question: marigold where the rung answers right, a grey headstone where it answers wrong. docs/make_cover.py draws it from the committed rows. The ladder, from graveyard.py bury, on 250 GSM8K test questions: | Rung | % of BF16 bytes | Right | Dead | Ghosts | Median tok/s | | :--- | :--- | :--- | :--- | :--- | :--- | | BF16 | 100.0 | 229 | 0 | 0 | 40.0 | | F16 | 100.0 | 229 | 1 | 1 | 39.6 | | Q8_0 | 53.3 | 227 | 4 | 2 | 64.3 | | Q6_K | 41.3 | 229 | 4 | 4 | 70.8 | | Q5_K_M | 39.0 | 🥇 231 | 3 | 5 | 78.7 | | Q4_K_M | 36.8 | 224 | 9 | 4 | 84.1 | | Q4_0 | 36.1 | 222 | 15 | 8 | 🥇 93.6 | | Q3_K_M | 34.4 | 191 | 42 | 4 | 74.2 | | Q2_K | 32.1 | 3 | 226 | 0 | 74.0 | Dead: BF16 right, this rung wrong. Ghosts: BF16 wrong, this rung right. --- Code xbill9/gemma-graveyard | Path | What | | :--- | :--- | | ladder.sh | Converts the checkpoint to BF16 and F16 GGUF, quantizes seven rungs from BF16, writes ladder.tsv | | f16_range.py | Counts the weights that overflow or underflow when BF16 is cast to F16 | | graveyard.py | run, bury, autopsy and repeat | | docs/think_derived.py | Thinking on against thinking off, question by question | | lazy_eval.py | Prompt, answer extraction and grading, reused from Week 3 | | run.md | Every command for the T4, in order, and how to resume after the VM stops | | runs/2026-10-27-ladder/run-ladder.sh | Runs the nine rungs in order, skips finished ones, then buries | | runs/2026-10-27-ladder/ | REPORT.md, the bury and autopsy tables, per-question tombstones | --- How I Built It At This Point You Should Have… - A GCE n1-standard-2 in us-west2-b with one Tesla T4, 15,360 MiB, driver 615.71.10, 2 vCPU and 7,436 MiB of RAM - llama.cpp built with CUDA for compute capability 7.5, at b11699 - google/gemma-4-E2B-it from Hugging Face, the BF16 checkpoint, text model only - google/gemma-4-E2B-it-qat-q4_0-gguf, Google's quantization-aware Q4_0 file - A clone: git clone https://github.com/xbill9/gemma-graveyard --- Step 1 — Build the Ladder ladder.sh converts the checkpoint twice, to BF16 and to F16, then quantizes seven rungs from the BF16 file with plain llama-quantize, with no importance matrix. That is the default a user gets. TMPDIR=/opt1/tmp ./ladder.sh /opt1/models/gemma-4-E2B-it /opt1/graveyard-gguf /opt1/llama.cpp rung file bytes sha256 BF16 /opt1/graveyard-gguf/e2b-BF16.gguf 9311305152 9fe40af9a72302151a5abc28a5057c5fe335eb80145e795ba745085fc73a2ad6 F16 /opt1/graveyard-gguf/e2b-F16.gguf 9311305152 5ab927e4b24a248de319e70467fc74d4ab87347ee9d6fb3ee60b293b12198cf8 Q8_0 /opt1/graveyard-gguf/e2b-Q8_0.gguf 4967497152 6b0301104c89a43c3bec5bae4b14d744f6000bc21e00e46e887fbbfe024de38a Q6_K /opt1/graveyard-gguf/e2b-Q6_K.gguf 3845346752 e800e0389f48646994d8ff6d3409dbf0c13b3fc217d0647b481385da33bd3d99 Q5_K_M /opt1/graveyard-gguf/e2b-Q5_K_M.gguf 3630288320 f6c3f4395a9d3839535d8c14ae48ab784096945cfc4485005853eede915cb4a9 Q4_K_M /opt1/graveyard-gguf/e2b-Q4_K_M.gguf 3427880384 dd9dc465bb535989205f158d77f28f6f8348e47d8ac8870920c152b5996cbc16 Q4_0 /opt1/graveyard-gguf/e2b-Q4_0.gguf 3360161216 2d84758474be87d0a74edfa98efbeb602484e26f2e5fa8695fd9641ad323c7b2 Q3_K_M /opt1/graveyard-gguf/e2b-Q3_K_M.gguf 3201351104 b10c1f38c93635ccee1c7f382e03f5d2f4e9516087d7d62a73ca7de0f52693ab Q2_K /opt1/graveyard-gguf/e2b-Q2_K.gguf 2989088192 65850f1a31efed2013ed148adc5dc937e973a8e79fd55cdba31edd53369fddb2 The reference is BF16 because that is the checkpoint's own dtype, so the BF16 GGUF holds the trained weights unchanged. F16 is a rung. --- Step 2 — Check What F16 Costs Before Running It F16 has more mantissa than BF16 and less exponent range, so the cast can overflow or round small weights to zero. f16_range.py counts both on the checkpoint, one line per tensor and a total: python3 f16_range.py /opt1/models/gemma-4-E2B-it/model.safetensors > runs/2026-10-27-f16-range/out.txt grep -E 'position_embedding_table|TOTAL' runs/2026-10-27-f16-range/out.txt {"tensor": "model.vision_tower.patch_embedder.position_embedding_table", "n": 15728640, "overflow": 0, "underflow": 5507733, "subnormal": 834716, "absmax": 0.431640625} {"tensor": "TOTAL", "tensors": 2011, "n": 5123178979, "overflow": 0, "underflow": 5523265, "subnormal": 7013885, "absmax": 900.0} No weight overflows. Of the 5,523,265 that underflow, 5,507,733 are in the vision tower's position table, which the text GGUF does not carry. --- Step 3 — Ask One Rung 250 Questions graveyard.py run starts its own llama-server on one GGUF and asks each question once: temperature 0, one request at a time, no prompt cache, thinking off, 1,024 max tokens. Only the reply's content is graded, in code: the number after Answer:, compared with the reference. ./graveyard.py run BF16 /opt1/graveyard-gguf/e2b-BF16.gguf data/heldout.jsonl runs/2026-10-27-ladder --llama /opt1/llama.cpp {"rung": "BF16", "n": 250, "right": 229, "errors": 0, "truncated": 4, "median_decode_tps": 40.0, "mean_tokens": 391.2} The whole model fits on the T4 at BF16: load_tensors: offloaded 36/36 layers to GPU load_tensors: CPU_Mapped model buffer size = 768.00 MiB load_tensors: CUDA0 model buffer size = 4384.87 MiB 🔎 Tip: Gemma 4 E2B keeps its embedding tables in CPU memory. The per-layer embedding table holds 2,348,810,240 of the checkpoint's weights, and llama.cpp leaves it memory-mapped on the CPU with -ngl 99. So each rung's GPU memory is about half its file size: 4,671 MiB in use at BF16 for a 9.3 GB file, 1,639 MiB at Q4_0, 1,287 MiB at Q2_K. --- Step 4 — Prove the Same Rung Gives the Same Answers A tombstone means something only if the same model on the same question always gives the same answer. BF16 on 50 questions, twice: for i in 1 2; do ./graveyard.py run BF16 /opt1/graveyard-gguf/e2b-BF16.gguf data/heldout.jsonl runs/2026-10-27-determinism-$i \ --llama /opt1/llama.cpp --limit 50 done The two runs compared in code, question by question: 0 mismatched extracted, 0 mismatched text, 46/50 both times Every answer and every character of text matched. The same holds below BF16. graveyard.py repeat compares a second full run of four rungs with the ladder's: ./graveyard.py repeat runs/2026-10-27-ladder runs/2026-10-28-repeat Q4_0 Q3_K_M Q8_0 F16 --common {"rung": "Q4_0", "n": 250, "mismatched_extracted": 0, "mismatched_text": 0, "mismatched_right": 0, "right_a": 222, "right_b": 222} {"rung": "Q3_K_M", "n": 250, "mismatched_extracted": 0, "mismatched_text": 0, "mismatched_right": 0, "right_a": 191, "right_b": 191} {"rung": "Q8_0", "n": 250, "mismatched_extracted": 0, "mismatched_text": 0, "mismatched_right": 0, "right_a": 227, "right_b": 227} {"rung": "F16", "n": 250, "mismatched_extracted": 0, "mismatched_text": 0, "mismatched_right": 0, "right_a": 229, "right_b": 229} A changed answer at a lower rung comes from the weights. --- Step 5 — Run the Ladder and Bury It runs/2026-10-27-ladder/run-ladder.sh runs the nine rungs in order and skips any rung whose results are complete, so it picks up again after the VM's evening stop. Then bury: ./graveyard.py bury runs/2026-10-27-ladder BF16 F16 Q8_0 Q6_K Q5_K_M Q4_K_M Q4_0 Q3_K_M Q2_K --json graveyard-ladder.json {"first_deaths_by_rung": {"Q2_K": 177, "Q3_K_M": 26, "Q4_0": 9, "Q8_0": 4, "Q5_K_M": 2, "Q4_K_M": 6, "Q6_K": 1, "F16": 1}, "tombstones": 226, "ghosts": 11} 226 tombstones and 11 ghosts: 226 different questions die at some rung, and 11 questions BF16 gets wrong come back right at one rung or more. The first-death counts add up to the tombstones, one per question; the per-rung dead counts in the ladder table add up to more, because one question can be dead at several rungs. Per-question detail is in graveyard-ladder.json. The one question that dies at F16: {'id': 'test-687', 'died_at': 'F16', 'answer': '13', 'said': 17.0, 'right_again_at': ['Q8_0', 'Q6_K', 'Q5_K_M', 'Q4_K_M']} "One twin is 7 years older than one triplet. If their combined ages are 44, how old is one of the twins?" BF16 answers 13, from two twins and three triplets. F16 adds up two twins and one triplet, and answers 17. Four lower rungs answer 13 again. --- Step 6 — Autopsy: How Each Rung Dies graveyard.py autopsy counts, per rung, answers cut off at 1,024 tokens, answers that repeat one line five or more times, and answers that put text in the thinking block: ./graveyard.py autopsy runs/2026-10-27-ladder BF16 F16 Q8_0 Q6_K Q5_K_M Q4_K_M Q4_0 Q3_K_M Q2_K {"rung": "Q3_K_M", "cut_off": 49, "looped": 0, "thought": 0, "empty_content": 0, "empty_right_in_reasoning": 0, "dead": 42, "dead_cut_off": 34, "dead_looped": 0, "dead_empty_content": 0, "first_deaths": 26, "first_deaths_right_again_lower": 0} {"rung": "Q2_K", "cut_off": 246, "looped": 228, "thought": 75, "empty_content": 63, "empty_right_in_reasoning": 2, "dead": 226, "dead_cut_off": 222, "dead_looped": 207, "dead_empty_content": 60, "first_deaths": 177, "first_deaths_right_again_lower": 0} Every rung: | Rung | Cut off | Looped | Thinking unprompted | Dead | Dead, cut off | | :--- | :--- | :--- | :--- | :--- | :--- | | F16 | 4 | 0 | 0 | 1 | 0 | | Q8_0 | 9 | 0 | 0 | 4 | 2 | | Q6_K | 4 | 0 | 0 | 4 | 1 | | Q5_K_M | 3 | 0 | 0 | 3 | 1 | | Q4_K_M | 6 | 0 | 0 | 9 | 3 | | Q4_0 | 10 | 0 | 0 | 15 | 8 | | Q3_K_M | 49 | 0 | 0 | 42 | 34 | | Q2_K | 246 | 228 | 75 | 226 | 222 | Three kinds of death: - F16 to Q4_0, a different road through the same problem. 23 first deaths; 16 of them are right again at some lower rung. Down to Q5_K_M most of the dead answers end normally with a wrong number: 4 of 12 were cut off. At Q4_K_M and Q4_0, 11 of 24 were. From F16 to Q5_K_M, ghosts (1 to 5 per rung) match deaths (1 to 4), so the score holds. From Q4_K_M down, deaths outnumber ghosts, 9 to 4 and 15 to 8. - Q3_K_M, out of room. 34 of its 42 deaths were cut off at 1,024 tokens, still working, with no repeated lines. BF16 used a median 466.5 tokens on the 44 questions Q3_K_M got wrong while cut off. - Q2_K, collapse. 246 of 250 answers run to the limit and 228 repeat a line five or more times. With thinking off, 75 answers open a thinking block anyway, and in 63 of them the whole answer lands there, leaving content empty. No other rung writes into the thinking block at all. 🔎 Tip: give two buries in one directory two names. bury writes its per-question tombstones to graveyard.json; --json picks the file, so the QAT comparison in Step 7 keeps the ladder's file intact. --- Step 7 — Google's QAT Q4_0 Beside the Plain Q4_0 Google publishes a Q4_0 GGUF of Gemma 4 E2B trained for 4-bit (quantization-aware training). Its weights differ from the base checkpoint, so it sits beside the ladder, against the same BF16 reference: ./graveyard.py bury runs/2026-10-27-ladder BF16 QAT_Q4_0 Q4_0 --json graveyard-qat.json {"rung": "QAT_Q4_0", "n": 250, "right": 221, "errors": 0, "truncated": 6, "median_decode_tps": 87.8, "mean_tokens": 391.9, "dead": 15, "ghosts": 7, "answer_changed": 27, "bytes": 3349516256, "gpu_mib_loaded": 1629, "gpu_mib_before": 0} {"rung": "Q4_0", "n": 250, "right": 222, "errors": 0, "truncated": 10, "median_decode_tps": 93.6, "mean_tokens": 430.4, "dead": 15, "ghosts": 8, "answer_changed": 27, "bytes": 3360161216, "gpu_mib_loaded": 1639, "gpu_mib_before": 0} Same number of deaths, same number of changed answers, one right answer apart. The graves differ: 6 of the 15 dead are the same question, 9 die only under QAT and 9 only under the plain Q4_0. QAT runs out of room less often: 2 of its deaths were cut off at 1,024 tokens, against 8 for the plain Q4_0. --- Step 8 — Bury the Ladder Again With Thinking On Gemma 4 can think before it answers. --think turns it on; the thinking goes to reasoning_content and only content is graded. Thinking needs room, so this pass allows 4,096 tokens in an 8,192-token context. It covers seven rungs, BF16 to Q3_K_M; F16, Q2_K and QAT Q4_0 were not run with thinking on. Each rung is buried against BF16 with thinking on, which gets 227 right. A second BF16 run of 50 questions matched the first exactly, 47 right both times. ./graveyard.py bury runs/2026-10-28-think BF16 Q8_0 Q6_K Q5_K_M Q4_K_M Q4_0 Q3_K_M --json graveyard.json {"rung": "BF16", "n": 250, "right": 227, "errors": 0, "truncated": 0, "median_decode_tps": 34.5, "mean_tokens": 842.2, "dead": 0, "ghosts": 0, "answer_changed": 0, "bytes": 9311305152, "gpu_mib_loaded": 4699, "gpu_mib_before": 0} {"rung": "Q3_K_M", "n": 250, "right": 208, "errors": 0, "truncated": 11, "median_decode_tps": 68.0, "mean_tokens": 1320.1, "dead": 26, "ghosts": 7, "answer_changed": 42, "bytes": 3201351104, "gpu_mib_loaded": 1517, "gpu_mib_before": 0} docs/think_derived.py lines the two passes up question by question: python3 docs/think_derived.py | tail -3 "q3_dead_cut_off_with_thinking_off": 34, "of_them_right_with_thinking_on": 21 } | Rung | Right, thinking off | Right, thinking on | Dead, off | Dead, on | Dead in both | | :--- | :--- | :--- | :--- | :--- | :--- | | BF16 | 229 | 227 | — | — | — | | Q8_0 | 227 | 🥇 230 | 4 | 2 | 0 | | Q6_K | 229 | 🥇 230 | 4 | 6 | 0 | | Q5_K_M | 231 | 🥇 230 | 3 | 6 | 0 | | Q4_K_M | 224 | 226 | 9 | 9 | 1 | | Q4_0 | 222 | 219 | 15 | 15 | 3 | | Q3_K_M | 191 | 208 | 42 | 26 | 12 | Three things change: - Q3_K_M recovers most of its loss. It gains 17 right answers. Of the 34 questions it lost by running out of room at 1,024 tokens, 21 are right with thinking on and 4,096 tokens. Both the thinking and the larger limit changed at once, so this run cannot say which one did it. - Above Q3_K_M the score holds and the graves move. Each rung stays within 3 answers of its thinking-off score, but the questions it loses are almost all new: 0 of the deaths at Q8_0, Q6_K and Q5_K_M are shared between the two passes, 1 of 9 at Q4_K_M and 3 of 15 at Q4_0. BF16 itself gets 218 questions right in both modes. - The cost is time. Answers use 2.08 to 2.18 times as many tokens, decoding runs at 0.79 to 0.93 of the thinking-off speed, and each rung takes 2.22 to 2.70 times as long on the T4. --- Compare and Contrast | | Q5_K_M | Q4_K_M | Q4_0 | QAT Q4_0 | Q3_K_M | | :--- | :--- | :--- | :--- | :--- | :--- | | Right of 250 (BF16: 229) | 🥇 231 | 🥈 224 | 🥉 222 | 221 | 191 | | Dead | 🥇 3 | 🥈 9 | 🥉 15 | 🥉 15 | 42 | | GPU MiB | 1,897 | 1,705 | 🥉 1,639 | 🥈 1,629 | 🥇 1,489 | | Median tok/s on the T4 | 78.7 | 🥉 84.1 | 🥇 93.6 | 🥈 87.8 | 74.2 | | Answers cut off | 🥇 3 | 🥈 6 | 10 | 🥈 6 | 49 | | Verdict | 🟢 same score as BF16 | 🟢 small loss | ⚠️ 15 dead | ⚠️ 15 dead, fewer cut off | ❌ 38 fewer right | --- So, Which One? To ship Gemma 4 E2B on a small GPU with BF16's score, take Q5_K_M: 39.0% of the bytes, 231 right against 229, and 1.97 times BF16's decode speed on the T4 (arithmetic from the medians). For the fastest file that stays within 7 answers, take Q4_0 or Q4_K_M. Below Q4, the losses stop being a few flipped answers: Q3_K_M saves 150 MiB of GPU memory over Q4_0 and costs 31 right answers, and Q2_K stops answering. With thinking on and 4,096 tokens, Q3_K_M gets 208 right, 11 short of Q4_0, at more than twice the time per question. --- Prior Work - Prompt, answer extraction and grading (lazy_eval.py) and the 250-question GSM8K split (data/, seed 0): from my Week 3 entry, lazy-gemma, whose grader came from my earlier Gemma 4 TPU work. - Open source used: llama.cpp (MIT) for conversion, quantization and serving; Gemma 4 E2B and Google's QAT Q4_0 GGUF (Gemma terms); GSM8K. --- Why Does Open Innovation Matter? A graveyard needs the same model at nine sizes, run the same way. - The weights are on the disk, so the ladder is built here. ladder.sh produces every rung from one BF16 file with the open converter and quantizer, and ladder.tsv records each file's sha256. A hosted API serves whichever quantization its provider chose, and it can change without notice. - The same answer twice. Greedy decoding on a fixed llama.cpp build gave identical text on every repeated run, BF16 to Q3_K_M, so every tombstone counts. That check needs control of the server, the seed and the build. - The failure is visible. Q2_K's answers in the thinking block show up only because llama-server returns reasoning_content and content separately and every row is kept. - No per-token bill. Ten models times 250 questions ran on one rented T4. --- The Spirit of It Every row in the graveyard is a grade-school maths problem, and each tombstone keeps its own epitaph: - test-687, twins and triplets, answer 13. Died at F16, said 17. Rose again at Q8_0, Q6_K, Q5_K_M and Q4_K_M. - test-2, a house flipped for profit, answer 70,000. Died at Q4_0, said 120,000. Never rose. - test-564, Judy's two weeks of reading, answer 240. Died at Q8_0, said 3. Rose again at every rung from Q6_K to Q4_0. Most graves above Q3_K_M have someone come back to them: 16 of 23. --- Prize Categories Gemma: Gemma 4 E2B, quantized locally with llama.cpp at nine levels and compared with Google's quantization-aware Q4_0 release, served on one Tesla T4. --- Summary The goal of this project was to find which answers Gemma 4 E2B loses at each quantization level, one question at a time. The key was a deterministic run: greedy decoding on one llama.cpp build, checked to give identical text twice at five rungs, so every changed answer comes from the weights. The results were: - 🟢 Q8_0, Q6_K and Q5_K_M stay within 2 right answers of BF16's 229 of 250; Q5_K_M scores 231 at 39.0% of the bytes. - 🟢 Every rung fits on the T4, BF16 included: all 36 layers on the GPU, 4,671 MiB in use, with the embedding tables in CPU memory. - 🟢 Above Q3_K_M, 16 of the 23 questions that die are right again at a lower rung. - ⚠️ Q4_K_M and Q4_0 lose 5 and 7, with 9 and 15 deaths. - ⚠️ Google's QAT Q4_0 scores 221 against the plain Q4_0's 222; the two share 6 of their 15 deaths. - 🟢 Q4_0, Q3_K_M, Q8_0 and F16 run a second time give the same text on all 250 questions. - ⚠️ With thinking on, rungs above Q3_K_M keep their score but lose different questions; at Q4_0, 3 of 15 deaths are shared. - ⚠️ Thinking on costs 2.1 times the tokens and 2.2 to 2.7 times the time per rung. - ❌ Q3_K_M loses 38; 34 of its 42 deaths are cut off at 1,024 tokens. With thinking on and 4,096 tokens it loses 19. - ❌ Q2_K gets 3 right: 246 answers hit the token limit, 228 loop, and 75 write into a thinking block with thinking off. Scope: one Tesla T4 on GCE, llama.cpp b11699, 250 GSM8K test questions per rung, greedy. Thinking off with a 1,024-token limit on all ten models; thinking on with a 4,096-token limit on seven rungs, BF16 to Q3_K_M. Determinism checked at BF16, F16, Q8_0, Q4_0 and Q3_K_M. No importance matrix, none of the other 1,069 GSM8K test questions, and no other machine. Speeds compare rungs on this T4 only. The strategy for burying Gemma 4's answers one quantization rung at a time was validated with an incremental step by step approach. --- References - Hacktoberfest Open-Source AI Challenge, Week 4: Ghost in the Machine - gemma-graveyard on GitHub - Run report: runs/2026-10-27-ladder/REPORT.md - lazy-gemma, the Week 3 entry - llama.cpp - google/gemma-4-E2B-it - google/gemma-4-E2B-it-qat-q4_0-gguf - GSM8K