One Packet to Tune, Two to Check: Tuning Scribe Without Fooling Myself
Scribe is a pipeline I'm building that reads handwritten answers off scanned paper forms with local models. Answers that look doubtful go to a person for review; the rest are accepted.
Part 2 ended with reading accuracy dropping sharply on handwriting the pipeline had never seen, while the review gate held. It also said part 3 would be the private held-out evaluation. That evaluation needs packets the pipeline has never seen, and it's still coming. This post answers the question that comes first: on a long form I could tune for, how far could tuning go, and how would I know the gains were real?
I moved Scribe to a 14-page questionnaire with 177 answers, on pages full of tick grids, yes/no underlines, timed tasks and tester-written scores. A packet here is one filled copy of those 14 pages. I tuned on one packet and kept two back as check packets.
There was no fine-tuning: no model weights changed. Tuning here means everything around the models: crops, prompts, the schema, review rules and serving settings. And most of what I learned was about checking rather than tuning. The first apparent gain was a scoring convention. The two worst bugs stayed silent until I looked at what the model actually saw. One change looked neutral on the tuning packet and hurt the check packets.
On all 531 answers across the three packets, here is the current setup (the last row of the table at the end) against where I started:
- 96% read exactly right, up from 93% (94.4% to 96.3% on the two check packets alone).
- 99.5% of the answers accepted without review were correct, up from 97.5%.
- Two wrong answers slipped past review, down from 11, but 9 of those 11 were on the tuning packet (2 to 1 on the check packets alone).
- About 22% of answers go to review, up from 18%: a second model that re-checks answers adds review work.
- A packet took about 37 minutes. Adding that second model had pushed it to about 70, and the speed work won the time back.
Answer by answer against the baseline, 16 answers went from wrong to right and none from right to wrong (exact McNemar p = 0.00003). On the check packets alone it was 7 and none (p = 0.016). Treat both as rough: one fix can flip several answers on the same page, and the first figure includes the packet I tuned on.
None of this counts toward part 1's targets: I tuned on one of these packets, and the check packets shaped some decisions along the way. The rest of this post is how I got there, what I threw away, and where the method leaked.
!How each change was tested and what it gained: baseline 93.2% exact with 11 escaped at about 52 minutes; tuned 95.3% with 4 escaped; with a second model 94.9% with 3 escaped at about 70 minutes, a dip that came from moved crops, not from the second model's flags; before caching 96.0% with 1 escaped at about 47 minutes; current, with prefix caching, 96.2% with 2 escaped at about 37 minutes, where caching also moved crops so the extra escape's cause is unresolved. Dropped along the way: cropping the tester-scored page, re-checking blank answers, 8 answers at once, removing the reading skills, and speculative decoding.
The setup
The pages always come in the same order: questionnaires, a page that a tester scores, and several pages of timed tasks and number fields. I built one schema for the packet. It now has 177 answer fields (201 before I dropped some), 89 checks between related answers, and a rule that a page's position in a complete packet decides which page it is.
I had three filled copies. They have never been published, so the models tested here cannot have seen these filled pages in training (blank forms of this kind may well be public).
The answer key was drafted automatically (two independent AI reads of each page, then an adjudicating pass), and two people, one of them me, then reviewed every value. The first review changed 2 of the 603 values in that first key, the two reviewers agreed on 99.2% (598 of 603), and I settled the five disagreements. People checking a pre-filled draft can miss what the draft missed, so the key is very good rather than perfect. It shrank to 531 answers when I dropped some fields, below.
The rules, which I settled after a first experiment had already been scored on all three packets:
- one packet for tuning (the hardest at baseline: 86% exact, against 94% on another)
- two check packets, kept out of the tuning loop and run to confirm a change
- every comparison is paired: each answer is scored in both runs, only the answers that changed count, and an exact McNemar test on the fixed and broken counts gives a rough p-value (reported, not used as a cut-off)
A fourth rule came later, from a result described below: a change stays only if it improves the tuning packet and holds on the check packets.
Fix the scorer before trusting a gain
My first experiment compared prompts with and without short "reading skills": form-agnostic notes on how pen marks work (underlines, crossings-out, handwritten digits, empty boxes). On the first packet scored, the skills version looked 1.5 points better.
It wasn't. The model writes [] for "nothing ticked", and my answer key writes a blank. Both mean the same thing, but the scorer counted them as different, and that one convention produced the whole gain. Scored fairly, the skills version was half a point behind on that packet, and across all three packets the two versions were within two answers of each other.
I check both scorings now and decide on the fair one; every number in this post uses it. Part 2's lesson again, from the other direction: before you trust a gain, check that your scorer isn't the thing that changed.
Look at what the model actually saw
Accuracy numbers told me which answers were wrong, not why. So I built a debugging view that shows, for every wrong answer, the image the model read next to where the answer actually is on the page.
It found two bugs that no metric would have explained:
- One page was never cropped at all. The step that finds each answer's box asks for every box on a page in one reply. On the tester-scored page that reply hit its length limit, the cut-off JSON failed to parse, and all 22 fields meant to be cropped silently fell back to the whole page. Fix: ask in batches of ten, and keep every complete box from a cut-off reply.
- A grid was read one column off. Each row of a tick grid is ticked under one of three column headings. Cropping a single row removed the headings, so the model miscounted columns and gave wrong answers with full confidence. Fix: read those rows from the whole page.
A third pattern, invented "yes" answers on blank rows of a timed-task page, turned out to be the crop catching a circle from the row above. Same fix.
On the tuning packet alone, three rounds of changes took exact accuracy from 85.6% to 89.1% and accepted-answer accuracy from 90.6% to 98.6%. Not all of that was better reading: the middle round also sent every tick on the tester-scored page straight to review, which takes the worst answers out of the accepted pile, and added an underline rule I'd noticed on these forms (the word where the underline starts is the one chosen) to the skills and to 26 yes/no hints.
The change that looked fine and wasn't
Once the box-finding worked, the tester-scored page was finally cropped. On the tuning packet that changed nothing (64% before, 64% after). I almost kept it.
On the two check packets it dropped that page from 94% to 86%. A crop shows only part of a row of ticks, and the old bug had been hiding the problem by never cropping at all.
I reverted it, and that's when "holds on the check packets" became a hard rule rather than a final formality. It also means the check packets have now shaped a decision; more on that below.
Read what you actually use
The tester-scored page has a tick per item and a handwritten score per section. The ticks were the problem: 15 of 66 wrong at the start, and once I sent every tick to review, 22 extra review items per packet (about 11% of all its answers). The written scores had one error across all three packets, and they are the values that get used downstream.
!A synthetic tester-scored page with invented items. Orange boxes mark the per-item ticks, including one over the box border, one crossed out and one faint; green boxes mark the handwritten section scores and the total. A legend below explains both.
A synthetic page with invented items, drawn for this post. Ticks can land over the border, be crossed out or be faint; the written section score is the tester's own count.
So the schema now reads only the written scores. It drops the tick fields, plus two fields that depended on them. That changes the task rather than improving the reading: the page's worst fields leave the scoring, review load drops by 22 items per packet, and each packet needs 24 fewer model calls. To keep comparisons honest, the summary at the top and every comparison from here on score both runs on the same reduced set of 531 answers; the figures before this point used the original 603.
With all the fixes so far, and no second model yet, exact accuracy across the three packets went from 93.2% to 95.3% (12 answers fixed, 1 broken), and wrong answers escaping review fell from 11 to 4. On the check packets alone the gain was smaller and not significant: 94.4% to 95.5% (5 fixed, 1 broken).
The second model, revisited
Part 1's best idea was a second, different model re-reading each non-blank answer the main model would accept. Adding it back on this form let one fewer wrong answer through (4 to 3), but the review load jumped from 17% to 27%.
Exact accuracy also dipped, from 95.3% to 94.9%. The second model only flags answers; it never changes one, so the dip came from somewhere else. This first version re-read each answer from the other source (the whole page for a cropped answer, a crop for a whole-page one), so the pipeline now had to find a box for every field. That changed how the box requests were batched and moved 129 of the main model's 270 crops. Eight of its answers changed as a result: 3 for the better, 5 for the worse. Something outside the main model had changed what it saw, again.
Looking at its disagreements, 92% were false alarms (55 of 60), and 42 of those were answers the main model had read from a crop that the second model re-read from the whole page. On a dense page, the small model often couldn't find the field.
I tried two fixes, one at a time:
- Also re-check blank answers. Dropped. Review load on the tuning packet rose from 27% to 36%, and every new flag was a false alarm: the small model "saw" answers on empty lines.
- Re-read the same image the main model read. Kept. False alarms dropped from 16 to 10 on the tuning packet, and the main model's answers there went back to exactly what they were without a second model.
So a second read is useful when it's a fair second look at the same thing, and much less useful as a way to confirm that something is empty. Its flags are still mostly false alarms: 92-95% in every version. Whether it earns its keep on this form is still open. In the current setup it stops 2 wrong answers that would otherwise be accepted, at the cost of 40 false alarms (about 7.5% of all answers). Two answers is more than nothing and less than proof.
Faster, checked answer by answer
A 70-minute packet was too slow. My condition for speed work: no change could make the answers worse, and every change got a paired run.
More than half the time went into reading the prompt rather than writing the reply: each whole-page read sends about 5,100 tokens, about 3,650 of them the page image.
| Change | Time saved | Answers |
|---|--:|---|
| read 4 answers on a page at once, and skip the page-type check when page order already settles it | 18% | 176 of 177 identical |
| also cap the short note the model writes after each answer at 60 characters | 24% in total | 2 changed, both for the better |
| reuse the processed page image across that page's whole-page questions (prefix caching) | 22% more per packet | 3 changed: 1 for the better, none for the worse; but 2 escaped review instead of 1 (below) |
The first two rows are from the tuning packet, the last from all three. Across all three packets, time went from about 70 minutes to about 47, then to about 37; the rest of the drop from 70 came from the same-image re-read, which no longer finds a box for every field. Before caching, the main model had 4 answers fixed and none broken compared with the tuned pipeline without a second model.
The extra escape with caching is the yes/no answer described below, and I can't say whether caching caused it. Finding the boxes is itself a whole-page request, so it gets the cache too, and caching moved 89 of the main model's 270 crop boxes, most by a few pixels. Other cached runs gave exactly the same boxes, so the moves come from caching itself, not chance. That answer's box was one of them, so the second model saw a slightly different image of it. Whether that is why it agreed would take repeat runs of the uncached setup, which I haven't done. I kept caching because none of the main model's answers got worse, and those repeat runs are next.
Things I tried and dropped:
- 8 answers at once instead of 4: 1% faster. The server was already saturated.
- No reading skills: 6-11% faster, but one more wrong answer escaped review on the check packets. One answer is too few to call on its own, but the cause was specific: on a yes/no question whose own hint doesn't mention the underline rule, the model read it wrong either way; with the skill it was unsure enough to send it to review. I kept the skills as a safety call, not a measured win, and fixing that hint now would be a fix found on a check packet.
- Speculative decoding (a small "drafter" proposes tokens for the main model to check in one step): on synthetic test images a single request ran at 20-23 tokens/s instead of 15, but in the pipeline it was 6.6% slower. In the serving library, a running speculative batch takes no new requests until its longest reply ends, and with a drafter loaded the library refuses the structured JSON output Scribe relies on. Along the way I found a bug in the library's batched speculative decoding: when one reply in a batch finishes before the others, the replies after it can pick up another request's position offset if their images differ in size. I patched it locally and reported it upstream. I could show wrong offsets but not changed answers, and the report says so.
The two that got through
Part 2 ended by auditing every answer that skipped review, so here are the two in the current setup.
- A free-text answer with an extra letter. The model added a plural "s" to one word of a two-word answer. The meaning is the same, but exact scoring counts it wrong, and the second model made the same misread, so nothing flagged it.
- A yes/no underline that ran under both words. The line starts under one option and carries on under the other. By the rule in the reading skills, where the line starts decides; the model went by where it ended, with high confidence. In the run just before caching, the second model disagreed and the answer went to review; with caching it agreed, and it slipped through. That run is the only setup in the table that caught it.
!A synthetic yes/no question, "Do you cycle to work? Yes / No", with a pen underline that starts under Yes and runs on under No. Labels mark where the line starts and where it ends.
A synthetic illustration of this kind of mark: an invented question and a generated pen mark, drawn for this post.
So one cosmetic error and one real flip, on a mark a person would hesitate over too. The fix I'd test next, on new packets rather than these: send any underline that spans both options straight to review.
What this doesn't show
- It's three packets. 531 answers is enough to see large effects, not small ones. Both models' answers shift a little when batching changes, and two of the changes here moved the main model's crops, so treat any one- or two-answer difference in this post as unresolved.
- The check packets aren't clean. The schema and answer key were built from all three packets, the first experiment was scored on all three, and three later decisions used check-packet results: reverting the crops, reading only the written scores (the tick error counts behind it covered all three packets), and keeping the reading skills.
- Few writers. Three packets can't show how this behaves across many writers and testers.
- It's not the held-out evaluation. Part 1 set the targets (at least 98% accuracy on accepted answers, and at most 2% of blank answers getting an invented value past review) to be measured only on a private held-out set. These packets don't qualify, and the untuned baseline was already above 98% on the check packets, so this says more about the form than about the tuning.
What I'm taking away
- A tuning/checking split pays for itself, even a leaky one. The clearest harmful change looked neutral on the packet I tuned on.
- Check the scorer, and look at what the model saw, before trusting a number. The first "gain" was a scoring convention; the two worst bugs were silent until I saw the crops; and adding the second model and turning on caching both moved the main model's crops.
- Read what is used, and score before and after on the same set.
- A second model should get a fair second look, not a harder question.
- Check speed work answer by answer, including what the model was shown, and count a speed-up only under the batching you actually run.
{% details The numbers, for engineers %}
Every Scribe run used the same machine as parts 1 and 2. Main reader Qwen3.8-27B (8-bit), second model Qwen3-VL-8B (8-bit), served locally with mlx-vlm 0.6.15. 531 answers across 3 packets (one for tuning, two for checking), scored fairly ([] = blank). Accepted acc = accuracy of answers accepted without review. Escaped = wrong answers that were not sent to review.
| Configuration | Exact | Accepted acc | Wrong | Escaped | Flags | Min / packet |
|---|--:|--:|--:|--:|--:|--:|
| Baseline (main model, crops by field type, no second model) | 93.2% | 97.5% | 36 | 11 | 18.1% | ~52 (older, larger schema) |
| Tuned (fixes above, written scores only; on the tuning packet, only the changed pages were re-run) | 95.3% | 99.1% | 25 | 4 | 17.3% | ~36 (estimate) |
| + second model (re-reads from the other source; moved 129 crops) | 94.9% | 99.2% | 27 | 3 | 26.7% | ~70 |
| + same-image re-read, 4 at once, no page-type check, note cap | 96.0% | 99.8% | 21 | 1 | 21.8% | ~47 |
| + prefix caching (current) | 96.2% | 99.5% | 20 | 2 | 22.2% | ~37 |
Part 1's second target counts invented answers (a value where the key has a blank) that escape review: 3 of 160 blank answers at baseline (1.9%), 0 in the last two rows. Escaped in this post is broader: any wrong answer that skipped review.
Check packets only (354 answers), exact / accepted acc / escaped: baseline 94.4% / 99.3% / 2; bold row 96.0% / 100.0% / 0 (of 275 accepted answers, which still allows a true error rate of about 1%); current 96.3% / 99.6% / 1.
Paired comparisons (exact McNemar on the answers that changed): current vs baseline 16 fixed / 0 broken, p = 0.00003 (check packets only: 7 / 0, p = 0.016); bold row vs baseline 15 / 0, p = 0.00006 (check packets only: 6 / 0, p = 0.03); tuned vs baseline 12 / 1, p = 0.003 (check packets only: 5 / 1, p = 0.22); bold row vs tuned 4 / 0, p = 0.125; bold row vs the first second-model run 8 / 2, p = 0.11, with 7 of the 10 on crops that run had moved; skills vs no skills p = 0.73. Read these as rough: the all-packet figures include the packet the changes were chosen on, one fix often flips several answers on the same page, and I ran many comparisons.
Prefix caching: the main model mixes two attention types, so mlx-vlm caches exact prompt snapshots, by default 16 tokens before the end of the prompt, a point no two questions share. A 4,000-token guard moves the snapshot to the end of the page image (cached prefix about 3,984 tokens; prompt processing for later whole-page questions about 6.6 s to 1.8 s). Cropped reads and the second model get no reuse. It isn't output-neutral, though: the main model's boxes moved for 89 of 270 cropped answers (median 2 px, largest 80 px), and 3 values changed.
Other rounds: no JSON constraint on its own, no gain; speculative drafter 89% of drafted tokens accepted on the synthetic probe, +6.6% time vs the same setup without the JSON constraint and about +9% vs the current configuration.
{% enddetails %}
---
This is part 3 of a series. Part 1: Confidence Is Theater: What I Learned Building Scribe. Part 2: One Imperfect Page per Hundred: Testing Scribe on Data I Didn't Curate.
Pipeline code and the public benchmarks from parts 1 and 2: kod201/scribe
Every Scribe run in this post ran locally on one machine.
3 comments
Mentions
New Best Local Image-To-3D Model Now Runs Natively on Apple Silicon Idol comes from Speridlabs and is Apache 2.0. Until now it needed an NVIDIA GPU with CUDA. The new port is part of MLX-VLM by Prince Canuma, and it reimplements the whole pipeline in MLX: the denoiser, the decoders, the DINOv3 and FLUX.2 VAE image encoders, and the background removal. What the Mac version does: - One command: give it an image and get a standard 3DGS .ply, plus the prepared input and the predicted camera - No CUDA, no conversion: it loads the original Idol weights (about 3.1 GB) directly - Same settings as the original: 50 steps and 262,144 Gaussians by default, adjustable from about 32K up to 1M - Photos or renders: a "real" mode for photos and a "synthetic" mode for renders and generated images - Local server: an /v1/meshes endpoint queues jobs and returns the PLY, if you want to call it from other tools The castle, factory and robot in the video were generated this way, at 50 steps and 262,144 splats each. You still need access to Meta's gated DINOv3 and the FLUX.2-dev VAE on Hugging Face. The output is a splat, not a mesh, so you'll need a splat viewer or a conversion step to use it as a regular model. github: Blaizzy/mlx-vlm
I think mlx-vlm should support both the Audio and the Omni model: Blaizzy/mlx-vlm and there should be native MLX exports for both here: - huggingface.co - huggingface.co
Support for Baidu Unlimited OCR 2 is cooking on mlx-vlm! I see PR #2601 Gonna test now DGX Spark with official repo vs mlx-vlm on M5 Max! Keep you posted! Official repo: baidu/Unlimited-OCR mlx-vlm PR: mlx-vlm#2601 t.co
MLX-VLM's trainer now supports MTP 🚀🚀🚀 mlx-vlm#2558
Finishing up the last two PRs for base VLM training for MLX-VLM by @Prince_Canuma to then start adding decision model training. And then……… Here are the current PRs: Multi-Token-Prediction: mlx-vlm#2558 Quantization Aware Training: mlx-vlm#2590
WHO WANTS MTP IN THE MLX-VLM TRAINER????????? mlx-vlm#2554
One Packet to Tune, Two to Check: Tuning Scribe Without Fooling Myself Scribe is a pipeline I'm building that reads handwritten answers off scanned paper forms with local models. Answers that look doubtful go to a person for review; the rest are accepted. Part 2 ended with reading accuracy dropping sharply on handwriting the pipeline had never seen, while the review gate held. It also said part 3 would be the private held-out evaluation. That evaluation needs packets the pipeline has never seen, and it's still coming. This post answers the question that comes first: on a long form I could tune for, how far could tuning go, and how would I know the gains were real? I moved Scribe to a 14-page questionnaire with 177 answers, on pages full of tick grids, yes/no underlines, timed tasks and tester-written scores. A packet here is one filled copy of those 14 pages. I tuned on one packet and kept two back as check packets. There was no fine-tuning: no model weights changed. Tuning here means everything around the models: crops, prompts, the schema, review rules and serving settings. And most of what I learned was about checking rather than tuning. The first apparent gain was a scoring convention. The two worst bugs stayed silent until I looked at what the model actually saw. One change looked neutral on the tuning packet and hurt the check packets. On all 531 answers across the three packets, here is the current setup (the last row of the table at the end) against where I started: - 96% read exactly right, up from 93% (94.4% to 96.3% on the two check packets alone). - 99.5% of the answers accepted without review were correct, up from 97.5%. - Two wrong answers slipped past review, down from 11, but 9 of those 11 were on the tuning packet (2 to 1 on the check packets alone). - About 22% of answers go to review, up from 18%: a second model that re-checks answers adds review work. - A packet took about 37 minutes. Adding that second model had pushed it to about 70, and the speed work won the time back. Answer by answer against the baseline, 16 answers went from wrong to right and none from right to wrong (exact McNemar p = 0.00003). On the check packets alone it was 7 and none (p = 0.016). Treat both as rough: one fix can flip several answers on the same page, and the first figure includes the packet I tuned on. None of this counts toward part 1's targets: I tuned on one of these packets, and the check packets shaped some decisions along the way. The rest of this post is how I got there, what I threw away, and where the method leaked. !How each change was tested and what it gained: baseline 93.2% exact with 11 escaped at about 52 minutes; tuned 95.3% with 4 escaped; with a second model 94.9% with 3 escaped at about 70 minutes, a dip that came from moved crops, not from the second model's flags; before caching 96.0% with 1 escaped at about 47 minutes; current, with prefix caching, 96.2% with 2 escaped at about 37 minutes, where caching also moved crops so the extra escape's cause is unresolved. Dropped along the way: cropping the tester-scored page, re-checking blank answers, 8 answers at once, removing the reading skills, and speculative decoding. The setup The pages always come in the same order: questionnaires, a page that a tester scores, and several pages of timed tasks and number fields. I built one schema for the packet. It now has 177 answer fields (201 before I dropped some), 89 checks between related answers, and a rule that a page's position in a complete packet decides which page it is. I had three filled copies. They have never been published, so the models tested here cannot have seen these filled pages in training (blank forms of this kind may well be public). The answer key was drafted automatically (two independent AI reads of each page, then an adjudicating pass), and two people, one of them me, then reviewed every value. The first review changed 2 of the 603 values in that first key, the two reviewers agreed on 99.2% (598 of 603), and I settled the five disagreements. People checking a pre-filled draft can miss what the draft missed, so the key is very good rather than perfect. It shrank to 531 answers when I dropped some fields, below. The rules, which I settled after a first experiment had already been scored on all three packets: - one packet for tuning (the hardest at baseline: 86% exact, against 94% on another) - two check packets, kept out of the tuning loop and run to confirm a change - every comparison is paired: each answer is scored in both runs, only the answers that changed count, and an exact McNemar test on the fixed and broken counts gives a rough p-value (reported, not used as a cut-off) A fourth rule came later, from a result described below: a change stays only if it improves the tuning packet and holds on the check packets. Fix the scorer before trusting a gain My first experiment compared prompts with and without short "reading skills": form-agnostic notes on how pen marks work (underlines, crossings-out, handwritten digits, empty boxes). On the first packet scored, the skills version looked 1.5 points better. It wasn't. The model writes [] for "nothing ticked", and my answer key writes a blank. Both mean the same thing, but the scorer counted them as different, and that one convention produced the whole gain. Scored fairly, the skills version was half a point behind on that packet, and across all three packets the two versions were within two answers of each other. I check both scorings now and decide on the fair one; every number in this post uses it. Part 2's lesson again, from the other direction: before you trust a gain, check that your scorer isn't the thing that changed. Look at what the model actually saw Accuracy numbers told me which answers were wrong, not why. So I built a debugging view that shows, for every wrong answer, the image the model read next to where the answer actually is on the page. It found two bugs that no metric would have explained: - One page was never cropped at all. The step that finds each answer's box asks for every box on a page in one reply. On the tester-scored page that reply hit its length limit, the cut-off JSON failed to parse, and all 22 fields meant to be cropped silently fell back to the whole page. Fix: ask in batches of ten, and keep every complete box from a cut-off reply. - A grid was read one column off. Each row of a tick grid is ticked under one of three column headings. Cropping a single row removed the headings, so the model miscounted columns and gave wrong answers with full confidence. Fix: read those rows from the whole page. A third pattern, invented "yes" answers on blank rows of a timed-task page, turned out to be the crop catching a circle from the row above. Same fix. On the tuning packet alone, three rounds of changes took exact accuracy from 85.6% to 89.1% and accepted-answer accuracy from 90.6% to 98.6%. Not all of that was better reading: the middle round also sent every tick on the tester-scored page straight to review, which takes the worst answers out of the accepted pile, and added an underline rule I'd noticed on these forms (the word where the underline starts is the one chosen) to the skills and to 26 yes/no hints. The change that looked fine and wasn't Once the box-finding worked, the tester-scored page was finally cropped. On the tuning packet that changed nothing (64% before, 64% after). I almost kept it. On the two check packets it dropped that page from 94% to 86%. A crop shows only part of a row of ticks, and the old bug had been hiding the problem by never cropping at all. I reverted it, and that's when "holds on the check packets" became a hard rule rather than a final formality. It also means the check packets have now shaped a decision; more on that below. Read what you actually use The tester-scored page has a tick per item and a handwritten score per section. The ticks were the problem: 15 of 66 wrong at the start, and once I sent every tick to review, 22 extra review items per packet (about 11% of all its answers). The written scores had one error across all three packets, and they are the values that get used downstream. !A synthetic tester-scored page with invented items. Orange boxes mark the per-item ticks, including one over the box border, one crossed out and one faint; green boxes mark the handwritten section scores and the total. A legend below explains both. A synthetic page with invented items, drawn for this post. Ticks can land over the border, be crossed out or be faint; the written section score is the tester's own count. So the schema now reads only the written scores. It drops the tick fields, plus two fields that depended on them. That changes the task rather than improving the reading: the page's worst fields leave the scoring, review load drops by 22 items per packet, and each packet needs 24 fewer model calls. To keep comparisons honest, the summary at the top and every comparison from here on score both runs on the same reduced set of 531 answers; the figures before this point used the original 603. With all the fixes so far, and no second model yet, exact accuracy across the three packets went from 93.2% to 95.3% (12 answers fixed, 1 broken), and wrong answers escaping review fell from 11 to 4. On the check packets alone the gain was smaller and not significant: 94.4% to 95.5% (5 fixed, 1 broken). The second model, revisited Part 1's best idea was a second, different model re-reading each non-blank answer the main model would accept. Adding it back on this form let one fewer wrong answer through (4 to 3), but the review load jumped from 17% to 27%. Exact accuracy also dipped, from 95.3% to 94.9%. The second model only flags answers; it never changes one, so the dip came from somewhere else. This first version re-read each answer from the other source (the whole page for a cropped answer, a crop for a whole-page one), so the pipeline now had to find a box for every field. That changed how the box requests were batched and moved 129 of the main model's 270 crops. Eight of its answers changed as a result: 3 for the better, 5 for the worse. Something outside the main model had changed what it saw, again. Looking at its disagreements, 92% were false alarms (55 of 60), and 42 of those were answers the main model had read from a crop that the second model re-read from the whole page. On a dense page, the small model often couldn't find the field. I tried two fixes, one at a time: - Also re-check blank answers. Dropped. Review load on the tuning packet rose from 27% to 36%, and every new flag was a false alarm: the small model "saw" answers on empty lines. - Re-read the same image the main model read. Kept. False alarms dropped from 16 to 10 on the tuning packet, and the main model's answers there went back to exactly what they were without a second model. So a second read is useful when it's a fair second look at the same thing, and much less useful as a way to confirm that something is empty. Its flags are still mostly false alarms: 92-95% in every version. Whether it earns its keep on this form is still open. In the current setup it stops 2 wrong answers that would otherwise be accepted, at the cost of 40 false alarms (about 7.5% of all answers). Two answers is more than nothing and less than proof. Faster, checked answer by answer A 70-minute packet was too slow. My condition for speed work: no change could make the answers worse, and every change got a paired run. More than half the time went into reading the prompt rather than writing the reply: each whole-page read sends about 5,100 tokens, about 3,650 of them the page image. | Change | Time saved | Answers | |---|--:|---| | read 4 answers on a page at once, and skip the page-type check when page order already settles it | 18% | 176 of 177 identical | | also cap the short note the model writes after each answer at 60 characters | 24% in total | 2 changed, both for the better | | reuse the processed page image across that page's whole-page questions (prefix caching) | 22% more per packet | 3 changed: 1 for the better, none for the worse; but 2 escaped review instead of 1 (below) | The first two rows are from the tuning packet, the last from all three. Across all three packets, time went from about 70 minutes to about 47, then to about 37; the rest of the drop from 70 came from the same-image re-read, which no longer finds a box for every field. Before caching, the main model had 4 answers fixed and none broken compared with the tuned pipeline without a second model. The extra escape with caching is the yes/no answer described below, and I can't say whether caching caused it. Finding the boxes is itself a whole-page request, so it gets the cache too, and caching moved 89 of the main model's 270 crop boxes, most by a few pixels. Other cached runs gave exactly the same boxes, so the moves come from caching itself, not chance. That answer's box was one of them, so the second model saw a slightly different image of it. Whether that is why it agreed would take repeat runs of the uncached setup, which I haven't done. I kept caching because none of the main model's answers got worse, and those repeat runs are next. Things I tried and dropped: - 8 answers at once instead of 4: 1% faster. The server was already saturated. - No reading skills: 6-11% faster, but one more wrong answer escaped review on the check packets. One answer is too few to call on its own, but the cause was specific: on a yes/no question whose own hint doesn't mention the underline rule, the model read it wrong either way; with the skill it was unsure enough to send it to review. I kept the skills as a safety call, not a measured win, and fixing that hint now would be a fix found on a check packet. - Speculative decoding (a small "drafter" proposes tokens for the main model to check in one step): on synthetic test images a single request ran at 20-23 tokens/s instead of 15, but in the pipeline it was 6.6% slower. In the serving library, a running speculative batch takes no new requests until its longest reply ends, and with a drafter loaded the library refuses the structured JSON output Scribe relies on. Along the way I found a bug in the library's batched speculative decoding: when one reply in a batch finishes before the others, the replies after it can pick up another request's position offset if their images differ in size. I patched it locally and reported it upstream. I could show wrong offsets but not changed answers, and the report says so. The two that got through Part 2 ended by auditing every answer that skipped review, so here are the two in the current setup. - A free-text answer with an extra letter. The model added a plural "s" to one word of a two-word answer. The meaning is the same, but exact scoring counts it wrong, and the second model made the same misread, so nothing flagged it. - A yes/no underline that ran under both words. The line starts under one option and carries on under the other. By the rule in the reading skills, where the line starts decides; the model went by where it ended, with high confidence. In the run just before caching, the second model disagreed and the answer went to review; with caching it agreed, and it slipped through. That run is the only setup in the table that caught it. !A synthetic yes/no question, "Do you cycle to work? Yes / No", with a pen underline that starts under Yes and runs on under No. Labels mark where the line starts and where it ends. A synthetic illustration of this kind of mark: an invented question and a generated pen mark, drawn for this post. So one cosmetic error and one real flip, on a mark a person would hesitate over too. The fix I'd test next, on new packets rather than these: send any underline that spans both options straight to review. What this doesn't show - It's three packets. 531 answers is enough to see large effects, not small ones. Both models' answers shift a little when batching changes, and two of the changes here moved the main model's crops, so treat any one- or two-answer difference in this post as unresolved. - The check packets aren't clean. The schema and answer key were built from all three packets, the first experiment was scored on all three, and three later decisions used check-packet results: reverting the crops, reading only the written scores (the tick error counts behind it covered all three packets), and keeping the reading skills. - Few writers. Three packets can't show how this behaves across many writers and testers. - It's not the held-out evaluation. Part 1 set the targets (at least 98% accuracy on accepted answers, and at most 2% of blank answers getting an invented value past review) to be measured only on a private held-out set. These packets don't qualify, and the untuned baseline was already above 98% on the check packets, so this says more about the form than about the tuning. What I'm taking away - A tuning/checking split pays for itself, even a leaky one. The clearest harmful change looked neutral on the packet I tuned on. - Check the scorer, and look at what the model saw, before trusting a number. The first "gain" was a scoring convention; the two worst bugs were silent until I saw the crops; and adding the second model and turning on caching both moved the main model's crops. - Read what is used, and score before and after on the same set. - A second model should get a fair second look, not a harder question. - Check speed work answer by answer, including what the model was shown, and count a speed-up only under the batching you actually run. {% details The numbers, for engineers %} Every Scribe run used the same machine as parts 1 and 2. Main reader Qwen3.8-27B (8-bit), second model Qwen3-VL-8B (8-bit), served locally with mlx-vlm 0.6.15. 531 answers across 3 packets (one for tuning, two for checking), scored fairly ([] = blank). Accepted acc = accuracy of answers accepted without review. Escaped = wrong answers that were not sent to review. | Configuration | Exact | Accepted acc | Wrong | Escaped | Flags | Min / packet | |---|--:|--:|--:|--:|--:|--:| | Baseline (main model, crops by field type, no second model) | 93.2% | 97.5% | 36 | 11 | 18.1% | ~52 (older, larger schema) | | Tuned (fixes above, written scores only; on the tuning packet, only the changed pages were re-run) | 95.3% | 99.1% | 25 | 4 | 17.3% | ~36 (estimate) | | + second model (re-reads from the other source; moved 129 crops) | 94.9% | 99.2% | 27 | 3 | 26.7% | ~70 | | + same-image re-read, 4 at once, no page-type check, note cap | 96.0% | 99.8% | 21 | 1 | 21.8% | ~47 | | + prefix caching (current) | 96.2% | 99.5% | 20 | 2 | 22.2% | ~37 | Part 1's second target counts invented answers (a value where the key has a blank) that escape review: 3 of 160 blank answers at baseline (1.9%), 0 in the last two rows. Escaped in this post is broader: any wrong answer that skipped review. Check packets only (354 answers), exact / accepted acc / escaped: baseline 94.4% / 99.3% / 2; bold row 96.0% / 100.0% / 0 (of 275 accepted answers, which still allows a true error rate of about 1%); current 96.3% / 99.6% / 1. Paired comparisons (exact McNemar on the answers that changed): current vs baseline 16 fixed / 0 broken, p = 0.00003 (check packets only: 7 / 0, p = 0.016); bold row vs baseline 15 / 0, p = 0.00006 (check packets only: 6 / 0, p = 0.03); tuned vs baseline 12 / 1, p = 0.003 (check packets only: 5 / 1, p = 0.22); bold row vs tuned 4 / 0, p = 0.125; bold row vs the first second-model run 8 / 2, p = 0.11, with 7 of the 10 on crops that run had moved; skills vs no skills p = 0.73. Read these as rough: the all-packet figures include the packet the changes were chosen on, one fix often flips several answers on the same page, and I ran many comparisons. Prefix caching: the main model mixes two attention types, so mlx-vlm caches exact prompt snapshots, by default 16 tokens before the end of the prompt, a point no two questions share. A 4,000-token guard moves the snapshot to the end of the page image (cached prefix about 3,984 tokens; prompt processing for later whole-page questions about 6.6 s to 1.8 s). Cropped reads and the second model get no reuse. It isn't output-neutral, though: the main model's boxes moved for 89 of 270 cropped answers (median 2 px, largest 80 px), and 3 values changed. Other rounds: no JSON constraint on its own, no gain; speculative drafter 89% of drafted tokens accepted on the synthetic probe, +6.6% time vs the same setup without the JSON constraint and about +9% vs the current configuration. {% enddetails %} --- This is part 3 of a series. Part 1: Confidence Is Theater: What I Learned Building Scribe. Part 2: One Imperfect Page per Hundred: Testing Scribe on Data I Didn't Curate. Pipeline code and the public benchmarks from parts 1 and 2: kod201/scribe Every Scribe run in this post ran locally on one machine.
trying out @PrismML Bonsai 2 2B mlx model on the GRPO trainer in MLX-VLM, I will compare it with unsloth and trl. mlx-vlm#2504