allenai

Can LLMs track state changes over long texts—like an object changing hands or a variable reassignment? On our new StateBench eval, frontier models including GPT-6 Sol & Claude Opus 5.5 fell to 13% accuracy or lower on the hardest, longest problems. 🧵 allenai/state-bench t.co

Unsiloed Achieves #1 Rank on olmOCR-Bench By Adnan Abbas and Aman Mishra. Originally published on Unsiloed AI. Published by Unsiloed AI | May 2026 Introduction Optical Character Recognition (OCR) remains the foundation of any document AI pipeline. Whether you are extracting data from invoices, parsing academic papers, or digitizing handwritten forms, the accuracy of your OCR layer determines the ceiling for everything downstream. The olmOCR-Bench benchmark, published by Allen AI (allenai/olmocr), is a rigorous, reproducible evaluation of how well modern AI models handle real-world OCR challenges. It runs 8,413 unit tests across 1,403 PDF pages, covering math equations, complex tables, headers/footers, multi-column reading order, long/tiny text, and old scans. We compared Unsiloed Parser against 18 other widely-used OCR services and document-AI platforms. Unsiloed Parser was evaluated by us on olmOCR-Bench using the unmodified upstream scorer and other services were also independently measured on the exact same harness (GPT-5.5, Claude Opus 4.7, LlamaParse (Agentic), Reducto (Agentic), Extend (Agentic), Landing AI, Azure Document Intelligence, AWS Textract and Unstructured. Full report and per-vendor runner code to reproduce the scores is at Unsiloed-AI/unsiloed-olmocr-benchmark). The scores for Nanonets OCR-3 and Nanonets OCR2+ couldn't be verified independently and was sourced from publicly-published olmOCR-Bench results, all run against the same upstream scorer, same dataset, and the same 7-category JSONL test files (verified md5-identical to our snapshot). Unsiloed Parser leads at 88.0 on the deterministic pass-rate. We also re-evaluated our failures using an LLM-as-a-Judge protocol (explained later in the post), which lifts the corrected score to 94.8 when semantically-equivalent rewrites are no longer counted as failures. Here are the results.  View full-size chart ------------------------------------------------------------------------ What olmOCR-Bench Tests The benchmark evaluates seven document-content categories using pass/fail unit tests across 1,403 single-page PDFs sourced from arXiv, the Internet Archive, and curated internal collections: Math on Research Papers (ArXiv Math) Can the model accurately convert complex mathematical notation like fractions, integrals, summation symbols, matrices into KaTeX-renderable LaTeX on clean, typeset research papers? One of the most demanding OCR tasks and a strong differentiator between models. Math on Scanned / Historical Pages (Old Scans Math) The same KaTeX-renderable-LaTeX requirement, but applied to mathematical content on scanned, low-DPI, or handwritten pages, the hardest math sub-task in the benchmark. Table Structure Preservation (Tables) Documents are full of tables with merged cells, nested headers, and irregular layouts. The benchmark tests whether models maintain logical structure, not just extract the text, but understand which cell belongs to which row and column. Header & Footer Handling (H&F) Page numbers, footnotes, running headers, and section markers create noise. The benchmark checks whether models correctly suppress these elements without mixing them into body text. Multi-Column Reading Order (Multi-Col) Academic papers, newspapers, and reports often use multi-column layouts. The benchmark evaluates whether models read text in the correct semantic order. Long / Tiny Text (Long/Tiny) Footnotes, captions, fine print, content that's small or runs across many lines. Tests OCR fidelity on edge cases that production documents are full of. Degraded / Historical Scans (Old Scans) Low-quality, low-DPI, or otherwise degraded images. The hardest category by a wide margin. ------------------------------------------------------------------------ How We Ran the Benchmark The bar chart and per-category breakdown below combine two sources, both scored with the same olmocr==0.4.27 benchmark code on the same allenai/olmOCR-bench dataset. Full benchmark results were obtained for Unsiloed Parser plus other services. - Unsiloed Parser - GPT-5.5 — OpenAI flagship VLM, full_no_document_anchoring prompt - Claude Opus 4.7 — Anthropic flagship via Vertex AI - LlamaParse (Agentic) — LlamaIndex's premium parse_page_with_agent mode - Reducto (Agentic) — Reducto's premium agentic pipeline (text + table + figure agents, with advanced_chart_agent=true), each scope receiving the same full_no_document_anchoring prompt - Extend (Agentic) — Extend AI's Parse API with parse_performance engine, agentic text + tables, formulas, and advanced charts enabled - Landing AI — Agentic Document Extraction (default tier) - Azure Document Intelligence — prebuilt-layout - AWS Textract — AnalyzeDocument (synchronous) - Unstructured — partition (hi_res strategy, legacy endpoint) For each commercial service the highest-quality publicly-documented configuration the vendor exposes was used i.e. premium / agentic / hi-res modes where they exist, full layout + table + math features turned on, and the vendor's flagship model where a choice was offered. Outputs were post-processed identically before scoring, so no system was disadvantaged by a default-only configuration. Full per-service config and runner source code is committed to our open-source benchmarks repo at Unsiloed-AI/unsiloed-olmocr-benchmark. Unsiloed Parser leads the field at 88.0 on the deterministic pass-rate, with a +0.6-point gap over the next closest competitor (Nanonets OCR-3 at 87.4) and +3.4 points over the next closest vision-LLM (GPT-5.5 at 84.6). Among dedicated document-AI services in our in-house run, Unsiloed Parser leads LlamaParse Agentic by 14.5 points, Landing AI by 18.5 points, and Azure DI / Textract / Unstructured by 39 points or more. ------------------------------------------------------------------------ Per-Category Score Breakdown  View full-size chart The full per-category table covers all 19 services. Unsiloed Parser leads or ties for first on 4 of 7 sub-categories: ArXiv Math (90.5), Long/Tiny (93.7), Multi-Col (87.9), and Old Scans (52.9). Nanonets OCR-3 wins Tables (94.2, with Unsiloed Parser second at 93.2), Nanonets OCR2+ wins H&F (96.8, ahead of Nanonets OCR-3's 96.6 and Unsiloed's 94.6), and LlamaParse Agentic wins Old Scans Math (89.7, ahead of Nanonets OCR-3's 88.9 and Unsiloed's 88.2). ------------------------------------------------------------------------ olmOCR-Bench Results | Rank | Service | Provider | Score | |:-----:|-----------------------------|-----------------|:--------:| | 1 | Unsiloed Parser | Unsiloed AI | 88.0 | | 2 | Nanonets OCR-3 | Nanonets | 87.4 | | 3 | GPT-5.5 | OpenAI | 84.6 | | 4 | Datalab Marker | Datalab | 83.2 | | 5 | Nanonets OCR2+ | Nanonets | 82.0 | | 6 | Claude Opus 4.7 | Anthropic | 81.9 | | 7 | GPT-5.4 | OpenAI | 81.0 | | 8 | Qwen3-VL-Plus | Alibaba | 77.9 | | 9 | Gemini 3 Pro | Google | 77.7 | | 10 | Claude Sonnet 4.6 | Anthropic | 73.9 | | 11 | LlamaParse (Agentic) | LlamaIndex | 73.5 | | 12 | Mistral Small 4 | Mistral AI | 69.6 | | 13 | Landing AI | Landing AI | 69.5 | | 14 | GLM-OCR | Zhipu AI | 68.4 | | 15 | Reducto (Agentic) | Reducto AI | 66.0 | | 16 | Extend (Agentic) | Extend AI | 64.0 | | 17 | Azure Document Intelligence | Microsoft | 48.7 | | 18 | AWS Textract | Amazon | 40.2 | | 19 | Unstructured | Unstructured AI | 39.9 | Scores were evaluated against allenai/olmOCR-bench (1,403 PDFs / 8,413 tests) using the unmodified upstream olmocr==0.4.27 scorer, with per-service runner code open-sourced at Unsiloed-AI/unsiloed-olmocr-benchmark. ------------------------------------------------------------------------ Why a Second Evaluation? Strict Pass-Rate vs. Semantic Correctness olmOCR-Bench's headline metric is the deterministic, unit-test-driven pass-rate that we just reported. It's the right way to compare systems publicly: it's reproducible, it requires no second model in the loop, and the rules are fixed. 88.0 is our number on that metric, and it's the one that matters for ranking against the public leaderboard. But once we had a leaderboard-leading number, we wanted to understand the remaining ~12% of "failures", what was actually wrong, and how much of it would matter to a downstream user reading the parsed document. What the strict tests actually flag olmOCR's strict tests are exact-string checks against a gold answer. By design, that strictness is what catches subtle errors but it also means any divergence from the reference string is recorded as a failure, even when the divergence is semantically harmless. In practice that includes things like: - A ground truth of "I am Name" (two spaces) vs. a prediction of "I am Name" (one space), scored as 0 even though no human reader would see a difference. - A math equation written with the literal Greek letter ε (Epsilon) rather than \epsilon, the LaTeX form passes, the rendered glyph fails, despite producing identical output downstream. - \frac{a}{b} vs. \dfrac{a}{b} vs. \tfrac{a}{b} (identical LaTeX rendering). - \cdot vs. \times for multiplication, or \coloneqq vs. :=. - "0.70" vs. "0. 70" inside a table cell. - twenty-five vs. twenty five (hyphenation), ASCII apostrophes vs. unicode curly quotes in transcribed handwriting. These aren't OCR errors, they're string-level divergences from the gold reference that don't change what a downstream reader, search index, or LLM consuming the parsed document would see. But under strict pass/fail rules, every one of them counts as a failure and pushes a benchmark score down without telling you anything useful about model quality. What LLM-as-a-Judge measures For our own system, we re-ran every strict failure through an LLM-as-a-Judge (GPT-5.5, reasoning_effort=medium, seed=42) and asked: "the strict rule failed, is the OCR output actually wrong, or did the rule reject a semantically-equivalent rewrite?" Each verdict is minor (rescue → count as pass) or real (genuine OCR miss). Strict tests catch the surface-level divergence; the judge tells us whether that divergence changes what a reader of the document would understand. This isn't a replacement for the strict score, it's a diagnostic on top of it that separates "rule said fail" from "the OCR was actually wrong." The result  View full-size chart Re-aggregating with the rescues applied to the new configuration, Unsiloed Parser's semantic-correctness score is 94.8% (with-baseline 8-group macro), a +6.8-pp lift over the strict 88.0 baseline. The breakdown by failure type (measured rescue rates on the current pipeline): - Tables: 64 of 70 failures (91%) are string-level divergences with no semantic impact; the table-to-string parser, not the OCR, is the bottleneck. - ArXiv Math: 243 of 277 failures (≈88%) are LaTeX-equivalent rewrites — \frac vs \dfrac, \cdot vs \times, spacing/brace variants. - Old Scans: the hardest category sees the largest +19-point lift (52.9 → 72.1), where many "failures" are hyphenation/whitespace differences in handwriting transcriptions. We didn't run the same LLM-as-Judge protocol against the other systems in this comparison, so we're not claiming a leaderboard rank on this metric. We share this number purely so readers can see what kind of failures remain at the top of the strict leaderboard, and judge for themselves whether those failures matter for their use case. ------------------------------------------------------------------------ Limitations of current benchmarks Looking past the leaderboard standings, our analysis surfaces fundamental problems with the way document-parsing systems are currently measured. Today's OCR benchmarks score systems against rigid string-level rules. As we showed in the LLM-as-a-Judge analysis above, the majority of "failures" at the top of the leaderboard are not OCR errors at all, they're whitespace, hyphenation, and notation rewrites that don't change document meaning. (On the new Unsiloed run, ~65% of strict failures by raw count and ~61% averaged per category are LaTeX/whitespace/punctuation-equivalent rewrites; the macro-weighted rescue rate is ~57%.) There is currently no widely-adopted benchmark that scores OCR systems on semantic correctness against the ground truth. Current OCR benchmarks were originally designed around academic-document workloads and may underrepresent emerging production document patterns. Existing benchmarks lean heavily on tables and equations content that was central to the academic-paper datasets these benchmarks were originally built from while underweighting the elements that dominate today's real-world document mix: charts and plots, multi-column layout, key-value pairs, forms, and structured fields. We're working on closing both gaps i.e. the semantic-aware evaluation and broader category coverage and plan to share more on this in the coming months. ------------------------------------------------------------------------ Try It Yourself If you want to skip the benchmark and just see Unsiloed Parser on your own documents, try the Unsiloed platform at unsiloed.ai. Upload a document and compare extraction results across models side-by-side. No code required. To reproduce the numbers or run your own vendor against the same harness, the full benchmarks repo, per-service runners, raw scorer logs, and JSON metrics is open-source at Unsiloed-AI/unsiloed-olmocr-benchmark. ------------------------------------------------------------------------ Logos and trademarks are the property of their respective owners.Use does not imply endorsement Data sourced using the unmodified olmocr==0.4.27 scorer (allenai/olmocr) on the allenai/olmOCR-bench dataset. Scores as of May 2026.

MolmoSpaces-CommonSense: Evaluating Commonsense in Robotic Manipulation Policies Real-world robotic manipulation requires reasoning about cluttered scenes, partially observable or hidden objects, and semantically appropriate actions, in addition to low-level execution within those scenes. However, existing simulation benchmarks primarily test the latter and remain dominated by comparatively clean environments with limited clutter and narrowly scoped task structure, leaving many forms of commonsense reasoning untested. We address this by introducing a taxonomy of reasoning processes for robotic manipulation grounded in developmental cognitive science. We operationalize this taxonomy through a new benchmark of robot tasks, MolmoSpaces-CommonSense, that requires reasoning about occlusion, containment, support relations, semantically correct grasp points, spatial affordances, and hidden objects. These evaluations require policies not only to recognize relevant scene structure but also to execute correct actions in them. We evaluate several zero-shot generalist manipulation policies. Across model types and architectures, we observe consistent failures in commonsense reasoning: success rates in cluttered scenes drop by over 50% as distractors increase, policies act on incorrect objects in cluttered settings 30% of the time, and affordance-sensitive grasping remains unreliable even with highly specific language prompts, with correct object part selection happening only about 50% of the time. Further, VLM-based success estimation struggles to reliably evaluate rollouts in these settings, suggesting that the challenge extends beyond control to the broader perception-reasoning-action stack. Our results suggest that existing benchmarks may substantially overestimate zero-shot real-world readiness. Code and policy leaderboard link at allenai/molmospaces. Comments: Project page: allenai/molmospaces

(11/n) Finally, a huge thanks to my coauthors and advisors @max_argus, @jjaesungpark, @DJiafei, @RanjayKrishna, and Ali Farhadi! 🌐 Leaderboard (CommonSense tab): molmospaces.allen.ai 💻 Code: allenai/molmospaces 📄 Paper: github.com/allenai/molmospaces/blob/commonsense-p…

Stop waiting for these companies to announce opportunities. Start watching their research GitHub organizations. Here are some worth following: Hardware & GPU research github.com/NVlabs github.com/NVIDIA github.com/ROCm github.com/AMD github.com/intel Big Tech research github.com/google-research github.com/google-deepmind github.com/microsoft github.com/facebookresearch github.com/amazon-science AI research labs github.com/openai github.com/anthropics github.com/deepseek-ai github.com/QwenLM github.com/allenai github.com/mistralai github.com/EleutherAI github.com/LAION-AI Cloud & infrastructure github.com/awslabs github.com/GoogleCloudPlatform github.com/cloudflare github.com/vercel-labs github.com/databricks

Before trusting a novelty judge, count the verdicts that move A judge can keep the same accuracy while changing its mind about half the examples. The average looks calm. The individual verdicts are doing parkour. 🤸 That is a synthetic counterexample, not a new model result. It is also the first diagnostic I would put beside a novelty-judge leaderboard after reading AllenAI’s Old Ideas, Novel Problems release. [1] !Four synthetic idea pairs have the same 75 percent accuracy before and after a prompt change, but pairs p2 and p4 change verdict, giving a 50 percent flip rate. An illustrative four-pair dataset. No LLM generated these decisions. The prompt changed more than its length AllenAI reports that GPT-5.4’s pairwise accuracy on its Human+Generated setting moved from 93.5% to 40.9% between two prompts. Those are the authors’ results, not a benchmark I reran. [1] The useful detail is in the released templates. One asks the judge to identify the idea that conference reviewers judged more novel. The other asks it to identify the idea that is more novel. [2] Those are related targets, but they are not identical. Predicting a reviewer-backed ordering can encourage a different decision rule from making an independent novelty assessment. The source change is a reason to inspect the evaluation contract, rather than treating prompt wording as a harmless wrapper. 🔍 I would not read this experiment as proof that one particular wording is universally better. The paper’s setup is part of the result. Two measurements, two questions Accuracy asks whether a decision agrees with a reference label. A flip rate asks whether the same pair receives a different decision under another setup. For an aligned set of pair IDs: flip rate = changed verdicts / compared pairs You can compute that disagreement without knowing which verdict is correct. That makes it useful when reference labels are expensive, incomplete or disputed. It cannot tell you which prompt to trust. Here is a small executable example. Both runs get three of four synthetic labels right. One fixes p4 while breaking p2, so the net accuracy movement is zero. The changed-verdict count is two. ✅ def audit(before, after, gold): if not before or before.keys() != after.keys() or before.keys() != gold.keys(): raise ValueError("Expected matching, nonempty pair IDs") if any(v not in {"A", "B"} for rows in (before, after, gold) for v in rows.values()): raise ValueError("Handle missing, tied, or invalid judgments explicitly") n = len(before) accuracy = lambda rows: sum(rows[k] == gold[k] for k in gold) / n flips = sum(before[k] != after[k] for k in before) return accuracy(before), accuracy(after), flips, n gold = dict.fromkeys(("p1", "p2", "p3", "p4"), "A") before = dict(zip(gold, ("A", "A", "A", "B"))) after = {"p4": "A", "p3": "A", "p2": "B", "p1": "A"} old_accuracy, new_accuracy, flips, n = audit(before, after, gold) print(f"Accuracy: {old_accuracy:.0%} -> {new_accuracy:.0%}") print(f"Verdicts changed: {flips}/{n} ({flips / n:.0%})") Output: Accuracy: 75% -> 75% Verdicts changed: 2/4 (50%) The second dictionary is deliberately in a different row order. The function aligns by ID, so sorting or asynchronous completion does not manufacture disagreement. I also checked unchanged decisions, completely reversed decisions, empty inputs, mismatched IDs and an invalid tie value. ⚠️ Do not silently join only successful rows. If a prompt produces missing or malformed judgments, dropping them changes the denominator. This small function rejects unequal coverage and nonbinary outputs so the caller has to decide how abstentions and failures are reported. For a real pipeline, reject duplicate IDs before turning records into dictionaries. Normalize any swapped A/B presentation back to stable idea identities. Otherwise a position swap can look like a changed scientific opinion. The labels deserve their own audit AllenAI separates Human-Only from Human+Generated data. The latter pairs reviewer-supported human contributions with ideas from a simple generator; the generated side is weakly labeled lower-novelty. The dataset documentation also says the human abstracts were edited to remove results, baselines and evaluation datasets. “Human” describes the idea’s origin, not a promise of untouched wording. [3] That matters when applying the benchmark elsewhere. A weak label can support a useful controlled study without becoming a universal verdict on every generated idea. The released expert annotation file is another easy denominator trap: its 30 cases contain 28 judge errors and two controls. It is a deliberately selected error analysis, not a random sample from which to estimate overall accuracy. [3] 💡 My practical reporting bundle would keep the reference-label metric, aligned flip rate, invalid-output counts and exact prompt versions together. It would also separate results by data setting rather than merge human and generated ideas into one reassuring average. This article checks source contracts and a synthetic audit program. I did not call any of the judges. A stable verdict is not necessarily a correct verdict, and a novelty score is not proof that no prior work exists. References and figure credit [1] AllenAI’s release and reported results. [2] Reviewer-referenced prompt and prompt without the reviewer reference. [3] Pinned dataset documentation, transformations and annotation sampling. Original synthetic verdict grid; no benchmark examples or reviewer excerpts redistributed. AI disclosure AI was used to draft the article.

🤖 How good are LLMs at judging scientific novelty? We built a benchmark to test them LLMs are increasingly used to judge whether AI-generated research ideas are novel. But are they any good at it? Our new eval reveals even frontier LLMs can reach different conclusions about similar ideas depending on how they're asked to evaluate them. 👇 At Ai2, we're building tools like AutoDiscovery to uncover unexpected patterns in data & Theorizer to propose candidate scientific laws from papers. As AI tools like these advance, we need reliable ways to tell whether the ideas they generate are genuinely new. To build our novelty-judge benchmark, we mined ICLR 2026 reviews for explicit comments on originality. We kept papers that reviewers agreed were (or were not) original & whose scores were unusually high or low within their research area, giving us examples of more and less novel work. We evaluated judge models on 2 sets of data: 👤👤 Human-only pairs (stronger vs. weaker evidence of novelty) 👤🤖 Human ideas w/ novelty evidence paired w/ basic LLM ideas, assumed less novel. Judges either picked the more novel idea in each pair or labeled single ideas as novel or not. On human–AI pairs, GPT-5.4 chose the human idea 93.5% of the time when asked which idea reviewers had judged novel. Asked simply which idea was novel, that share fell to 40.9%. Prompt wording had a smaller effect on which idea models chose in human-only pairs. Prompt changes that helped one judge could hurt another, & more reasoning or access to related papers didn't consistently improve results. And 2 purpose-built novelty judges performed worse than a simple prompted Claude Opus 4.6 baseline. We hope our benchmark helps researchers build more reliable novelty judges, while our reusable pipeline makes it easier to create new eval sets as AI-generated research advances. 💻 Blog: allenai.org 👨🏻‍💻 Code: allenai/novelty-eval 📊 Data: huggingface.co 📄 Paper: arxiv.org

📄 Paper: arxiv.org 🤗 Data: hf.co 💻 Code: allenai/novelty-eval With @its_sshahid @peterjansen_ai @dsweld @Siangliulue @Hoper_Tom

Choose a parser before feeding PDFs to AI | 6 open-source choices Bad parsing can ruin good AI answers. Six options, with different use cases and adoption checks. Docling, PaddleOCR, olmOCR, Tesseract, EasyOCR, Unstructured. Explore the original cards: channel profile → AI Open Source Radar → Collections. Choose the collection for this task and inspect the project cards. useaistation.com Sources checked 23 Oct 2026. This is a project-selection guide, not hands-on testing or a live ranking. These are different choices, not a verified integrated stack. Software licenses, model weights, hardware and external service costs must be checked separately. Community/core and enterprise features may differ. Developer disclosure: we build AI Open Source Radar, not the featured projects. Narration: Microsoft free Aria neural voice. Project and License sources (observed branch references, subject to change): Docling: docling-project/docling License: raw.githubusercontent.com PaddleOCR: PaddlePaddle/PaddleOCR License: raw.githubusercontent.com olmOCR: allenai/olmocr License: raw.githubusercontent.com Tesseract: tesseract-ocr/tesseract License: raw.githubusercontent.com EasyOCR: JaidedAI/EasyOCR License: raw.githubusercontent.com Unstructured: Unstructured-IO/unstructured License: raw.githubusercontent.com #OpenSource #AITools #DocumentAI

One setup detail: the Docker images include the dependencies, but Olmo-core itself still needs to be installed. The images are tested on Allen AI’s H100 clusters. Different hardware, drivers or CUDA versions may require adapting the Dockerfile. GitHub: allenai/Olmo-core