Before trusting a novelty judge, count the verdicts that move
A judge can keep the same accuracy while changing its mind about half the examples. The average looks calm. The individual verdicts are doing parkour. 🤸
That is a synthetic counterexample, not a new model result. It is also the first diagnostic I would put beside a novelty-judge leaderboard after reading AllenAI’s Old Ideas, Novel Problems release. [1]
!Four synthetic idea pairs have the same 75 percent accuracy before and after a prompt change, but pairs p2 and p4 change verdict, giving a 50 percent flip rate.
An illustrative four-pair dataset. No LLM generated these decisions.
The prompt changed more than its length
AllenAI reports that GPT-5.4’s pairwise accuracy on its Human+Generated setting moved from 93.5% to 40.9% between two prompts. Those are the authors’ results, not a benchmark I reran. [1]
The useful detail is in the released templates. One asks the judge to identify the idea that conference reviewers judged more novel. The other asks it to identify the idea that is more novel. [2]
Those are related targets, but they are not identical. Predicting a reviewer-backed ordering can encourage a different decision rule from making an independent novelty assessment. The source change is a reason to inspect the evaluation contract, rather than treating prompt wording as a harmless wrapper. 🔍
I would not read this experiment as proof that one particular wording is universally better. The paper’s setup is part of the result.
Two measurements, two questions
Accuracy asks whether a decision agrees with a reference label. A flip rate asks whether the same pair receives a different decision under another setup.
For an aligned set of pair IDs:
flip rate = changed verdicts / compared pairs
You can compute that disagreement without knowing which verdict is correct. That makes it useful when reference labels are expensive, incomplete or disputed. It cannot tell you which prompt to trust.
Here is a small executable example. Both runs get three of four synthetic labels right. One fixes p4 while breaking p2, so the net accuracy movement is zero. The changed-verdict count is two. ✅
def audit(before, after, gold):
if not before or before.keys() != after.keys() or before.keys() != gold.keys():
raise ValueError("Expected matching, nonempty pair IDs")
if any(v not in {"A", "B"} for rows in (before, after, gold) for v in rows.values()):
raise ValueError("Handle missing, tied, or invalid judgments explicitly")
n = len(before)
accuracy = lambda rows: sum(rows[k] == gold[k] for k in gold) / n
flips = sum(before[k] != after[k] for k in before)
return accuracy(before), accuracy(after), flips, n
gold = dict.fromkeys(("p1", "p2", "p3", "p4"), "A")
before = dict(zip(gold, ("A", "A", "A", "B")))
after = {"p4": "A", "p3": "A", "p2": "B", "p1": "A"}
old_accuracy, new_accuracy, flips, n = audit(before, after, gold)
print(f"Accuracy: {old_accuracy:.0%} -> {new_accuracy:.0%}")
print(f"Verdicts changed: {flips}/{n} ({flips / n:.0%})")
Output:
Accuracy: 75% -> 75%
Verdicts changed: 2/4 (50%)
The second dictionary is deliberately in a different row order. The function aligns by ID, so sorting or asynchronous completion does not manufacture disagreement. I also checked unchanged decisions, completely reversed decisions, empty inputs, mismatched IDs and an invalid tie value.
⚠️ Do not silently join only successful rows. If a prompt produces missing or malformed judgments, dropping them changes the denominator. This small function rejects unequal coverage and nonbinary outputs so the caller has to decide how abstentions and failures are reported.
For a real pipeline, reject duplicate IDs before turning records into dictionaries. Normalize any swapped A/B presentation back to stable idea identities. Otherwise a position swap can look like a changed scientific opinion.
The labels deserve their own audit
AllenAI separates Human-Only from Human+Generated data. The latter pairs reviewer-supported human contributions with ideas from a simple generator; the generated side is weakly labeled lower-novelty. The dataset documentation also says the human abstracts were edited to remove results, baselines and evaluation datasets. “Human” describes the idea’s origin, not a promise of untouched wording. [3]
That matters when applying the benchmark elsewhere. A weak label can support a useful controlled study without becoming a universal verdict on every generated idea.
The released expert annotation file is another easy denominator trap: its 30 cases contain 28 judge errors and two controls. It is a deliberately selected error analysis, not a random sample from which to estimate overall accuracy. [3]
💡 My practical reporting bundle would keep the reference-label metric, aligned flip rate, invalid-output counts and exact prompt versions together. It would also separate results by data setting rather than merge human and generated ideas into one reassuring average.
This article checks source contracts and a synthetic audit program. I did not call any of the judges. A stable verdict is not necessarily a correct verdict, and a novelty score is not proof that no prior work exists.
References and figure credit
[1] AllenAI’s release and reported results.
[2] Reviewer-referenced prompt and prompt without the reviewer reference.
[3] Pinned dataset documentation, transformations and annotation sampling.
Original synthetic verdict grid; no benchmark examples or reviewer excerpts redistributed.
AI disclosure
AI was used to draft the article.
2 reactions
🤖 How good are LLMs at judging scientific novelty? We built a benchmark to test them
LLMs are increasingly used to judge whether AI-generated research ideas are novel. But are they any good at it?
Our new eval reveals even frontier LLMs can reach different conclusions about similar ideas depending on how they're asked to evaluate them. 👇
At Ai2, we're building tools like AutoDiscovery to uncover unexpected patterns in data & Theorizer to propose candidate scientific laws from papers. As AI tools like these advance, we need reliable ways to tell whether the ideas they generate are genuinely new.
To build our novelty-judge benchmark, we mined ICLR 2026 reviews for explicit comments on originality. We kept papers that reviewers agreed were (or were not) original & whose scores were unusually high or low within their research area, giving us examples of more and less novel work.
We evaluated judge models on 2 sets of data:
👤👤 Human-only pairs (stronger vs. weaker evidence of novelty)
👤🤖 Human ideas w/ novelty evidence paired w/ basic LLM ideas, assumed less novel.
Judges either picked the more novel idea in each pair or labeled single ideas as novel or not.
On human–AI pairs, GPT-5.4 chose the human idea 93.5% of the time when asked which idea reviewers had judged novel. Asked simply which idea was novel, that share fell to 40.9%. Prompt wording had a smaller effect on which idea models chose in human-only pairs.
Prompt changes that helped one judge could hurt another, & more reasoning or access to related papers didn't consistently improve results. And 2 purpose-built novelty judges performed worse than a simple prompted Claude Opus 4.6 baseline.
We hope our benchmark helps researchers build more reliable novelty judges, while our reusable pipeline makes it easier to create new eval sets as AI-generated research advances.
💻 Blog: allenai.org
👨🏻💻 Code: allenai/novelty-eval
📊 Data: huggingface.co
📄 Paper: arxiv.org
30 points · 1 comment
Mentions
Before trusting a novelty judge, count the verdicts that move A judge can keep the same accuracy while changing its mind about half the examples. The average looks calm. The individual verdicts are doing parkour. 🤸 That is a synthetic counterexample, not a new model result. It is also the first diagnostic I would put beside a novelty-judge leaderboard after reading AllenAI’s Old Ideas, Novel Problems release. [1] !Four synthetic idea pairs have the same 75 percent accuracy before and after a prompt change, but pairs p2 and p4 change verdict, giving a 50 percent flip rate. An illustrative four-pair dataset. No LLM generated these decisions. The prompt changed more than its length AllenAI reports that GPT-5.4’s pairwise accuracy on its Human+Generated setting moved from 93.5% to 40.9% between two prompts. Those are the authors’ results, not a benchmark I reran. [1] The useful detail is in the released templates. One asks the judge to identify the idea that conference reviewers judged more novel. The other asks it to identify the idea that is more novel. [2] Those are related targets, but they are not identical. Predicting a reviewer-backed ordering can encourage a different decision rule from making an independent novelty assessment. The source change is a reason to inspect the evaluation contract, rather than treating prompt wording as a harmless wrapper. 🔍 I would not read this experiment as proof that one particular wording is universally better. The paper’s setup is part of the result. Two measurements, two questions Accuracy asks whether a decision agrees with a reference label. A flip rate asks whether the same pair receives a different decision under another setup. For an aligned set of pair IDs: flip rate = changed verdicts / compared pairs You can compute that disagreement without knowing which verdict is correct. That makes it useful when reference labels are expensive, incomplete or disputed. It cannot tell you which prompt to trust. Here is a small executable example. Both runs get three of four synthetic labels right. One fixes p4 while breaking p2, so the net accuracy movement is zero. The changed-verdict count is two. ✅ def audit(before, after, gold): if not before or before.keys() != after.keys() or before.keys() != gold.keys(): raise ValueError("Expected matching, nonempty pair IDs") if any(v not in {"A", "B"} for rows in (before, after, gold) for v in rows.values()): raise ValueError("Handle missing, tied, or invalid judgments explicitly") n = len(before) accuracy = lambda rows: sum(rows[k] == gold[k] for k in gold) / n flips = sum(before[k] != after[k] for k in before) return accuracy(before), accuracy(after), flips, n gold = dict.fromkeys(("p1", "p2", "p3", "p4"), "A") before = dict(zip(gold, ("A", "A", "A", "B"))) after = {"p4": "A", "p3": "A", "p2": "B", "p1": "A"} old_accuracy, new_accuracy, flips, n = audit(before, after, gold) print(f"Accuracy: {old_accuracy:.0%} -> {new_accuracy:.0%}") print(f"Verdicts changed: {flips}/{n} ({flips / n:.0%})") Output: Accuracy: 75% -> 75% Verdicts changed: 2/4 (50%) The second dictionary is deliberately in a different row order. The function aligns by ID, so sorting or asynchronous completion does not manufacture disagreement. I also checked unchanged decisions, completely reversed decisions, empty inputs, mismatched IDs and an invalid tie value. ⚠️ Do not silently join only successful rows. If a prompt produces missing or malformed judgments, dropping them changes the denominator. This small function rejects unequal coverage and nonbinary outputs so the caller has to decide how abstentions and failures are reported. For a real pipeline, reject duplicate IDs before turning records into dictionaries. Normalize any swapped A/B presentation back to stable idea identities. Otherwise a position swap can look like a changed scientific opinion. The labels deserve their own audit AllenAI separates Human-Only from Human+Generated data. The latter pairs reviewer-supported human contributions with ideas from a simple generator; the generated side is weakly labeled lower-novelty. The dataset documentation also says the human abstracts were edited to remove results, baselines and evaluation datasets. “Human” describes the idea’s origin, not a promise of untouched wording. [3] That matters when applying the benchmark elsewhere. A weak label can support a useful controlled study without becoming a universal verdict on every generated idea. The released expert annotation file is another easy denominator trap: its 30 cases contain 28 judge errors and two controls. It is a deliberately selected error analysis, not a random sample from which to estimate overall accuracy. [3] 💡 My practical reporting bundle would keep the reference-label metric, aligned flip rate, invalid-output counts and exact prompt versions together. It would also separate results by data setting rather than merge human and generated ideas into one reassuring average. This article checks source contracts and a synthetic audit program. I did not call any of the judges. A stable verdict is not necessarily a correct verdict, and a novelty score is not proof that no prior work exists. References and figure credit [1] AllenAI’s release and reported results. [2] Reviewer-referenced prompt and prompt without the reviewer reference. [3] Pinned dataset documentation, transformations and annotation sampling. Original synthetic verdict grid; no benchmark examples or reviewer excerpts redistributed. AI disclosure AI was used to draft the article.
🤖 How good are LLMs at judging scientific novelty? We built a benchmark to test them LLMs are increasingly used to judge whether AI-generated research ideas are novel. But are they any good at it? Our new eval reveals even frontier LLMs can reach different conclusions about similar ideas depending on how they're asked to evaluate them. 👇 At Ai2, we're building tools like AutoDiscovery to uncover unexpected patterns in data & Theorizer to propose candidate scientific laws from papers. As AI tools like these advance, we need reliable ways to tell whether the ideas they generate are genuinely new. To build our novelty-judge benchmark, we mined ICLR 2026 reviews for explicit comments on originality. We kept papers that reviewers agreed were (or were not) original & whose scores were unusually high or low within their research area, giving us examples of more and less novel work. We evaluated judge models on 2 sets of data: 👤👤 Human-only pairs (stronger vs. weaker evidence of novelty) 👤🤖 Human ideas w/ novelty evidence paired w/ basic LLM ideas, assumed less novel. Judges either picked the more novel idea in each pair or labeled single ideas as novel or not. On human–AI pairs, GPT-5.4 chose the human idea 93.5% of the time when asked which idea reviewers had judged novel. Asked simply which idea was novel, that share fell to 40.9%. Prompt wording had a smaller effect on which idea models chose in human-only pairs. Prompt changes that helped one judge could hurt another, & more reasoning or access to related papers didn't consistently improve results. And 2 purpose-built novelty judges performed worse than a simple prompted Claude Opus 4.6 baseline. We hope our benchmark helps researchers build more reliable novelty judges, while our reusable pipeline makes it easier to create new eval sets as AI-generated research advances. 💻 Blog: allenai.org 👨🏻💻 Code: allenai/novelty-eval 📊 Data: huggingface.co 📄 Paper: arxiv.org
📄 Paper: arxiv.org 🤗 Data: hf.co 💻 Code: allenai/novelty-eval With @its_sshahid @peterjansen_ai @dsweld @Siangliulue @Hoper_Tom