Ai2 examines how prompts change AI judgments of novelty

A model can generate an idea, and another model can judge its novelty. But how stable is that judgment? Ai2’s October 27 study examines scientific-idea originality assessment. It also offers a useful caution for creators using LLMs to shortlist concepts: a confident verdict may depend heavily on the assessment prompt.

How the study was built

The authors used ICLR 2026 reviews to identify explicit signals about originality. They did not treat paper acceptance as proof of novelty, since a paper can be accepted for other strengths. They compared human-authored ideas with each other and with ideas from a basic LLM generator.

Generated ideas used weak labels rather than verified truth for every example. This matters: the work investigates assessment stability, not a universal measure of creative value.

What changed the verdicts

The team varied instructions, comparison format, reasoning budget and access to related literature. In one prompt change, GPT-5.4 accuracy on Human+Generated fell from 93.5% to 40.9% for the same pairs. These figures describe that experiment, not general model accuracy.

The authors found that more retrieval or computation did not consistently improve results. A judge appearing stable on human research could behave differently on generated ideas. Evaluate a judge on the kind of material it will actually assess.

Implications for creative selection

Applying this to scripts or advertisements is editorial reasoning, not a result demonstrated by the scientific benchmark. The study does not establish that models are useless for creative work. It supports caution around an automatic originality score and encourages separating criteria.

Clarity, brief compliance, feasibility and novelty ask different questions. A single overall rank can let polished wording obscure a weaker concept. Request explanations for each criterion and inspect the actual features supporting each assessment.

A suggested editorial procedure

• Define the objective and criteria before comparing entries.
• Present ideas in the same structure and comparable length.
• Repeat with the order swapped while preserving the criterion.
• For disputed decisions, test a reasonable rewording of the question.
• Keep a person responsible for choices affecting budget or the rejection of an important concept.

This procedure can expose instability; it does not guarantee a correct ranking. Agreement between several answers is not proof of novelty either. Scientific novelty requires literature assessment, while advertising concepts need their own contextual research. This article does not introduce an automated originality judge into VIZURAI.