FreeLLMCan we trust LLMs to judge whether a research idea is novel?
Six leading LLMs give wildly different novelty assessments, with accuracy swings over 50 points after minor prompt tweaks.
The benchmark uses two pools: 154 high‑novelty ideas from top‑rated ICLR papers and 145 low‑novelty ideas from rejected papers, plus a matched set of AI‑generated ideas.
Small prompt changes altered a model’s accuracy by more than 50 percentage points.
Adding retrieval or extra reasoning steps did not consistently improve performance, and two purpose‑built novelty systems performed worse than a simple Claude Opus 4.6 baseline.
The authors warn that current novelty judges are unstable, making it hard to rely on LLMs for automated scientific discovery.
What it means for you
Researchers cannot yet trust LLMs to filter genuinely new ideas, so human review remains essential for scientific evaluation. Users of free or locally hosted models—such as Kimi K3, GLM‑5.3, MiMo‑V2.6‑Pro, DeepSeek‑V4.1‑Flash, and GLM‑5.3‑Flash—should be aware that novelty assessment tools built on these models may inherit the same instability.
On FreeLLMArena
From our rankings on 27 Oct 2026, based on Arena's blind votes:
- Kimi K3 (Max): #1 among free chat models
Sources
- Can we trust LLMs to judge whether a research idea is novel?, Hugging Face (official), 27 Oct 2026
Written automatically from the sources above, then checked by our code: every number and name must appear in them, and no sentence may copy a source. Spotted a mistake? Report it.
Which AI can you use for free today?
Every free AI model, ranked by blind human votes, with the memory and speed it needs on your own GPU.