FreeLLMCan we trust LLMs to judge whether a research idea is novel?
Six leading LLMs give wildly different novelty assessments, with accuracy swings over 50 points after minor prompt tweaks.
The benchmark uses two pools: 154 high‑novelty ideas from top‑rated ICLR papers and 145 low‑novelty ideas from rejected papers, plus a matched set of AI‑generated ideas.
Small prompt changes altered a model’s accuracy by more than 50 percentage points.
Adding retrieval or extra reasoning steps did not consistently improve performance, and two purpose‑built novelty systems performed worse than a simple Claude Opus 4.6 baseline.
The authors warn that current novelty judges are unstable, making it hard to rely on LLMs for automated scientific discovery.
O que isso significa para você
Researchers cannot yet trust LLMs to filter genuinely new ideas, so human review remains essential for scientific evaluation. Users of free or locally hosted models—such as Kimi K3, GLM‑5.3, MiMo‑V2.6‑Pro, DeepSeek‑V4.1‑Flash, and GLM‑5.3‑Flash—should be aware that novelty assessment tools built on these models may inherit the same instability.
No FreeLLMArena
Dos nossos rankings em 31 out 2026, baseados nas votações às cegas da Arena:
- Kimi K3 (Max): #1 among free chat models
Fontes
- Can we trust LLMs to judge whether a research idea is novel?, Hugging Face (official), 27 Oct 2026
Escrito automaticamente a partir das fontes acima, depois verificado pelo nosso código: todo número e nome deve aparecer nelas, e nenhuma frase pode copiar uma fonte. Encontrou um erro? Reporte‑o.
Qual IA você pode usar grátis hoje?
Todo modelo de IA gratuito, classificado por votos humanos cegos, com a memória e velocidade que ele precisa na sua própria GPU.