Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers

The preprint operates within the established paradigm of using large language models (LLMs) for various Natural Language Processing (NLP) tasks, including idea generation. It specifically addresses the emergent area of research ideation using LLMs, an area still solidifying its own potential paradigms.

This preprint presents a large-scale human evaluation of LLM-generated research ideas compared to those from expert NLP researchers. While the study design and execution fall under 'normal science' within the broader LLM paradigm, the results indicate a potential shift in how research ideation is perceived and conducted. The findings of LLMs generating statistically more novel ideas challenge the existing model of relying solely on human expertise for ideation. This does not constitute a full 'model crisis' or 'revolution' yet, as the LLM approach has limitations (lack of diversity, feasibility concerns), and the evaluation is limited to prompting-based NLP research. The study identifies these limitations and proposes further research directions, suggesting it contributes to 'model drift', refining and potentially expanding the current LLM paradigm in research. Further investigation, particularly the proposed execution phase comparing realized outcomes of LLM- vs. human-generated ideas, is needed to confirm the degree of drift or whether this constitutes a stronger shift toward a new model of research ideation.