An image of a speech bubble embedded in another speech bubble. The inner bubble has a line graph, octopus, and paperclip emoji followed by a question mark.
The Results Are In!

About the Survey

What do NLP researchers think about controversial NLP issues? And perhaps more interestingly: What do NLP researchers think other NLP researchers think about controversial NLP issues?

Recent rapid changes in the NLP landscape have raised many questions. For example:

  • Are we devoting too many resources to scaling up?
  • Do language models understand language? Will they ever?
  • Is the traditional paradigm of model benchmarking still tenable?
  • What kinds of predictive models are ethical for researchers to build and release?
  • Will the next most influential advances come from industry or academic labs?

On these and similar questions, it’s often unclear what counts as general consensus, what is still actively debated, or what is widely considered overturned. We believe it would benefit the field to have a better understanding of the current landscape of community opinions and beliefs.

To this end, we conducted the NLP Community Metasurvey from May to June 2022. This survey covers over 30 potentially controversial positions, asking not only whether you agree, but what percentage of the community you think agrees with the position, exposing the difference between what people think and what people think people think. This idea was inspired by the PhilPapers Surveys.

The results and analysis are descriptive, not prescriptive, as such issues cannot be resolved by majority vote. By necessity, we cover a subjectively chosen set of questions and reduce many complex issues into simplified scales, but we hope that the outcome can serve as a reference point for community discussion and for future surveys. This is not the final word in any debate, but we hope it will spark new discussions as an initial study of the range of positions people hold and ways in which the community may mis-model itself.

We analyze the results in our report and you can explore them yourself in the online dashboard.

Lots of NLP researchers answered our call, and for each one, we donated $10 to one of several non-profits that participants chose at the end of the survey. Bcause of your participation, we donated a total of $4,570:

Thanks for taking part in something we hope will help us all understand our community better!

—The NLP Community Metasurvey Team