About the Survey
What do NLP researchers think about controversial NLP issues? And perhaps more interestingly: What do NLP researchers think other NLP researchers think about controversial NLP issues?
Recent rapid changes in the NLP landscape have raised many questions. For example:
- Are we devoting too many resources to scaling up?
- Do language models understand language? Will they ever?
- Is the traditional paradigm of model benchmarking still tenable?
- What kinds of predictive models are ethical for researchers to build and release?
- Will the next most influential advances come from industry or academic labs?
On these and similar questions, it’s often unclear what counts as general consensus, what is still actively debated, or what is widely considered overturned. We believe it would benefit the field to have a better understanding of the current landscape of community opinions and beliefs.
To this end, we conducted the NLP Community Metasurvey from May to June 2022. This survey covers over 30 potentially controversial positions, asking not only whether you agree, but what percentage of the community you think agrees with the position, thus exposing the difference between “what people think” and “what people think people think.” This idea was inspired by the PhilPapers Surveys.
The results and analysis are descriptive, not prescriptive, as such issues cannot be resolved by majority vote. By necessity, we cover a subjectively chosen set of questions and reducing many complex issues into simplified scales, but we hope that the outcome can serve as a reference point for community discussion and for future surveys. This is not the final word in any debate, but we hope it will spark new discussions as an initial study of the range of positions people hold and ways in which the community may mis-model itself.
The results are available in our report and will soon be accessible on this website (suitably aggregated to prevent deanonymization).
Lots of NLP researchers answered our call, and for each one, we donated $10 to one of several non-profits that participants chose at the end of the survey. Bcause of your participation, we donated a total of $4,570:
- $1,650 to the GiveWell Maximum Impact Fund
- $830 to GiveDirectly
- $950 to the WHO COVID-19 Solidarity Response Fund
- $1,140 to the Distributed AI Research Institute
Thanks for taking part in something we hope will help us all understand our community better!
—The NLP Community Metasurvey Team