Last week, Lab Land put out a Twitter poll, touching on the cognitive distortions that make it difficult to do high-quality science. Lots of people (almost 50) responded! Thank you!
We had to be vague about where all this came from, because it was before the publication of the underlying research paper. Ray Dingledine, in Emory’s Department of Pharmacology, asked us to do the Twitter poll first, to see what answers people would give. Dingledine’s paper in eNeuro is now published, so we can explain.
Raymond Dingledine, PhD
The paper is titled “Why Is It So Hard To Do Good Science?” Basically, Dingledine argues, our cognitive biases get in the way. eNeuro summarizes the take-home message this way: “Improving experimental design and statistical analyses alone will not solve the reproducibility crisis in science.”
When designing their experiments, Dingledine says, scientists need to take account of “the law of small numbers”—the distortions random variation can introduce when sample sizes are small – along with other cognitive biases.
In the 1960s and 1970s, psychologists Daniel Kahneman and Amos Tversky demonstrated that people tend to engage in “fast thinking” — relying on preconceived notions and emotions — when making decisions in the face of new information. In his update of this research, Dingledine found that scientists of all career stages are subject to the same biases as undergraduates when interpreting data.
The findings reinforce the roles that two inherent intuitions play in scientific decision-making: our drive to create a coherent narrative from new data regardless of its quality or relevance, and our inclination to seek patterns in data whether they exist or not. Moreover, we do not always consider how likely a result is regardless of its P-value.
Each of the Twitter poll questions corresponds to questions that Dingledine asked graduate students, postdoctoral and senior research staff, and principal investigators across three basic science departments. We had to tweak a couple of the questions to fit the format. And to be sure, Lab Land’s goal was to promote discussion of the research and awareness of the issues behind it, rather than to gather research-quality data about cognitive biases. [We would not be offended at the term “push poll.”] It does look like that respondents to the Twitter poll were clued in. See below.
Answers: 1) Smaller 2) small samples/variance 3) yes, and for 4) remember that healthcare is a huge chunk of the economy, even if the stereotype of engineers fits. Lab Land thanks Dingledine and SFN’s David Barnstone for willingness to roll with this experiment.
About the author
