Editorial: Debunk bad science and embrace the good

Follow @csteditorials

We always thought that study about cheating was fishy.

Did you happen to read about it? Maybe somebody tweeted it to you. According to news stories awhile back, a supposedly scientific study concluded that people who read a short essay saying their behavior is predetermined were more likely to cheat when they then took a test.

Could a lifetime of moral training be so easily undermined?

But now another study — a much bigger one that sounds a lot less fishy, published Thursday in the journal Science — says that the cheating study and plenty of others in the field of psychology are basically bunk. The researchers tried to reproduce 100 such studies, all of which had been published in three leading psychological journals, and found that more than half the time the findings did not hold up.

Follow @csteditorials

The report in Science raises questions about the integrity of research in psychology and, more generally, in all science. This is no small matter in a nation that increasingly seeks “data-driven solutions” — as it should — for everything from cancer to criminal justice reform to poverty. Research of the highest quality is essential.

Should an insurance company cover medical costs for a particular psychological disorder? Let’s see what the studies say. Should trans fat be banned because it is bad for our health? Let’s look at the research.

There’s a Penn & Teller feel to this report in Science, a sense of pulling back the curtain to debunk magical claims. In 62 of the 100 studies the researchers tried to duplicate, the findings did not hold up.

Other researchers, according to the New York Times, point to several explanations for why many studies may be flawed and why those flaws may go undetected.

For one, they said, there can be “dysfunctional incentives,” such as the problem of funding coming from a group that has an agenda. The classic example would be the many studies once funded by tobacco industry that found no link between smoking and cancer.

And while it is best for a study to be duplicated at least once or twice to make sure the results hold up, researchers are rewarded for pushing forward. “Scientists have pointed to a hypercompetitive culture across science that favors novel, sexy results and provides little incentive for researchers to replicate the findings of others,” writes the Times.

For non-scientists like us, there are lessons to drawn:

• Take every new study with a grain of salt, especially if the findings appear to contradict established science or common sense.
• Look for overwhelming evidence. The research that did hold up when duplicated, according to the report in Science, had the most robust results to begin with.
• Put your trust in research that has been challenged, duplicated and confirmed repeatedly. The science that confirms the reality and danger of climate change, most obviously, meets this test, consisting of a body of work by thousands of respected scientists and institutions around the world.
• Consider the source. If a study that claims global warming is a myth was funded by the coal industry, consider just that — the study was funded by the coal industry.

Science is not a set of immutable facts, but a methodology for teasing out the truth. It never stops. If it is true that much of the scientific research in psychology is deeply flawed, it is also true that scientists themselves have done the debunking.

And, for all we know, that big study in Science is flawed, too. Let’s do it again.

Follow the Editorial Board on Twitter: Follow @csteditorials

Tweets by @CSTeditorials