philip lelyveld The world of entertainment technology

Tech Firms Hire ‘Red Teams.’ Scientists Should, Too

Another botched peer review—this one involving a controversial study of police killings—shows how devil's advocates could improve the scientific process.

THE RECENT RETRACTION of a research paper which claimed to find no link between police killings and the race of the victims was a story tailor-made for today’s fights over cancel culture.

First, the authors asked for the paper to be withdrawn, both because they’d been “careless when describing the inferences that could be made from our data” and because of how others had interpreted the work. (In particular they pointed to recent op-ed in The Wall Street Journal with the headline, “The Myth of Systemic Police Racism.”) Then, after two days of predictable blowback from those decrying what they saw as left-wing censorship, the authors tried to clarify: “People were incorrectly concluding that we retracted due to either political pressure or the political views of those citing the paper,” they wrote in an amended statement.

No, the authors said, the real reason they retracted the paper was because it contained a serious mistake. In fact, that mistake—a misstatement of its central finding—had been caught soon after the paper’s initial publication in the Proceedings of the National Academy of Sciences in July 2019, and was formally corrected in April of this year.

...Without a doubt, there are jerks in science, and not all critiques are well-intentioned. But if we strip away the nastiness of Reviewer #2s, and the notion that their quibbles amount to spiteful sabotage, they start to look a bit like Red-Team leaders. Their more vigorous approach to doing peer review could help clean up the scientific record by making sure fewer incorrect conclusions are published. Isn’t that worth the effort?

See the full story here: https://www.wired.com/story/tech-firms-hire-red-teams-scientists-should-too/