Scientists are failing to replicate AI studies.
Missing code and data make it tough to compare AI work. That may hurt progress.
The problem: Science reports that of a sample of 400 papers from AI conferences in recent years, only 6 percent of presenters shared code. Just a third shared data.
Why it matters: Without access to that information it’s hard to reproduce a study’s findings, making it impossible to benchmark tools that could push the field forward.
At the AAAI meeting, Peter Henderson, a computer scientist at McGill University in Montreal, showed that the performance of AIs designed to learn by trial and error is highly sensitive not only to the exact code used, but also to the random numbers generated to kick off training, and to "hyperparameters"—settings that are not core to the algorithm but that affect how quickly it learns. He ran several of these "reinforcement learning" algorithms under different conditions and found wildly different results.
Henderson's experiment was conducted in a test bed for reinforcement learning algorithms called Gym, created by OpenAI, a nonprofit based in San Francisco, California. John Schulman, a computer scientist at OpenAI who helped create Gym, says that it helps standardize experiments.
How to solve it: There’s a clear culture of keeping such details under wraps. Some meetings and journals are now encouraging sharing. Perhaps more ought follow.
Psychology has dealt with its reproducibility crisis in part by creating a culture that favors replication, and AI is starting to do the same.
Yet AI researchers say the incentives are still not aligned with reproducibility. They don't have time to test algorithms under every condition, or the space in articles to document every hyperparameter they tried.
See the full story here: http://www.sciencemag.org/news/2018/02/missing-data-hinder-replication-artificial-intelligence-studies