Artificial Intelligence Confronts a ‘Reproducibility’ Crisis
Machine-learning systems are black boxes even to the researchers that build them. That makes it hard for others to assess the results.
Pineau is trying to change the standards. She’s the reproducibility chair for NeurIPS, a premier artificial intelligence conference. Under her watch, the conference now asks researchers to submit a “reproducibility checklist” including items often omitted from papers, like the number of models trained before the “best” one was selected, the computing power used, and links to code and datasets.
