I spent the last five days at ICERM (the new Math institute at Brown University) attending the workshop Reproducibility in Computational and Experimental Mathematics. The workshop was focused on discussing how mathematicians can ensure that their computations are reproducible, in order to ensure correctness and facilitate their use by others. It’s a topic dear to my heart and one that I’ve blogged about before.
My hat is off to the organizers for managing to assemble a highly diverse group of experts, including not only academic luminaries from both pure and applied math, open source software gurus, and leaders from companies like Github and Google (yes, Peter Norvig himself attended). Most of the talks were excellent. Many included live demos of great tools, and others introduced me to things that I never thought you could do with computation – like discovering new formulas for pi.
Going into the workshop, I felt that I already knew a lot about reproducibility and had relatively good habits in this regard. So what did I learn? I picked up a new tool, Andrew Davison’s Sumatra, which I had heard of before but now have begun to use in earnest (more on that in a future post). I was impressed with Lorena Barba’s Reproducibility PI Manifesto and learned a new trick from her: put your figures up on Figshare before submitting a paper in order to retain copyright on the figures. I marveled at Greg Wilson’s goal of reaching 20% of all scientists with his Software Carpentry courses, and I determined to host such a course at KAUST in the near future.
I also learned that the reproducibility movement in computational science and mathematics involves a wide range of opinions and concerns. For instance, some consider that the primary motivation for reproducibility is to ensure correctness of results, while others feel that it is scientific productivity. There is disagreement about how much value should be placed on code development, on how reproducibility should be taught, and on ways in which journals and funding agencies should encourage reproducibility. In the end, we had difficulty even agreeing on a well-defined terminology for concepts related to reproducibility. Nevertheless, there is broad agreement that we need to improve our habits in recording and presenting our computational work. On the final day, in a spurt of crazy massive Google Doc collaboration (have you ever edited a documented live with 30 others at once?) we drafted a report that I’ll link to here once it appears.
If you want to know more, take a look at the great thought pieces submitted and the rest of the material on the wiki.
