Recently, Dr. Tony Wilson from CUNY Brooklyn tried to recreate my analysis, so that he could figure out how it worked and apply it to his own data ... he couldn’t quite recreate some of my core results.
I dug up my original code, sent it to him, and after a couple of back-and-forth emails we found my error.
So: how many results, negative or positive, that enter the published literature are tainted by a coding mistake as mine was. We just don’t know. Which raises an important question: why don’t we review code (or other custom software) as part of the peer-review process?
When software is not visible, it is often excluded from peer review
Lack of visibility means that incentives to produce high-quality, widely shared, and collaboratively developed software are lacking