Policy and Protocol on Third-Party Verifications

 


This protocol describes how third parties can, at the request of the AEA Data Editor, conduct a reproducibility check.

Alternate protocols are possible, but should be verified with the AEA Data Editor prior to engaging any resources.

Preliminaries

  • The author(s) should provide a complete and exhaustive archive, ready for publication, to the AEA Data Editor.
    • The archive does not need to be public at this stage, as long as it can be shared privately.
    • The archive does not have to contain the data necessary for the reproducibility check if data is confidential or proprietary. However, the archive must contain a publishable description of how an independent researcher can access the data ("data availability statement") once all conditions for access have been met.
  • The AEA Data Editor will verify the archive.

What or Who Is a Third-Party Replicator

The third-party replicator is a person not affiliated with the AEA editorial offices who has access to the (public-use, confidential or restricted) data used in the author's paper, and who

  • has not been involved with the author's research project
  • has disclosed any conflicts of interest
  • promises to conduct an arms-length reproducibility exercise with no direct interaction with the author (other than to undertake specific steps to access the data)

Steps for the Third-Party Replicator

  • Download the author's replication archive(s) from the location provided by the AEA Data Editor (public or privately shared).
    • Do not obtain any code or instructions directly from the author.
  • Ensure access to any confidential files that are described in the replication archive's README.
    • Consider whether a person not familiar with the precise data environment could reasonably find and access the data, based solely on the instructions in the README.
  • Follow the checklist to conduct the reproducibility exercise, relying exclusively on the README for instructions and guidance.
  • Write a report.
  • Send the report to the AEA Data Editor.
  • Report any interactions with the author in the course of conducting the reproducibility exercise (help, assistance, clarifications).