Policy and Protocol on Third-Party Verifications

• Preliminaries
• What or Who Is a Third-Party Replicator
• Steps for the Third-Party Replicator

This protocol describes how third parties can, at the request of the AEA Data Editor, conduct a reproducibility check.

Alternate protocols are possible, but should be verified with the AEA Data Editor prior to engaging any resources.

Preliminaries

• The author(s) should provide a complete and exhaustive archive, ready for publication, to the AEA Data Editor.
  • The archive does not need to be public at this stage, as long as it can be shared privately.
  • The archive does not have to contain the data necessary for the reproducibility check if data is confidential or proprietary. However, the archive must contain a publishable description of how an independent researcher can access the data ("data availability statement") once all conditions for access have been met.
• The AEA Data Editor will verify the archive.
  • Aside from the data itself, all other materials specified in the Data and Code Availability Policy, such as data citations, must be included.
    
    

What or Who Is a Third-Party Replicator

The third-party replicator is a person not affiliated with the AEA editorial offices who has access to the (public-use, confidential or restricted) data used in the author's paper, and who

• has not been involved with the author's research project
• has disclosed any conflicts of interest
• promises to conduct an arms-length reproducibility exercise with no direct interaction with the author (other than to undertake specific steps to access the data)
  
  

Steps for the Third-Party Replicator

• Download the author's replication archive(s) from the location provided by the AEA Data Editor (public or privately shared).
  • Do not obtain any code or instructions directly from the author.
• Ensure access to any confidential files that are described in the replication archive's README.
  • Consider whether a person not familiar with the precise data environment could reasonably find and access the data, based solely on the instructions in the README.
• Follow the checklist to conduct the reproducibility exercise, relying exclusively on the README for instructions and guidance.
• Write a report.
• Send the report to the AEA Data Editor.
• Report any interactions with the author in the course of conducting the reproducibility exercise (help, assistance, clarifications).