Graduate Students Should Be Able to Specialize In Replication

Now that the need for more replication has forced its way onto the scientific agenda we should begin thinking about how to build systems to support its growth and institutionalization. New publications and conferences are all good steps, but we should go beyond relying on a loosely organized group of scientists who dedicate time to something that has collective utility but little individual utility.

I think one solution is to create well-defined graduate programs that focus on replication. The programs would vary depending on the fields they are designed to police. For example, the psychology replication program would train students in methods for replicating psychology studies and identifying methodological weaknesses that might make a study a good candidate for replication. The programs would effectively be mechanisms for credentialing official science cops.

The programs would accomplish a number of things relative to current initiatives:

1. Create the sense of an unbiased authority. The current system relies on researchers policing the “competition” or policing themselves. Both situations incentivize skewed results. With “Replicators” there would be fewer conflicts of interest — even if they conducted some of their own original research — because official credentials would establish them as unquestioned authorities and put their work under the microscope. Replicators would also make it harder for people to deem replication attempts illegitimate and launch a war in the court of public opinion (see: Bargh, John).

2. Support the innovation of replication best practices. If Replicators do choose to do “original” research a good portion of it could focus on improving the methods for judging the validity of a study. (For example, Gregory Francis has a new paper on using effect sizes to test for publication bias.) A lot this research would overlap with statistics research, but there are also qualitative advances that can be made.

3. Create a self-sustaining system. The ponzi scheme structure of academia is designed so that fields can grow exponentially. Making replication a part of the academic system will create a future supply of Replicators.

The one downside is that by concentrating official responsibility in the hands of a few you exempt everyone else from doing better science. Unfortunately, until changes to the tenure process alter publishing incentives I’m skeptical that a loosely organized system of individual responsibility would be better. Of course there’s no reason we can’t have journals, conferences, individual efforts, and academic programs. As every presidential candidate says, we need an “all of the above” approach.
Francis, G. (2012). Publication bias and the failure of replication in experimental psychology Psychonomic Bulletin & Review DOI: 10.3758/s13423-012-0322-y

2 Responses to Graduate Students Should Be Able to Specialize In Replication

  1. Bashir says:

    I am very confused. This would be an entire graduate program? The students would do nothing but replicate (or not) other work. Nothing but that?


  2. Eric Horowitz says:

    Not necessarily — it could be a specialization or a small curriculum within a particular program. I think it’s important to realize that, depending on their scope, these programs might not be a good fit for the standard Ph.D student who has visions of wining prizes based on their own original research. The programs would attract a different type of student — somebody who’s more interested in ensuring the integrity of science as a whole than in doing original research. If it seems that nobody currently in academica would be interested in this, that might be right. I assume these programs would be attractive to certain people who want to do something research-related but can’t find the right fit in the current environment.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s