Reproducibility: Which Levers?

I was reading about health behavior change today and I was reminded that there is a difference between a complicated system and a complex system (D.T. Finegold and colleagues) and it crystalized for me why  the confident pronouncements of the reproducibility folks strike me as earnest but often misguided. If you think about it, most laboratory experiments are complicated systems that are meant to be roughly linear: There may be a lot of variables and many people involved in the manipulation or measurement of those variables but ultimately those manipulations and measurements should lead to observed changes in the dependent variable and then there is a conclusion; by linear system I mean that these different levels of the experiment are not supposed to contaminate each other. There are strict rules and procedures, context-specific of course, for carrying out the experiment and all the people involved need to be well trained in those procedures and they must follow the rules for the experiment to have integrity. Science itself is another matter altogether. It is a messy nonlinear dynamic complex system from which many good and some astounding results emerge, not because all the parts are perfect, but in spite of all the imperfection and possibly because of it. Shiffren, Börner and Stigler (2018) have produced a beautiful long read that describes this process of “progress despite irreproducibility.” I will leave it to them to explain it since they do it so well.

I am certain that the funders and the proponents of all the proposals to improve science are completely sincere but we all know that the road to hell is paved with good intentions. The reason that the best intentions are not going to work well in this case is that the irreproducibility folks are trying to “fix” a complex system by treating it as if it is a complicated problem. Chris Chambers tells a relatively simple tale in which a journal rejects a paper (according to his account) because a negative result was reported honestly which suggests that a focus on positive results rewards cheating to get those results and voilà: the solution is to encourage publication without the results. This idea is fleshed out by Nosek et al (2018) in a grand vision of a “preregistration revolution” which cannot possibly be implemented as imagined or result in the conceived outcomes. All possible objections have been declared to be false (bold print by Chris Chambers) and thus they have no need of my opinion. I am old enough to be starting my last cohort of students so I have just enough time to watch them to get tangled up in it. I am a patient person. I can wait to see what happens (although curiously no objective markers of the success of this revolution have been definitively put forward).

But here’s the thing. When you are predicting the future you can only look to the past. So here are the other things that I read today that lead me to be quite confident that although science will keep improving itself as it always has done, at least some of this current revolution will end up in the dust. First, on the topic of cheating, there is quite a big literature on academic cheating by undergraduate students which is directly relevant to the reproducibility movement. You will not be surprised to learn that (perceived) cheating is contagious. It is hard to know the causal direction – it is probably reciprocal. If a student believes that everyone is cheating the likelihood that the student will cheat is increased. Students who cheat believe that everyone else is cheating regardless of the actual rate of cheating. Students and athletes who are intrinsically versus extrinsically motivated are also less likely to cheat so it is not a good idea to undermine intrinsic motivation with excessive extrinsic reward systems, especially those that reduce perceived autonomy. Cheating is reduced by “creating a deeply embedded culture of integrity:” Culture is the important word here because most research and most interventions target individuals but it is culture and systems that need to be changed. Accomplishing a culture of integrity includes (perhaps you will think paradoxically) creating a trusting and supportive atmosphere with reduced competitive pressures while ensuring harsh and predictable consequences for cheating. The reproducibility movement has taken the path of deliberately inflating the statistics on the prevalence of questionable research practices with the goal of manufacturing a crisis, under the mistaken belief that the crisis narrative is necessary to motivate change when it is more likely that this narrative will actually increase cynicism and mistrust, having exactly the opposite effect.

The second article I read that was serendipitously relevant was about political polarization. Interestingly, it turns out that perceived polarization reduces trust in government whereas actual polarization between groups is not predictive of trust, political participation and so on. It is very clear to me that the proponents of this movement are deliberately polarizing and have been since the beginning, setting hard scientists against soft, men against woman and especially the young against the old (I would point to parts of my twitter feed as proof of this I but I don’t need to contaminate your day with that much negativity, suffice to say it is not a trusting and supportive atmosphere). The Pew Center shows that despite decades of a “war against science” we remain one of the most trusted groups in society. It is madness to destroy ourselves from within.

A really super interesting event that happened in my tweet feed today was the release of the report detailing the complete failure of the Gates Foundation $600M effort to improve education by waving sticks and carrots over teachers with the assumption that getting rid of bad teachers was a primary “lever” that when pulled would spit better educated minority students out the other end (seriously, they use the word levers, it cracks me up; talk about mistaking a complex system for a complicated one). Anyway, it didn’t work. The report properly points out that that the disappointing results may have occurred because their “theory of action” was wrong. There just wasn’t enough variability in teacher quality even at the outset for all that focus on teacher quality to make that much difference especially since the comparison schools were engaged in continuous improvement in teacher quality as well. But of course the response on twitter today has been focused on teacher quality: many observers figure that the bad teachers foiled the attempt through resistance, of course! The thing is that education is one of those systems in our society that actually works really well, kind of like science. If you start with the assumption that that the scientists are the problem and if you could just get someone to force them to shape up (see daydream in this blog by Lakens in which he shows that he knows nothing about professional associations despite his excellence as a statistician)…well, I think we have another case of people with money pulling on levers with no clue what is behind them.

And finally, let’s end with the Toronto Star, an excellent newspaper, that has a really long read (sorry, its long but really worth your time) describing a dramatic but successful change in a nursing home for people with dementia. It starts out as a terrible home for people with dementia and becomes a place you would (sadly but confidently) place your family member. This story is interesting because you start with the sense that everyone must have the worst motives in order for this place to be this bad—care-givers, families, funders, government—and end up realizing that everyone had absolutely the best intentions and cared deeply for the welfare of the patients. The problem was an attempt to manage the risk of error and place that goal above all others. You will see that the result of efforts to control error from the top down created the hell that the road paved with good intentions must inevitably create.

So this is it, I may be wrong and if I am it will not be the first time. But I do not think that scientists have been wasting their time for the last 30 years as one young person declared so dramatically in my twitter feed. I don’t think that they will waste the next 30 years either because they will mostly keep their eye on whatever it is that motivated them to get into this crazy business. Best we support and help each other and let each other know when we have improved something but at the same time not get too caught up in trying to control what everyone else is doing. Unless of course you are so disheartened with science you would rather give it up and join the folks in the expense account department.

Post-script on July 7, 2018: Another paper to add to this grab-bag:

Kaufman, J. C., & Glǎveanu, V. P. (2018). The Road to Uncreative Science Is Paved With Good Intentions: Ideas, Implementations, and Uneasy Balances. Perspectives on Psychological Science, 13(4), 457-465. doi:10.1177/1745691617753947

I liked this perspective on science:

“The propulsion model is concerned with how a creative work affects the field. Some types of contributions stay within the existing paradigm. Replications,1 at the most basic level, aim to reproduce or recreate a past successful creation, whereas redefinitions take a new perspective on existing work. Forward or advance forward incrementations push the field ahead slightly or a great deal, respectively. Forward incrementations anticipate where the field is heading and are often quite successful, whereas advance forward incrementations may be ahead of their time and may be recognized only retrospectively. These categories stay within the existing paradigm; others push the boundaries. Redirections, for example, try to change the way a field is moving and ake it in a new direction. Integrations aim to merge two fields, whereas reinitiation contributions seek to entirely reinvent what constitutes the field.”

 

Advertisements
Leave a comment

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: