“What can we learn from van Wolferen et al.’s replication attempt? Unfortunately, not much”

This post is really short not so short anymore but I just could and I still cannot fit everything in one tweet. I recently published my first article (with Yoel Inbar and Marcel Zeelenberg). We report 3 direct failures to replicate an ‘protection effect’ where a reminder of insurance led people to believe misfortune was less likely (read more about the paper in a previous post)

The original author (of the paper we failed to replicate) just published a successful conceptual replication of this effect and responds to our paper (she calls it the ‘insurance effect’ but we mean the same thing).

Two things are noteworthy.

1. The response, with data, has been published super fast. The March issue of Judgment and Decision Making is the first to appear after the January issue where we published our paper. This is a 100% win for the SJDM Journal because academic discourse and publishing is stereotypically perceived to be slow.

2. Unfortunately, the original author thinks our replication attempts are not of great value:

“Recently, van Wolferen, Inbar and Zeelenberg (2013) reported their failure to directly replicate the Tykocinski’s 2008 health-insurance field study in three experiments of their own. What can be learned from the failure to replicate this field study?
Unfortunately, not much.”

You should scroll to page 177 of the paper to read why she thinks so.

Of course, I disagree that our replication attempts do not teach us much about the insurance effect and I plan to make a proposal in which we work together to resolve the apparent inconsistencies between her and our paper(s).

EDIT: Here’s why I disagree:

1. We supposedly “overlooked crucial contextual aspects“. Like approaching participants who might have been in a hurry in Study 2.
— First, people only participated if they had time. Second, I am not sure why the study should take long. In the response paper that reports a conceptual replication, the manipulation was 2 questions in a phone call. That cannot take longer than 1 minute–and that’s generous estimate. It is thus unlikely that ‘length’ of manipulation explains our failure to replicate. In addition, the manipulation in the original study was 3 questions on health care insurance: not a lengthy manipulation.

2. Another aspect we apparently missed was: “for an affect-based mechanism to operate, the population studied must have realistic concerns regarding the issue in question“. For that reason the student subjects in Study 1a (likely to be unconcerned about health) may not show the effect.
— OK. This might be true. But we also do not find the effect in Study 1b and 1c, where we even test if older people–who might be concerned about their health–are more prone to the insurance effect. We find no evidence that this happens. In addition, in Study 2a and 2b we test the insurance effect (and a tempting fate effect) with students in a context where they do have realistic concerns: being called upon to present in front of class. Again, we find no evidence for the protection effect (but do find a tempting fate effect).

3. We did not attempt to replicate the lab studies reported in the original article, and the suggestion is that we should have.
— Study 1 in the original paper is the only experiment that allows for a ‘clean’ test of the insurance effect. The other studies could also be explained by tempting fate effects alone. Therefore, we decided to only replicate Study 1. I guess my interpretation of the other studies in the original paper is debatable, so that’s something I should work out with the original author. Nevertheless, I think we created a fair test of the insurance effect in the five studies we report.

(end of edit)

If you have any thoughts on what could cause the difference between our results, please share your thoughts. Also, any advice on how to ‘do’ adversarial collaborations is welcome!

References

Original paper (study 1): Tykocinski, O. E. (2008). Insurance, risk, and magical thinking. Personality & social psychology bulletin, 34(10), 1346–56. doi:10.1177/0146167208320556

Our replication attemptsVan Wolferen, J., Inbar, Y., & Zeelenberg, M. (2013). Magical thinking in predictions of negative events : Evidence for tempting fate but not for a protection effect. Judgment and Decision Making, 8(1), 45–54.

Conceptual replicationTykocinski, O. E. (2013). The insurance effect : How the possession of gas masks reduces the likelihood of a missile attack. Judgment and Decision Making, 8(2), 174–178.

Advertisements

About Job van Wolferen

PhD-student at TIBER / Tilburg University. Research: Moral hazard and fraud in the insurance industry. Interested in behavioral economics and JDM
This entry was posted in All and tagged , , , . Bookmark the permalink.

2 Responses to “What can we learn from van Wolferen et al.’s replication attempt? Unfortunately, not much”

  1. Pingback: Replication in Psychological Science | Job van Wolferen

  2. Pingback: Articles I Like | Job van Wolferen

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s