I recently gave a lecture on meta-analyses as part of a Drug Literature Evaluation course. One example I typically use to illustrate the importance of critically evaluating literature combined with the potential impact of the media is, “Is Cybermedicine Killing You” – The Story of a Cochrane Disaster. It is an editorial published back in 2005 in response to a fatally flawed Cochrane paper that was ostensibly published as a review of, “interactive health communication applications” (IHCAs). The review’s plain language summary described it as “computer-based programmes for people with chronic disease”.
In my mind, there are four key observations about that situation:
1. The nature of the research being reviewed was mischaracterized from the beginning.
As pointed out by Eysenbach and Kummervold, the studies included in the meta-analysis were all interventional in nature (i.e., in addition to an informational piece, there was an accompanying, structured aspect of decision support/behavioral change). However, most of the media headlines generated from that Cochrane review omitted the interventional aspect and presented it in terms of consumers passively looking at health-related information on the Internet.
2. The authors of the Cochrane review arrived at a flawed conclusion and admitted it.
The authors of the JMIR editorial pointed out several “stunning” mistakes in the review including: data extraction errors, and that conclusions about positive outcomes from the eHealth interventions were misinterpreted as harmful. The editorial re-examined the work and actually found that only 2 of the 11 study interpretations were correct. Some of them were startlingly basic errors such as classifying a reduction in hemoglobin A1c (HbA1c) in the intervention group and an increase in HbA1c in the control group as a bad outcome.
3. The retraction acknowledging the erroneous findings barely caused a ripple.
After the flurry of media stories about the supposed dangers associated with looking online for health information, very few news outlets publicized the retraction. This is despite the retraction being issued in only 13 days. Further, the revised conclusion archived in the Cochrane Library stated the opposite findings from the original that, “IHCAs appear to have largely positive effects on users, in that users tend to become more knowledgeable, feel better socially supported, and may have improved behavioural and clinical outcomes compared to non-users.” Nothing nefarious here; it is pretty common to see sparse coverage allotted for retractions versus the “if it bleeds, it leads” initial coverage. But it is still regrettable.
4. What could have been an early blow to eHealth and empowered patients was avoided.
Given the authority of Cochrane reviews, the unease of many clinicians with patients seeking health information (even more pronounced five years ago), and the relatively early stage of consumer health informatics – this inaccurate condemnation could have proved detrimental to the advancement of the specialty and efforts by what are currently referred to as e-patients. Fortunately, a few researchers, authors, and advocates discovered the situation and acted swiftly to correct it.
It always surprises me that this story is not related more often, especially among those involved with its related fields or movements. There are not a lot of situations that can serve so aptly as both a cautionary tale and one of promise and encouragement.
Links for all relevant, full-text publication are embedded in the text. Here are the written citations just in case.
Eysenbach G, Kummervold PE. “Is Cybermedicine Killing You?”–The story of a Cochrane disaster. J Med Internet Res 2005;7(2):e21.
Murray E, Burns J, See Tai S, Lai R, Nazareth I. Interactive Health Communication Applications for people with chronic disease. Cochrane Database Syst Rev 2005; (4):CD004274.