Cooking the books against homeopathy?
A negative review of homeopathy published in 2005 reverberates still in blogs, journals, and press releases.
A reader comments, and I wax philosophical. But let’s start with summary of the issues.
In 2005, a comparison of homeopathy and allopathy (mainstream medicine) studies was published in The Lancet.
- 110 homeopathy and 110 matched mainstream-medicine studies were compared.
- Smaller and lower-quality studies showed more benefits than larger and higher-quality studies.
- When the analysis was restricted to the largest studies (about 6% of the total studies), the authors concluded that homeopathy was inferior.
According to the authors, their conclusion was justified because “This finding is compatible with the notion that the clinical effects of homeopathy are placebo effects.”
“Notion”?
Yes, it’s always comforting when your findings conform to popular opinion.
After that article appeared, the Swiss Association of Homeopathic Physicians published an open letter to the editor that was critical of the conclusions. This year, others have weighed in with their criticisms here and here, which focus on the lack of transparency in divulging the criteria used to select and evaluate the studies.
Its complicated and technical, and enough to make you go for a massage. However, I find that this recent article by 2 reviewers in Germany quickly cuts to the problem — the methods used to analyze the homeopathy studies probably lead to the erroneous, albeit comforting conclusions.
- The studies selected for the review differed greatly.
- Number of patients treated
- Type of homeopathy used
- Type of publication (some studies were unpublished)
- Medical conditions treated
- Overall, homeopathy showed a significant benefit compared to placebo.
- But restricting the analysis to successively larger studies resulted in progressively less statistical significance for homeopathy.
- Ultimately, negative conclusions from the analysis of the 8 largest homeopathy studies were influenced by 1 negative study of arnica to prevent muscle soreness in 400 long-distance runners.
The bottom line?
Were the authors lazy, unknowledgeable, or beset by bias?
Maybe, but it’s really not important. The failure here is with the journal in meeting its responsibility as gatekeeper for high-quality, peer-reviewed studies.
The Lancet is really more of a newspaper than a medical journal. Rapid publication is one of its attractive features for researchers. Considering the number of methodological deficiencies in the article, one wonders if the editors even bothered to have it peer reviewed.
11/12/08 18:32 JR
DanaUllmanMPH said:
on November 13, 2008 at 1:22 pm
John,
Thank you for your comments above. In case some of your readers want more detail about the “junk science” that the original Lancet article contained, here’s that information…
Homeopaths and skeptics of homeopathy should agree on one thing: the “science” behind the Shang paper was junk science. It is more than a tad ironic that people who normally consider themselves to be “defenders of conventional science and conventional medicine” are actually defending the questionable data of the Shang paper as well as the questionable ethics of the authors for their biased reporting and questions the integrity of the journal, The Lancet, for publishing this questionable report.
The Ludtke/Rutten paper in the Journal of Clinical Epidemiology IS important, as is its companion paper in the October 2008 issue of Homeopathy (published by Elsevier) by Rutten/Stopler.
Both new articles show that Shang’s results (that the effects from homeopathic medicines are the same as that of a placebo) are “less definite” as they have been previously presented. The Lancet editors have even asserted this study now closes the case on homeopathy by publishing an editorial called “The End of Homeopathy.” However, these new analyses show how fatally flawed this study was and how embarrassingly biased its editors have been, as the below assertions prove.
Shang omitted certain high quality studies in homeopathy (was it a coincidence that the vast majority of these omitted studies had a positive result?), how they defined what is “high quality” is open to question (initially, the authors didn’t even report which studies were defined as “high quality,” and today, there is no clarity on the point score for each study), their decision to never evaluate or compare all of the “high quality” studies (the authors assert that high quality randomized, double-blind and placebo controlled studies are actually “biased” unless they are over 98 subjects in homeopathic studies but magically conventional medical trials are only biased if they are under 146 subjects).
Even the use of different criteria for the two different systems of medicine throws the comparison into question. In fact, Rutten and Stopler assert that choosing these different numbers was a decision made “post-hoc,” which ultimately questions the integrity of the science and the ethics of the authors.
If Shang evaluated only those clinical trials that his own group defined as “high quality” (the 21 homeopathic trials and the 9 allopathic trials), there is a statistically significant difference between those patients given a homeopathic medicine and those given a placebo.
The new re-analysis notes that 2 studies by Reilly, one by de Lange-de-Klerk, and one by Hofmeyr were not defined by Shang as “high quality” but they were defined as such in a major meta-analysis of homeopathic clinical trials conducted by Linde, et al which was published in the Lancet (1997) (in fact, the Reilly papers were published in the Lancet and the BMJ, and both of these journals published editorials that acknowledged the high quality nature of these studies . Three of the above mentioned four trials showed a positive effect towards homeopathic treatment.
It was interesting to note that Shang excluded Wiesenauer’s chronic polyarthritis study (N=176) because no matching trial could be found (Linde, 1997, defined this study as “high quality”). And yet, because none of the trials (!) in the final evaluation matched each other in any way, omitting inclusion of this study was the result of bias from the authors.
Also, three of the eight large and high quality conventional medical trials tested drugs that were deemed to be “effective” and yet, these medical treatments have been withdrawn from medical use due to the serious side effects that later research confirmed. I was also pleased that the Rutten/Stolper article made note of the fact that Shang acknowledged that their study disregarded adverse effects (how convenient).
(continued below)
JR said:
on November 14, 2008 at 4:27 pm
(continuation of Dana Ullman ’s comment)
Four (!) of the 21 high quality homeopathic trials sought to evaluate the prevention or treatment of muscle soreness. These three of the four trials had negative results, and if all of these trials were omitted from the analysis, there was a highly significant difference between homeopathic treatment and the placebo (P<0.007). All of the conventional medical studies that evaluated this condition found negative results, though none (!) of these studies were deemed by Shang to be “high quality,” thus further skewing the results.
The new re-analysis of the Shang review did not question the outcome data that was extracted from the clinical trials. However, the authors did note that Shang reported on a study of traumatic brain injuries by Chapman, which found that reported on only one outcome measure as “negative” even though this study reported that 2 of the 3 outcome measures were “positive”. Also, the Shang review surprisingly included a “weight-loss” study, and Shang extracted data from day 1, but day 2 had been defined as the main outcome parameter (whereas the day 1 results were “negative,” the day 2 results were “positive” (in any case, this study should not have been included in the analysis because they had never undergone previous preliminary trial to deem their worthy of a larger clinical trial).
(continued below)
JR said:
on November 14, 2008 at 4:29 pm
(continuation of Dana Ullman ’s comment)
In addition to all of the above serious concerns about the data report, let’s assume that the Shang paper was perfect. Although Shang’s paper asserts that the effect from homeopathic treatment is very small, even their skewed data show that the odds ratio (OR) from the 8 large and high quality homeopathic trials found an effect of OR = 0.88, which was the same as a meta-analysis of statin treatment and the occurrence of haemorrhagic stroke.
The two new reanalyses of the Shang review of homeopathic research provide the old cliché, GIGO. Junk data indeed creates junk science which creates junk and meaningless results.
Finally, a press release from the Lancet in 2005 when the Shang article was published quoted from one of its senior editors, Zoë Mullan, who acknowledged an inherent conflict on the part of the authors: “Professor Eggers stated at the outset that he expected to find that homeopathy had no effect other than that of placebo. His ‘conflict’ was therefore transparent. We saw this as sufficient.”
It was ethically sound for Eggers and team to acknowledge their assumptions and prejudices prior to submitting the article, and therefore, it was the duty of the Lancet and its editors to hold the authors to a high standard and to confirm that these biases didn’t creep into their article. Here is where the Lancet failed good science and good journalism.
The Shang team has a known history of skepticism against homeopathy. They were neither a good or reasonably objective source for this analysis.
JR said:
on November 14, 2008 at 4:31 pm
Aside from the detail that Dana adds to the discussion, his post points out a sad truth about science, politics, divorce, and most everything else in life.
The effort required by those who attempt to correct an erroneous statement is always, always much greater than the effort expended by the perpetrator of the falsehood.
What’s worse, most of the people who were influenced by the error will never hear the correction.
JR