The first two randomized controlled trials of treatments of chronic medical conditions are much more likely to overstate any beneficial effects when compared to later clinical research, according to a group led by researchers from the Mayo Clinic, which urged regulators to be wary of data from early studies in a product’s chain of evidence.
Early RCTs in conditions such as stroke, chronic obstructive lung disease, diabetes, coronary artery disease and kidney disease can demonstrate extreme, unpredictable fluctuations in the measurement of treatment effects, along with large exaggerations of benefits, they wrote.
Dubbed the Proteus phenomenon, subsequent clinical trials and replications tend to contradict or moderate treatment effects over time. Researchers are encouraged to take many early RCTs with a grain of salt.
“Clinical and policy decisions made using the early exaggerated effect may be misguided and based on incorrect estimates of benefits and harms,” they wrote, in their paper published in Mayo Clinic Proceedings. “Considering the increasing morbidity and mortality of chronic medical conditions, decision makers should act on early evidence with caution.”
The researchers examined 70 meta-analyses, published between January 2007 and June 2015, that covered 930 clinical trials with an average follow-up of 24 months. The meta-analyses were selected from the 10 general medical journals with the highest impact factor. Those with fewer than five RCTs were excluded.
They found that 37 percent of the meta-analyses demonstrated exaggerated early effects in the first two trials — with an effect size almost three times larger than the overall size, on average — or showed the strongest heterogeneity between the first and second trial.
The effect was more frequently demonstrated in endocrinology, compared to cardiology, oncology, nephrology or pulmonary medicine, and was much more frequently seen when evaluating a medication, as opposed to a procedure or other intervention.
The causes of this are unclear, with the exaggerated effects not being significantly associated with several different aspects of clinical trials potentially linked to a risk of bias — such as the study’s size, length of follow-up, number of sites, blinding, funding sources or whether the trial was stopped early. Eighteen percent of RCTs were funded by nonprofit or governmental sources, and two-thirds were conducted in outpatient settings.
Researchers stated that possible explanations include shifts over time from efficacy evaluation in earlier trials to effectiveness evaluation in later trials, and that subsequent trials have wider inclusion criteria compared with earlier trials that may be highly selective in their enrollment.
With about one-in-three meta-analyses demonstrating the Proteus effect, regardless of the cause, users of early evidence in patient care should consider this effect when weighing harms and benefits, the researchers said.
And with recent increases in the rate of global clinical trials, even a small number of questionable trials could pose dangers.
“This is even truer for clinical trials published early in the chain of evidence, because a new breakthrough usually has more impact and media attention,” they wrote.
