The unpredictability paradox: review of empirical comparisons of randomised and non-randomised clinical trials

We identified studies from the Cochrane Review Methodology Database,6 other methodological bibliographies, Medline, and SciSearch, and by hand searching journals, personal communication with methodologists, and checking the reference lists of relevant articles

R. Kunz; A. D Oxman

2011

Scholarcy highlights

  • Observational evidence is clearly better than opinion, but it is thoroughly unsatisfactory
  • To summarise comparisons of randomised clinical trials and non-randomised clinical trials, trials with adequately concealed random allocation versus inadequately concealed random allocation, and high quality trials versus low quality trials where the effect of randomisation could not be separated from the effects of other methodological manoeuvres
  • Eleven studies that compared randomised controlled trials with non-randomised controlled trials, two studies that compared trials with adequately concealed random allocation and inadequately concealed random allocation, and five studies that assessed the relation between quality scores and estimates of treatment effects, were identified
  • Failure to use randomisation or adequate concealment of allocation resulted in larger estimates of effect due to a poorer prognosis in non-randomly selected control groups compared with randomly selected control groups
  • Failure to use adequately concealed random allocation can distort the apparent effects of care in either direction, causing the effects to seem either larger or smaller than they really are
  • 89% of new treatments were rated as improvements, but only non-randomised COTs detected a significantly higher “treatment gain” from the new treatment compared with RCTs
  • It is a paradox that unpredictability is introduced into the design of clinical trials by using random allocation to protect against the unpredictability of the extent of bias in the results of non-randomised clinical trials
  • The size of these distortions can be as large as or larger than the size of the effects that are to be detected

Need more features? Save interactive summary cards to your Scholarcy Library.