Analysis of clinical trials including prognostic models
Randomised controlled trials (RCTs) are one of the most important tools to estimate effects of medical interventions. However, there are a number of issues relating to the statistical analysis of RCTs over which there is much debate. We aim to consider and raise discussion of a number of these issues, and suggest possible statistical approaches for dealing with these in the analyses of RCTs.
RCTs typically provide unadjusted comparisons of patient outcomes according to treatment. However, trials often adjust for important predictors of outcome to allow for chance imbalances between treatment groups at baseline1 although such adjustment often makes little difference to the results or conclusions. More recently, it has been proposed that adjustment using appropriate co-variates can also be used to improve the power of an RCT irrespective of any baseline imbalance (also considered in the context of the potential to reduce the required sample size)2,3.
Under this theme we aim to consider both the impact of co-variate adjustment in the presence of chance baseline imbalance, and the potential for co-variate adjustment to improve power of an RCT. In discussing these issues possible approaches to such adjusted analyses and their interpretation will be explored including the choice of the co-variates and how adjusted effect measures should be interpreted (for example the adjusted odds ratio in comparison to an unadjusted odds ratio)4.
Current work is ongoing evaluating the impact of co-variate adjustment on power in trials of traumatic brain injury and also cardiovascular disease.
As well as adjusting the treatment effect for other independent prognostic factors there is interest in assessing the actual association of other co-variates on outcome in their own right. A number of prognostic models in different disease areas have been developed to enable the prognosis of patients to be more accurately assessed and appropriate treatment and risk management given, for example in traumatic brain injury5. The development and impact of these models will be considered as will approaches to simplifying them to be readily used by clinicians and patients. The development of these models is also relevant to the discussion on approaches to subgroup analyses mentioned below.
The analysis of RCTs by subgroups of individuals (e.g. according to age, gender or medical history) remains controversial and often misunderstood6,7. It is recognised that such analyses should be limited to a few key baseline factors which are specified prior to any analyses being undertaken. In addition an appropriate analysis typically calculates effect estimates and confidence intervals within such subgroups together with an overall interaction test rather than calculating separate p-values within each category of the subgroup.
(1) Subgroup analysis by risk
However, patients vary considerably often presenting with multiple risk factors for the outcome of interest. An alternative approach to analysing subgroups to account for this has been proposed8 whereby patients are analysed according to their underlying risk rather than individual characteristics and a single interaction test performed between underlying risk and treatment. These issues will be discussed more fully including possible approaches to such subgroup analyses. Current work in this area has been undertaken in the context of treatment for non-ST-elevation acute coronary syndrome9 and will be further explored in trauma, smoking cessation and other disease areas.
(2) Bayesian approach to subgroup analysis
When two treatments are compared in many subgroups, the chances of making “false discoveries” (Type-I error) increase. As more subgroups are looked at it becomes easier to find some difference just by chance. This is a threat of great concern to statisticians. For this “statistical” reason subgroup analysis is often discouraged. Also, real qualitative interactions in sub-group analyses (where the direction of treatment benefit differs in the different subgroups) are considered by many trial methodologists to be extremely unusual and some recommend that they should generally be disbelieved.
However the medical point of view on subgroup analysis is different. If the effect of the intervention changes in particular subgroups, this is likely to have important implications for clinical practice and policymaking. Even when the interaction is not qualitative but quantitative (where the direction of treatment benefit is the same but the magnitude of the effect differs in the different subgroups) the change in the effect can be clinically relevant and cannot be ignored. Furthermore, from a biological point of view it is not always sensible to think that a treatment is going to act in the same way in any subgroup of patients, regardless of their many other circumstances. Rather the contrary, we would expect to find differences, arguably making subgroup analysis the logical thing to do.
Health research should be guided by relevant health issues and questions. Effects of treatments in subgroups of patients is clearly an interesting and very relevant question to health professionals. There are certainly important statistical issues when interpreting results from subgroup analysis, but rather than discouraging subgroup analysis, we should warn about and explain these issues to the investigators and so that they can interpret the results prudently. And above all, we should try to develop better techniques to overcome the current “statistical” problems of subgroup analysis.
A recent paper10 suggested eleven criteria to assess the credibility of subgroup analysis; among them one is about the indirect evidence or “biological rationale”, basically the premise is that we are more likely to believe an effect for which we have some “other evidence”. Bayesian analysis could be a statistical tool to formally and explicitly assess this criterion. Priors believes of a specific subgroup effect could be elicited among experts and they can be “updated” with the trial data to a posterior. Current work is evaluating this approach in a randomised controlled trial in trauma.
In the context of clinical trials of conditions with high risk of mortality, the evaluation of non-fatal outcomes is challenging because of the potential competing risk of death that may interfere with the probability of experiencing the non-fatal outcome of interest. The importance of disentangling the effect on non fatal outcomes is relevant for decision making, and to understand potential mechanism of action that could inform future research.
We will first consider what analytical approaches would be appropriate were the time to both competing outcomes available. Then we will investigate whether estimates of treatment effects on the competing non-fatal event can be made with such data, and if so, under what assumptions. Current work is evaluating this approach in a randomised controlled trial in trauma.
1 Altman DG. Adjustment for covariate imbalance. Biostatistics in Clinical Trials. Chichester: John Wiley & Sons; 2001: 122-7.
2 Hernández AV, Steyerberg EW, Habema JDF. Covariate adjustment in randomized controlled trials with dichotomous outcomes increases statistical power and reduces sample size requirements. Journal of Clinical Epidemiology 2004; 57: 454-460.
3 Roozenbeek B, Maas AIR, Lingsma HF et al. Impact Study Group. Baseline characteristics and statistical power in randomized controlled trials: Selection, prognostic targeting, or covariate adjustment? Critical Care Medicine 2009; 37(10): 2683-90.
4 Steyerberg EW, Eijkemans MJC. Heterogeneity bias: The difference between adjusted and unadjusted effects. Medical Decision Making 2004; 24: 102-4.
5 Perel P, Arango M, Clayton T et al. Predicting outcome after traumatic brain injury: practical prognostic models based on large cohort of international patients. British Medical Journal 2008; 336: 425-9.
6 Assmann SF, Pocock SJ, Enos LE, Kasten LE. Subgroup analysis and other (mis)uses of baseline data in clinical trials. Lancet 2000; 355: 1064-9.
7 Wang R, Lagakos SW, Ware JH, et al. Reporting of subgroup analyses in clinical trials. New England Journal of Medicine 2007; 357: 2189-94.
8 Pocock SJ, Lubsen J. More on subgroup analyses in clinical trials. New England Journal of Medicine 2008; 358: 2076.
9 Fox, KA, Clayton TC, Damman P et al. Long-term outcome of a routine versus selective invasive strategy in patients with non-ST-segment elevation acute coronary syndrome a meta-analysis of individual patient data. Journal of the American College of Cardiology 2010; 55: 2435-45.
10 Sun X, Briel M, Walter SD, et al. Is a subgroup effect believable? Updating criteria to evaluate the credibility of subgroup analyses. BMJ 2010;340:c117.