2.6 Trial results: additional analyses

Page last updated: September 2016

2.6.1   Subgroup analyses

If only some of the participants from the whole trial population would be eligible for treatment according to the proposed listing, present a subgroup analysis to show the relative effectiveness of the proposed medicine in eligible participants.

Ensure that the participant characteristics and treatment details have been presented in Subsection 2.4 for the whole trial population, each relevant subgroup and its complement (ie all the participants who are not included in the subgroup).

Justification for the use of subgroups

The PBAC prefers submissions based on the whole population of a randomised trial. If a submission seeks listing of a medicine for a particular subgroup within a trial, clarify why the trial enrolled a broader population than the subgroup, and why the proposed medicine should not be available to the patients in the complement of the subgroup.

Provide the following information to support a subgroup analysis:

• The plausibility of a variation in treatment effect for the subgroup, as it relates to the pharmacological, biological or clinical rationale for using the medicine. An unexplained variation is difficult to interpret in the absence of such plausibility (cross-reference Subsection 1.1, if appropriate).
• Whether the subgroup analysis was prespecified and whether randomisation was stratified by the subgroup. Cross-reference the appropriate section in the trial protocol (or other source) that discusses prespecified subgroups, justification for the selection of subgroups, the precise method for defining subgroups and a clear justification for any threshold used to define subgroups.
• The number of subgroup analyses originally conducted and any statistical adjustment for multiple comparisons.

Results of subgroup analyses

For each outcome relevant to the submission, present the relative and absolute treatment effect measures for the whole trial population, the subgroup and the complement. The data to present will differ according to the type of outcome (see example tables in Subsection 2.5.1, which may be adapted to report subgroups). An example using dichotomous outcomes is shown in Table 2.6.1.

Include relative and absolute treatment effect measures for the subgroup, the complement of the subgroup and the total trial population. Test for interaction between the subgroup and its complement to support and quantify the association between the treatment effect and the covariate defining the subgroup. If the subgroup is defined by a continuous variable, particularly if the subgroup was not prespecified, present a sensitivity analysis on the threshold value chosen to define the subgroup for different thresholds.

Use a random effects meta-analysis for pooling data, if feasible (see Subsection 2.6.2 for further guidance on meta-analyses). See Subsection 2.6.3 for subgroup analysis in an indirect comparison of randomised trials.

– = not required; CI = confidence interval; k = number of studies contributing to the pooled estimate of effect; n = number of participants with event; N = total participants in group; OR = odds ratio; P = probability; RD = risk difference; RR = relative risk

Present adverse event data as for dichotomous data (refer to Subsection 2.5.2 for guidance). Take care when testing for interaction where the average period at risk per participant varies substantially between the relevant subgroup and its complement.

2.6.2   Meta-analyses

Where more than one trial reports a particular outcome, present a meta-analysis of the aggregated results of each trial that reported the relevant outcome, if this is feasible.

State the software used for the analysis. The Cochrane Collaboration’s RevMan¹⁹ succinctly conveys the requested array of meta-analysed information in a suitable format. Stata software²⁰ is an acceptable alternative.

Use a DerSimonian-Laird random effects model to pool group-level trial data. Explain and justify any other method used. Provide adequate detail of all sources of information relied on for these analyses, and document and reference the methods used to make them independently reproducible and verifiable.

Where individual patient data are meta-analysed or used in a pooled analysis, ensure that the trial in which each individual was randomised is included as a covariate in the analysis.

Justify any decision to not present a pooled result (eg because there is significant clinical heterogeneity between studies).

Publication bias

Assess the risk of publication bias.²¹ Where there are sufficient trials, present a funnel plot, and a statistical test such as the Begg test²² or Egger test,²³ if possible.

Results of meta-analyses

Adapt Tables 2.5.1–2.5.3 to present the pooled estimates with their 95% CIs. For example, Table 2.5.1 would be presented with Table 2.6.2. For dichotomous outcomes, separately present analyses for the relative risk, odds ratio and risk difference.

Report results for the extent of statistical heterogeneity observed using a Cochran Q statistic, degrees of freedom, chi-square test for heterogeneity, and the I-squared statistic with its 95% uncertainty interval.

For each outcome, clearly state the number of trials providing data to that outcome as a proportion of the total number of trials identified in Subsection 2.2. Discuss the implications of substantial differences in duration of follow-up or time at which patients are at risk of an event.

CI = confidence interval; df = degrees of freedom; k = number of studies contributing to the pooled estimate of effect; OR = odds ratio; P = probability; RD = risk difference; RR = relative risk

Comment on the consistency of treatment effects across the trials. Include a forest plot of relative and absolute treatment effects where it is important to interpret the data or if subgroups are presented. Discuss the results for each outcome. If the forest plot is not important to interpret the data, include it as an attachment.

Assess the clinical and statistical heterogeneity in the meta-analyses

Discuss and explain any heterogeneity of treatment effect across trials and the I-squared statistic. Unexplained heterogeneity, depending on its direction and magnitude, generally makes the summary estimator less meaningful. Where there are strong biological or methodological grounds for heterogeneity, consider presenting sensitivity analyses that explore the impact of these factors. Discuss any implications of factors that may cause heterogeneity of treatment effect with regard to the proposed target population.

Consider the factors in Appendix 4 when assessing heterogeneity. If appropriate, present Table A4.1 from Appendix 4, and outline where trials are similar and where they differ. Cross-reference the table if it is elsewhere and discuss the differences in the context of the meta-analysis.

Present the pooled incidence rate differences, if there is a risk of heterogeneity because the trials have different periods of follow-up.

Pooled time-to-event outcome data

Where multiple trials report on a time-to-event outcome, present the pooled results across the trials, the number of trials contributing to the forest plot and the proportion of trials among the total number of trials included in the submission. Data from multiple trials involving a particular time-to-event outcome may be statistically combined in a number of ways. The preferred method is to pool individual patient data from a Cox proportional hazards model. Justify and reference the method(s) used. Describe the methods and provide sufficient data as an attachment to allow the results to be reproduced and verified independently.

Ensure that the pooling method includes the trial as a covariate. If individual patient data are not available, pool the hazard ratios from the trial-level data to present the pooled hazard ratio with its 95% CI. If hazard ratios with their standard errors are not all available, pool dichotomised data based on a common duration of follow-up. A biostatistician can provide expert advice about pooling the integral between Kaplan–Meier curves.

Adverse event data

Present the meta-analysis of adverse event data as for dichotomous data (see Tables 2.5.1 and 2.6.2). Report the duration over which adverse events were recorded for each trial. If events per period at risk have been analysed (eg using straight Poisson regression or negative binomial approach, as appropriate), pool these results across trials.

Meta-analyses of subgroups

When the submission relies on a subgroup analysis, present this meta-analysis in Subsection 2.6.1 with the subgroup analysis. Justify the omission if a meta-analysis of the whole trial population or of the complement to the subgroup has not been presented

2.6.3   Indirect comparisons

Baseline characteristics, treatment details, outcomes and outcome definitions for the included trials that are relevant to the assessment of an indirect comparison are presented in Subsection 2.4. Results for the individual included trials are presented in Section 2.5. Cross-reference to these subsections where relevant.

Indirect comparison methodology

Describe the method(s) used for the indirect comparison, such as the Bucher single pairwise method,²⁴ matching-adjusted indirect comparison,²⁵ simulated treatment comparison,²⁶ network meta-analysis or mixed treatment comparison. Where there are multiple common comparators in the network, perform pairwise comparisons for each possible pathway in the network. The Bucher method²⁴ is widely used; it describes how to indirectly compare the odds ratios from randomised trials that share a common reference arm. This method has been extended to include other treatment effect measures, such as relative risk, absolute risk and hazard ratio.²⁷

More complex methods, such as network meta-analyses, may be presented as supplementary analyses. For network meta-analyses, present the results of pairwise comparisons for each link in the network. Although some methods consider nonrandomised studies in a network, avoid including nonrandomised studies. Where nonrandomised studies must be included, present the results of the network meta-analysis both with and without the nonrandomised studies.

Unadjusted indirect comparisons (such as a naive comparison between single arms), or indirect comparisons where differences in trial characteristics may affect the transitivity of the trials in the comparison, are difficult to interpret and reduce the confidence of the PBAC in decision making. Where patient-level data are available for at least one study in the comparison, use matching-adjusted indirect comparisons or simulated treatment comparisons to correct for trial differences to improve the transitivity of the comparison.

When considering complex approaches (eg matching-adjusted indirect comparisons, simulated treatment comparisons, network meta-analyses, mixed treatment comparisons), balance the additional information requests and challenges these approaches may present with any reduction in uncertainty they may deliver. Provide sufficient detail to repeat the analysis, including programming code for statistical software such as Stata, R, SAS or WinBUGS. For methods that require individual patient data (matching-adjusted indirect comparison or simulated treatment comparison), attach the individual patient dataset in a spreadsheet. Justify where this is not possible.

Transitivity assumption

Transitivity implies that the treatment comparisons within the indirect comparison do not differ with respect to the distribution of known treatment effect modifiers. Table 2.6.3 provides guidance on the key steps in assessing the transitivity assumption for indirect comparisons. These steps are further described below.

Assessing factors that may cause heterogeneity of comparative treatment effects

Studies with substantial heterogeneity may have been excluded in Subsection 2.2. For studies retained in the analysis, identify any differences in trial characteristics or patient characteristics (see Appendix 4). If Table A4.1 of Appendix 4 was completed during an assessment of heterogeneity for the inclusion of studies in Subsection 2.2, cross-reference the table; otherwise, present the table as an attachment, to compare factors within and across trial sets. Cross-reference Subsection 2.4 (baseline characteristics, treatment details and outcome definitions presented for the individual trials) and discuss any differences.

Summarise any differences within and across trial sets, and briefly state the likely effect, if any, of differences on the comparative treatment effect. Where trials are heterogeneous for characteristics that have no impact on treatment effect, these differences do not affect the transitivity of the indirect comparison.

If an indirect comparison includes confounders, adjustment using a meta-regression may be appropriate. However, meta-regression usually requires at least 10 trials per adjustment variable to achieve stability in the meta-regression results.²⁸ An alternative approach is to present a matching-adjusted indirect comparison or a simulated treatment comparison, in addition to the pairwise comparisons (ie Bucher method).

Assessing event rates in the common reference groups

Compare the event rates across the common reference arms of the pairwise comparisons. Cross-reference if this has been presented elsewhere (eg Subsection 2.2). Report and discuss the implications of any differences in the event rates. Where event rates differ, and this is likely to be because of differences in patient baseline risk, present evidence of a constant relative (or sometimes absolute) treatment effect across baseline risks. This may improve the validity of the indirect comparison.

Assessing the impact of the measure of comparative treatment effect on statistical heterogeneity

Where the indirect comparison is based on multiple A versus C and/or B versus C trials, present the statistical heterogeneity within the meta-analyses of each trial set using both an absolute and relative outcome measure. Specify which outcome measure (odds ratio, relative risk, absolute risk difference) results in the smallest amount of statistical heterogeneity and apply this outcome measure in the indirect comparison, or describe and justify an alternative outcome measure. The choice of outcome measure should minimise the variation in the comparative treatment effect within each and all sets of included randomised trials – that is, be least affected by differences between trials in terms of baseline risk or other factors. Discuss the evidence to support a constant treatment effect using the nominated outcome measure across the indirect comparison.

Particularly where the desired final outcome is an absolute risk difference yet a relative outcome measure is more consistent across trials, perform the indirect comparison using an odds ratio and convert this to an estimate of relative risk or absolute risk difference.²⁹

Results of the indirect comparison

Present the results of the indirect comparison:

• For dichotomous outcomes, present the results of each individual randomised trial as the odds ratio, relative risk and absolute risk difference with 95% CIs between the common reference, and the proposed medicine and the main comparator (this will likely require three separate tables).
• For time-to-event outcomes, present the results of each individual randomised trial as the hazard ratio with its 95% CI between the common reference, and the proposed medicine and the main comparator. Also report the median event-free survival in each arm of the common reference, proposed medicine and main comparator.
• Where there is more than one randomised trial in a set, separately pool the treatment effect results between the common reference and the proposed medicine, and between the common reference and the main comparator. Present the relevant outcome measures with 95% CIs using the random effects model (Subsection 2.6.2 discusses how to present meta-analyses).
• Calculate the indirect estimate of effect, and present the estimate as a relative risk and odds ratio (or the ratio of hazard ratios) with its 95% CI or, if previously justified, the absolute risk difference.
• Where there are multiple common reference arms that allow multiple pairwise indirect comparisons, present these and compare the indirect comparative treatment effects. Discuss any differences, noting that unexplained differences in treatment effects are difficult to interpret. Present a supplementary network meta-analysis to synthesise the available data, if appropriate.
• Where trials or trial sets have been excluded in Subsection 2.2, include sensitivity analyses in which these trials are included, if possible. Similarly, if trials or trial sets have been included that may be increasing heterogeneity, include sensitivity analyses in which these trials are excluded, if possible.

An example summary table for dichotomous outcomes is shown in Table 2.6.4. Adapt Tables 2.5.1–2.5.3 for other types of outcomes.

– = not required; CI = confidence interval; n = number of participants with event; N = total number of participants in group; OR = odds ratio; RR = relative risk

Indirect comparisons of subgroups

See Subsection 2.6.1 for presentation of a subgroup analysis. Present, where possible, an indirect comparison for the whole trial population, the subgroup and its complement. Discuss the results. Explain when this is not possible.

Additional methods to quantify results

Clearly document and reference any additional methods used to quantify the results of the indirect comparison in terms of the magnitude of effect and its 95% CI (eg network meta-analyses, mixed treatment comparisons, meta-regressions, matching-adjusted indirect comparisons or simulated treatment comparisons). Ensure that any additional documented or referenced methods are reproducible and independently verifiable.

Follow these steps to establish the comparative treatment effect:

• Explain the method used.
• Present the statistical code used for the comparison (or the equation in the case where a simple method is used), and explain the variables included in the model. Where continuous variables have been translated to categorical or dichotomous covariates for the model, explain and justify the choice of threshold. Where the choice is arbitrary (eg median age, reduction of 10 mmHg), present a sensitivity analysis where the threshold is varied.
• Present the assumptions required for each approach, how the assumptions were tested and the results of such testing.
• Describe and justify the priors where Bayesian methods have been used.
• Present the results, and CIs or intervals to capture the uncertainty in the approach.
• Present heterogeneity statistics or bias statistics.
• Interpret the results and explain any uncertainties.
• Compare the results from the simple indirect comparison method (Bucher’s method) and explain any difference.
• Present the individual patient data if these are required by the statistical approach (eg matching-adjusted indirect comparison), or justify their omission.

Where appropriate, assess the implications for the conclusions of the indirect comparison if trials that are considered to be less comparable (eg in terms of trial populations or doses) are excluded.

2.6.4   Adjustment for treatment switching

Adjustments to correct for treatment switching may reduce the PBAC’s confidence in the estimate of the treatment effect in the absence of switching, and evidence without treatment switching is preferred.

Where one or more of the included studies has participants that switched treatments, check whether the pattern of switching is consistent with current clinical practice for the comparator arm and/or future clinical practice for the intervention. If not, the observed comparative treatment effect may not reflect the expected treatment effect in the Australian population. In these cases, adjustment may be appropriate.

Methods for adjusting the treatment effect for treatment switching may rely on assumptions that are difficult to validate; ensure that the approach provides an estimate of comparative treatment effect that has a low risk of overstating the true comparative treatment effect.

Preferred approach

Describe the mechanism of treatment switching for each arm of each relevant trial. For each arm, explain:

• the medicine(s) to which switching occurred
• the extent of the switching (see Table 2.6.5)
• whether the treatment switching from the comparator arm reflects current clinical practice (or how it differs)
• whether the treatment switching from the intervention arm will reflect clinical practice if the proposed medicine is listed.

If switching (and the likely proportion of patients switching) resembles current (comparator arm) or future (intervention arm) clinical practice, adjustment for treatment switching in this arm is not appropriate and no further information is required.

If switching (or the extent of switching) does not reflect clinical practice, describe the differences and address the following issues:

• State whether treatment switching and/or specific analyses to adjust for treatment switching were prespecified in the trial protocol. Reference the section of the protocol that discusses this.
• Present the baseline characteristics of switchers and nonswitchers, as well as the characteristics of participants just before switching. Cross-reference the appropriate table in Subsection 2.4 if this has already been discussed, and summarise the differences here. If participants switched primarily as a result of disease or condition progression, present the characteristics of the participants who were at risk of switching (progressed) but did not switch and compare them with those who did switch.
• Provide the reasons for switching (eg disease or condition progression, toxicity) and the patient numbers for each category.
• Complete Table 2.6.5 to report the extent and timing of treatment switching.

cx = number switched from the medicine to the comparator at time point x; N = number randomised; sx = number at risk of switching at time point x; tx = number switched from the medicine to any subsequent therapy at time point x

a Patients at risk of switching are usually those who stop the assigned treatment and remain alive (eg disease or condition progression, or medicine intolerance).

Several methods can be used to adjust survival estimates for treatment switching.³⁰ Using simple methods is acceptable when the estimate of comparative treatment effect is clearly towards the null. More complex methods (eg inverse probability of censoring weights, a rank-preserving structural failure time model, 2-stage methods) have assumptions that can be difficult to validate. Provide details on the approach, the assumptions and how they have been tested, and justify the selection of the approach including a rationale supporting how the assumptions used by each method are reasonable. Provide additional evidence or discussion that will reduce the uncertainty associated with the estimate of the treatment effect following adjustment. If complex methods are used, present the results of several commonly used methods, and clearly justify why a method is not used. Where more complex methods are presented, also present the results of simpler methods as a reference.

Where the discussion of methods is necessarily detailed, present this in a technical attachment.

Results of adjustment for treatment switching

For each of the methods used to adjust the treatment effect for treatment switching, present the adjusted treatment effect and the 95% CI. Explain any heterogeneity of treatment effects across the different methods for adjustment. Present the treatment effect and the 95% CI in the absence of switching for comparison.

Where possible, present a Kaplan–Meier graph with curves for each treatment arm with adjustments for treatment switching.³¹ Display 95% CIs for each arm, and include a risk table with the graph to display the numbers of censored patients and patients still at risk in each arm across regular time points for the trial’s follow-up period.

Where complex statistical approaches for adjusting for treatment switching have been used, search the literature for studies that report on the treatment arms in the absence of switching (eg historical controls). Discuss the applicability of the findings from the identified studies to the key trials in the submission. Compare the Kaplan–Meier curves of the nonswitched studies with the modelled Kaplan–Meier curves and discuss where they differ.

In addition, where there is a largely uncontaminated estimate of an outcome that occurred before switching (eg progression-free survival), discuss whether the outcome is a valid surrogate for the clinically relevant outcome (eg overall survival) in Subsection 3A.4. For example, where progression-free survival is a justifiable surrogate for overall survival, compare the estimate of overall survival by transforming progression-free survival with the overall survival determined by statistical methods used above to adjust for switching.

If possible, use a number of different statistical approaches to adjust for switching. A similar result from a number of analyses will reduce uncertainty and increase confidence in the result. Comparison with historical controls or with overall survival calculated from a surrogate measure will also improve confidence in the statistical approaches.

Adjustment for treatment switching in trials that rely on subgroups or indirect comparisons

There is a risk of bias associated with the use of subgroups, indirect comparisons and adjustment for treatment switching. Approaches that combine adjustment for treatment switching with either subgroup analyses or indirect comparisons (or both) may be regarded as poor-quality evidence. Therefore, do not combine these approaches or, if unavoidable, ensure that the results can be clearly interpreted by the PBAC as conservative.