Appendix 5 Translating comparative treatment effects of proposed surrogate measures to target clinical outcomes

Page last updated: September 2016

Introduction

The PBAC prefers submissions that do not rely on proposed surrogate measures (PSMs) to inform effectiveness in terms of patient-relevant or clinically relevant outcomes. Where possible, present evidence from direct randomised trials of the treatment effect of the proposed medicine on clinically relevant outcomes.

Where no such evidence is available, establish the likely comparative treatment effect on clinically relevant outcomes by transforming the comparative treatment effect of a surrogate measure.

A surrogate measure is a biomarker that is intended to substitute for one or more target clinical outcomes (TCOs). Although a surrogate measure may or may not have clinical relevance, it is not the key purpose for treatment, which is to affect the severity of, or the transition to, future TCOs.

Relevant to the PBAC, the relationship between a PSM and a TCO is one that quantifies the change in the TCO as a consequence of a change in the PSM. Throughout this appendix, the transformation of the PSM to the TCO should be interpreted as the transformation of the comparative treatment effect on the PSM to the comparative treatment effect on the TCO.

This appendix takes the following approach:

• A5.1 – Define the PSM and the TCO.
• A5.2 – Establish the biological reasoning for the link between the PSM and the TCO, including how pivotal the PSM is to the causation pathway of the TCO, and present epidemiological evidence to support this.
• A5.3 – Present randomised trial evidence to support the nature of the PSM-TCO comparative treatment effect relationship.
• A5.4 – Translate the comparative treatment effect on the PSM from the studies included in Part A, Subsection 2.2, to an estimate of the comparative treatment effect for the TCO.

When interpreting the evidence to identify the relationship between the PSM and the TCO (Section A5.2 of this appendix), and the relationship between the comparative treatment effect on the PSM and the comparative treatment effect on the TCO (Section A5.3 of this appendix), present indications of causality. That is, the PSM (and the comparative treatment effect on the PSM) always precedes the TCO (and the comparative treatment effect on the TCO), and their associations are strong, measured with high precision, and maintained after adjustment for confounders (if there are sufficient numbers of trials with sufficient information to enable such adjustment).

Use the following types of evidence to analyse a PSM-TCO relationship (listed from strongest to weakest):

1. multitrial meta-regression
2. single trial or small number of randomised trials where individual patient data are available (including multicentre analysis where participants were randomised by centre)
3. one randomised trial – no individual patient data or not randomised by centre
4. no randomised trial data.

Given the uncertainty associated with transforming PSMs to TCOs, ensure that the treatment effect observed on the PSM is robust and unbiased. Bias may result from, for example, issues of study quality, imbalances in baseline characteristics, loss to follow-up, discontinuations, inappropriate dosing, subgroup analysis or adjustments for crossover. Where an unknown proportion of the comparative treatment effect on the PSM may be the result of bias, the estimate of the comparative treatment effect on the TCO will be uncertain. In the absence of a robust estimate of the comparative treatment effect on the PSM, transformation to a comparative treatment effect on the TCO is not informative.

The approach taken in this appendix has been informed by the Surrogate to Final Outcomes Working Group report, and this remains a useful resource when additional explanation is required.

A5.1 Definition, selection and measurement

A5.1.1 Proposed surrogate measure

Where an intervention may have multiple benefits (eg avoiding multiple strains of a virus or multiple forms of cardiovascular events), a PSM that captures the overall intended clinical outcome is more persuasive. Ensure that the PSM is responsive and able to be measured with reliability and validity.

Define and describe the PSM, with reference to the epidemiological and randomised trial evidence identified in this appendix, by including the following:

• the units of measurement
• the measurement tool(s) or criteria used
• the evidence of reliability from test to test
• the variability across observers or different measurement tools
• the measurement of the comparative treatment effect (eg odds ratio, standardised mean difference).

Ensure that the definition and method of measurement of the PSM are consistent across the evidence. Report and discuss any discrepancies when presenting evidence in this appendix.

A5.1.2 Target clinical outcome

Ensure that the choice of TCO is patient-relevant and captures the key purposes for intervening in a disease process. The goal of treatment may be to improve quality of life, or prevent or slow a medical condition in the long term. Ensure that the TCO is consistent with the health states defined in the natural history of the disease or condition. In some cases, more than one TCO may be required to capture the effects of the proposed medicine on the disease or condition process. Justify if the nominated TCO does not capture an outcome of the disease or condition, or an adverse outcome of the treatment. There may be evidence that the proposed medicine has a positive treatment effect for one TCO (eg myocardial infarction) and a negative treatment effect for another TCO (eg haemorrhagic stroke).

With reference to the epidemiological and randomised trial evidence identified in this appendix, ensure to:

• justify the choice of the TCO and justify the exclusion of other potentially relevant TCOs (particularly those for which the proposed medicine may have a negative treatment effect)
• describe how the TCO is patient-relevant and nominate, with evidence, the extent of change that would be considered meaningful (see Subsection 2.4.3)
• state whether the TCO is reversible
• state whether the TCO is itself a substitute for a more clinically relevant outcome (multistep transformation to a subsequent TCO is discouraged)
• provide the units of measurement
• list the measurement tools or criteria used
• provide evidence of reliability from test to test
• explore variability across observers or different measurement tools
• describe the measurement of the comparative treatment effect (eg odds ratio, standardised mean difference).

Ensure that the definition and method of measurement of the TCO are consistent across the evidence. Report and discuss any discrepancies when presenting evidence in this appendix.

A5.1.3 Relationship between the proposed surrogate measure and the target clinical outcome

When exploring the nature of the PSM-TCO relationship in subsequent parts of this appendix, comment on the following:

• Is the nature of the PSM-TCO relationship still current?
• Have there been changes to treatments or health care systems over time that may have affected the PSM-TCO relationship?
• Is there any evidence of resistance or tolerance to a medicine, or a waning treatment effect over time? Consider and explain any waning treatment effects, and any effects of having no long-term randomised trials that capture the PSM and the TCO.

Derive the PSM-TCO comparative treatment effect relationship from randomised trials that measure both the PSM and the TCO. If this type of evidence is unavailable, it is difficult to quantify the link between changes in the PSM and changes in the TCO. Ensure that the epidemiological evidence in Section A5.2 of this appendix is unequivocal and robust.

A5.2 Biological reasoning and epidemiological evidence

A5.2.1 Biological reasoning

The information request for biological reasoning concerns the disease pathogenesis and disease or condition pathways, and how the PSM and the TCO relate to them, independent of medicine actions. (Mechanisms of action are presented in Section A5.4 of this appendix.) To provide confidence that altering the PSM provides clinical benefit, clearly explain the biological relationship between the PSM and the TCO.

Present and discuss the disease or condition pathway, clearly linking the PSM to the TCO. State whether the PSM is a necessary step in the development of the TCO, and discuss how close the development of the PSM is, in both temporal and pathological terms, to the development of the TCO.

A5.2.2 Epidemiological evidence

Epidemiological or observational studies support a claimed biological plausibility of the PSM-TCO relationship. Reasons for examining any association are also relevant for investigating the association between the PSM and the TCO.

Describe in detail the epidemiological evidence identified, which may include in vitro studies, animal studies, case reports, cross-sectional observational studies, ecological association studies, retrospective observational cohort studies, non–population based prospective observational cohort studies, or population-based prospective observational cohort studies.

Describe the limitations of the evidence with reference to the study design (eg individual-based associations from observational studies are more convincing than ecological associations).

Present the statistical associations, including the nature or shape of the association, the strength of the association and the precision (95% confidence interval [CI]). Report all relevant statistical outputs, such as regression coefficients and R-squared.

Describe and explain any contradictory findings, primarily where the direction of effect changes, or there is a large difference in the magnitude of effect.

A5.3 Randomised trial data for all medicines

A5.3.1 Identifying relevant trials

Review the literature systematically to find randomised trials that explore the relationship between the PSM and the TCO, irrespective of the medicine or class of medicines examined. Present the search terms, inclusion criteria and the PRISMA flowchart, clearly showing the exclusion of trials. List the excluded trials and reasons for exclusion in an attachment.

From the list of included trials, compile a list of the medicines, categorised by mechanism of action or class, that act on the PSM (see Table A5.1). Present the extension studies associated with the identified trials.

For each mechanism of action, discuss the biological reasoning for the effect of the medicine on the PSM. Discuss whether the mechanism of action of the medicine is the same as, or similar to, the pathological mechanism of the disease or condition. Rationalise any lag in onset of the treatment effect and the implications for the PSM or the TCO, or both.

A5.3.2 Trial characteristics

For each of the included trials or meta-analyses, discuss the following factors that may affect the estimate of the relationship between the comparative treatment effect on the PSM and the comparative treatment effect on the TCO:

• The quality of the included trials or meta-analyses (present an assessment of the internal validity of the included trials according to the guidance provided in Part A, Subsection 2.3, in an attachment).
• Whether relevant trials have been excluded from any meta-analyses or meta-regressions.
• Whether the analysis of the PSM was designed prospectively or retrospectively.

Present the characteristics of each of the trials as per Table A5.2.

A5.3.3 Trial results

Present the results of the randomised trials and the proposed relationship between the comparative treatment effect on the PSM and the comparative treatment effect on the TCO (Table A5.3). Where multiple trials exist for a class of medicine, clearly show the results of a meta-analysis for individual studies. Present the results of any meta-regressions, including the intercept and coefficient (and their 95% CIs), the R-squared for trials and for individuals (if individual patient data are available), and the surrogate threshold effect as determined by prediction bands. Justify where a meta-regression has not been presented.

Discuss the PSM-TCO comparative treatment effect relationship. Include details of the shape of the relationship (eg linear, exponential) and whether there is any evidence of a floor or ceiling effect, below or above which the comparative treatment effect on the PSM no longer predicts a comparative treatment effect on the TCO.

PSM = proposed surrogate measure; TCO = target clinical outcome

^a Where the PSM is a continuous variable, present the mean baseline and mean final value for the PSM, separated by treatment arm. Where the PSM is a dichotomous variable, such as progression-free survival, this column may be adapted to show the proportion in each arm achieving the PSM.

^b Where the trial has included a placebo, no treatment or best supportive care arm, report the absolute number of TCO events in that arm to give an indication of the baseline risk. A long-standing comparator may also be used as an adequate reference for baseline risk.

Where available, present results of the relationship between the comparative treatment effect for the PSM and the TCO across different trial dates, disease or condition stages, treatment settings and patient populations. State which particular subpopulations (or subpopulations are not included in the overall trial populations) do not have trial evidence available. Where these subpopulations would be treated according to the proposed listing in Part A, Subsection 1.4 of the submission, strongly justify the extrapolation of the PSM-TCO relationship to this subpopulation in Section A5.4 of this appendix.

Discuss where the relationship of the comparative treatment effect for the PSM and the TCO differs across trials, medicines or mechanisms of action. Discuss possible causes of the heterogeneity – for example:

• mechanism of action of the medicine
• population characteristics
• disease or condition characteristics, or severity
• treatment settings
• definition or measurement of the PSM
• definition or measurement of the TCO
• quality of the trial
• nature of the proposed relationship (eg linear, asymptotic, floor or ceiling effects).

A5.3.4 Multiplicity of pathways

Although unexplained heterogeneity is difficult to interpret, heterogeneity that can be linked to a characteristic will require further consideration, particularly if the cause of the difference in the relationship between the PSM and the TCO differs according to mechanism of action of a medicine, population characteristics, or disease or condition characteristics. Where differences in the relationship between the PSM and the TCO are present, it is likely that the TCO can be affected by an alternative pathological pathway that is more or less prevalent across differences in the included trials. Where the PSM-TCO comparative treatment effect relationships differ according to the:

• mechanism of action, explain why different medicines with similar effects on the PSM may result in different effects on the TCO
• patient characteristics, or disease or condition characteristics, explain why similar changes in the PSM in these subpopulations may result in different effects on the TCO.

Alternative pathological pathways that do not involve the PSM undermine the validity of the PSM. Therefore, where appropriate, exclude trials with medicines or populations in which the alternative pathway is present if:

• there is compelling evidence of the existence of the alternative pathway (such evidence may be randomised trial evidence linking an alternative PSM with the TCO)

and

• the alternative pathway is not present for the proposed medicine (and the main comparator) or the population in which listing is being sought.

Present evidence to support these claims.

Where trials are removed that have medicines of different mechanisms of action or populations that do not reflect the proposed listing, present the estimate of the PSM-TCO comparative treatment effect relationship with all trials included as the base case. Remove less-relevant trials through a sensitivity analysis.

A5.3.5 Validity of results

For each of the trials, meta-analyses and meta-regressions, compare the observed TCO comparative treatment effect with the predicted effect on the TCO if calculated according to the epidemiological evidence presented in Section A5.2 of this appendix (Table A5.4).

PSM = proposed surrogate measure; TCO = target clinical outcome

Discuss differences between the observed and predicted comparative treatment effect on the TCO.

A5.3.6 Summarising the evidence

Several parameters of the evidence presented are critical to understanding and interpreting the translation of the PSM for the proposed medicine to an estimate of the TCO (Table A5.5). These are general conditions, outside of which the translation of the PSM to the TCO becomes less certain.

IQR = interquartile range; PSM = proposed surrogate measure; TCO = target clinical outcome

^a Placebo, no treatment or best supportive care arms, or long-standing comparator

Where more than one estimate of the relationship between the comparative treatment effect on the PSM and the comparative treatment effect on the TCO has been established, justify the selection of one estimate for the base case, and present the remainder as sensitivity analyses.

A5.4 Applying the relationship between comparative treatment effects to the proposed medicine

A5.4.1 Mechanism of action

When applying the PSM-TCO comparative treatment effect relationship to the trial evidence for the proposed medicine, it is critical that both the proposed medicine and the main comparator have the same mechanism(s) of action as medicines for which the PSM-TCO comparative treatment effect has been established in Section A5.3 of this appendix. When a medicine is not of a class of medicines presented in Section A5.3 of this appendix, it is not possible to determine to what extent the TCO is affected by changes in the PSM, and to what extent it is affected by alternative pathological pathways or by negative physiological effects. Therefore, where one or both of the proposed medicine and the main comparator are not represented by the mechanism(s) of action in Section A5.3 of this appendix, the comparative treatment effect on the PSM may have a very different relationship to the comparative treatment effect on the TCO. Where this is the case, the transformation of the PSM to the TCO will be uncertain.

Explain the mechanism(s) of action and the biological reasoning for the mechanism(s) of action of the proposed medicine and the main comparator on the PSM and the TCO. Identify differences between the mechanism(s) of action of the proposed medicine, and the main comparator and the medicines identified in the trial evidence in Section A5.3 of this appendix. Clearly explain how any differences will not result in a different measurement of the PSM-TCO comparative treatment effect relationship.

Where the proposed medicine and the main comparator are within the same class of medicines identified in Section A5.3 of this appendix, it is still important to identify differences in physiological effects, and discuss whether different effects can alter the disease or condition process and, hence, the PSM-TCO comparative treatment effect relationship.

A5.4.2 Applicability of the evidence

As outlined in Section A5.3 of this appendix, the applicability of the results of the relationship between the treatment effect on the PSM and the treatment effect on the TCO to different populations and stages of disease is not guaranteed. However, evidence of consistency across different populations and stages of disease is supportive. Compare the patient population, disease or condition stages and circumstances of use for the proposed medicine and the studies identified in Section A5.3 of this appendix. If there are differences, justify why the relationship between the treatment effect on the PSM and the treatment effect on the TCO identified in Section A5.3 is applicable to the clinical trial(s) of the proposed medicine.

The PSM-TCO comparative treatment effect relationship is uncertain beyond the observed ranges for the PSM presented in Section A5.3 of this appendix. Compare the baseline values of the PSM and the comparative treatment effect on the PSM presented in Section A5.3 with that observed for the key trials of the proposed medicine, and discuss.

A5.4.3 Estimate the comparative treatment effect for the proposed medicine

Present the proposed medicine’s comparative treatment effect (with CIs) on the PSM for each trial and for a pooled analysis. Translate this using the relationship proposed in Section A5.3 of this appendix. The comparative treatment effect on the PSM and the estimate of the PSM-TCO relationship will have a degree of uncertainty; thus, capture this in the statistical approach and present as a 95% CI around the estimated comparative treatment effect on the TCO. Do not simply translate the upper and lower CIs of the comparative treatment effect for the PSM observed in the key trial by the point estimate of the relationship established in Section A5.3 of this appendix, because this does not adequately capture the uncertainty in the estimate of the comparative treatment effect on the TCO.

Discuss the implications of any surrogate threshold effect identified in Section A5.3 of this appendix.

State whether there are any concerns about the duration of the treatment effect.

 

 

 

[1] www.pbs.gov.au/info/industry/useful-resources/pbac-feedback