3A.9 Uncertainty analysis: model inputs and assumptions

Page last updated: September 2016

Information Requests

  • Explain the methods used to represent the uncertainty around the model’s input parameters, translations and structure. For each, define the uncertainty or alternatives to be tested in sensitivity or scenario analyses (Subsection 3A.9.1)
  • Present and discuss the univariate sensitivity and scenario analyses (Subsection 3A.9.2)
  • Present and discuss relevant multivariate analyses and any probabilistic sensitivity analysis (Subsection 3A.9.3)
  • Summarise the findings of the uncertainty analysis (Subsection 3A.9.4)

3A.9.1 Identifying and defining uncertainty in the model

Present univariate deterministic sensitivity analyses for all uncertain input parameters, or natural groups of input parameters (eg cost or utility weights for all target clinical outcomes). The following requests are based on good-practice guidelines for model parameter estimation and uncertainty analysis.38

Parameter uncertainty

Use commonly adopted statistical standards to represent the uncertainty around the true value of each uncertain input parameter. For example, beta distributions are a natural match for transition probabilities; log-normal for relative risks or hazard ratios; logistic distributions to calculate odds ratios; and gamma or log-normal for costs and utility parameters.

Justify using alternative distributions. Use interval estimates (eg 95% CIs) derived from fitted probability distributions to define the ranges of the parameter values tested in the deterministic sensitivity analyses.

Where there is very little information on a parameter, adopt a conservative approach by defining a broad range of possible parameter values. Never exclude parameters from uncertainty analysis on the grounds that there is insufficient information to estimate uncertainty.

Consider correlation between input parameter values. If applicable, represent the joint uncertainty around the true values of two or more input parameters in the uncertainty analyses. In particular, it is preferable to represent the joint uncertainty around transition probabilities in the intervention group and the comparator group through the application of a relative treatment effect parameter. If a relative treatment effect parameter is not applicable, individual-level data for the comparator and intervention could be bootstrapped to provide more realistic estimates of the joint uncertainty between these.38

The joint estimation of multiple input parameters when using regression analysis produces relevant correlation parameters. Otherwise, model calibration methods may be used to represent joint uncertainty around the true value of model input parameters.

Translational uncertainty

Where clinical data have required translation for applicability issues, transformation or extrapolation for incorporation into the model, systematically consider the assumptions incorporated into the translation and identify any uncertainty in these assumptions. Identify plausible alternatives for testing in scenario analysis.

Examples of analyses that can be used where the data or outcome translations are incorporated into base-case analysis are presented in Table 3A.9.1.

Table 3A.9.1 Examples of potential sources of translational uncertainty in the economic model and suggested scenario analyses

Translations incorporated into base-case analysis

Suggested uncertainty analysis

Transformation of continuous outcome data to a dichotomous outcome

Alternative thresholds (Subsection 2.5.1)

Treatment effect with adjustment for switching

Treatment effect without adjustment for switching, and/or using an alternative adjustment technique (Subsection 2.6.4)

Treatment effect based on translation (eg subgroup analysis) following applicability study

Treatment effect based on intention-to-treat population (Subsections 2.6.1, 3A.3.2)

Selected source(s) of data for treatment effect

Alternative available source(s) of data, and/or meta-analysis of data as source of treatment effect (alternative analyses presented in Subsections 2.5 and 2.6)

Transformation of a surrogate to a final outcome

Range of alternative plausible values (as derived establishing STFO relationship; Subsection 3A.4.2)

Extrapolation of data beyond the trial

Alternative data truncation point(s), alternative choices of parametric model, or alternative assumptions regarding ongoing treatment effect (Subsection 3A.4.3)

Pooled within-trial data to estimate utility values (or alternative approach)

Estimates based on individual arms (or the alternative approach; Subsection 3A.5.1)

Externally sourced utility values

Alternative values or sources (Subsection 3A.5.1)

STFO = surrogate to final outcome

Structural uncertainty

If multiple plausible model structures are defined, assess the potential impact of the alternative structures on the model outputs. If a substantial impact is predicted, use a formal approach to characterise the structural uncertainty. Parameterise structural assumptions where there is sufficient clinical evidence or expert opinion to do so. Alternatively, use scenario analyses to assess the impact of assumptions around the structure of the economic model. Report the results of each set of plausible structural assumptions.

Describe and justify the inclusion and exclusion of potential scenario analyses when making alternative assumptions about data translation and model structure.

Include an analysis of the impact of the time horizon.

Use other scenario analyses to assess the effects of substantial use of the proposed medicine beyond the intended population and circumstances of use defined in the requested restriction. This wider population or circumstances are expected to have demographic and patient characteristics and circumstances that differ from the target population and circumstances.

3A.9.2 Presentation of univariate sensitivity and scenario analyses

Tabulate all parameter values and assumptions included in the model, and present the results of univariate sensitivity and scenario analyses in a similar format to Table 3A.9.2.

Use a tornado diagram to represent the relative effect of the uncertainty around alternative input parameters on the base-case incremental cost-effectiveness result.

Identify the input parameters and model assumptions to which the incremental cost-effectiveness results are most sensitive.

3A.9.3 Presentation of multivariate and probabilistic sensitivity analyses

Use multivariate sensitivity analyses to test the combined effects of the uncertainty around the true values of input parameters to which the base-case incremental cost-effectiveness result was shown to be sensitive in the univariate analyses.

Describe the multivariate sensitivity analyses to be undertaken, and present the results. Justify the inclusion and exclusion of parameters in these analyses.

A probabilistic sensitivity analysis (PSA) may be provided in addition to deterministic sensitivity analysis. Although PSA can usefully characterise parameter uncertainty, it cannot address translational or structural uncertainty.

If undertaking a PSA on a cohort-based state transition model, the number of iterations (sets of randomly sampled input parameter values included in the analysis) should provide stability in the model outputs across multiple analyses using alternative random number seeds. Provide the random seed associated with the presented results to enable replication, and also ensure that the model permits alternative seeds.

If undertaking a PSA on an individual-level model (eg a discrete event simulation), the number of iterations may be selected to balance stability of model outputs and a reasonable time required to undertake a PSA (eg a few hours, rather than a few days).

Use cost-effectiveness planes and acceptability curves to present the results of a PSA, as well as the tabulated presentation of the interval estimates for the ICER or the incremental net benefits of the proposed medicine.

3A.9.4 Summary of the uncertainty analysis

Describe and justify a likely range of values within which the true estimate of the incremental cost-effectiveness of the proposed medicine is likely to lie, identifying the key sources of uncertainty. This range may be informed by a formal PSA, or by subjective interpretation of the presented deterministic sensitivity and scenario analyses.

Discuss the implications of the sensitivity and scenario analyses with respect to the certainty of the base-case ICER estimate.

Discuss the likely overall effect of deficiencies in the evidence base on the reported cost-effectiveness of the proposed medicine.

Table 3A.9.2 Results of the sensitivity and scenario analyses characterising the uncertainty around the ICER

Variable or assumption

Base-case value

Plausible alternative(s) or range of values

Incremental outcomes

Incremental costs

ICER

Description of impact on ICER

Base case

 

 

[base case]

[base case]

[base case]

 

Discounting rate

Outcomes and costs = 5%

Outcomes and costs = 3.5%
Outcomes and costs = 0%

[alternative estimates]

[alternative estimates]

[alternative estimates]

[describe as required]

Plausible range of treatment effect, if modelled as a variable (eg hazard ratio or relative risk)

[add]

[eg upper and lower 95% confidence intervals around estimate]

[alternative estimates]

[alternative estimates]

[alternative estimates]

[describe as required]

Altered patient characteristics, if relevant

[add]

[eg different average age, disease or condition severity]

[alternative estimates]

[alternative estimates]

[alternative estimates]

[describe as required]

Transition or event probabilities

[add]

[add]

[alternative estimates]

[alternative estimates]

[alternative estimates]

[describe as required]

Outcome-related assumptions or variables [Recommended examples:

  • alternative estimates of the STFO relationship
  • alternative methods or sources of utility weights]

[add]

[add]

[alternative estimates]

[alternative estimates]

[alternative estimates]

[describe as required]

Cost-related assumptions or variables

[add]

[add]

[alternative estimates]

[alternative estimates]

[alternative estimates]

[describe as required]

Alternative extrapolation variables or assumptions [Recommended examples:

  • start point
  • choice of parametric model
  • assumption regarding ongoing treatment effect]

[eg maximum follow-up]

[eg median follow-up]

[alternative estimates]

[alternative estimates]

[alternative estimates]

[describe as required]

Any other translation assumptions [eg use of intention-to-treat/nonadjusted data]

[add]

[add]

[alternative estimates]

[alternative estimates]

[alternative estimates]

[describe as required]

Alternative assumptions regarding model structure

[add]

[add]

[alternative estimates]

[alternative estimates]

[alternative estimates]

[describe as required]

Time horizon

[add]

[eg trial based; 5, 10, 20 years, as appropriate]

[alternative estimates]

[alternative estimates]

[alternative estimates]

[describe as required]

Plausible alternatives for other variables or assumptions [eg including leakage beyond the requested restriction]

[add]

[add]

[alternative estimates]

[alternative estimates]

[alternative estimates]

[describe as required]

ICER = incremental cost-effectiveness ratio; STFO = surrogate to final outcome