2.3 Trial design and execution
Page last updated: September 2016
Information Requests
- Assess risk of bias (internal validity) using an approach relevant to the design of the studies included in the submission (Subsections 2.3.1–2.3.3)
2.3.1 Internal validity
Assess the risk of bias to provide the PBAC with a clear idea of which trials are of greater scientific rigour. Information needed to inform an assessment of the risk of bias differs for randomised trials and nonrandomised studies, and the two approaches are described in Subsections 2.3.2 and 2.3.3, respectively.
Where the rarity of the disease or condition prohibits the use of a traditional parallel-group randomised controlled trial, alternative trial designs may be acceptable (eg randomised crossover trials, including n-of-1 trials and trials with a randomised adaptive design). Such trials require a protocol, a clinical trial registry number or identifier, and a design that involves a randomisation procedure. Where a submission is based on such a trial, risk of bias can be addressed as for randomised trials.
The best approach to assessing the validity of single-arm studies will depend on the design of the study. Justify the approach (or modifications to the approaches below) taken to capture the key limitations of the study design.
2.3.2 Risk of bias assessment for randomised trials
The preferred approach for randomised trials is described in Chapter 8 of the Cochrane handbook for systematic reviews of interventions (version 5.1.0).3 Complete Table 2.3.1 for each included trial.
Present factual information about the design and conduct of the trial – such as how the participants were allocated to groups, or whether or not participants or assessors were blinded. After the table, provide additional information about the following aspects that may influence an assessment of risk of bias (state if this information is not relevant or not available):
- Unmasking. Discuss whether the medicine or comparator has any effects (such as adverse events) that may result in the participant, the investigator or the outcome assessor ‘guessing’ the treatment allocation of the participant.
- Treatment decisions. Describe the decision-making processes (including responsible personnel) for decisions such as either stopping treatment or starting a new or concomitant treatment in response to adverse events, treatment failure or inadequate treatment response. Discuss whether these decisions could affect the measurement of any of the key outcomes.
- Testing decisions. Discuss whether the investigator or person caring for the participant can request tests that are not part of the protocol or that occur at different times than prescribed in the protocol. Discuss whether these tests may affect the measurement of key outcomes or adverse events.
- Nature of outcomes. Regardless of whether the trial is blinded or open-label, discuss whether any of the key outcomes could be affected by a participant’s, investigator’s or outcome assessor’s knowledge of treatment allocation.
- Missing data. Discuss the reasons for any loss to follow-up or missing data. Summarise how missing data have been imputed and the assumptions surrounding the use of these methods (cross-reference Subsection 2.4). Discuss whether the characteristics of the participants who were lost to follow-up are similar to, or different from, those remaining in the trial, and state whether there is a differential loss to follow-up or discontinuation across the arms. Discuss whether missing data are expected to affect the treatment effect, and if the effect is likely to be overestimated or underestimated.
Where the information provided in the submission implies a risk of bias, describe the likely effect that the bias may have on the direction of the comparative treatment effect.
Present the flow of participants in Table 2.3.2.
|
Type of bias |
Trial |
Description |
Source(s): page number(s) of clinical study report/publication |
|---|---|---|---|
|
Selection bias: random sequence generation and allocation concealment |
Trial 1 |
[Describe the method used to generate the allocation sequence in sufficient detail to allow an assessment of whether it should produce comparable groups. Describe the method used to conceal the allocation sequence in sufficient detail to allow an assessment of whether intervention allocations could have been foreseen in advance of, or during, participant enrolment] |
[insert source] |
|
Trial 2 |
[insert description] |
[insert source] |
|
|
Trial 3 |
[insert description] |
[insert source] |
|
|
Performance bias: blinding of participants and personnel |
Trial 1 |
[Describe all measures used, if any, to blind trial participants and personnel from knowing which intervention a participant received. Provide any information relating to whether the intended blinding was effective] |
[insert source] |
|
Trial 2 |
[insert description] |
[insert source] |
|
|
Trial 3 |
[insert description] |
[insert source] |
|
|
Detection bias: blinding of outcome assessment |
Trial 1 |
[Describe all measures used, if any, to blind outcome assessors from knowing which intervention a participant received. Provide any information relating to whether the intended blinding was effective] |
[insert source] |
|
Trial 2 |
[insert description] |
[insert source] |
|
|
Trial 3 |
[insert description] |
[insert source] |
|
|
Attrition bias: incomplete outcome data |
Trial 1 |
[Describe the completeness of outcome data for each main outcome, including attrition and exclusions from the analysis. State whether attrition and exclusions were reported, the numbers in each intervention group (compared with total randomised participants), and the reasons for attrition/exclusions, where reported] |
[insert source] |
|
Trial 2 |
[insert description] |
[insert source] |
|
|
Trial 3 |
[insert description] |
[insert source] |
|
|
Reporting bias: selective reporting |
Trial 1 |
[State how the possibility of selective outcome reporting was examined, and what was found] |
[insert source] |
|
Trial 2 |
[insert description] |
[insert source] |
|
|
Trial 3 |
[insert description] |
[insert source] |
|
|
Other sources of bias |
Trial 1 |
[State any important concerns about the study design and execution that are not addressed elsewhere in this table] |
[insert source] |
|
Trial 2 |
[insert description] |
[insert source] |
|
|
Trial 3 |
[insert description] |
[insert source] |
Note: Adapted from the Cochrane Collaboration’s tool for assessing risk of bias (Chapter 8 of the Cochrane handbook for systematic reviews of interventions, version 5.1.0)3
|
Trial ID |
Intervention arm |
No. randomised |
Did not receive intervention |
Lost to follow-up |
Discontinued |
Analysed |
Source of information |
|---|---|---|---|---|---|---|---|
|
Trial 1 |
Proposed medicine |
N |
n (%) |
n (%) |
n (%) |
n (%) |
Reference the source of this information |
|
Main comparator |
N |
n (%) |
n (%) |
n (%) |
n (%) |
Reference the source of this information |
|
|
Trial 2 |
Proposed medicine (high dose) |
N |
n (%) |
n (%) |
n (%) |
n (%) |
Reference the source of this information |
|
Proposed medicine (low dose) |
N |
n (%) |
n (%) |
n (%) |
n (%) |
Reference the source of this information |
|
|
Main comparator |
N |
n (%) |
n (%) |
n (%) |
n (%) |
Reference the source of this information |
|
|
Trial 3 |
[etc] |
[etc] |
[etc] |
[etc] |
[etc] |
[etc] |
[etc] |
Systematic reviews and meta-analyses
Assess the risk of bias for included individual trials within an identified systematic review or meta-analysis. Where individual trials are not able to be retrieved and the submission relies on a pooled treatment effect from the published systematic review and meta-analysis, clearly report the risk of bias assessment undertaken by the authors of the systematic review; also assess the quality of the systematic review using a validated tool (eg AMSTAR4 or ROBIS5).
2.3.3 Risk of bias assessment for nonrandomised studies
Nonrandomised studies are at high risk of bias. Methods for mitigating the risks associated with the differential distribution of known confounders because of nonrandom treatment allocation (such as matching and controlling for confounders in the analysis) cannot adjust for the differential distribution of unknown confounders.
Present an assessment of risk of bias appropriate for the study design and conduct of the included nonrandomised studies. The internal validity of a nonrandomised study can be elicited by reference to how the study design or conduct differs from that of a well-designed, double-blind randomised controlled trial. Discuss whether there is a risk of bias as a result of the study design or the conduct of the study, and describe any measures taken to mitigate the risk of bias. Potential sources of bias include:
- imbalances in baseline or post-baseline characteristics that are potential confounders (see Appendix 4)
- treatment switching or imbalances in the use of later-line or concomitant therapies
- patients who are selected into the study and are already receiving the intervention (or comparator), where these patients are different to those who are not, or who have started then stopped, the intervention, and where these two groups may have different expected outcomes
- a definition of the intervention or comparator (doses, duration, setting) that is too broad or ambiguous, and where allocation of intervention status may be influenced by the knowledge of outcomes
- missing data that are not missing at random, not balanced across groups and of sufficient magnitude to affect the estimate of the outcome or the method of accounting for missing data affects the estimate of the outcome irrespective of whether data are missing at random
- outcome measures that are subjective, or outcome assessors who are not blinded to treatment allocation
- timing of measurement of outcomes, or the method of determining outcomes, that differs between study arms
- reporting of only some of the results that were pre-defined in the protocol
- reporting of outcomes, time points or subgroups that were not pre-defined in the protocol.
The Risk Of Bias In Non-randomized Studies – of Interventions (ROBINS-I) tool6 provides guidance on how to identify and report study characteristics that may impact on the comparative treatment effect in nonrandomised studies. Use the domains identified in the ROBINS-I tool to organise a discussion of the risk of bias. It is not necessary to complete the ROBINS-I tool. If another tool is used, or an alternative approach is taken, describe the approach.
Present factual information on the design and conduct of the study. Provide references to support the information.