Once a literature review is complete, each individual study must be vetted for its relevance to the topic. Does it address the outcomes of interest? Does it apply to the particular practice setting? Does it apply to the particular patient population? Not all studies will address all outcomes. However, the evidence for all important patient outcomes in a specific clinical situation must be evaluated. For example, in Stage IV rectal cancer when considering a palliative resection versus long-term chemotherapy, evidence for each management strategy must be evaluated for both quality and quantity of life. In addition the risk of a poor outcome as viewed by the patient due to either surgical or medical complications must be considered. For many questions a structured review or CPG exists that covers most of the outcomes of interest but a primary literature search may be required to supplement evidence for specific outcomes.
Stratifying Evidence
Once the evidence is collected, it is initially stratified by study methodology. Well designed structured reviews and meta-analysis based on well-designed RCTs are the highest order of evidence, followed by well designed RCTs themselves, lower quality RCT studies with methodological limitations and finally observational studies (cohort and case control). Within the GRADE system, expert opinion is not viewed as evidence in and of itself. In other words, while an expert is required to interpret evidence, expert opinion may or may not be based on best evidence.
Random Error and Systematic Error (Bias)
All studies are subject to error, which may to a greater or lesser extent affect the results of a study and our confidence in the stated results. Error can be classified into two major categories: random error and systematic error or bias. Random error is the variation in outcomes due to chance alone. Studies are performed on sample populations from the population at large, thus the results of each study are estimates of the actual effect of an experimental intervention on the overall population. If a study is performed on 20 different sample populations replicating strict methodology each time, the final results of each trial will be closely approximated but will vary due to chance, much like a coin toss performed multiple times will not always add up to exactly 50 % heads and 50 % tails. Random error is by definition variable and can occur in either direction, (you can toss 7 heads or 6 tails in a row), resulting in a positive or negative effect on the estimate of an outcome of interest. It can be minimized through the use of large sample sizes either in individual studies or by combining similar smaller studies in a meta-analysis. A well designed prospective study should have a sample size calculation for a specific outcome as part of its methodology.
Systematic error or bias results in a systematic or fixed effect on a study. This type of error is not affected by sample size as it is related to study methodology. Virtually no study is devoid of all bias. However, when evaluating a study one must try to determine whether the effect from systematic error or bias is large enough to significantly alter the observed effect of an experimental intervention.
Methodological Limitations (Bias)
There are four levels of evidence in the GRADE system; high quality, moderate quality, low quality and very low quality (Table 2.1) [7]. Evidence from RCTs starts out as high quality evidence but may be down graded to moderate or even low quality if bias or methodological issues are identified. Similarly, although evidence from observational trials is generally classified as low or very low quality, it may be upgraded under certain circumstances (Fig. 2.1).
Table 2.1
GRADE: levels of evidence and definitions
Category | Definition | Examples |
|---|---|---|
High | We are very confident that the true effect lies close to that of the estimate of the effect | Randomized trials without serious limitations |
Well performed observational studies with very large effects | ||
Moderate | We are moderately confident in the effect estimate: the true effect is likely to be close to the estimate of the effect, but there is a possibility that it is substantially different | Randomized trials with serious limitations |
Well-performed observational studies yielding large effects | ||
Low | Our confidence in the effect estimated is limited: the true effect may be substantially different from the estimate of the effect | Randomized trials with very serious limitations |
Observational studies without special strengths or important limitations | ||
Very low | We have very little confidence in the effect estimate: the true effect is likely to be substantially different from the estimated of effect | Randomized trials with very serious limitations and inconsistent results |
Observational studies with serious limitations | ||
Unsystematic clinical observations (case series or case reports) |
Bias in randomized trials can occur in three parts of a study; differences observed at the start of a study, differences that arise as a study progresses and differences at the completion of a study [16] (Table 2.2). Blinding should be present at all levels of a trial starting with allocation and randomization, and including the patient, the care giver, the assessors and the data analysts. When absent, the results usually favor an overestimation of effect. Differences in treatment or exposure to confounding treatments in the experimental arm, incomplete follow up or loss to follow up and failure to adhere to the intention to treat principle in superiority trials are also associated with over estimation of effect. Loss to follow up takes on greater importance when the number of events in either the experimental or control group is small relative to the percentage lost to follow up or if the loss to follow up is imbalanced between the two groups.
Crohn’s Colitis and Ileal Pouch Anal Anastomosis
IBD: Management of a Painful Anal Fissure and Skin Tags in Patients with Crohn’s Disease
Deciding on an IRA vs. IPAA for FAP
Utility of Primary Tumor Resection in Asymptomatic, Unresectable Metastatic Colon and Rectal Cancer
Anal Fissure: Recurrence After Lateral Internal Sphincterotomy
Which Patients with Fecal Incontinence Require Physiologic Workup?
Table 2.2
Study limitations in randomized trials
1. Lack of allocation concealment |
Those enrolling patients are aware of the group to which the next enrolled patient will be allocated (e.g., “pseudo” randomized trials with allocation by day of the week, birth date, chart number etc.) |
2. Lack of blinding |
Patient, care givers, those recording outcomes, those adjudicating outcomes or data analysts are aware of which arm patients are allocated |
3. Incomplete accounting of patients and outcome events |
Loss to follow-up and failure to adhere to the intention-to-treat principle in superiority trials; or in noninferiority trials, loss to follow-up and failure to conduct both analysis considering only those who adhered to treatment, and all patients for whom outcome data are available |
4. Selective outcome reporting bias |
Incomplete or absent reporting of some outcomes and not others on the basis of results |
5. Other limitations |
Stopping early for benefit |
Use of unvalidated outcome measures (e.g. patient reported outcomes) |
Carryover effects in crossover trial |
Recruitment bias in cluster randomized trials |
Adapted from Balshem et al. [7]
Related posts:
Crohn’s Colitis and Ileal Pouch Anal Anastomosis
IBD: Management of a Painful Anal Fissure and Skin Tags in Patients with Crohn’s Disease
Deciding on an IRA vs. IPAA for FAP
Utility of Primary Tumor Resection in Asymptomatic, Unresectable Metastatic Colon and Rectal Cancer
Anal Fissure: Recurrence After Lateral Internal Sphincterotomy
Which Patients with Fecal Incontinence Require Physiologic Workup?
Stay updated, free articles. Join our Telegram channel
Full access? Get Clinical Tree
