The use of large databases to enhance quality: The National Healthcare Quality Report example

Large databases result from data collected for administrative purposes, from surveys performed for developing national agendas and other reasons, or from disease or procedure-specific registries. The authors are familiar with the use of these large databases for epidemiologic studies, including studies of the burden of gastrointestinal disease. The National Healthcare Quality Reports (NHQR) are an example of use of these sources for discovery of quality and of the trends in quality care.

The NHQRs have resulted from a 1999 congressional mandate to the Agency for Healthcare Research and Quality (AHRQ) and have been developed in coordination with the Department of Health and Human Services. The most recent report available containing measures for colorectal cancer screening is the 2008 report, which includes data from 2005 with comparison data from as early as 1999, depending upon the measure. The 2008 NHQR used 33 databases ( Box 1 ) to report on process and outcome measures aggregated at the national and state level. As defined in this report, process measures “track the receipt of medical services”; whereas, outcome measures “in part reflect the results of medical care.” Of 40 core report measures and 9 composite measures, 3 core measures for colorectal cancer are reported every other year along with a fourth noncore measure endorsed by the National Quality Forum concerning surgical resection of colorectal cancer. The 2008 report measures and findings include the following information:

• •
  

  Colorectal cancer screening: adults aged 50 years and older who ever received colorectal cancer screening (colonoscopy, sigmoidoscopy, proctoscopy, or fecal occult blood test)

  
  
  
  
  • 
    

    In 2005, 55.5% of adults aged 50 years and older (49.2% in individuals aged 50–64 years, 63.1% for adults aged 65 years and older) had received screening for colorectal cancer. This result was an increase from 49.8% in 2000.

  
  
  
  
• •
  

  Advanced stage colorectal cancer: colorectal cancer diagnosed at advance stage (tumors diagnosed at regional or distant site) per 100,000 population aged 50 years and older

  
  
  
  
  • 
    

    In 2005, 80.8 advanced stage colorectal tumors were diagnosed per 100,000 population. There has been a steady decrease in this number since first reported in 2000 at 95.2 per 100,000 population.

  
  
  
  
• •
  

  Colorectal cancer mortality: colorectal cancer deaths per 100,000 population per year

  
  
  
  
  • 
    

    The United States Healthy People 2010 goal for colorectal cancer deaths is 13.7 per 100,000. A steady downward trend has occurred since 1999 with 17.5 per 100,000 in 2005.

  
  

The application of large databases for quality enhancement requires significant resources and analytical expertise. The quality of these data sources must be judged in the same ways as all data found in clinical and research settings, and accurate interpretation of the data demands availability of analysts with appropriate expertise with both the sources’ database structure and content. The possible effects of such reporting, however, can be seen in system changes after feedback of measurement results to providers; national, state, and local administrative units and health care organizations; and from the informed implementation of clinical decision support systems that support just-in-time quality care.

Challenges in the use of administrative data

The credibility of administrative data, which is usually insurance claims data, for quality reporting and research has long been the subject of debate. The term powerful is often used to describe the role of administrative data in studying population-based patterns of health, disease, and medical care. This powerful data has potential advantages for research and quality, including the ability to measure large samples of geographically dispersed patients, the ability to assemble longitudinal records of both inpatient and outpatient care and across providers, and the fact that the data is already collected and inexpensive for investigators to obtain relative to the collection of clinical data through chart abstraction.

Administrative data, however, is not collected for the purpose of measuring quality and its validity for this purpose has been questioned. Of particular concern with US administrative databases is the use of 2 coding systems: the International Classification of Diseases, 9th Revision, Clinical Modification (ICD-9-CM), which is used for coding diagnoses by both hospitals and physicians and for coding procedures by hospitals; and the American Medical Association’s Current Procedural Terminology (CPT), which is used by physicians for coding procedures. For insightful review of the use of administrative data for assessing quality across a decade, see the review and editorial by Iezzoni. This author describes how administrative files often explicitly aim to minimize data collection by limiting the number of slots available for codes that may affect completeness of comorbidity documentation, and that coding inaccuracy may occur in an effort to maximize payment, as from the Medicare Prospective Payment System using diagnosis-related groups, or to minimize the perception of poor quality through underreporting of complications. The coding systems for procedures (ICD-9-CM for hospitals and CPT for physicians) do not readily link, “hindering comparisons between hospital-generated and physician-generated data.” Until 2007, diagnoses present on admission could not be differentiated from complications occurring during hospitalization in Medicare billing data.

Administrative data has been shown in some circumstances to lack either specificity or sensitivity for identifying conditions important in quality measurement. Jollis and colleagues demonstrated that claims data failed to identify more than 50% of patients with any of several important conditions when compared with a clinical information system. Other studies reported by Jollis found that clinical criteria for acute myocardial infarction were found in only 43% to 87% of records having this diagnosis coded at discharge. Keating and colleagues used 3 accepted measures of quality care for diabetes and compared the use of administrative data alone with administrative data plus medical record data from 3 health plans. Underdetection of quality indicators was noted for all 3 health systems using administrative data alone, and, importantly, the ranking of the health systems changed for some indicators when data from both sources was used. In addition, the degree of underdetection varied by patient age and race, pointing out the complexity of accurate quality reporting.

In quality measurement, inclusion of cases in the denominator of a measure that do not accurately represent the condition being tested can falsely conclude underperformance on the measure. Solberg and colleagues explored algorithms for improving the positive predictive value of case identification for 3 conditions using administrative records, after discovering that using a single code to identify cases of diabetes might include up to 80% of cases without diabetes. The algorithms developed for accurate case identification required for all conditions that more than a single code be used. For example, accurately identifying cases with diabetes required either 2 or more outpatient or 1 inpatient ICD-9 code for diabetes in a year or a filled prescription for a diabetes-specific medication (excluding a medication that may be used for diabetes and other conditions), and assumed health plan enrollment for at least 11 months of the year being studied.

Incomplete physician documentation can lead to omissions in administrative codes. Davenport and colleagues compared data on surgeries performed at a single institution as recorded in 2 volunteer systems; the University HealthSystem Consortium Clinical Database, which relies on administrative data, primarily ICD-9 codes entered by hospital billing coders; and the National Surgical Quality Improvement Program, which requires that nurses extract the data from the medical record and code it to risk and outcome variables specifically designed for risk-adjusting surgical outcomes. To justify billing, the billing coders are required to code only conditions and procedures that are clearly documented by physicians, sometimes with specific wording, and therefore cannot use conditions and procedures that are identifiable in nursing and other notes.

Ultimately, the lesson learned is that for administrative data to be used for enhancing quality, its use is subject to the same caveats as for successful conduct of research using any large databases; the investigator must have adequate knowledge of the database structure and its contents, in particular, knowledge of the completeness and accuracy of the data contained and must be cautious in the conclusions drawn. Comparison with clinical databases, including chart validation of a subset of cases of interest, should be considered before determining the accuracy of the codes used, although this may be untenable for many administrative databases, especially those in the public domain without the ability to link to identified health records.

A recent study on the association of colonoscopy and death from colorectal cancer (CRC) highlights some of the advantages and limitations of using administrative databases in research. The design was a population-based case-control study in Ontario, using administrative claims to ascertain exposure to colonoscopy and diagnosis of CRC. Exposures in more than 10,000 case subjects (with CRC) were compared with controls matched for age, sex, geographic region, and socioeconomic status. The analysis found that subjects with prior colonoscopy had a 67% reduction in risk of death from left-sided CRC compared with controls and no reduction in risk of death from right-sided cancer. The study raised serious concerns about the potential benefits of screening with colonoscopy.

The strengths of this study include the large number of cases with matched controls. Cases and controls came from typical practice settings, suggesting that data could be generalizable. There is some biologic plausibility to the outcome; cancers of the right colon may be biologically different from distal cancers.

The limitations are also significant and are related to the administrative nature of the data. First, the indications for colonoscopy are unknown. It is unlikely that most examinations were screening examinations during the time period of the study (1992–2003), when little screening colonoscopy was performed in Canada. If rectal bleeding was a common indication, the procedures may have been more likely to detect left-sided lesions because these may present with bleeding, and exerted a more protective effect in the left colon because of selection bias. A second limitation is that we cannot determine the quality or completeness of the colonoscopy examination from the administrative database. For example, it is quite possible that some examinations coded as colonoscopy were incomplete, and did not fully visualize the proximal colon. It is also possible that the bowel prep was poor in the proximal colon. In both situations, the outcome could be a poor quality examination of the proximal colon, which could result in failure of colonoscopy to exert a protective effect. Finally, it is not known if there were polyps seen but not removed at colonoscopy. Based on the National Polyp Study, polypectomy was associated with a lower than expected rate of CRC during surveillance, suggesting that one of the primary benefits of colonoscopy is the detection and removal of cancer precursor lesions. Using this administrative database, the authors cannot determine if all detected polyps were removed. The limitations of this study are largely caused by the lack of data precision in the administrative database, and potentially compromise the outcome. The authors cannot determine if the result is caused by poor quality colonoscopy, or true biologic differences in the behavior of proximal and distal CRC.

Challenges in the use of administrative data

The credibility of administrative data, which is usually insurance claims data, for quality reporting and research has long been the subject of debate. The term powerful is often used to describe the role of administrative data in studying population-based patterns of health, disease, and medical care. This powerful data has potential advantages for research and quality, including the ability to measure large samples of geographically dispersed patients, the ability to assemble longitudinal records of both inpatient and outpatient care and across providers, and the fact that the data is already collected and inexpensive for investigators to obtain relative to the collection of clinical data through chart abstraction.

Administrative data, however, is not collected for the purpose of measuring quality and its validity for this purpose has been questioned. Of particular concern with US administrative databases is the use of 2 coding systems: the International Classification of Diseases, 9th Revision, Clinical Modification (ICD-9-CM), which is used for coding diagnoses by both hospitals and physicians and for coding procedures by hospitals; and the American Medical Association’s Current Procedural Terminology (CPT), which is used by physicians for coding procedures. For insightful review of the use of administrative data for assessing quality across a decade, see the review and editorial by Iezzoni. This author describes how administrative files often explicitly aim to minimize data collection by limiting the number of slots available for codes that may affect completeness of comorbidity documentation, and that coding inaccuracy may occur in an effort to maximize payment, as from the Medicare Prospective Payment System using diagnosis-related groups, or to minimize the perception of poor quality through underreporting of complications. The coding systems for procedures (ICD-9-CM for hospitals and CPT for physicians) do not readily link, “hindering comparisons between hospital-generated and physician-generated data.” Until 2007, diagnoses present on admission could not be differentiated from complications occurring during hospitalization in Medicare billing data.

Administrative data has been shown in some circumstances to lack either specificity or sensitivity for identifying conditions important in quality measurement. Jollis and colleagues demonstrated that claims data failed to identify more than 50% of patients with any of several important conditions when compared with a clinical information system. Other studies reported by Jollis found that clinical criteria for acute myocardial infarction were found in only 43% to 87% of records having this diagnosis coded at discharge. Keating and colleagues used 3 accepted measures of quality care for diabetes and compared the use of administrative data alone with administrative data plus medical record data from 3 health plans. Underdetection of quality indicators was noted for all 3 health systems using administrative data alone, and, importantly, the ranking of the health systems changed for some indicators when data from both sources was used. In addition, the degree of underdetection varied by patient age and race, pointing out the complexity of accurate quality reporting.

In quality measurement, inclusion of cases in the denominator of a measure that do not accurately represent the condition being tested can falsely conclude underperformance on the measure. Solberg and colleagues explored algorithms for improving the positive predictive value of case identification for 3 conditions using administrative records, after discovering that using a single code to identify cases of diabetes might include up to 80% of cases without diabetes. The algorithms developed for accurate case identification required for all conditions that more than a single code be used. For example, accurately identifying cases with diabetes required either 2 or more outpatient or 1 inpatient ICD-9 code for diabetes in a year or a filled prescription for a diabetes-specific medication (excluding a medication that may be used for diabetes and other conditions), and assumed health plan enrollment for at least 11 months of the year being studied.

Incomplete physician documentation can lead to omissions in administrative codes. Davenport and colleagues compared data on surgeries performed at a single institution as recorded in 2 volunteer systems; the University HealthSystem Consortium Clinical Database, which relies on administrative data, primarily ICD-9 codes entered by hospital billing coders; and the National Surgical Quality Improvement Program, which requires that nurses extract the data from the medical record and code it to risk and outcome variables specifically designed for risk-adjusting surgical outcomes. To justify billing, the billing coders are required to code only conditions and procedures that are clearly documented by physicians, sometimes with specific wording, and therefore cannot use conditions and procedures that are identifiable in nursing and other notes.

Ultimately, the lesson learned is that for administrative data to be used for enhancing quality, its use is subject to the same caveats as for successful conduct of research using any large databases; the investigator must have adequate knowledge of the database structure and its contents, in particular, knowledge of the completeness and accuracy of the data contained and must be cautious in the conclusions drawn. Comparison with clinical databases, including chart validation of a subset of cases of interest, should be considered before determining the accuracy of the codes used, although this may be untenable for many administrative databases, especially those in the public domain without the ability to link to identified health records.

A recent study on the association of colonoscopy and death from colorectal cancer (CRC) highlights some of the advantages and limitations of using administrative databases in research. The design was a population-based case-control study in Ontario, using administrative claims to ascertain exposure to colonoscopy and diagnosis of CRC. Exposures in more than 10,000 case subjects (with CRC) were compared with controls matched for age, sex, geographic region, and socioeconomic status. The analysis found that subjects with prior colonoscopy had a 67% reduction in risk of death from left-sided CRC compared with controls and no reduction in risk of death from right-sided cancer. The study raised serious concerns about the potential benefits of screening with colonoscopy.

The strengths of this study include the large number of cases with matched controls. Cases and controls came from typical practice settings, suggesting that data could be generalizable. There is some biologic plausibility to the outcome; cancers of the right colon may be biologically different from distal cancers.

The limitations are also significant and are related to the administrative nature of the data. First, the indications for colonoscopy are unknown. It is unlikely that most examinations were screening examinations during the time period of the study (1992–2003), when little screening colonoscopy was performed in Canada. If rectal bleeding was a common indication, the procedures may have been more likely to detect left-sided lesions because these may present with bleeding, and exerted a more protective effect in the left colon because of selection bias. A second limitation is that we cannot determine the quality or completeness of the colonoscopy examination from the administrative database. For example, it is quite possible that some examinations coded as colonoscopy were incomplete, and did not fully visualize the proximal colon. It is also possible that the bowel prep was poor in the proximal colon. In both situations, the outcome could be a poor quality examination of the proximal colon, which could result in failure of colonoscopy to exert a protective effect. Finally, it is not known if there were polyps seen but not removed at colonoscopy. Based on the National Polyp Study, polypectomy was associated with a lower than expected rate of CRC during surveillance, suggesting that one of the primary benefits of colonoscopy is the detection and removal of cancer precursor lesions. Using this administrative database, the authors cannot determine if all detected polyps were removed. The limitations of this study are largely caused by the lack of data precision in the administrative database, and potentially compromise the outcome. The authors cannot determine if the result is caused by poor quality colonoscopy, or true biologic differences in the behavior of proximal and distal CRC.