Surgeons are under enormous pressure from multiple healthcare stakeholders to measure and improve their performance. Government regulators are publicly reporting patient outcomes and satisfaction scores.¹ Payers are reducing reimbursements based on quality measurements.² Licensing boards and professional societies are revising member certification to increasingly include performance evaluation.³ Patients are now searching online for information about surgeon outcomes to guide where they seek care.⁴ Surgeons themselves have created quality collaboratives to share best practices and improve their own performance.^5,6 In short, we are in an era of unprecedented focus on evaluating and reporting the work of surgeons.

Despite the widespread interest in measuring and improving surgical quality, little consensus exists on what measures to follow or which strategies to implement. In this chapter, we describe the general principles of performance measurement in surgery, including how to choose among measures. We then outline the benefits and drawbacks of different performance improvement strategies.

Establishing accurate measurement of surgical quality is essential to any attempt at improving performance. The following sections describe the key principles to understanding the underlying methodology and options for performance measurement.

Understanding Variation in Outcomes

Some hospitals and surgeons seem to simply do better than others, and this reality creates an opportunity to learn and improve from the best performers. However, reliably and fairly identifying high and low performers can be challenging. In addition to the quality of care provided, patient outcomes can also be highly influenced by chance and case mix. To understand how best to measure quality, it is important first to explore why outcomes vary across hospitals and surgeons.

SAMPLE SIZE AND THE PROBLEM OF CHANCE (“JUST BAD LUCK”)

Variation in outcomes across surgeons and hospitals may be the result of good or bad luck. The role of chance becomes important in low-volume procedures (eg, pancreatectomy) or when the event rate is low (eg, death after a cholecystectomy). Good or bad luck can result in either a type 1 or type 2 error.

Type 1 errors occur when extreme outcomes—good or bad—are attributed to quality when they actually are simply due to chance. Consider, for example, the “zero-mortality paradox” observed in Medicare claims data.⁷ A hospital with a 0% mortality rate 3 years in a row for pancreatic resection might be considered the highest quality; however, in a subsequent year, it might have a 30% higher mortality rate than other hospitals. The apparent paradox is explained by the fact that most hospitals with a 0% mortality rate simply have a low case volume and good luck, and thus, this rate does not accurately reflect the quality provided at these hospitals. In other words, the difference between a low-volume hospital (eg, 5 pancreatectomies a year) having a 0% mortality rate (no deaths) or 20% mortality rate (1 death) is more likely due to chance rather than quality. Thus, reporting a mortality rate of either 0% or 20% does not accurately represent the quality provided at a hospital.

Type 2 errors occur when real differences in quality are difficult to detect because of limited sample size. Widely recognized in clinical trials as being “underpowered” (ie, sample is not large enough to find differences), limited sample size is commonly overlooked in surgical quality improvement initiatives. For example, a review of quality indicators recommended by the Agency for Healthcare Quality and Research found that only a small minority of hospitals have adequate surgical volume to detect meaningful differences in mortality rates.⁸ Thus, although there may be real differences in quality between hospitals, a type 2 error prevents them from being detected.

THE ROLE OF CASE MIX (“BUT MY PATIENTS ARE SICKER”)

When presented with their own outcomes data, surgeons with worse outcomes often intuitively reply, “But my patients are sicker.” Without question, patient characteristics, including their comorbid conditions, functional status, procedure indications, and so on, play a role in patients’ outcomes and should be accounted for when measuring outcomes. How much patient characteristics matter, however, depends on the comparison being made.

Adjustment for case mix (the type or mix of patients) is most important when there are strong underlying differences in the patients being compared. For example, comparing groups of patients in 2 different surgical intensive care units should be adjusted for case mix. The age, Acute Physiology and Chronic Health Evaluation (APACHE) score, and health profiles of the patients in an intensive care unit may vary widely and contribute significantly to the variation in their outcomes. Similarly, adjusting for case mix is appropriate when comparing the outcomes of a tertiary referral hospital (which treats many complex cases) to those of a smaller community hospital (which may only operate on generally healthier patients requiring less complex procedures).

The importance of case-mix adjustments may be overstated when making procedure-specific comparisons. For example, the unadjusted coronary artery bypass grafting (CABG) mortality rates in the state of New York in 2001 ranged from <1% to 4%. When the outcomes were risk-adjusted for patient factors, the variation remained essentially unchanged.⁹ In other words, little of the variation in mortality rates could be explained by the underlying case mix because patients who undergo CABG have a relatively similar profile. This is not meant to downplay the role of risk adjustment but only to point out that, in many cases, case mix has a much smaller role than previously thought.

What Performance Should Be Measured? The Structure, Process, Outcomes Framework

Described first by Donabedian in 1998, the “Structure, Process, Outcomes” model is the most common framework used for quality improvement in health care.¹⁰ Each category has its own benefits and limitations, which are described below in the context of surgery (Table 4-1).

STRUCTURE

Structure refers to measurable attributes of a surgeon (eg, years of training, specialty service availability) or hospital (eg, number of inpatient beds, procedure volume). The primary benefit of this approach is that the data are easily collectable. Studies that described the association of better pancreas surgery outcomes with high-volume centers used this approach. Although the structure approach does well to predict outcomes across hospitals, it provides little actionable information within a hospital or about individual providers.

PROCESS

Process describes the measurable steps involved in the patient’s care. Performing certain process measures should translate to improved patient outcomes (eg, giving preoperative heparin to reduce risk of a postoperative thromboembolism). Administrators are particularly drawn to these measures because they are readily actionable and measurable. Despite the anticipated benefit, very few “high-yield” process measures have been identified to correlate with improved patient outcomes.

OUTCOMES

Outcomes represent the end result of care. In surgery, the most common outcomes of interest are mortality and postoperative complications. These tend to have the most face validity with surgeons who often care most about the “bottom line.” Unfortunately, comparing outcomes fairly requires high case volume (which many hospitals or individuals do not have) and detailed patient information to appropriately risk adjust. For example, comparing a process measure between 2 hospitals may only require, for example, the percentage of patients who appropriately received heparin before surgery. However, to compare the outcome (eg, rates of thromboembolism), one would need not only data on the outcome but also data on known risk factors (eg, obesity, physical inactivity, history of thromboembolism) to allow for risk adjustment and fairer comparisons. Thus, more data (and, therefore, often more resources) are required to compare outcomes performance.

Choosing the Right Measurement Approach

With limited resources to collect data, it can be challenging to choose where efforts should be focused. Should a surgeon be evaluated based on process measures such preoperative antibiotic administration? Or should we instead focus on the “bottom line” of the outcomes and assess surgical site infection rates? Although there are judgment calls about which should be valued over the other, there are also real statistical limitations that should help inform the choice.

Choosing the right measure will depend on the characteristics of the procedure and our ability to find meaningful differences. Statistically speaking, the more often something occurs, the easier it is to detect. Therefore, one should ask the following 2 questions: (1) How often is the procedure performed? (2) How often does the adverse event occur? Consider the following 4 categories and when structure, process, or outcomes would be an appropriate measurement approach (Fig. 4-1).¹¹