Price Is Observable. Is Quality?

The measurement problem

Trilliant Health compares commercial negotiated rates with hospital mortality and readmission measures and finds little consistent relationship between the two. The quality measures examined do not explain much of the observed price variation. They are not raw performance scores, and they should not be read as one.

CMS does not report raw mortality and readmission rates as direct measures of hospital performance. Its outcome measures are risk-standardized and use hierarchical models intended to account for differences in patient characteristics and to stabilize hospital estimates. CMS also reports uncertainty around those estimates and has historically classified many hospitals as no different from the national rate rather than forcing a precise ranking.

An observed patient outcome reflects underlying patient risk, treatment, care environment, factors that were not measured, and random variation. Risk adjustment reduces some of those differences. The residual is not thereby the provider’s contribution. Observing an outcome, risk-adjusting it, and attributing what remains to provider quality are three different steps.

Credibility and accountability

Actuarial credibility provides a useful way to think about how much accountability an observed result can support. ASOP No. 25 defines credibility as the predictive value attached to a particular set of data. Classical credibility methods formalize a simple principle: experience receives more weight as it becomes sufficiently large and stable, and less weight when the available experience is thin.

The relevance to provider quality is the size of the comparison being asked to carry the decision. A purchaser may want to compare one hospital with another for a particular condition, procedure or service line. As the comparison becomes more clinically specific, the number of relevant observations declines.

That creates a credibility constraint. The observed result may contain useful information without supporting a fine distinction between providers. Risk adjustment and hierarchical modeling can improve the estimate, but they do not create observations that are not there.

The problem also works in the other direction. Expanding the population increases the number of observations but combines patients with different diagnoses, severity, comorbidities and treatment pathways. Narrowing the population improves clinical comparability and reduces statistical stability.

The level at which provider accountability becomes clinically meaningful may also be the level at which the available experience becomes least credible.

More observations do not eliminate the problem

More observations generally improve stability. In healthcare, getting them usually means combining patients who are less alike. A broad hospital population adds cases and adds variation in diagnosis, severity, comorbidity, and treatment. A narrower cohort improves clinical comparability and gives random fluctuation more influence.

Actuaries use similar constraints: the predictive value of experience depends partly on its homogeneity, and the analyst has to weigh homogeneity against the size of the data set. Pooling several years raises N and can mix different periods of hospital performance. Expanding the population raises N and adds heterogeneity. Splitting patients into narrower cohorts improves comparability and reduces precision.

CMS’s measurement choices are a response to that tradeoff. Hierarchical models pull hospital-specific estimates toward the mean where volume is thin. CMS has also expanded some measures to include Medicare Advantage beneficiaries to improve reliability, and shortened performance periods so the result reflects more recent performance. Reliability studies of hospital outcome indicators find that a substantial share of observed variation can be statistical noise rather than a stable difference in performance, and that reliability varies by measure, condition, and hospital volume.

What a weak price-quality correlation establishes

A negotiated rate is a dollar amount. A hospital mortality or readmission rate is an estimate with uncertainty around it. If the quality estimate is noisy, its correlation with price is attenuated even when price and underlying quality are related. A correlation near zero therefore cannot separate two possibilities:

• little relationship between price and true quality; and

• too little signal in the measure to detect a relationship that exists.

Trilliant’s scatterplots support the narrower conclusion. The selected quality measures do not explain the observed price differences. They do not show that lower-priced and higher-priced hospitals provide equivalent clinical quality. That is the point at which an unexplained price becomes a different claim from a steer to a named provider.

Retrospective performance and prospective purchasing

The core review separates retrospective measurement from prospective purchasing. The sample-size constraint shows up on both sides of that line. The provider needs enough relevant historical experience for the quality estimate to carry weight. The purchaser needs enough predictable future volume for the distinction to change where cases go. Geographic concentration, and elective or high-frequency services, make the second condition easier to meet. Rare, emergent, and clinically complex events usually do not. A stable look backward does not supply the forward volume.

Evidentiary Ceiling

Provider outcomes contain information about quality. Risk adjustment, hierarchical models, and longer measurement periods can make that information more usable. None of them makes provider quality observable in the way a negotiated price is observable.

How much accountability a measure can support depends on its credibility, the population it represents, and the decision in front of the purchaser. The same estimate can flag an unusually poor result, feed a quality-improvement review, or sit inside a broader provider assessment, and still be too noisy to identify the hospital that should receive the next patient.

Price transparency makes the price coordinate substantially more precise. The quality coordinate remains an estimate. A calculation of provider value inherits that uncertainty. Prospective purchasing adds a second uncertainty: where the next cases will occur, and how many of them can be moved.