Monday, April 29, 2013

Does measurement improve performance?

Like it or not, measuring physician performance is now a key part of the conventional wisdom on improving our health care system. Borrowing from management guru Peter Drucker’s mantra “You can’t manage what you can’t measure” health care policy makers have embraced performance measurement as being central to managing our heretofore unmanageable health care system.  But there is a small but seemingly growing group of Don Quixote-like dissenters who are tilting at the performance measurement windmill, arguing that these measures will not achieve the ends of improving quality and saving money and may instead do considerable harm.

Dr.  Bob Centor, author of DB’s Medical Rant blog, is one of them.   (Disclosure: Dr. Centor is chair-elect of ACP’s Board of Regents, although the views he expresses in his blog are his own, not ACP policy).  One of his posts, titled “What has performance measurement wraught?” calls them madness:

“Most readers know that I am obsessed with performance measurement and why it not only rarely works but often causes negative unintended consequences.  As I have pondered this question recently, computers cannot replace physicians as diagnosticians.  And the same misunderstanding of medicine that would advocate such a position drives the performance measure movement.

Physician decision making requires a complex weighing of disease severity, number of diseases, social situation, cost of medications, the patient's desires and willingness to address issues and more that you can imagine.  To think that we can apply simple rules to such decision making represents an unjustifiable conceit that patient care is simple and can therefore be broken down into RULES.

The unintended consequences of this movement are many.  We now have nonsensical report cards and, here the author gasps, public reporting.  If we could define excellence, then public reporting would make sense.  But we cannot define excellence through rules that cover only selected diseases and only one aspect of doctoring.

How do we stop this madness???"

In an earlier post, he cites  a commentary in the Journal of the American Medical Association (JAMA) which suggests that poorly-designed performance measures can cause harm to patient care.  “Too often we have measures based on a religious belief (e.g. lowering HgbA1c is always the proper goal) and not based upon good prospective data,” he writes.   “Whenever we have to struggle to meet a performance goal, we run the risk of unexpected consequences.  This irresponsible process likely harmed patients.  Let me repeat that sentence. This irresponsible process likely harmed patients.  The reasons now are clear.  Some, including the authors of this commentary complained bitterly back in 2006.  We allow organizations to establish performance measures without expecting the same rigorous testing that any other intervention must have prior to approval.  We would not approve a new drug without careful testing for both efficacy and safety.  Should we not hold performance measurement to the same standard?”

But is it possible to improve clinical performance without measuring it? The Institute for Healthcare Improvement, formerly headed by ex-CMS administrator Don Berwick, MD, says that “Measurement is a critical part of testing and implementing changes; measures tell a team whether the changes they are making actually lead to improvement.”    In 2008, Dr. Berwick co-authored an article published in Health Affairs that presented the Institute’s now widely accepted Triple Aim of improving individual patient outcomes, improving population outcomes, and lower per capita costs.  He writes that “in general, opacity of performance is not a major obstacle to the Triple Aim. Many tools are in hand to construct part of a balanced portfolio of measures to track the experience of a population on all three components. At the Institute for Healthcare Improvement (IHI), for example, we have developed and are using a balanced set of systemwide measures closely related to the Triple Aim.  A more complete set of system metrics would include ways to track the experience of care in ambulatory settings, including patient engagement, continuity, and clinical preventive practices.”

Measurement for the purpose of helping groups of physicians assess how well they are doing in achieving the triple aim may be challenging enough, paying based on performance measures raises a whole host of other issues.  All payment systems create a mix of potentially good and potentially bad results.  Fee-for-service achieves the potentially good outcomes of creating incentives for physicians to actually see their patients and not undertreat them, because FFS pays them on how many patients they see and how many procedures they do, but it can also have the undesirable outcomes of “rushed” assembly line visits and over-testing and over-treatment. Capitation achieves the potentially good outcomes of encouraging physicians to be more efficient and not over-treat their patients, since they are paid the same amount per patient no matter how many procedures or visits provided, but it can also have the undesirable outcome of incentivizing physicians to not see patients enough, not treat sicker patients, or undertreat them.  Payment systems linked to performance measures can have the desirable outcomes of creating incentives for physicians to organize their care to achieve better outcomes for their patients, better care of the patient population they see, and maybe, lower costs (the Triple Aim), but also the undesirable outcomes of “treating to the measure” (paying attention only to things being measured, and less to things not being measured), and creating disincentives for physicians to take care of sicker patients and those with lower socioeconomic status because such patients may adversely affect their performance “score.”

Performance measures though could help level out the potentially undesirable incentives existing in FFS or capitation:  FFS tied to performance measures could help counter the incentives for over-treatment because physicians who over-treat with no improvement in outcomes wouldn’t score as well on measures of individual, population or per capita cost outcomes.  Capitation tied to performance measures—if accompanied with appropriate risk adjustment-- could help counter incentives for physicians to under-treat patients, since under-treatment would result in poorer “scores” on individual and population-based health outcomes and patient experience with the care provided.

My sense is that the performance measurement genie is out of the bottle and isn’t going away.  We live in an era where just about everything and everybody is being measured and held accountable for getting better results as efficiently as possible.  Health care is so damn expensive that the public (through government) and insurance company shareholders will want to know if physicians are achieving the best possible results and the lowest possible cost—how can they know what results they are getting without measuring it?

But as measurement becomes increasingly imbedded in our health care system, we should pay attention to potential unintended consequences. We should insist on meaningful measures that are based on the best available science through a transparent process, not measurement for the sake of measurement.  We should test measures whenever possible before they are widely adopted, just as we do for new drugs, and withdraw measures that turn out to be harmful, just as the FDA withdraws newly approved drugs if they are found to have unforeseen harmful side effects.  We need to be very careful as we design payment models that incorporate performance measure so that what is best for the patient, not what is best for the measure, always comes first.  All of these, and more, safeguards are called for in ACP policy on performance measurement

And rather than starting with measurement as the be-all and end-all goal, we should begin by defining how best to organize care to achieve the best possible results for patients, through models like Patient-Centered Medical Homes, then determine a payment model that best supports those models, and then build and incorporate measures that actually help the physicians in these systems monitor and achieve the best possible outcomes for their patients—not the other way around.

If we really believe, as ACP does, that a well-trained internist, in a system of care designed to achieve the best outcomes for patients, will be shown to be the best bargain in American medicine, then performance measures can be our friends—but only if they are the right measures, measuring the right things, for the right reasons, and with the right oversight.    And we should always keep in mind the cautionary note from sociologist William Bruce Cameron, sometimes misattributed  to Albert Einstein, “That not everything that can be counted counts, and not everything that counts can be counted.”

Today’s question: do you think performance measures will improve or harm health care outcomes?


james gaulte said...

As long as you are adding cautionary notes to your discussion let me suggest adding Goodhart's law. This states that when a performance measure become a target it looses its value as a target.Think about teachers "teaching to the text" and the untoward consequences of the four hour pneumonia rule.

Jay Larson MD said...

We can't change behavior unless we know where we are starting at. With that said, performance measures are just one more burden to add to the crippled primary care system.

james gaulte said...

Oops, I misquoted Goodhart
obviously it should have read. When a performance measure becomes a target it loses its value as a performance measure.

Harrison said...

If we don't know where we are going why should we be in a hurry to move and why would we want to measure progress?

I am stressed all the time by measurements and things that I perceive as over sight.

Every time I see comments from individual patients about my office and my practice, I am pleased.
But when Press Ganey sends out its surveys, I am reminded that my raw scores of 89 to 90 are in the lower half most of the time nationally. While a raw score of 91 is way up in the upper half.
What is being measured and how am I supposed to get to it?
I don't really know.
There are all kinds of suggestions and some of them may help me, but...
But in the meantime it costs me money.
Potentially a couple of thousand each year.

And in the hospital I know I'm measured by how frequently I guess right on whether a pt will be discharged in 24 hours. If I say they should be on observation status, but then that is converted, the hospital loses money. And I am told about it.
On the other hand, there is the potential for fines if I make too many pt inpt's when they leave the next day.

As far as I can tell nobody is measuring the Neurologists for over ordering MRI's and MRA's. Nobody is measuring the ER for overusing CT angio's to rule out PE every time someone comes in with chest pain and shortness of breath.
Nobody is measuring orthopedic surgeons for over using blood products after their bloodless hip procedures.

But readmissions are now being measured.

I don't know the goal.

If we are trying to treat patients individually then should we really concern ourselves with resources?
If we are advocating for our patients then should our goal not be to do what is right for them, and then if resources are scarce, well then we may run into that, but that doesn't keep us from suggesting it and advocating for it.

The goal of course being that we should communicate well with our patients and do what is right for them.
And if we do that, then we should roughly fall in the right places as far as the measurements are concerned.
That is if we are following evidence based guidelines.

A patient centered medical home, or advanced primary care model should help us. We would work as a team with our office staff and with our local specialists to keep patients in touch with the local system. We would keep them up to date on screenings and on vaccines. We would seek to keep them on evidence based treatment plans for chronic illnesses. And we would teach them what to expect and who to call when they have questions or problems.

How many measurements do we need to know that that is happening?

Right now with so many people in our country going to ER's for primary care it is a good bet that it is not happening.

It was that problem that the ACA, now called Obamacare, was aimed at correcting.

We can measure access.
And we should.


PCP said...


When you say:

"We should insist on meaningful measures that are based on the best available science through a transparent process, not measurement for the sake of measurement. We should test measures whenever possible before they are widely adopted"

Does this also apply to the onerous and time sapping MOC requirements being foisted upon us by the ABMS/ABIM also? Or does evidence not matter there.
Because the deafening silence from ACP as these requirements become more and more intrusive, expensive and time consuming is quite revealing.
I just completed the MOC process and far from feeling like it was a useful and refreshing process, it felt like a laborious process designed to appease some entity who is above the need to prove or explain itself due to its power convincing the payers and others about their seal of approval and now using that monopolistic position to extract its pound of flesh.
No sooner had i completed that process did I get an email letting me know that henceforth in 2yr cycles it will be reported whether I am "meeting requirements for MOC". Where is the evidence any of this works or makes for a better physician? Or is asking that too much. Are we now just to shut up and do as we are told?