A study published in the current issue of JAMA concludes that self-reported Leapfrog hospital safe practices scores just don't correlate with risk-adjusted inpatient hospital mortality rates. Better Leapfrog scores don't mean lower risk-adjusted mortality rates. If borne out, this undercuts the whole pay-for-performance premise of the Leapfrog Group and others. The authors of this study noted that theirs is the first peer-reviewed study to examine the correlation so, of course, "further study is needed."
The number of NQF measures tracked for this program was cut in half relatively recently, as folks have tried to home in on the key predictive indicators (see the HealthBlawger's interview with Leapfrog Group CEO Leah Binder from last fall), apparently to no avail.
(There's been a lot of noise coming from JAMA lately, but one would hope that should not impugn this study or its authors.)
The question is: If Leapfrog-style P4P is a bust, what's the next big thing going to be on the cost containment / quality improvement / value-based purchasing front?
Is it EBM-driven, measuring inputs rather than outcomes? Seems to me that would make sense, but only if we have some killer data linking the inputs to the desired outcomes.
Update 4/1/09: See Leah Binder's — and the study authors' — comments on this post and see also another study looking at a broader slice of the Leapfrog P4P experience in a more positive light, footnoted in the JAMA study.
David Harlow
The Harlow Group LLC
Health Care Law and Consulting
Leah Binder says
April 1, 2009 at 10:48 amDavid,
I encourage you to more closely examine the JAMA study on Leapfrog’s use of certain of the NQF-endorsed safe practices before you draw such broad conclusions that I am certain the study’s authors did not intend. The study examines only one element among many elements of the Leapfrog Survey, and as the study’s authors point out, the other elements of the survey have been well established in the literature for reducing mortality and improving outcomes. In Leapfrog’s own P4P program, which I discussed with you in an earlier interview, the subset of safe practices this study addresses are weighted at 15% of the total score, with the rest of the score based on other elements of the survey that are linked to mortality rates. Whether or not this study’s conclusions are correct–and the jury is still out, since this study examined only 155 hospitals in 2005, when the safe practices were very new–it still makes sense to reward hospitals for good performance on those practices. The safe practices in the study include guidelines on handwashing, nurse workforce adequacy, patient consent procedures, etc. These practices matter deeply to patients, their families, and purchasers. Our statement on this study appears on our website: http://www.leapfroggroup.org.
Leah Binder, CEO, The Leapfrog Group
David Harlow says
April 1, 2009 at 1:36 pmLeah, Thanks for your comments. Point taken. I’ve linked back to the study form last year that looked at a broader slice of the Leapfrog P4P experience and portrayed it in a more positive light. The question remains open for me, though: Are there other measures that we should be looking at that would allow us to home in on better-performing institutions more easily, or is the narrowed set of NQF measures now being tracked by Leapfrog a good proxy for overall quality of care? Also, is risk-adjusted mortality rate the besst measure we have to measure quality of care in a hospital?
adams.dudley@ucsf.edu says
April 1, 2009 at 3:44 pmDavid,
Thanks for making people aware of this topic. We’d like to offer what, from our perspective, is a bit of a clarification. To conclude from our study that it “undercuts the whole pay-for-performance premise of the Leapfrog Group” is perhaps throwing out the baby with the bathwater. Our study results only raise questions about one of the four “leaps” on the Leapfrog Survey, the Safe Practices Leap. In our JAMA article, we noted that another recent study found that performance on the first 3 leaps did correlate with inpatient mortality (see Jha, AK, et al. Joint Commission Journal on Quality and Safety, June 2008; 34(6):318-25).
We ourselves do believe that the one leap we studied, the Safe Practices leap, is important, and Leapfrog is certainly on the right track in pioneering efforts to assess hospital performance in this domain in addition to the leaps it already addresses that are linked to inpatient mortality and outcomes. That being said, it seems that the leap we studied may not be designed in a way to optimally assess adherence.
When coaching hospital management on how to successfully engender a culture of safety and quality improvement, experts often stress the importance of using constructive criticism rather than blame. The health care quality community should keep the same in mind when discussing the implications of our results for Leapfrog. Leapfrog, to their credit, took a certain risk in allowing external, independent evaluation of this component of their survey. We need to all think about constructively moving forward in this process of figuring out how to help hospitals measure and improve their safety and quality, rather than simply blaming Leapfrog for one element of their survey not correlating with mortality.
We hope that helps. It’s an important topic for people to discuss.
Leslie P. Kernisan, MD
R. Adams Dudley, MD, MBA
Philip R. Lee Institute for Health Policy Studies
University of California, San Francisco
David Harlow says
April 1, 2009 at 4:49 pmThanks very much for the clarification — and shame on me for blogging late at night.
Leah Binder says
April 2, 2009 at 9:33 amDavid,
Thanks for a productive discussion of Pay for Performance and the Leapfrog Hospital Survey. I agree we do need to ask tough questions about which measures are most valuable and why. Leapfrog continually updates our survey based on these public conversations and new research, to assure we include the evidence-based measures that have the greatest impact on quality. That’s why Leapfrog’s transparency is a cornerstone of our mission. Unfortunately, it makes us unique in the healthcare world. I hope you will work to encourage greater transparency from others in the healthcare world so we aren’t always alone in this. Thanks, Leah Binder