Editor’s note: This is the fourth and final story in a series on hospital safety. If you have other questions or story ideas about patient safety, please email alexis.allison@fortworthreport.org.
When Patricia McGaffigan needed orthopedic surgery last year, she assessed the Leapfrog safety grades of hospitals near her in Boston. She also consulted with the hospitals’ orthopedic surgeons, asking about their experiences with that kind of operation.
In the end, she chose to go to a surgeon at a B-rated hospital over a surgeon at an A-rated hospital. The former surgeon had performed the procedure she needed many times over, while the latter had only performed a handful.
McGaffigan is well-informed when it comes to assessing patient safety. She’s a nurse and the vice president of safety programs at the Institute for Healthcare Improvement. She also leads the board overseeing an international patient safety certification exam.
She knows the safety grades, which The Leapfrog Group calculates twice a year for hospitals across the country, don’t tell the full story of a hospital’s commitment to safety — no rating system does.
“Being able to simply say, ‘Well, an organization got this grade or this rating by a rating agency’ can be helpful,” McGaffigan said. “But it’s not an absolute marker … of the care that someone will have or the experience that they’ll have.”
Fort Worth hospitals that didn’t do well on this spring’s scorecard further critique the nonprofit’s methodology — noting that it’s limited or lacking, and more representative rating systems exist.
The spring safety grades
While JPS Health Network and Medical City Fort Worth earned a B and an A this spring, respectively, the three Texas Health Resources hospitals in Fort Worth and Baylor Scott & White All Saints Medical Center – Fort Worth didn’t crack a B.
The middling grades aren’t new. In the most recent reporting cycles, all four hospitals garnered consistent C’s, along with about one-third of hospitals ranked across the country. This spring, though, Baylor Scott & White All Saints Medical Center – Fort Worth’s grade dropped to a D. Only about 7% of the hospitals in the U.S. received the second-lowest score.
Texas Health Resources declined to be interviewed for this story, but shared a written statement with the Report. The statement is published in full below.
When Michael Sanborn took the helm as president of Baylor Scott & White’s Fort Worth hospital in 2016, he remembers hearing concerns about the validity of the Leapfrog scores. The Leapfrog Group calculates the grade using multiple data sources, and its methodology has shifted since its first grade report in 2012.
These days, the group relies heavily on publicly reported data that hospitals submit to the U.S. Centers for Medicare and Medicaid Services. Hospitals can also choose to submit supplemental data through Leapfrog’s yearly survey.
The inclusion of the latter source feels worrying, Sanborn said, because that data is self-reported. A 2017 study published in Medical Care concluded that voluntarily reported Leapfrog data skews “toward positive self-report” in comparison with mandated Medicare reporting.
Hospitals that don’t respond to the survey still receive a grade, “but the grade that you get is influenced by participation by the survey,” Sanborn said.
Missy Danforth, the vice president of health care ratings at The Leapfrog Group, told the Report that hospitals can receive A’s-F’s — regardless of whether they submit the survey.
For example, Baylor Scott & White Medical Center Centennial in Frisco did not submit 2021 survey data to Leapfrog, according to The Leapfrog Group website; however, the hospital earned an “A” grade this spring. Baylor Scott & White All Saints Medical Center – Fort Worth has previously earned B’s without submitting supplemental data.
The Leapfrog Group uses charts that look like gas gauges to help patients understand the data. Sanborn pointed out inconsistencies between what the gas gauges show and what he’s seeing in the hospital.
For example, the Baylor Scott & White hospital performs “below average” when it comes to doctors ordering medications through a computer, rather than by hand, according to The Leapfrog Group. But that rating doesn’t align with hospital practice, Sanborn said.
He said it’s almost impossible for physicians to order prescriptions without a computer, except for in the case of extreme emergencies.
That metric comes from voluntarily submitted supplemental data. But because the hospital doesn’t submit the data, The Leapfrog Group replaces it by assigning the hospital a score for that metric based on hospitals like it, according to the group’s scoring methodology. The Group only does this for two metrics, however.
Other gas gauges depict the prevalence of infections patients contract in the hospital, preventable problems after surgery, or other injuries like bed sores, falls and blood clots. These come directly from the U.S. Centers for Medicare and Medicaid Services data.
Sanborn told the Report that he and his colleagues are assessing those gas gauges, but that Leapfrog’s “red areas” don’t align with the hospital’s performance. He’s not sure why that is yet.
The discrepancy might exist because data from the U.S. Centers for Medicare and Medicaid Services lags months or years behind reality. For the spring Leapfrog grade, for example, the majority of data from the federal agency came from a reporting period that began in 2018.
The strain that acutely ill COVID-19 patients placed on the hospital in the first years of the pandemic might also be a factor, he said. The Baylor Scott & White hospital treated fewer COVID-19 patients in 2020 and 2021 than JPS Health Network or Texas Health Harris Methodist Hospital Fort Worth, according to the U.S. Department of Health and Human Services.
However, as a health system, Baylor Scott & White is the largest ECMO transfer program in Texas. ECMO, or extracorporeal membrane oxygenation, is a treatment that bypasses the body’s heart and lungs when the body’s own organs start to fail. The Baylor Scott & White hospital received COVID-19 patients across the country for ECMO when other medical options failed.
“And so we got the sickest of the sick, transferred here for ECMO support,” Sanborn said.
For the first time since the pandemic began, the spring safety grades include data from the first two years of the pandemic. The Baylor Scott & White hospital mostly received “above average” ratings for those metrics.
However, when it came to patient experience during the pandemic, a metric that’s measured by surveys given to patients after overnight stays, the hospital scored poorly on provider communication and staff responsiveness. The low grades aren’t unique to the Baylor Scott & White hospital. Hospitals across the country received lower patient experience scores in the first year of the pandemic.
“We are still investigating,” Sanborn said. “And regardless, are always focused on continuous improvement.”
Read more about patient safety:
Should medical errors be criminalized? This Fort Worth medical school offers another way.
For the first time, hospital safety grades showcase impact of pandemic on patient experience
Dashboards, huddles and gloves: How Medical City Fort Worth crafts a culture of safety
John Peter Smith Hospital initially scored a D in patient safety. Now, the county hospital is making B’s and an A.
Rating the raters
Although his hospital doesn’t focus on The Leapfrog Group’s grades, Sanborn said he’s thankful for them. “There are absolutely merits to Leapfrog,” he said, “and I actually applaud them for elevating the importance of patient safety.”
The Baylor Scott & White hospital measures its own success with other scoring systems.
U.S. News & World Report, for example, evaluates how hospitals across the country perform when it comes to treatment of patients with life-threatening or rare conditions, as well as more common procedures like hip replacements.
According to those rankings, the Baylor Scott & White hospital is “high-performing” when it comes to six procedures, including colon cancer surgery and treatment for kidney failure.
Sanborn also pointed to a 2019 article published in NEJM Catalyst, back then an idea-sharing platform of the New England Journal of Medicine.
The article rated four hospital rating systems, including U.S. News & World Report and Leapfrog. None scored an A, but U.S. News & World Report outranked the others with a B, and was deemed least likely to “misclassify hospital performance.” The Leapfrog Group scored a C-; the article criticized it partly for its inclusion of self-reported data.
But The Leapfrog Group’s safety grades have their own merits, according to Marianne Fazen, executive director of the Dallas-Fort Worth Business Group on Health. The Business Group is a coalition of more than 60 employers — including the city of Fort Worth — who’ve committed to improving health care in north Texas.
First, the grades are what Fazen calls patient-oriented. “The Leapfrog scores are very simple, easy to understand by ordinary people,” she said.
Leapfrog is the only ranking system that focuses solely on patient safety, according to its website. It’s also the only hospital safety rating whose methodology has been peer-reviewed and published in the Journal of Patient Safety, said Jandis Price, the Business Group’s project manager.
The Business Group’s mission is to help employers and employees make informed health care decisions and incentivize health care providers to improve their performance.
The group has also served as a regional partner for The Leapfrog Group for more than 20 years, Fazen said. The partnership means the Business Group educates employers, employees and health plans about the Leapfrog grades and encourages area hospitals to pay attention to them.
The Business Group encourages employers who pay for their employees’ health insurance to choose hospitals with higher safety grades. That’s because patient safety issues like hospital-acquired infections and medical errors can lead to extended hospital stays and increased health care costs, not to mention death, Price said.
“So obviously, it helps their bottom line,” Price said. “They want to keep their employees healthy.”
How should patients use the ratings?
- In an emergency situation, seek care regardless.
- Consider the ratings, then dig deeper. On Leapfrog’s website, for example, look at the gas gauges specific to the procedure you may need.
- Consult with a physician in the hospital before agreeing to a procedure there, and come prepared — and with a companion. Ask three questions: “What’s my main problem? What do I need to do? Why is it important for me to do that?” Don’t be afraid to ask “How many of these specific procedures have you done in the last year?” Finally, ask if there’s a patient who’s had a similar procedure who may be able to share their experience with you.
– From Patricia McGaffigan, Institute for Healthcare Improvement
Regardless of the rating system a hospital follows, its leaders need to transparently communicate to the public how they’re addressing patient safety, said McGaffigan, with the Institute for Healthcare Improvement.
The Institute doesn’t recommend a specific rating system, she said, but she does use The Leapfrog Group as a data point in her own health care decisions. She noted that no system is perfect, but she’s heartened by their ongoing evolution — one that mimics how hospitals should think about patient safety.
“Safety is a dynamic, ever-changing property of a system,” McGaffigan said. “Even in the highest rated of organizations, things can go wrong. This is daily work.”
Alexis Allison is the health reporter at the Fort Worth Report. Her position is supported by a grant from Texas Health Resources. Contact her by email or via Twitter. At the Fort Worth Report, news decisions are made independently of our board members and financial supporters. Read more about our editorial independence policy here.
Noncommercial entities may republish our articles for free by following our guidelines. For commercial licensing, please email hello@fortworthreport.org.
