Making predictions during a pandemic

Permanente physicians play key role in projecting hospital staffing, space, and supply needs during COVID-19

By Janet Byron
The Permanente Federation

A year ago, as a novel coronavirus spread quickly through communities across the United States, health care systems such as Kaiser Permanente were faced with a multitude of critical unknowns.

Will our Atlanta hospital have enough ICU space for COVID-19 patients? How many nurses should be scheduled to staff the emergency department in Portland? And how soon will the Seattle medical center need to locate more respirators?

“Predicting the future is never easy,” says Scott Young, MD, senior medical director for quality at The Permanente Federation, “but the physician voice, and being informed by physicians, has been one of the most crucial aspects of forecasting during a pandemic.”

As the reality of the impending pandemic set in and states and local municipalities began implementing shelter-in-place orders to try to contain the coronavirus’s spread, Dr. Young convened Kaiser Permanente’s COVID-19 Forecasting and Surveillance Guidance Group. Permanente physicians partnered with Kaiser Permanente researchers, national infectious disease leaders, and the organization’s predictive analytics group to develop the best data possible for guiding clinical decision-making and operations under the rapidly evolving conditions of the COVID-19 pandemic.

“What we started with was one-size-fits-all simulations across the program,” says group member Vincent Liu, MD, an intensivist with The Permanente Medical Group and researcher with the Kaiser Permanente Division of Research in Northern California. “But what we soon found was that regional practices differed, for example, with how we deliver call center care, or the way we use clinics or urgent care centers. As a result, our use of leading indicators evolved to reflect differences in how each region tailors its care and practice to its members.”

The predictive analytics group is co-chaired by Dr. Young, Dr. Liu, and James Bellows, PhD, managing director of evaluation and analytics with the Care Management Institute, until his retirement last November. They began meeting every other day in March 2020 to develop methods for predicting how many COVID-19 cases each Kaiser Permanente region could expect as well as their severity, and how caseloads and surges would affect hospital capacity, staffing, and supply needs.

Data drives decision-making

Predictive analytics is a science that identifies the likelihood that something will happen in the future, based on information about what happened in the past; researchers use statistical algorithms and machine-learning techniques to analyze “big data” sets. In large health care systems with robust electronic health records such as Kaiser Permanente, physician researchers use predictive analytics to identify health care gaps, trends, and disparities to help solve care delivery challenges.

“In the middle of a pandemic or any public health emergency, data drives decision-making. This situation was no different,” says Felipe Lobelo, MD, PhD, FAHA, senior physician consultant for population health research with The Southeast Permanente Medical Group, and a member of the predictive analytics group. “The only thing constant during a pandemic is that things change. We need to be able to adapt to new scenarios.”

Sabrina G. Dahlgren, Kaiser Permanente managing director for performance analytics and forecasting, says that input from Permanente physicians was key to developing useful information needed for planning hospital intensive-care unit and bed capacity, as well as staffing and supply needs, in the organization’s 39 medical centers and for the more than 80,000 physicians and nurses providing care.

“The product we deliver must be centered on the people who will use it, so that the clinician community has what it needs in order to be effective,” says Dahlgren, who leads the Kaiser Permanente Insight team that developed a standard set of COVID-19 forecasts for all regions and medical centers.

Space, staff, and ‘stuff’

The group created 480 forecasts, which are updated with local data on hospital censuses, COVID-19 testing rates, and other indicators. Every day, 7 different models are run for each forecast — for a total of about 3,360 models — and the best-performing one is selected. Additionally, a combination of short-term models, 20-day models, and scenarios for longer time periods (30 days, 90 days, and year) provide possible scenarios for the path of the pandemic.

Trends identified in the forecasts give clinical and operational leaders in each region and service area the information they need to manage any medical center’s 3 S’s: “space, staff, and stuff.”

In the middle of a pandemic or any public health emergency, data drives decision-making.

— Felipe Lobelo, MD, PhD, FAHA

While group members say that the earliest predictions tended to be overly conservative, by summer 2020 regions were able to identify “epidemiological hotspots” — places where COVID-19 cases could be expected to soon strain hospital resources — several weeks in advance, with a high degree of accuracy.

“There’s a famous saying in the forecasting community, which I’ll paraphrase: All models are wrong, but some of them are useful,” Dr. Young says. “Seeing the future with an unknown, novel virus that is racing around the world — you’re inevitably going to be wrong at times. But our forecasting community has been able to continually adapt and improve our models to give care delivery teams needed information to care for our patients and members.”

Other regional representatives in the COVID-19 Forecasting and Surveillance Guidance Group include Ben Broder, MD, Southern California Permanente Medical Group; Claudia Steiner, MD, Colorado Permanente Medical Group; Dana Tom, Hawaii Permanente Medical Group; Michael Horberg, MD, Mid-Atlantic Permanente Medical Group; Lucy Savitz, PhD, Northwest Permanente; and Michael L. Jackson, PhD, Washington Permanente Medical Group; David Grossman, MD, Debbie Cunningham, Margo Gordon, PhD, Taposh Roy, and Vivian Tan, Kaiser Permanente; and Sumita Bhatia, The Permanente Federation.