WATCH: Digital health and Al webinar

From ambient AI scribes and predictive analytics to “digital twin” technology and remote monitoring, health systems are racing to adopt new tools that promise to transform care. But which digital health and AI solutions truly add value for patients and clinicians — and which are just hype?

Hosted by Stephen Parodi, MD, executive vice president at The Permanente Federation and The Permanente Medical Group, with insights from national leaders in health care:

• Kristine Lee, MD, Associate Executive Director, Technology, Innovation, Virtual & Desktop Medicine, and the Virtual Medical Center, The Permanente Medical Group
• Caroline Pearson, Executive Director, Peterson Health Technology Institute

Together, they explore:

– How leaders decide which digital health tools to adopt

– Real-world results from ambient AI scribes and their impact on physician burnout, workflow, and recruitment

– What the evidence shows about digital tools for chronic disease management (hypertension, diabetes, virtual PT, remote monitoring)

– How to measure ROI, value, and patient experience—beyond time savings

– The role of federal policy and new models like CMS’s ACCESS initiative

– Where automation and AI can safely support clinical decision-making while keeping humans in the loop

Podcast transcript

Transcript is autogenerated. Although edited for clarity, it should not be considered an exact replication of the podcast and may also be updated as needed.

Stephen Parodi, MD: Hi everyone, and welcome to our Permanente Live webinar, Digital Health and AI: Hope, Hype and Hard Truths. I’m Steve Parodi, an infectious disease physician and executive vice president with The Permanente Federation. There is a lot of buzz around digital health and the use of AI in healthcare. Ambient scribes have become ubiquitous throughout our many practices. Generative AI solutions are enhancing our predictive analytic capabilities and talk of agentic AIs promise to automate complex tasks including diagnostics, therapeutics, and back-office functions. Given that rapid change, how do we as clinicians, leaders, patients, and families separate useful innovation from overhyped tech? Are we actually seeing true value? Those are some of the questions and more that our two esteemed guests today are going to tackle. We are joined by Caroline Pearson, executive director of the Peterson Health Technology Institute, where she leads initiatives and grantmaking to fulfill the center’s mission to create high-performing health systems that deliver better care at lower costs. We’re also joined by Dr. Kris Lee, an associate executive director with The Permanente Medical Group, where she leads technology innovation, virtual and desktop medicine, and has responsibility for the newly branded virtual medical center as well as The Permanente Medical Group’s consulting services. Thank you both for joining us today.

Kristine Lee, MD: Thank you for having us, Steve.

Caroline Pearson: Great to be here.

SP: And as we jump in, we’re going to explore these different technologies and see whether they can enhance rather than lose our basic mission to provide human-centered and design care. So before we dive in, just a little bit of housekeeping. You can join the conversation on social media by using the hashtag #PermLiveLeadership. Please submit any questions using the Q&A function on the Zoom meeting. We’re going to definitely ask some of your questions a little bit later. So let’s go ahead and dive in here. And Kris, I’m going to start with you. How can we as leaders, determine which digital health solutions to adopt? And I’d love to hear about what you’re up to here at Kaiser Permanente.

KL: No, that’s a great question. I think it really comes down to being very, very clear about the problem that you’re trying to solve. One of the things that we face all the time are vendors coming with very pointed solutions, but really for the operational leaders and us as physicians to understand what problem are we trying to solve. And when I think about our journey with AI and digital technology adoption, a couple of things really rise up to the top. One of them is physician burnout. So after sprinting through the pandemic and then having to resume the marathon, our physicians were telling us they’re very burned out. And then we all sort of faced the Great Resignation, which I think hit every industry in the states and maybe globally. So really thinking also about workforce capacity and also our workforce shortage. There is a shortage of healthcare workers in the US and we are not immune to that either.

And so really these are the big problems that we’re thinking about. How could we solve those? And then really understanding the why of the problem and coming up with different hypotheses about how might a new technology or a new tool be able to address that problem. We have really tackled physician burnout as our number one pressing problem to solve. And I will also say that there’s not, there’s no such thing as one tool that’s going to solve that problem by itself. So we’ve been taking it piece by piece and day by day and tool by tool and really doing that evaluation and measuring the ROI in terms of what impact is this making? So really having a very clear measurement strategy is also very important.

SP: Well, Kris, that sort of tees up the next sort of question here, which is that both of your organizations have put out research about ambient AI listening tools and other digital health and AI solutions. So I’d love to hear from both of you about how you evaluate the solutions, where you’re looking at ROI, commonalities, and approach design and integration. So Caroline, maybe I can turn to you for that to start.

CP: Absolutely. Well, I love what Kris said about being clear on what problem you’re trying to solve. That is one of the first questions that we ask folks when we work with ’em. So early in the year, early last year, last year, we started with about a dozen health systems that we’re in the process of piloting and rolling out ambient solutions. And so we began to just gather early data on what their experience was, how they were measuring those tools and what they were seeing in early evaluations. And I think when you ask health systems, what problem were you trying to solve? We saw an evolution throughout the implementation. So many of the systems began by saying, we’re really focused on efficiency and ROI, these are going to deliver a lot of time back to our providers. And then over the course of our work with them, we very quickly saw lots of hospitals moving from pilot phase to full implementation.

It was one of the fastest rollouts that we’ve seen, I think, in health care ever of technology. And so we said, well, what made you change? And they said, oh my gosh, the provider demand is high. And so you just saw that the user experience for clinicians was so compelling. I think that they really sort of experienced the value of the tools and to this burnout point, it made the paperwork burden feel lighter. And I think everybody believes that that may also translate into a better patient experience. We don’t have great evidence on that yet, but certainly we understand that if a provider turns around away from is able to turn away from a computer screen, look their patient in the eye and feel confident that they can listen and not just be trying to capture everything that’s going on in the room, that that’s likely to translate to a better experience.

So I think that the initial evidence is very compelling on those provider metrics across all the studies that we’ve seen. What’s a little less clear is how much time we are saving. They’re certainly saving some time, but it’s spread throughout a provider’s day, right? Small portions of time. And every note is sometimes hard to recover. And so it is likely that we’re going to see that accrue in the form of less burden on those providers, but maybe not necessarily a huge increase in the number of patients that we’re seeing per clinic or the amount that we can deliver in terms of access.

SP: So Kris, I am struck by the rollout of this AI technology in ambient in particular. And you know what it reminds me of when I first got my manual of how to do word processing and it was a big old book, and then the next time around it was a little software update and here’s a tear sheet. Can you talk a little bit about how you did ambient AI, because it was different than rolling out the EHR.

KL: Oh, very different. So the first thing we did was who are we going to include in a brief pilot? So we did a pilot, I want to say we piloted for maybe 34 or four weeks at the longest. So no “pilot-itis” happening here. And I included the most reticent folk in the pilot because that’s really the big test. Will people adopt the technology and use it? And we all kind of know who those people are on our team. So I didn’t start with just the really super technology enabled folks. I really took people who would were going to be resistant to the technology and put them in the pilot group. And the initial results were just astounding. And Caroline, to your point, we were sitting on a gold mine. And so it wasn’t like, oh, we’re going to pilot for 6 months and do a business case.

We knew it was a gold mine, especially with our first tester group. And the way we rolled it out was we pushed a little app out on everybody’s phone and we had some at-the-elbow training from our technology folks at each of our medical buildings. But really it was the word of mouth and the people that we thought might be reluctant to adopt the technology became the loudest, the biggest loudspeaker to everybody else. And I will never forget, I was in clinic on the day that we turned it on for everybody, and I literally saw doctors exiting their offices and running to their office mate’s office being like, you’ve got to see this, this is amazing. We had a one-page because who has time to read the whole manual? To your point, Steve, it was a one-page instructions on how to do this. We did have a video because we understand that people consume and learn information in different ways, but it was really that peer-to-peer spread that accelerated adoption of the tool. The results speak for themselves.

SP: So I mean, that’s an amazing story right now, Caroline, I heard a little bit of course support for that. That’s a great thing. On the other hand, it’s like, and I’ll use the dirty word, what kind of efficiency are we going to see here? What kind of ROI, and I guess it’s early days, but what do you think that looks like? And you’re interfacing with a lot of different stakeholders here, so you’re not just talking to health care leaders, you’re talking to employers, people paying for the health care. What are you hearing on that front?

CP: Well, I think the first question to ask is, ROI to whom? So in terms of thinking about the health system as the most common purchaser, there’s 2 theories on where we could see revenue gains as a result of the solution. One theory was if notetaking gets a lot faster, we if expand clinics right, can you basically get more patients into every clinic. I would say we’ve seen very little adoption of that as a solution. I think generally you’re not seeing time savings enough in sort of consolidated chunks enough that would cause folks to do that. And I think particularly to Kris’ point, given the burnout, most systems have not been focused on trying to make visits shorter and add patients to every clinic. But we do see early emerging evidence that some of the coding around those notes may be more complete.

And so providers may be incrementally increasing their billing for every patient as you’re getting better capture of the encounters in the room. And that really flows through the coding from a system point of view. That’s obviously something that as a philanthropy focused on health care costs, we want to watch, we both believe that we want every visit to be coded accurately and completely and ensure that patients are getting the care that they need. But we also don’t want that to create an escalation of billing that just increases total spending without actually materially changing the care folks are getting. So from a cost point of view, that’s the ROI to the hospital versus the system is something we need to watch.

SP: And Kris, I know that no project goes without financial consideration. So I’m curious again, where ambient is today and where do you see that or what do you see it enabling in the future?

KL: Yeah, well, the technology’s developing so quickly, very quickly. The product that we piloted here in Northern California is different than the product that we’re using today. And a lot of that is based on our experience and our influence with the vendor to say, these are the needs of these different operational areas. Could we do this? Could we add more configurability so that more people can find it more useful to what suits their practice in patients? Very different than an ambulatory setting. A mental health therapist might have a very different encounter than a primary care physician or an orthopedic surgeon. So having that flexibility with the technology has been absolutely fantastic. In terms of what you and Caroline were just talking about in ROI type of situation, what have we seen? We’ve actually seen physician voluntary turnover go down to pre-pandemic levels from an all-time high down to pre-pandemic levels. And I’m even seeing this now as a recruiting tool. So we get medical students applying for residencies and we get graduating residents applying for jobs with us in our workforce. And this is absolutely one of the questions they asked, do you have this type of technology available for me? So we’re seeing it in a lot of different areas. It’s not just necessarily the time savings, but it’s also the quality of the interactions that physicians are looking for to have with their patients.

SP: Yeah, I mean, you’re so right, Kris. I mean, I was just recently at a physician trade meeting where there were a lot of residents and fellows there. And in some ways I appreciate you saying it’s a recruitment tool. In many ways it’s turning into table stakes. Folks are saying, I’m not going to work at a place that doesn’t have this. So it is really been an interesting shift in just literally like a year. I’m going to pivot us here for a second and move us to talk a little bit about chronic disease management that holds a whole ton of promise. So Kris, maybe you could talk a little bit about what Kaiser Permanente is doing around digital health tools to better support patients with chronic conditions.

KL: Yeah, well, to understand we have millions of patients that we take care of, and an average physician might have a couple of thousand of patients that they take care of. How do you manage the chronic conditions of that? It just sounds completely overwhelming. So it’s using these digital health tools to look at aggregates, population aggregates, make that completely transparent and visible and surface up actionable items, right? Care gaps. Oh, you haven’t done your colon cancer screening or you haven’t done your cervical cancer screening in the right window. Let’s get that booked for you. So that’s one way that we’re releasing that technology of saying, okay, what is the overall population? And then having that radical transparency and not just to necessarily a primary care physician to the whole system. So we have physical therapists booking mammograms as an example, because we’re really thinking about the population as a whole, but I think it’s really with the actionable insights and really surfacing those up at the front. So a patient may come in for foot pain, but actually really I need to look at, oh, hey, what’s your A1C like over here? So I think the responsibility, especially in primary care, is that we’re holding the whole patient all the time, even though they may come in with a completely, completely different issue. So really leveraging the technology to, one, understand the population, two, deliver actionable insights to the doctor that’s taking care of that patient. That’s really been our approach here at TPMG.

SP: And Caroline, can you talk a little bit about it from the perspective of the broader industry and how they’re thinking about it? Kris is giving sort of perspective from a integrated health system, but this is permeating I think throughout the entire industry.

CP: Absolutely. We saw such a rapid adoption of new technologies following the COVID pandemic. And so you just had this boon of shift to virtual care, to digital treatments, to home monitoring, all of which was really long overdue. But then we had a lot of health plans and employers and providers saying, gosh, we’ve adopted all of these things and we don’t really know if they’re working. And particularly when we at PHDI think about what works, first and foremost is clinical outcomes. If you’re a person with type 2 diabetes engaging with a chronic disease management app, we believe that you expect that solution is going to improve your hemoglobin A1C, not just that it’s going to deliver you a nice experience. And so as these solutions have had more years of existence, we’ve started to be able to ask much more thoughtful questions about which of these things work.

And so we really try to call balls and strikes. So we’ve done now about 6 assessments of different categories of digital health solutions and really trying to say, what’s the mechanism of action? And you see a few different things. Some of these are taking established care delivery models and translating them to virtual or home-based settings that are more convenient, that enable more scale, that are often more efficient. So you can think about things like virtual PT actually is very effective and in fact works. Not only is it clinically effective, but it really makes it much easier often for older adults who may have trouble with transportation. So they can do the PT exercises, but they can’t get to PT. And we know PT is clinically effective and that not nearly enough people do it and they get a lot of unnecessary care as a result.

So how do we translate that to a virtual setting? Another example that we’ve looked at is hypertension. In any provider panel, you’ve got about half of your patients that are experiencing hypertension, and many of them may not be in blood pressure control for any number of reasons. Some of them aren’t coming in regularly, but often they’ve got other health conditions that may be a higher priority when that primary care doc is sitting in the room with them. And you’ve seen a lot of innovation around home blood pressure cuffs and virtual care teams that can really expand the capacity of a primary care physician to be able to treat more of their patients, get more folks onto hypertension meds and titrated and into blood pressure control, often in 3 or 4 months versus a year of going in and doing all the visits. So I think we’re seeing a lot of innovation around what works here in digital health, and we’re finally getting enough data to say, okay, not everything works. Technology isn’t magical pixie dust. You don’t just sprinkle some technology on health care problems and everything gets better. But when we know that there are proven care models that we can begin to translate into other delivery modes, virtual delivery modes, that’s where we see really big effect. And so that that’s some of what we’re seeing throughout all of health care.

KL: Can I piggyback on what you just said, Caroline? Because you brought to mind this example that this is a real-life experience that I had. I’m a primary care physician by training, and I still practice and see patients, and I happen to be at the consumer electronic show in Las Vegas and went to the health care section. They have all these different sections. And one of the things that I saw was, it’s unusual, but there was a commode set up on a stage and I was like, oh, this is weird. What is this? And it was, oh, we’ve developed the technology where we can do a chemical analysis of a person’s urine. And I said, okay, well what do you do with that data once it’s collected? Oh, we just send it to the primary care physician. And as a primary care physician, I was like, well, what am I going to do with that information? And then what if everybody has one? I’ll expand a little bit more too. Think about the Oura ring. A lot of people wear these rings to monitor their sleep. Could you imagine if all of that data somehow then came to me as the primary care physician? What would I do with that? And so there’s a lot of these technology solutions that are being developed that really may not have any practicality in their current state right now to the overall health of the patient.

CP: There’s 2 things about what you just said, Kris, that I think are so important. One is we can generate a lot of data with technology, but if we don’t have an action to take at the end of that data, we are not going to affect change. And this is one where we’ve seen some real differences actually. So virtual monitoring, remote monitoring for hypertension versus diabetes have varied the different clinical results. And some of that just ties back to the basics of those care plans, which is we have pretty effective medications and treatments for hypertension. If we know somebody is experiencing high blood pressure and we have the human capacity to intervene, we can get that. We can get them into blood pressure control. Diabetes is much more complicated. You have a ton of diabetes monitoring data hitting PCPs, and unfortunately all of that data does not yield lower blood sugar and better hemoglobin A1C in most cases.

So we have to really think about that. And then the second thing you said, Kris, that I loved is this integration. For a long time, digital health has just sort of floated around, not fully connected and integrated with the rest of the care delivery, and I think increasingly we’re seeing some of the best care models tightening those things. So how do you get the orthopedic surgeon or the PCP to refer to virtual PT before you send them for a cortisone shot or a back surgery versus just hoping somebody’s going to sign up for that on their own.

SP: This is such a great conversation. I just want to remind our audience that if you have any questions for Caroline or Chris, please submit them using the Q & A function. Kris, you opened the door. So I’m going to have to ask this question. So in the idea of things aren’t pixie dust, you actually have to study them, I’d be curious about you talking about the latest program, the pilot that you’re doing. I believe you’re doing it with a vendor and you are looking at data points and remote patient monitoring and baking that all in and looking at food. So I’m curious, what are you up to? Why are you doing it? Are you going to know if it works?

KL: Yeah, so we are exploring digital twin technology. So using remote wearables such as a CGM, a wrist device that monitors your activity and your sleep blood pressure cuff and a scale with the patient logging their food into an app. So collecting really over 3,000 data points per day on an individual person and seeing, really studying and learning their metabolism. So there’s an AI algorithm in the background that’s really learning. How do you respond to that cheese enchilada at 3 PM in a way that a primary care physician would never be able to really deeply understand a person and creating a digital twin of that person and then surfacing insights back to that patient that are very, very personalized, such as, oh, okay, you just logged that cheese enchilada. Can you tell, I like cheese enchiladas. But so maybe you should go walk for 15 minutes after this meal because we think your blood glucose is going to go up from this. So we want you to walk to bring that blood glucose curve down.

I’m really looking at this and thinking about this as an adjunct to our population health management programs. So we have our chronic conditions management teams that are doing a fabulous job, but maybe I only get 90% of my patients with high blood pressure under control. What about the other 10%? So could this be a way of really thinking about coming up alongside what we do traditionally in our population health management systems and really personalizing this to the patient? So far we’re about 60 days in, we’re seeing great results, we’re seeing A1Cs come down, seeing people’s weight falling also, which is great, and more to come. So we’re in the middle of this pilot right now. Very exciting, also very exciting to see the adoption from the patients. So the patients really like it so far and they seem to be hungry for it.

SP: So Caroline, we’re talking about pilot within the system. It sounds like the federal government may be getting in the mix here, so maybe you could talk a little bit about that. What are they thinking about? Why are they thinking about it, from your perspective?

CP: Absolutely. Well, I think we all adopt technology every day in our regular non-health care lives. So it is a little silly to think that we wouldn’t equally begin to integrate it in proven ways within health care. But because of the way that many of these digital health tools have come to market and get sold, in fact, they have not had any coverage in Medicare. So while you have a lot of Medicare advantage plans and then commercial plans covering digital health tools, traditional Medicare really has very few reimbursement models for that. And so there was a big new program announced last week called the access model in which CMS is really going to try to create for the first time ever a reimbursement model for some of these digital health solutions that we’ve been talking about, which is wonderful. I mean, as a provider, it’s very hard to keep track of which of your patients might have access to what.

And if you think there’s a really great tool, you certainly would want to extend that to your Medicare beneficiaries too. So I think it’s exciting and it’s likely to really help stimulate the market and the growth of more digital health tools. But what’s most exciting for me as an evaluator is the way that they’re setting up the payment structures, which it’s all going to be outcomes-based payments. So there will be Medicare payment for any digital health tool that wants to participate, but some of that payment is going to be withheld and the solutions are going to be responsible for delivering those clinical benefits that are pre-specified in order to get the full payment. And ultimately, as you’re asking Chris, how do you know what works, the very best way to do that is to tie those payments to clinical outcomes, both saying to the vendor, this is what we expect in terms of performance and then generating the data that enables you to really sort out which solutions are most effective. So I’m excited to see all of the data and all of the adoption that comes off of that program.

SP: It’s pretty interesting, right, because they’re almost flipping the fee-for-service model on its head and injecting value-based principles and using technology as a way to make that happen. It’s very interesting. Kris, you referenced earlier the patient experience, and I’m curious because it sounded positive people are liking this stuff, but how are you measuring it and do we need to measure it? And how do we need to understand that? I know that there’s a lot of concern also from other folks, regulators and others about the implementation of the technology.

KL: So we regularly survey our patients to ask them what they think. And another thing that we do is when we do these sort of different pilots of things, we make sure to include our patients in the design of these pilots. So they are a piece of how are we going to implement this? How might this lens, what should we consider? We shouldn’t assume that we know what they’re thinking. At The Permanente Medical Group, we also have a lot of discussions about consenting patients, transparency to our patients, about our use of different digital health technology. So in the ambient scribe example, every patient is consented every time. So they know that we’re using this, it’s going to help me write my notes. What we’ve seen is 95% acceptance rate with our patients. So something less than 5% of our patients actually say, oh no thank you.

And they actually think it’s pretty cool when we ask them, they’re like, wow, you guys are cutting edge. So that was our findings within the first year of using and adopting this ambient scribe technology. There are bigger conversations about, okay, well what if we’re using AI tools in an acute care setting like the hospital? How do you consent to patient who’s potentially having some life or critical event happening? How do you do that? And so there’s a lot of debate, I think also in the community about what is the level of disclosure of these things do we even disclose at all? So I think that those areas are still being tested out, but patients seem to be very willing to allow us to use AI to augment their care.

SP: We got a lot of great questions coming in. I just want to encourage people to keep submitting them. Caroline, I’m going to ask you one quick question before we switch over to the audience, which is what do you see as the biggest barriers to implementation or adoption of digital health tools like the ones we’re talking about?

CP: I think a lot of it right now is around, it’s like business stuff around how do you contract for it, how do you pay for it, how do you integrate it on provider side? How do you integrate it with the EMR? I think that’s all getting easier every year, but as Chris said at the beginning, there’s still definitely point solution fatigue. And so on the one hand why I like point solutions is actually you get really clear data about what’s best-in-class for the specific condition. But from a patient point of view, if you’ve got 3 comorbidities, you don’t want to touch 3 different solutions. And from a provider or purchaser point of view, it’s hard to manage. So I think we’ve got a ways to go in terms of making that experience better. Interestingly, I don’t think it’s resistance on the form of the patients. I mean, as Kris said, patients really trust their doctors. And so when doctors are saying, this is a tool that is helping me deliver care well to you, I think you get a lot of trust in that typically from patients.

SP: So I’m going to cut over to the audience questions, and Kris, I’m going to tee up the first one for you here. So do you think, this is kind of a trick question here. Do you think there’s going to be a time in the future where AI is going to be used in surgery or used in robotics? And maybe more broadly, to build on the question, where do you see AI not touching?

KL: Yeah, I actually think it’s going to have broad infiltration. So there’s already AI being used in different procedures. I’ll give an example, colonoscopy. So if you have a scope that’s augmented with intelligence, maybe you can be like, Hey, look at that polyp over there. That one looks like it could be a higher-risk polyp than that polyp that you just snared over there. So this is existing technology. There’s a lot of AI that’s already embedded in existing diagnostics. Imaging is probably the furthest advanced. So looking at this mammogram, oh, actually this looks like the highest risk area. This is where you should be focused on to do your biopsy or whatnot. So it’s already embedded in that technology. I absolutely could see it in surgery as well. Risk prediction, I could see it in that. Imaging during the actual live procedure, I could see it being used in that, taking the pretest probability or the presurgical probability and the postsurgical probability and saying, here’s how you can increase your postsurgical success rate for this specific patient. So I can see it being used in many ways. And I’m sorry if I forgot the second half of that question, but I do see it being broadly, potentially being broadly utilized. Will it replace the surgeon? I don’t think so.

SP: That’s really great. And Caroline going to play off of this with this next question. We were talking about this a little bit before the session, which is where is AI maybe not performing right now? And the question really focused on mental health care. And so I’m curious, just given the early days experience with chatbots and others, your thoughts on that in terms of its use in the mental health sphere?

CP: So we have looked at digital mental health tools specifically for people with mild to moderate depression and anxiety. And the first takeaway there is actually digital solutions, including AI-enabled solutions, are quite clinically effective for mild to moderate symptoms. So they really can deliver comparable benefits to live psychotherapy. So they should be part of the toolkit, but they have to be AI chatbots that are trained specifically for mental health. So the big concern right now really is unfortunately you’ve got the availability of things like ChatGPT and other general models that people are going and seeking therapeutic support from. There’s high rates of suicidal ideation being professed to those models every day, and they are not actually trained or structured to challenge negative thoughts, redirect thinking, all of the things that a psychotherapist would be trying to do in a session to help someone succeed. And so you really do want a chatbot that’s designed specifically and tested for mental health purposes. And I think the interesting question right now is how do you get those 2 different types of chatbots to work together? How do you transition someone who is needing mental health care into something that’s going to be clinically appropriate for them? And that’s really an important question that we’re grappling with right now.

SP: This is another interesting question. When you think of AI, when you really strip it down and you’re talking about large language models at the end of the day, Kris, maybe I’ll ask you first, how is language access influencing improvements in AI models? And in particular, how are we thinking about this from a non-English speaking perspective so that we make sure that we have entire swaths of our population that we’re responsible or caring for are not left behind?

KL: Yeah, I would say for me as the operator or the person that’s helping to deploy these technologies, I have to very deeply understand the data set that any tool that’s using generative AI has been trained on. Let’s say there was a hypertension tool that was trained only on native Hawaiians, and then I’m going to go and try to deploy that in a Latino population somewhere else. That tool is not going to be as successful with managing that Latino population because it was trained on a dataset of people that look completely different, that have different physiology, and different behaviors. So really looking at the data sets, and you can’t be too small of a data set, but you really need to understand what are the potential biases that may be propagated really just based on that information alone.

So I think in this area, like tools, if you’re really thinking about, oh, I really want to help this segment of this population, what is the dataset that we could train on to develop a tool specific to that population, that group where you’re trying to make an intervention? Is this happening all the time? No, because it is very specific. So if you really want to have a tool that has the highest performance, you need to have that data lake, that committed data set to really address that population. And I just don’t see that that’s necessarily happening right now. I think people are looking for the biggest data sets and trying to train tools that would be broadly applicable and that in these early stages of AI development. And I could see in the future, hey, people might want to go after this market and be like, oh, I want to be the preeminent tool to help people of, I don’t know, Afghani ancestry have great hypertension control. So just giving some examples there.

SP: So I almost feel like putting up a picture of the Terminator here, but so the question here relates to a lot of the focus has been ensuring that humans are in the loop with the decision making when it comes to an AI, and I paint that across the board right now. Do you see a feature where there are activities that get fully automated? I’ll use that term. There may be other terms to use. So Caroline, maybe I’ll turn to you so

CP: Many. I do. I hope so, and I’m a bit of an outsider. The typical rhetoric when you have AI conversations is that folks, leaders will say the AI is not going to take your job, but the person trained with the AI will take the job from the person not trained with the AI. Personally, we have a workforce shortage, a clinical workforce shortage. We have an aging population, and we have an unaffordable health care system right now that doesn’t deliver great care to everybody who needs it in the country. And so we have to embrace technology and empower it to do some things autonomously or independently, of course, with the appropriate controls and oversight and humans in the loop. And I think we’re at the start of making those decisions. But there will be facets of care that I think can be automated and would urge us to embrace that. Hypertension, we would talked a lot about hypertension today, hypertension medication titration to someone who’s taking regular home readings with their blood pressure cuff is I think a really good candidate for early adoption. Mental health care is another one where you’ve got a tremendous access problem, and we need to think really carefully about how we can be thoughtful, but empower some of those tools to allow us to use our finite workforce to the greatest and best use of their skills.

SP: So we’re almost out of time. I’m going to ask both of you a rapid fire question, Kris, and you only pick one thing. What AI use case most excites you right now?

KL: Right now, I would see this digital twin technology with the remote combining a bunch of other different technologies into a great technology ecosystem solution is the thing that excites me the most immediately. And you only allowed me to have one answer. So that’s what I’m most excited about right now.

SP: And Caroline, if I could turn to you.

CP: Automation of simple clinical decision making. Think urgent care. Think about the number of things a primary care doctor does that could be automated and would allow them to spend more time focused and talking to their patients about different issues. I think there’s a lot we should automate.

SP: So Caroline and Chris, thank you so much for joining us today. And I want to thank everyone who joined this webinar and listened in. Want to point out, look for the link to the webinar recording in your email, and don’t forget to share it with your network. And also be sure to follow Permanente Medicine on social media to learn about the future programs and check out permanente.org for our library of past videos and podcasts.

The development of new health care tools and technologies will continue at a rapid clip. As leaders, clinicians, health economists and analysts, we need to remain focused on delivery of high-quality care and not get distracted by the latest trends. Separating what offers value and defining what value means from what it doesn’t, will help us successfully integrate digital health tools and improve our patients’ outcomes, increase physician satisfaction, and decrease costs. Ensuring that humans are committed and remain in the loop will require dedication, work, innovation, and ingenuity. And that’s what has always taken us as physicians to help us fulfill our mission, to make the lives of our patients and fellow clinicians just a little bit better. Thank you all for joining us today.