From Needle-in-a-Haystack to 95%: AI, Goals of Care, and Systemwide Change

Why This Episode Matters

Goals-of-care conversations can profoundly shape serious illness care, but in many health systems they remain difficult to find, inconsistently documented, and hard to measure. In this episode, Matthew Gonzales and Deborah Unger describe how Providence treated serious illness communication as a systemwide quality problem, combining leadership commitment, clinician training, nursing engagement, informatics, and AI to make “what matters” conversations more visible and actionable across 51 hospitals.

Key Ideas Explored

• Why goals-of-care documentation became a “conversation in the haystack” problem 
• How Providence made serious illness communication a system priority, not a palliative care side project 
• Why training physicians alone did not move the needle, and how nurses became critical to implementation 
• The tension between standardized documentation and preserving the humanity of the conversation 
• How AI helped identify meaningful goals-of-care conversations without relying on checkboxes or dot phrases 

Takeaways for Quality Leaders

• Treat important clinical conversations as part of system design, not just individual clinician skill. 
• Build measurement only after defining what meaningful quality looks like in practice. 
• Engage the disciplines closest to the workflow; nursing involvement may reveal implementation paths leaders miss. 
• Avoid designing metrics that reward documentation behavior while missing the underlying clinical purpose. 
• Look for AI use cases where language, workflow burden, and quality measurement intersect. 

Continue the Conversation

Dr. Gonzalez -  

Email: Matthew.Gonzales@providence.org

Dr. Unger - 

Email: Deborah.Unger@providence.org
Bluesky: @qoflmd.bsky.social

Resources & Frameworks Referenced

• Providence Institute for Human Caring
• Ariadne Labs Serious Illness Conversation Guide
• Guide Successful Strategies for Operationalizing Goals-Of-Care Documentation - NEJM Catalyst
• Finding the Conversation in a Haystack: Leveraging AI to Detect Goals-Of-Care Documentation - Journal of Pain and Symptom Management

Leading Quality is a podcast for healthcare leaders committed to improving systems, culture, and outcomes.

If you found this episode valuable, follow the show, rate and review the podcast, or share it with a colleague working to improve care.

Connect with Jason Meadows on LinkedIn for more insights on healthcare quality and leadership.

Help us build this podcast  community from the ground up: share your top insight from this episode and where you’re seeing it in your own work. I read every response and will share what we’re learning over time in future episodes and other ways.

New episodes published every other Thursday at 7AM Eastern Time.

Credits:

Host, Writer, and Executive Producer
Jason Meadows, MD

Produced by
Thrive Healthcare Improvement

Edited by
Milan Milosavljevic

Transcript

Why Goals Of Care Gets Lost

SPEAKER_01 0:00

We ran this pilot project with ENI. First, we tested it against expert annotated notes and we saw how well it performed. It found notes that were written by nurses and residents and radiation oncology and obstetrics on high-risk pregnancies that we never would have found.

SPEAKER_02 0:27

Welcome to Leading Quality, the podcast spotlighting the people moving healthcare forward from the front lines to the C street. I'm your host, Jason Meadows.

unknown 0:45

Dr.

SPEAKER_03 0:46

Deborah Unger and Dr. Matthew Gonzalez are palliative care physicians at Providence, who have been deeply involved in the Institute for Human Caring's work to improve serious illness care at scale. Matt helps lead this work across Providence. And Deborah brings what I suspect is a rare combination of palliative care expertise and informatics training to the problem of how important clinical conversations are documented, found, and used. In this episode, we talk about goals of care conversations, those deeply important discussions about what matters to patients and families in the context of serious illness. Clinicians have these conversations every day, but in most health systems, they can be almost impossible to find later, buried across progress notes, consult notes, telephone notes, and narrative documentation. The deeper issue is how a health system moves from hoping these conversations happen to designing a reliable system that supports them, measures them, and makes them visible without reducing them to a checkbox. Providence's work has involved executive leadership, clinician training, nursing engagement, documentation redesign, and now AI models that can read millions of notes and identify meaningful goals of care documentation across 51 hospitals. This conversation is really about what happens when serious illness communication is treated not just as an individual clinician skill, but as a quality informatics and system design problem. This conversation is really about what happens when serious illness communication is treated not just as an individual clinician skill, which it is, but as a quality informatics and system design problem. Matt and Deborah bring the lived experience of building that system, learning from early missteps, and pushing towards a future where patient goals are easier to document, easier to find, and ultimately easier to honor.

unknown 2:45

Dr.

SPEAKER_03 2:45

Gonzalez and Dr. Unger, welcome to the show.

unknown 2:49

Thanks.

SPEAKER_00 2:50

Thank you so much.

Two Paths Into Palliative Care

SPEAKER_03 2:51

So I'd like to start as I often do, uh, with a little kind of 10,000-foot view of both of your uh career paths up until now. Maybe, Matt, can you just kind of start me off with you know what drew you into palliative care, serious illness communication, any background you want to give me about uh your path so far?

SPEAKER_00 3:10

Yeah, totally. Um I appreciate the question. Uh if you had asked me what I end up here in this space 30 years ago, uh no. I was not some like foretold thing where I was for many years longing to be in the the space of providing better serious illness care. I I think for me though, like I've had a few fundamental experiences. One when I was was actually very young, I had my my favorite uncle die um when he was just 27 and I was just 13. And um his experience of like serious illness and dying uh was pretty awful. And so I think very naively thought and took up the mantle really of motivating my life towards trying to make sure that other people didn't have that kind of suffering. And the way that I thought about that was more around like trying to prevent people from getting ill. But as I've matured, as I've like gone through medical training, I was struck very much by the fact that despite all of our advances, despite everything that that modern medicine can do, and it is, to be sure, incredible, we really haven't like changed the mortality rate of our species. Like it still remains at 100%. And so ultimately, like there is something that will happen to all of us. And um, while some people may find that depressing, I find that very connecting because it means it's a universal human experience. And so for me, uh I've I've reframed and realized over many decades that you know his suffering, uh my suffering at the time was yes, of course, about his illness, but it was also about the terribleness of his experience and a lot of um the emotional, the psychosocial, the spiritual distress that was just really unattended to at the time. And so for me, I've dedicated my life to not trying to help us all live forever, which I know some folks are doing, um, but more around trying to recognize that when you're seriously ill, it takes a village and that people have needs beyond you know, their anti-cancer therapy or their heart failure goal-directed therapy. Um and so I I love that part of medicine and being able to sit with people and help systems be able to think about how we provide better holistic whole person care for folks.

SPEAKER_03 5:58

Deborah, I wonder if um I'm curious to hear kind of your your rendition of of how you got here to this point.

SPEAKER_01 6:05

So I I feel like I I also would not have guessed that I would be here if you'd ask my like high school self. Um I come from uh a family of people in healthcare. My father is a physician, pulmonary critical care, who ended up taking up the mantle of doing palliative care um kind of before it really existed as a specialty, um, and was a hospice physician for many years before he retired. Um and my mother is a nurse. She ended up getting her PhD in nursing and nursing education, but was also a hospice nurse when I was young. Um so I grew up with discussions of healthcare in the family, and I was very adamantly against going into healthcare. Um, when people asked me, like, what do you want to be when you grow up? I was like anything but a doctor or a nurse because they work too hard. Um but then kind of college age, I started thinking about what I wanted to do with my life and being of service and combining the technology and the science of medicine really appealed to me. But it was kind of interesting going through medical training with the lens of palliative care being something that existed, but it's not really well taught in a lot of our medical education, especially, you know, 10 to 15 years ago. And so it was it was sometimes difficult, you know, in medical school when I had a class where you're like, oh, memorize all the nerves in the hand and then regurgitate it. But I'm like, but what about people? Can we take care of people? Um so I came out of that and decided to pursue palliative care full-time and did my fellowship, did some uh research year and then the clinical year and fellowship. And then I went and was a clinical attending and faculty for a fellowship in Nashville for several years, but was always interested in technology. So that's kind of how I met Matt and then ended up here at Providence with the Institute for Human Caring about four years ago.

SPEAKER_03 8:32

Yeah, so I was gonna ask you about that, but I'm glad that you uh that you brought that into the story, the the connection um that you and Matt share. So maybe Matt, you can can you elaborate a little bit more on kind of what the Institute for Human Caring is um and what uh what your your role is in it and how you guys kind of got started together.

SPEAKER_00 8:52

Totally.

Building The Institute For Human Caring

SPEAKER_00 8:53

Um so the Institute for Human Caring um is was founded uh gosh 12 years ago now, um, by Dr. Ira Bayok. Of course, many folks um listening will know that Ira was one of the pioneers of the field of hospice and palliative care. Um he wrote uh a best-selling book, Dying Well, back in the 90s, that has been maintained um a huge degree of um popularity and resonance. Um and, you know, 12 years ago, he was looking and trying to figure out how we can uh uh cause meaningful change at scale. And so talked to a bunch of health systems across the country who focused really on the West Coast. Um, but at the end of the day, Providence, um, decided to step up and fund this big audacious goal of creating something called the Institute for Human Caring. It's really a group, a collection of folks that come from all different um disciplines, and not just clinical disciplines, um, but like professional walks of life from from all you know, marketing folks, communication folks, software folks, data scientists, health system educators, clinicians, the goal of really trying to look at the experience of serious illness uh from all different angles and trying to understand what the true barriers are to getting it right. Because while we've had a lot of success, we know that there are a lot of barriers to getting it right. And so our team really looks at it from multiple angles, trying to work within our whole system, figuring out what those barriers are, how we create resources, how we create um meaningful change, and and that sort of like culture change package that we've been working on for the last 12 years is is super fun. Um but as you as you might imagine, right, like it's not just all of those those disciplines that I I listed, but increasingly over that time, our world has become more and more digital, you know, and we're we're not going backwards. We're we're not our our society is not becoming more analog. Um so for us, one of the heavy investments that we've really made is in the digital sphere, recognizing that we are continuing down that path as a society within healthcare. And so trying to figure out and get get out ahead of it and do that well felt really important. And that's been a challenge, to be totally honest. Um, it's gotten a little bit easier and more exciting over the last three years with the the uh advent of like modern AI and large language models. Um but it's it's it's been a key part of our journey really since the founding. And that's meant that we've needed to really bring in experts, really wonderful people that are um not only trained in things like data scientists, but folks like Deb who are you know duly um trained and qualified to be able to do work both in the clinical space, so understanding what's really you know meaningful and like how to really affect patient care. You can't go through training and take care of patients without getting some sense of that, but also um understanding and knowing how technology works and how um best to interface with it and how to think about it. So I would say it's just it's been a real pleasure to um that's been a part of our team now for four years. We we met on Twitter back when it was Twitter, um, and we're exchanging tweets back then back and forth about like, wouldn't it be cool if and um amazingly when when Ira retired from the institute um and I stepped into his role, it was a really easy decision to bring Deb on and to be able to help us move further faster.

SPEAKER_03 12:54

That's great. And uh yeah, it's funny, I as you were saying that I think I still called it Twitter, maybe I shouldn't, but that's um I just I'm I'm still learning to with it's X, right? Um That's great, and so I guess to bring a little more concrete detail to the the what the meat of what we've been talking about today, you mentioned the uh the Institute for Human Caring um is tasked with has tasked itself with with removing a lot of barriers to this this uh you know uh serious illness care that that um you know is is so important.

The Needle In The Notes Problem

SPEAKER_03 13:38

Well a lot of what we're gonna talk about today is is around goals of care conversations, um, how they're done well, how they're documented, how we know that they were documented, um, you know, how we actually find the conversation in the haystack. Um and uh maybe you can, uh Matt, just staying with you for a second, can you kind of set the scene of where and when this kind of documenting of goals of care project uh started and and uh how and why it started.

SPEAKER_00 14:06

Yeah, totally. Uh you know, it I think you very well described it as a needle and a haystack problem. Um you know, all of us clinicians that have these types of conversations, whether you're a palliative care clinician like we are, or hospitalists, oncologists, no matter who you are, we're all having conversations around what matters to patients and families in the context of their serious illness. And that's really how we define a goals of care conversation within our health system. It's those really important what matters conversations when you're seriously ill. Um, and it's not just like um a sharing of prognosis, it's not just um a conversation around what happens if your heart stops, it's like the more meaningful stuff. Like, you know, what does this mean in the context of your life? What does that cause you to worry about? What are the trade-offs that you're willing to go through for the chance of more time? All of those things are really deeply important. Um and we know clutches are having those conversations, but they're really hard to find. You know, before we did this work, the down button was like my favorite button in um our electronic health record because you're just like scrolling down through every single note, right? Like when I come to meet somebody before I consult on them, I'm like, what have they already talked about with other people? And so you're reading the last like 20, 50, 100, 500 notes, you know, depending on where you practice. And this just it's a lot of data for us to sort through. And um, and it used to be very, very hidden. So we recognized really at the outset back in 2015 that one of the key things that we needed to do was make it easy to find these conversations and also make it easy to measure whether those conversations were happening. Because it wasn't just that on the clinical side it was hard to find. We also couldn't find them on the back end to be able to say, are we really, really causing change to happen? Are we moving the needle on this? And so we developed really back in 2015 a simple standalone way to be able to know if those conversations happened and to be able to find them on the electronic health record. And you know, the looking back on the technology that we used 11 years ago, uh, it was the right technology at the time. Is it much better now? As you'll hear about yes, 100%. But fundamentally, that shift that happened in in 2015 when we roll this out, essentially being able to have a separate place to put these conversations and the single source of truth that collated all of them in one place was huge because it meant that I didn't need to go and read all those 500 notes anymore. I could reliably go and within three clicks of logging into someone's chart within the electronic health record, be able to find those conversations in one place. And that that allowed us to be able to deliver, I think, better patient folk uh centered care.

SPEAKER_03 17:07

Yeah, absolutely. I I'd love to claim credit uh as well for the uh the conversation in the haystack uh metaphor, but I think I did steal that from Deborah's one of Deborah's publications, uh, a uh an abstract you published last year. So I just wanted to give credit where credit was due. And uh and yeah, it's uh it's an incredibly difficult problem. I think there's uh probably so many clinicians listening who have had the experience of of trying to pour through notes and find, you know, what was discussed, where was it discussed, where in the note was it documented, does the substance of the note match the substance of the conversation? And I imagine that's a lot of the you know the work that you've had to do. You know, I I understood, and I'll include links to the uh to the relevant articles, including the the New England Journal uh catalyst article, but I I understand that one of the probably one of the things that moved the needle in this work was to have Providence place this as a system priority rather than it being kind of a just a palliative care side project. And I'm really curious to know how that came to be and and who had to, you know, how you kind of convinced the right people to uh to make this a system-wide priority.

SPEAKER_00 18:24

Jason,

Leadership Buy In And System Metrics

SPEAKER_00 18:25

I think you're totally right to say that that that was critical. And I think um when we talk nationally about this, people are always like that that is a big question that we get asked. Like, how did you do that? Um probably maybe from the outside, if you're sitting in a different organization, it felt like a larger ask. But at the time in particular, you know, at the founding of the institute, the institute for human caring was founded directly at the CEO level for a reason. And so there was institutional will, there remains institutional will, I should say, because this really aligns with our mission, vision, and values of this organization. You know, there's a saying within Providence, it's um they call it the Providence Promise cultists that says, uh, know me, care for me, ease my way. What I would say is I think most of our employees would be able to say that sort of tagline, as it were, but it really resonates with people that in order to really best take care of folks, we need to know and understand who they are. That's the know-me part. And um so it it it aligned very much with sort of the high-level vision of the organization, and it happened at a time where it was increasingly being recognized that people in American healthcare were dying badly, that we weren't attending to what needed to be talked about, we weren't attending to their symptoms, and so it was a um it was actually relatively easy to say we need to make this better for patients and families. Um, so a lot of it was was organizational tie-in will and alignment with mission.

SPEAKER_03 20:16

I wanted to pause for a moment to invite you into something I'm starting with this podcast. If something from this episode connected with your own experience, where you've seen it work or not work, I'd love to hear about it. There's a short link in the show notes where you can share what you're seeing in your own work. It takes about a minute. I read every response, and over time I'll be sharing what we're learning together in future episodes and in other ways, and giving shout-outs during future episodes to people in the community who share ideas that really move the conversation forward. If you're up for it, I'd really value your perspective. Thanks. You know,

Documentation Design Without Extra Burden

SPEAKER_03 21:05

what did goals of care documentation look like in kind of the early days? What was problematic about the way things were structured from an informatics perspective and how has that evolved over time?

SPEAKER_01 21:21

Yeah. As you know, Matt was talking about one of the biggest problems from a data science and informatics perspective around these conversations is that there's no sort of one way to find them on the back end. They're they're written narratively in text. It can be in your history and physical note, it can be in your progress note for the day, it can be in your consult note, it can be in a telephone note. Um, it's not just palliative care, it's the hospice and the oncologist and the surgeon and the primary care doc. So it's not inpatient or outpatient. It's not just physicians, it's chaplains and social workers and nurses and respiratory therapists and physical therapists. So it's not one discipline. So how do you find these? Um, and so that's that was sort of the first impetus of creating these standalone goals of care notes that use the technology at the time, which was essentially a form of checkboxes that was based on the serious illness conversation guide out of Ariadne Labs, um, which the institute partnered with, um, to create one place where people could document these conversations, but then we could also find them later. We could find all the notes that were goals of care notes. And that I think was also an important part of almost the perfect storm or the opposite of a perfect storm, to allow for this to be an institutional metric that was recognized at the executive level, because you can't ask for a metric to be created if you can't measure it. Um, and so we were able to say, hey, not only can we find these documents where people talked about these important conversations and put them in front of clinicians, which is probably the most important thing, but we can also take it and show which patients had them, at what point did they have them, who's having them. Um, and so we can actually show what quality measures are associated with having those conversations. Because if you can't find them, you can't associate them with any outcomes. Uh, so that was a really big part of how it kind of came to be so recognized as an institutional priority as well.

SPEAKER_03 23:55

Yeah. And so this uh the the serious illness conversation um borrowed from Ariadne Labs, and you mentioned kind of how it had a uh a very nicely kind of standardized templated version of this. I'm trying to picture this in my mind's eye as you're describing it. Is this, you know, as a clinician, is this like where I go to create my notes? So I'm I'm creating a note now as I would any other day, and I do a smart phrase, dot phrase, you know, pick your lingo, and it pops into a note. Is it like a whole different section? Um, and I have a follow-up question too.

SPEAKER_01 24:36

Yeah, so when it was originally rolled out, it was a whole other section. So it was not where you would do your normal notes. Uh, we had created, as Matt kind of said this, a single source of truth where we put everything related to advanced care planning. So we had information around code status, we had information around post forms, post or most or kpops or whatever you call it, um, and advanced directives and healthcare agents. But then importantly, we also had all of the documentation around goals of care. And from that, you could create the goals of care note. Now, I don't know if this is part of your follow-up question, but one of the pieces of feedback we got was that that was burdensome to clinicians, that it was not as efficient and easy as people would like it to be to go to another place to document their note. They didn't want to have to do that double documentation. They didn't have to write, want to have to write their progress note for the day and then go fill out a separate sort of template around goals of care. Um, so our second iteration of of documenting goals of care was to integrate it with uh sort of your daily documentation through a nested layer of tools, but essentially is a smart phrase dot phrase. Uh pick your smart term trademark epic. Um but uh you could build that into your uh you know progress note template, or you could add it, you know, just by writing dot GOC. And then there were a couple other uh uh quality initiatives we did building on top of that, but that was kind of the direction we went originally. We started with a standalone note and then we moved to an integrated documentation method.

SPEAKER_03 26:32

Yeah, you you covered it pretty well. So yeah, really it was around my my follow-up question, it was just around clinician reaction, clinician adoption. That's kind of usually where my QI brain goes is like what's this workflow gonna look like and how happy are people gonna be with it?

SPEAKER_00 26:50

Um this is the right question to ask because like we all I mean, like we're one group in a large health system. Uh, if similarly, like the cardiology group, the hospitalist group, the nephrology group, like if everybody has like one thing that they want us to, each like clinician to do, it becomes unmanageable. And we're all struggling with too short of an amount of time to be able to see our patients and interact with them and take care of them in the way that we want to. So it's it's exactly the right thing. And why I would say uh even those technologies have had their limits with us, and that's why I'm like so grateful and in awe every day of the work that Deb's been doing to integrate AI into our work because I I am excited that we it feels like we're finally at the place where we can like jettison and abandon some of the um teaching people how to use the electronic health record correctly and to really just get back to like taking really good care of patients.

SPEAKER_03 27:55

Yeah, there's definitely a lot of uh as you say, training people how to use the the EMR you know correctly or or the new version of correctly that we can kind of leapfrog when when things become a little bit more real-time, interactive, and intuitive. And I'm curious. So

Training Clinicians Then Empowering Nurses

SPEAKER_03 28:12

I think if if I remember this correctly, um early on, a lot of the work focused on training, I think training clinicians to do better goals of care conversations. And that I think that remained part of the work, but maybe there was more added to it. I wonder if you can go into that, Matt.

SPEAKER_00 28:30

Yeah, uh 100%. It's um really critical. So as I sort of alluded to, I have lots of different folks on our team. Um, and informatics is a key part of that, but we also recognized one of the barriers were was that clinicians didn't get enough training uh in their like formative training, you know, med school, nursing school, whatever, to have these conversations and feel confident about them. I myself like to med school in the early 2000s and I got 15 minutes of role-play education on this total. And like that's what I was expecting to be able to go and then break bad news to patients if we have. Um you know, I would suggest there's there there really needs to be a like a broader shift, and I know there is some broad shift at the like uh medical education level, a nursing education level that's happening. You know, the truth is that medications have changed dramatically in the past 20 years, and yet communication is still really essential. And so I I think that fundamentally we need larger policy change around helping people to communicate better. Just because you can talk doesn't mean that you can communicate well. And so we have spent a large amount of time in infrastructure. We've trained um nearly 5,500 of our clinicians in the serious illness conversation guide um over the last 10 years. We call it um not serious illness conversation guide, we kind of custom branded it for ourselves. We call it um at advanced communication training. We specifically chose those words because it turned out uh when you call it communication training, nobody really wants to show up for that, but you add the word advanced in front of it, and it suddenly everybody feels much more comfortable coming to a class because now they're not attending basic communication training. Um, so we we have nuanced it over many years. We used to do a two and a half hour live in-person class. We've now pivoted um several generations later from that to some online interactive modules and um sort of like triadic role play via um Teams or Zoom or whatever the platform folks are on. Um, most recently we're we're in another iteration generation change of rolling out a um AI-based conversation simulator um that has AI sort of trained patients to respond as a patient would so that clinicians can interact and really practice these techniques. Try saying, I wish, worry, wonder, or um teach back in the safety of role-play-like environment, whether that be human-to-human or human-to-ai, feels really essential to getting this right. Like the last thing we wanted to do was create some big quality project where we said we have to measure this. We we're expecting these communications or expecting these conversations to happen and not better equipping our clinicians to have those conversations. It felt like doing that would quite frankly set up patients for the potential for harm.

SPEAKER_03 31:39

Yeah, and and so the the training uh and and kudos on the on realizing that advanced was gonna be the uh the unlock there, I imagine the attendance probably uh shot right up as soon as as soon as you added the uh the advanced on there. I uh you know, I I'm curious then, once this training was up and running, did you notice that that um did that move the needle enough? Were there more layers that had to kind of come on top of that to keep things going?

SPEAKER_00 32:09

Yeah, there's definitely many layers to each of these things, and I think that's a lot of what we focused on in the catalyst paper of like it took we we had a lot of we've had a lot of success. So just to say um last year on the start health system, we had 120,000 of these goals of care conversations recorded. Uh over the last number of years, 85% of patients with an ICU length of stay of five days or more had a goals of care conversation that met quality conversation definition. We're a large health system, right? Like we have 50, 51 hospitals across seven different states. So it is it's a huge, huge effort, and it's taken many different levers. Um one thing that I found particularly fascinating was, and this was, I'll admit, like an early mistake for us, and part of it was the narrow vision-minded vision of like myself and others around the leadership of the institute being a little more physician-focused. And that's what we focused early on on physician training. And that did not move the dial at all. Um, and we did everything we thought we were supposed to do. We like paid uh docs to come in, we bought some fancy meals, we provided CME and like MOC point, like anything you could possibly think of. But um they were kind of voluntole to be there. And so the barrier to resistance was relatively high. What shifted the needle for us in our pilot hospitals where we started was training uh the floor nurses. That fundamentally shifted things. And over two years um in those two hospitals, we trained 91% of the acute care nurses with floor step-down ICU at the adult and on the adult side of the hospital. And that shifted things. And I think it's because nurses are really good at quality improvement. Nurses are really good at process, and so um, for instance, uh embedding in um reports, the sort of report structure within the hospital for nurses at the end of their 12-hour shift or the beginning, depending on your perspective. The question uh were goals of care discussed for this patient or later, what were the goals of care that were discussed? Report at Cosmone at shift change. So I admittedly, like if folks are listening to this and wanting to copy, yes, of course, the informatic is super important to be able to find and measure. Yes, of course, having leadership buy-in and being able to ensure that this is actually something that people care about. But um approaching it from a true interdisciplinary spirit and recognizing that like you can't just see it from your own discipline perspective. Like we have to reach across and figure out like how to do this well with others.

SPEAKER_03 35:04

It's exciting to hear how different I can just kind of imagine the feeling, like how how different things became once the nurses started to get involved, and how that may have had ripples out to other people that those nurses got to be in contact with.

SPEAKER_00 35:20

Yeah, and I had the privilege of like being in the hospital at the time. Like I was kind of deployed as like working in those hospitals and trying to um figure out what the real real problems were. It was fascinating to me the shift that happened. And we got really creative, you know. So one of the things that we did for a few, I think it was like four months, we had a nurse that was like really talented at these conversations. And unfortunately, you know, she was injured and was on modified duty and wasn't able to like do all of the normal bedside nursing care. And so we worked with the hospital leadership and she did goals and care conversations all day. And she was quite happy to do that because it meant that she was still getting to do something more than like typical modified duty of like restocking, you know, the back closet. She was still interacting with patients and families, she was still like able to do things and like helped coach actually some of the other nursing staff around doing this. So I think another key example of successful for us was having people that really cared about the initiative as like clinician champions in the actual hospital to learn like what works well and what doesn't.

SPEAKER_03 36:28

So I'm curious, and this maybe goes to both of you, but I'll think I'll start with with Deb.

Moving Beyond Checkboxes With AI

SPEAKER_03 36:32

Um one of the tensions that I think that that can you can kind of naturally imagine, uh, I think a lot of people would would think when they're when they're trying to be templated or having you know checkboxes about something that is so deeply human, um that there's a bit of a tension there. I'm curious how the two of you think now and and maybe if you're thinking has has changed at all over time on how you document these conversations in a standardized way without reducing the conversation to a checkbox.

SPEAKER_01 37:08

Yeah, it's such a good question. Um I spend a lot of my time actually thinking about that question because I I think we all know that as soon as you reduce a process to a metric and you put a checkbox on it, that you're very you're you're at risk for making it just a checkbox, that people check a box and they're like, I'm done, and abdicate like the rest of their responsibility. So what we did is this has been a very iterative process as culture has changed, as technology has changed. But when we first started, when we started with that sort of integrated note tool, one of the things that we quickly realized is that it is it is very tempting to do that checkbox do the minimum. And so when people were putting in notes, you know, sometimes we would get things like, oh, the patient was sleeping, or I tried to call their family and no one answered. And we're like, well, that's not a goals of care conversation. Um so the the next iteration included two check boxes that went through uh an extreme round of of governance and uh clinician input around what would be the most meaningful information, what was sort of the minimum requirement for what we would consider a quality goals of care conversation. And what came out of that is that you pretty much had to document who was there, who was part of the conversation to at least satisfy that you had a conversation with someone. Um, and an overall goal for their medical care, which could range anywhere from do all the offered aggressive medical interventions to focus on comfort with some middle ground for time-limited trials or, you know, uh some interventions that would not be acceptable. And then we kept the rest of that documentation completely narrative full text. So as long as you kind of documented at least those two pieces and then had some details of the conversation, we were capturing that information. And that really helped because it got rid of the I went by the room and no one was there, kind of um piece. And really got people to realize this had to be a full conversation, but it kept that flexibility and fluidity of still being able to document in a narrative format. Now, I will say over the years, to your point and your question, there's still some pushback around, well, you know, we would get feedback saying basically, oh, we have these conversations all the time. We just we were busy and we didn't use the dot phrase, or, you know, it's a new attending, they didn't, they didn't know to use the right template, or it was a fellow and they're just rotating here for a week and they didn't know about it. And how are we supposed to train them to do that? And so that is where we were kind of stuck for a few years. And then there was this revolution with AI and large language models. And so what we decided to do is leverage that technology because the whole idea behind large language models is they're so good with words and language that we essentially set up an AI model with a prompt. So we said, Hey, AI, you are now a palliative care doctor. I want you to read this note for me in my place and tell me whether the note has goals of care conversation documentation within it. And we gave it some boundaries, some rules. We said, you know, if it's just a conversation around code status, that doesn't count. If you're talking about a conversation that happened, you know, yesterday or a week ago, that doesn't count. It's only if it happened today. Um and we kind of defined a little bit about what we wanted for goals of care based on our prior quality work. You know, who someone had to be present for the conversation. There had to be some details about the plan of medical care. And we ran this pilot project with the AI. First, we tested it against expert annotated notes, where basically we had an interdisciplinary team of clinicians, two physicians, social worker, and a uh palliative care program manager read notes and say ourselves whether they contained goals of care information. And then we had the AI read it and we saw how well it performed. After we saw that it did pretty well with that, we we turned it into a pilot project here at Providence and ran it for every inpatient note written for six months at four of our hospitals. And then we we manually reviewed the notes that came out of that uh that uh the AI had tagged as containing goals of care information that did not contain our old method of documentation. And um, it did really well. It did surprisingly well. It found notes that were written by nurses and residents and radiation oncology and um obstetrics on high-risk pregnancies that we never would have found because we don't we don't always think to train people in using the.goc in those specialties or disciplines, or um, we know there are education gaps. And it was just amazing what came out of that. Um, we actually did some estimates of if the AI ran from our entire system for a year, it would have found not only all the regular notes that would have been documented in our old dot phrase way, but an additional 7,000 notes, goals of care notes for uh for the year. And so with that, we got full institutional buy in and we launched it um Providence wide, so to all 51 of our hospitals last December. And we're continuing to monitor it. It's done an amazing job. I will Say we in the first month, month and a half, it had read something like 5.7 million notes, found 30,000 goals of care conversations documented. And it was doing it not only in traditionally documented, typed or dictated notes, but also in ambient scribe created notes. So we're kind of creating this really cool full loop where you can walk into the room, your conversation is heard by the ambient scribe. The scribe writes your note, and then the AI determines the quality pieces of it that are present without you having to use a single checkbox, a single thought phrase, a single template. And it's really cool. I just think that it has a lot of potential for some of our other projects. But yeah, we're really proud of it.

SPEAKER_03 45:00

Yeah, I mean, it's exciting that there is this what to me, I guess as another palliative care clinician amongst the two of you, an unlikely friendship and partnership between, you know, AI, palliative care, informatics, who the heck ever thought that all of these different fields were going to join forces in such a like a actually in retrospect, probably an obvious way, right? Like the thing that the large language models do best is language, and that's the thing that we have that's been so elusive. And it also sounds like what I picked up on there as you were describing it is that using these AI models helped you to realize you didn't even have the right denominator because the the goals of care template you were using was the denominator until we realized there was a whole bunch of other non-dotes out there. Um what a what a cool experience. Matt, is there more to add? I you know the initial question was on how we document without reducing to checkbox. I think we went so far beyond that, but I'm curious to hear your thoughts if there's anything else there.

SPEAKER_00 46:11

Yeah, I think I think we've just gone so far beyond this at this point in time, which is wonderful. And I alluded to this earlier, but um I am so happy to stop like this teaching people the right way to put things in. I would have so many conversations of like, well, I had the conversation, but I didn't put it in the way that was like the approved way, and like, so you're just measuring the checkbox, you're not actually measuring whether I did it. And this this goes beyond that. Or um I had the conversation, but I just I forgot to like type dot goals of care. So like I you can't tell me that I'm not doing it. Um, and it turns out when we use an AI tool, those clinicians that said they were doing it, they're they're not doing it as often as they think they are, um, because it's not meeting the same sort of quality definition. I I will also point out for those that are are listening and think, oh, well, like I could never do this. That's gotta be um really hard to do. There were some challenging pieces for sure, but clearly they're solvable. We're running this model across, you know, all of our hospitals and it's reading millions of notes. Um and it's not that costly. So our cost estimate for the year is that it's gonna cost us $27,000 a year to run this, which yes, is uh uh dollars for sure. But for a health system of our size, $27,000 is not a lot of money to spend on trying to ensure that the stories that what matters to people that that is is delivered to clinicians and in one place so that people can read it and understand it and figure out how best to care for people. So it is um solution that feels like it wins for us around a lot of a lot of I'm gonna say checkboxes because it gets us away from the checkboxes. And now the question I have is like, how soon can we deploy this across all of our like outpatient clinics? Because that that's that's the next space that we need to move into.

SPEAKER_03 48:18

Oh, that's so cool. Uh and as

Costs Model Changes And The Next Frontier

SPEAKER_03 48:20

I was um reading through uh in background for this interview, I think the at least in your publication that you described using, I think, Chat GPT 4.0. And uh I, like probably many, many people out there, are are paying attention to the version numbers as they come out with uh 5.5 just coming out uh six days, I guess, before we recorded this. It was April 23rd, I believe. And that's a that's a big that's a number of steps up and uh and a big uh leap from from what 4.0 was, even though it was it was really great. Um are we like uh and even even with 4.0, you were getting you know incredible uh specificity and accuracy. Are are you guys in a place to be nimble enough to to use new new models as they're coming out? And and is that reaping any additional gains, or is it you know was it already good enough?

SPEAKER_01 49:16

Yeah, that's definitely um on our plan to do is to to test the validation with uh further models. Uh we haven't gone beyond 4.0 at the time because it it there is a delay in terms of when those versions come out for sort of commercial public use and using it in healthcare because it has to be approved to use for all of our you know privacy concerns and behind the firewalls and that it works with all of the um the infrastructure we have. Um we did we actually had a a little a little oopsie because um we had our model at one point auto updated to a new version, and then one of our like one of the data definitions didn't match because it had updated in the new version and we were like, oh, whoops. So we it is something that we're very aware of. We're trying to do deliberately though, not uh letting it sort of do that auto updating any further. But I think one of the big things is that we were get, as you said, we were getting such a good performance from the models that we had that there isn't a huge push to use the newer versions, particularly because if we continue to use a version that did well and is integrated, you know, into our systems and the price drops, that's even a better, you know, deal for us because as the newer versions come out, the price per token and things like that go down on the older versions. So it's it's kind of a win-win for us.

SPEAKER_03 51:11

Oh, that's so cool. So, you know, I'm also thinking, you know, I noticed that you you set kind of different levels of aspirations for uh how what percentage of patients meeting the definition of of ICU stay at least five days, um, have these documented goals of care conversations. And it was a more modest number to begin with, and then scaled up, I believe, at 90% at the highest level, which Matt, you already said was was it met and exceeded at uh system-wide and I think almost at every at every individual hospital, um, which is just you know really incredible. It makes me wonder like you had this great leadership support to begin with. The project has been evolving and has really made some great gains. Does that, I mean, does that start to change a lot of like the leadership conversations? You know, there's a whole lot of metrics we can point to that don't see such incredible success. And the fact that there's been this meteoric rise from, you know, certainly much less than 50% to greater than 90, 95%. Is that, I mean, does that really, you know, change the leadership conversations that are happening?

SPEAKER_00 52:21

It does. And I I would say so we we graduated actually just a year ago from being a system level metric, not because people didn't think it was important, but because we we had exceeded the goals uh for so many years in a row that people were like, well, why are we still measuring this? Like if we're gonna put quality bonuses on this look, we're already we're we're we expect to meet it rather than stretching in other areas. Uh I was nervous when that happened. Uh because when something falls off like that and it's been on for years, people assume, but I would say um we have continued to uh be able to uh meet those expectations. And I think it's um in large part culture change. Uh, you know, I didn't want to like you know say that years ago because I wasn't sure we were there yet, but it feels like culture has shifted, and that's not to say that we get it right every time, but I think we get it right more often than we don't. And I I also think um because we at the institute have a great team. I mean, you're listening to Deb, she's brilliant. Um, Melissa Forbin, who's our senior program manager that oversees this program, is incredible and smart and passionate and keeps working on this every day. And so you know, I had a mentor that that's shared that uh life is short and that you gotta work with people that you like or even love. And so I think part of our our secret sauce too is that um we've created a great team and have really great people that are passionate about this, and um, we continue to keep moving forward.

SPEAKER_03 54:02

There's so many more avenues that I I wanted to go down with this conversation, and I think we could we could keep it going for for quite a while, but I I wanted to round out the conversation to hear uh from both of you um what you think this work looks like five years from now, if it's reaching its its you know, its highest five-year aspirations. Maybe Deb will start with you.

SPEAKER_01 54:28

I I have a lot of aspirational thoughts when it comes to this, but I think we're really reaching this inflection point in both our documentation processes and then how we measure outcomes and quality from that documentation. Because as I said, I think it's becoming a much more seamless circle. I used to talk when I was talking about quality and documentation, I said there are three things that have to happen. First, for like we're talking goals of care conversations, the conversation has to happen. And then the second is that it has to be documented, and then the third is that you have to be able to find it later. And without any three of those three things, it it's a broken system. But with this advance in technology, all of those become seamless. You walk into a room, the conversation is recorded, so you don't have to say later whether it happened or not. It either did or it didn't. The documentation happens without you having to sit down and then be, you know, uh interrupted by another consultant and, you know, shorten your note and be like, I talked to the patient instead of writing down the full quality of that, the richness of that conversation. And then you don't have to use a checkbox for us to be able to find it later. And so I think we're gonna go into these systems where the way we look at quality fundamentally changes. And I think it's also gonna be amazing to see the ways that that then feeds back into our education and even real-time coaching. You know, Matt mentioned earlier we were working on uh in an AI um conversation, communication skills training product. What if in real time, as you're having these conversations, you could have a little bug in your ear being like, hey, you're using a lot of medical jargon and they seem really, they seem really mistrustful. What if you tried saying this? Or, hey, you know, two days ago someone had a conversation and they were very adamant they didn't want to talk about prognosis. Why don't you not bring that up today? And you had a little, you know, helper in your pocket for improving these kinds of conversations. Um, because as Matt said, this is something that we haven't really changed our training on in years, but it's so important. Um, you know, people are gonna, I think, you know, the classic quote is that, you know, people don't, you know, remember what you said, but they'll remember how you said it. People will remember and respond when we communicate well. And so having something that actually helps with that, be that either in our education or training or in sort of that real-time format, I think it's just gonna be a complete change in the way that we look at this.

SPEAKER_00 57:42

Yeah, I can't wait for that vision to happen. I'm excited about it for sure. And I agree, all of that is potential. Um I think a lot about to the future and the um the gaps that we still have, which is we can say a lot of these conversations happened, but we also can't yet demonstrate what that means for patients and families. And that is the gap that I think that like really bears closing with all of these techniques. And I think we're really close. For the first time, like truthfully, in my career, it feels like we or someone else will be able to accurately and adequately measure whether goal concordant care was provided. And that that's that's really exciting.

SPEAKER_03 58:32

Wow. I mean, what a what an incredible success and what a high bar to reach um on on both the the informatics technical fronts and and the human-centered care fronts.

Where This Goes Next And How To Connect

SPEAKER_03 58:44

I'm really grateful that that you guys would take this time to have have this conversation, and I want to give our listeners the opportunity if they if they wanted to connect with you or to follow your work to know where they should do that, and we can include appropriate links. Um Deb, where would be the the right place for people to find you?

SPEAKER_01 59:03

Probably email is actually the the easiest, but I also have a blue sky account, so I will provide that link.

SPEAKER_00 59:12

Yeah, for surprisingly, for technical folks, like we're like email is still like our where we mostly are and not on socials, but like at least you call it like by fax machine, Matt. Okay, that's fair. Our fax number is 888-555-1234.

SPEAKER_03 59:31

So you uh same same question for you, Matt. The uh email, is there anywhere else that we should uh I think email is great.

SPEAKER_00 59:38

Um I'm occasionally on on Twitter still, but um that's just mostly very, very, very ad hoc. So um we'll we'll definitely send your emails, Jason when folks can reach it there.

SPEAKER_03 59:49

Beautiful. Um well thank you both for for this uh deep reflection on what has been a really incredible effort, as you said, since since 2015, Matt, and really excited to uh see where this goes. Uh so uh, you know, the intersection of QI, palliative care, and and AI uh was not on my radar a few months ago, but I'm so glad we connected for this conversation. Um thank you both so much.

SPEAKER_00 1:00:15

Same, thanks for your time. Thank you.

SPEAKER_03 1:00:19

Thanks so much for listening to today's episode of Leading Quality. If you enjoyed the show, please take a moment to like, subscribe, and share it with someone who might find it useful. You can find all our episodes at leadingquality.budsprout.com or in your favorite podcast app. The show is written and hosted by me, Jason Meadows, edited by Milan Milosavievich, and produced by Thrive Healthcare Improvement. See you next time.