image: thumbnail.jpg
With a little help from my friends: Tools for developing & deploying algorithms in the hospital
Speaker bio:
Chloé Pou-Prom is a data scientist with the Data Science and Advanced Analytics (DSAA) team at Unity Health Toronto. The DSAA team uses high quality healthcare data in innovative ways to catalyze communities of data users and decision makers in making transformative changes that improve patient outcomes and healthcare system efficiency.
Timestamps:
2:37 - Start of presentation
4:41 - Projects their team works on
7:39 - Team is split into 4 teams (data integration & governance, advanced analytics, product development, and support)
8:00 - Successful deployment example (CHARTwatch - early warning systems for detecting patient deterioration)
12:09 - How this was running before (one person's laptop, no logging, no dev environment)
13:23 - Moving from before to now - What do you need for development?
13:40 - Connections to databases
16:25 - Reproducible environment
17:50 - Package based development
19:30 - Dev, staging, production environments
20:24 - What do you need for deployment?
30:21 - end of fire alarm break :)
33:07 - Authentication with Posit Connect (with hospitals Active Directory)
34:30 - Scheduling scripts on Posit Connect
35:35 - Receiving alerts when there's downtime (internal package for monitoring applications)
38:07 - Using internally developed packages with Posit Package Manager
40:58 - Implementation plan
49:29 - Monitoring
54:25 - Summary of development and deployment tools needed
56:12 - Start of Q&A
Helpful resources:
⬢ Slides: https://lnkd.in/ePEH5k6Z
Tools for Development:
⬢ Database connections [https://lnkd.in/eFDHR4cF]
⬢ Reproducible environment that lets multiple people collaborate on a project: [Renv: https://lnkd.in/e_DHddeA]
⬢ Package-based development [Functions and utilities that you re-use: https://r-pkgs.org/]
⬢ development, staging, production environments [https://lnkd.in/ezczn2Zz]
Tools for Deployment
⬢ Authentication: [Posit Connect: https://docs.posit.co/rsc/ ]
⬢ Downtime protocols
⬢ A secure way to download internally-developed packages [Posit Package Manager: https://docs.posit.co/rpm/]
⬢ Implementation plan
⬢ Monitoring ["Monitoring Model Performance": https://lnkd.in/eSUgRYNM &
"MLOps with vetiver in Python and R": https://lnkd.in/efN4vcrg]
About Data Science and Advanced Analytics at Unity Health Toronto:
⬢ https://unitynet.unity.local/departments-programs-services/corporate-services/data-science-and-advanced-analytics/
About CHARTwatch
⬢ "Preparing a Clinical Support Model for Silent Mode in General Internal Medicine": https://lnkd.in/eJAWmYWk
⬢ "Implementing Machine Learning in Medicine": https://lnkd.in/eFesPijY
⬢ "From Compute to Care: Lessons Learned from Deploying an Early Warning System into Clinical Practice": https://lnkd.in/e-D7aDDf
Want to join us at a future event? pos.it/community-events
Transcript#
This transcript was generated automatically and may contain errors.
Hi, everybody. Happy Tuesday. Welcome to the Posit Enterprise Community Meetup. If you just joined now, feel free to say hi through the chat window and maybe where you're calling in from. I'd also like to say, if you want to connect with each other in the chat, that's all for you, too. So feel free to use the chat to meet other people who are joining us today as well.
Special welcome to those joining us for their first time today. This is a friendly and open meetup environment for teams to share use cases, teach lessons learned, and just meet each other and ask questions. These happen on Tuesdays at 12 Eastern Time, not every single Tuesday, but we do have a community event calendar, which I'll share in the chat in just a second here. You can also use the link shared on the screen here, too. Together, we're all dedicated to making this space inclusive and open for everybody, no matter your experience, industry, or background.
During the event, we'd love to hear what questions you have as well, so you're able to ask questions on YouTube Live and also anonymously through Slido, which I will share on the screen in just a second here as well, the link, and I'll put it in the chat. To address one common question up front, yes, the recording will be available, and it will actually be ready right away, like immediately as the presentation ends at the same exact YouTube Live link.
At one of the recent data science hangouts, which happened on Thursdays at noon Eastern Time, I was able to get a little bit of a view into the amazing work the team at Unity Health Toronto is doing to bring predictive analysis to different clinical problems and resource problems in the hospital. And I'm so excited to be able to dive into that deeper today with Chloe Pupron.
Chloe is a data scientist with the Data Science and Advanced Analytics team at Unity Health Toronto. Their team uses high-quality healthcare data in innovative ways to catalyze communities of data users and decision makers in making transformative changes that improve patient outcomes and healthcare system efficiency. I'd love to bring Chloe up here on stage with me, our virtual stage here. Thanks so much for joining us, Chloe.
Thank you for the introduction, Rachel, and thank you all for joining. I'm really excited to be talking about tools and insights for developing and deploying algorithms in the hospital.
So actually, as I was developing the slides, there were a few alternate titles I had considered. Another one that I had considered was this one right here, which is how we got our model out of one person's computer and deployed it to the hospital, which I think is an apt description of the things I'll be talking about for the next hour or so.
So I guess for the next while, I'll spend a bit of time going over DSAA, who we are, what we do, where we started versus where we are now, and really just diving into the main content of the talk, which is where are the things that you need for development and where are the things that you need for deployment? And all of this, of course, is in the context of working in the hospital, working in the health care setting.
About DSAA and Unity Health Toronto
Yeah. So hi, everyone. I'm Chloe and I'm a data scientist for the DSAA team at Unity Health Toronto. So DSAA or rather the Data Science and Advanced Analytics is a health care data analytics group at Unity Health Toronto, also known as UHT. Throughout the slides, there are a few acronyms. I'll try my best to expand them. I think acronyms might be like a data science thing or maybe a hospital thing. I'll try my best to expand all of them.
So basically, DSAA's aim is really to work with Spittel, work with our collaborators and also our partners to make better decisions, increase hospital efficiency and improve patient care and patient outcomes. So in particular, because we're working in the hospital, the projects that we work on will require process and change management. So that's why we work directly with clinicians and with administrative decision makers to develop and deploy our solutions.
Now, what kind of solutions do we work on? A little bit of everything. So we have solutions that will rely on statistics, some that focus more on artificial intelligence, some on machine learning, some on optimization, like operations research, for example. Currently, we have more than 40 active solutions at Unity Health Toronto, and these are, for example, solutions for predicting patient outcomes for enhanced clinical management, planning for hospital bed capacity. We have some medical imaging AI tools and we also have some solutions for assignment and scheduling.
Here is an example of some of the things that we've worked on. So right here, I have a screenshot of our COVID dashboard. So here we're reporting are the counts of people who are COVID positive, people who have been exposed to COVID, as well as people who previously had COVID and are now recovering. And in this case, we're showing the counts for the three different sites that make up our hospital network.
So Unity Health Toronto is based in the Greater Toronto area and it is made up of three different hospitals. So we're showing the numbers for those three hospitals.
Here we have an example of our warfarin dosing tool. This is an application for calculating the warfarin dose based on various input features. So warfarin is a medication, so it's a blood thinner and we're calculating the dosing based off different input features. And then we have our emergency department forecasting tool. So here we're counting the number of patients that are arriving to the ED. So on the right hand side, you can see we have our forecasted numbers and we're also reporting the historical numbers. So you can get a sense of how well the model is doing.
So who are the people that work within the data science and advanced analytics group? So our team is actually made up of four different teams. So we have a data integration and governance team. So they are the ones who will take the data and will connect to the database, transform the data in a way that is amenable for data science. And then we have the advanced analytics team. So they're the ones who are taking the data and building a model on top of it. And next we have the product development team. So they're the ones who are deploying the model.
So they're also the ones that are handling any kind of like downstream integrations. So, for example, if you have a model that is supposed to send a push notification, the product development team will be the ones who are taking care of that. And finally, we have the support team who are really the glue that hold everything together. So they're the ones who take care of project management and coordination at both an internal and external level. So because we're collaborating with lots of different groups within the hospital, this is where having the support team is really helpful.
ChartWatch: an early warning system
Okay, so I briefly talked about some of the different solutions that we worked on. I want to kind of dive in and focus on one solution in particular, which is a successful deployment known as ChartWatch. So in August 2020, we deployed ChartWatch, which is an early warning system for detecting patient deterioration. So it's been running since August 2020. And to this day, it's still running in the hospital. ChartWatch was deployed to the general internal medicine ward within the hospital.
I guess, as an aside, why is it called ChartWatch? Really, it's a branding thing. So before becoming known as CSAA, we were actually known as Chart, which is the Center for Healthcare Analytics Research and Training. So one thing you may notice throughout the talk is like I do mention a thing known as Chart, and it's really because of who we were initially known as.
So we developed ChartWatch. It's an early warning system. And really, it's a model that predicts which patients are at risk of deterioration. So what I mean by deterioration is any of the following. So either transfer to the intensive care unit or ICU, death in the hospital, or transfer to the palliative care unit.
So the way that the ChartWatch model was trained is it was trained on about 20,000 patient visits. And these data consisted of laboratory values, vital measurements and demographics. So these are all things that you would typically find in an electronic health record. And the way the model works is it outputs a risk group. So for every patient, you would generate a risk group. So the patient will be classified as either being high risk versus medium risk versus low risk. And then these risk groups are then delivered to different end users. And you have different methods of delivery based on who the predictions are going out to.
So, for example, we have emails. These are sent to our charge nurses, our palliative care team. And what these emails contain is basically a list of patients with their risk group listed. So if you look at the column, the third column from the right, you'll see the ChartWatch risk groups are included here. Another method of delivery is updates to the front end tool. So here we have a screenshot of our electronic sign up tool, which is typically used by clinicians. And if you'll notice here, the screenshot has a column for ChartWatch. So once again, the third column from the right, you'll notice there is a ChartWatch column and this contains the ChartWatch risk groups. And in this case, the risk groups have been color coded. So high risk group color coded as red, low risk group as green and the medium risk group as yellow. So we're following the traffic light system in this case.
And then finally, the last method of delivery is alerts that are sent to phones. So for patients that are flagged as high risk, we actually sent out an alert to clinician phones to let them know that the patient is at a high risk of deterioration. So going through the different methods of delivery, really, they're varying the level of urgency based off the end users that they're going to and based off the information that's being communicated. And you can read more about it in these links below. The slides will be available to share and you'll be able to access these links. And also at the very end, I have slides with all of the links, resources and papers I shared throughout the talk.
From one person's laptop to a deployed system
OK, so I've gone over ChartWatch, but I have this slide just to really pause, take a step back and reflect on how we got from where we were to where we are now. I should probably title this slide back to the past because we are going to the past. So just to give a bit of context, this is where things were at before. So before we had scripts that were running from one person's laptop. It works, but lots of ways in which this can fail. So, you know, what if this person is sick? What if they go on vacation? Most of the laptops we use are Windows laptops. What if we get that Windows update? You know, like the laptop will be out for a while. We have no logging. We have no development environments. That's how things are before.
Now, where things are at now. Well, you know, things are much better. We have service accounts. So basically, this removes the reliance on one person. We log all the things. We have staging versus production environments and so much more. So how do we get from before to now?
Now, where things are at now. Well, you know, things are much better. We have service accounts. So basically, this removes the reliance on one person. We log all the things. We have staging versus production environments and so much more.
I'd say really it's a combination of hard work, trial and error, lots of talking, talking to the right people, figuring out what works and also what doesn't work. And I'd say out of all the tools and findings, I think we can summarize them into two categories. So we have the tools for development and also the tools for deployment. So let's dive right in. What do you need for development?
Tools for development
So the first thing that we need for development is really connections to databases. So within the hospital, there are various data systems. So, for example, we have a different system for labs, a different one for surgical procedures and so on. So in order to help the data science team work better, we developed a package called ChartDB. So ChartDB is an internal R package to interact with hospital databases.
And with regards to connecting to databases, we wrote a bunch of connection functions and they all follow the same pattern. So, for example, if I want to connect to database A, I just need to give us input, my username and my password. If I want to connect to database B, I also give us input, my username and my password. And if I want to connect to the hospital's enterprise data warehouse, or rather the EDW, I would give, as you expect, my username and my password as input to the function.
So internally, this is actually what's happening. Internally, we have a bunch of retries because, you know, the connection may not work the very first time. We also have a bunch of checks, so making sure that the connection is successful and that what's being returned is what you expect. So ChartDB offers connection functions and it also offers other utility functions.
So, for example, I have this get patient ADT function right here. And what ADT stands for is Admit Discharge Transfer. So, basically, when I call this function, it will return a list of patient movement events. So I have a row for when the patient first gets admitted to the hospital, when they transfer, for example, from the emergency department to a medical ward, and then when they get transferred. And, you know, knowing where a patient is located in the hospital is pretty useful in nearly all of our projects.
Reproducible development environment with renv
All right, so we have database connections. Another thing that we need for development is an environment that lets multiple people collaborate on projects. A really reproducible development environment.
So in order to help with that, we ended up using renv. Here I have a link to renv and here's a screenshot of the documentation. As I said before, at the end, all of the links will be included in the slides. So renv is an R package for R dependency management.
And how are we using our project? Well, here's an example of our project structure. So, you know, we have our main folder, we have a bunch of files, a bunch of subfolders. And really, I want to bring attention to that last file at the end, which is the renv.lock file, which is where all of the renv magic happens.
So this is what the renv.lock file looks like. And basically what it does is it specifies the different packages that your project relies on, as well as the different package versions. So, for example, here, my project relies on deep IR and I'm using version 1.0.7.
Package-based development
Okay, so I've highlighted a few things for development. So I have my database connections, I have my reproducible environment, which is facilitated through renv. Another thing that I need is functions and utilities that are reused. So really what I mean by that is package-based development.
So here I have a link to the R packages textbook, as well as a screenshot of documentation. I highly recommend it. So really, why write a package? There are a bunch of reasons, but I'd say that for us specifically, what we found the most relevant was being able to share code and knowledge with others, as well as having no more copy-pasting.
So I find that as I'm developing, if I'm copy-pasting the same bit of code over and over, kind of like, you know, it rings an alarm. You're like, maybe I should take a step back and think of what I'm doing.
So I find that as I'm developing, if I'm copy-pasting the same bit of code over and over, kind of like, you know, it rings an alarm. You're like, maybe I should take a step back and think of what I'm doing.
So if we revisit the project structure that I introduced a few slides ago, this R subfolder right here is where you have all of your package functions. And then there is the description file right here, which is where you would define the attributes of your package.
So for example, this is a description file that I have for ChartWatch. So, you know, it's listing out a bunch of attributes like the package name right here. So it's ChartWatch. The package version 1.11.10. And at the very end, I have the imports right here, which specifies the packages that my package relies on. So the dependencies.
Working across multiple environments
Okay, so our development checklist is, we're adding more to it. So we have our database connections. We have an environment that allows multiple people to collaborate. We have package-based development. Another thing I'm going to add is environments, environments, environments. So the reason I say the same word three times is because we actually work with three different environments. We have the development environment, the staging environment, and the production environment.
So the development environment, this could be your local computer. But in our cases, we usually use a development server. So what our development server includes is GPUs, which allow for any projects that require deep learning. And also, the development server works nicely with ChartDB. So that means that anyone who wants to develop something, you know, they're guaranteed that database connections will work for them. So they'll be able to access the data that they need for their projects.
Next, you have the staging environment, which is as close to the real deployment environment as possible. So for example, when we need to make updates to ChartWatch, we would first apply it to the staging environment to make sure everything is working as we expect it. And then finally, there's a production environment, which is where things actually get deployed.
ChartWatch deployment story
All right, so things I need for development. The last thing I added was a thing about environments. Briefly talk about the production environment, which I think segues nicely into the next point, which is what do you need for deployment? So actually, before I dive in, I'm just going to take a small intermission so I can reflect a bit on ChartWatch's deployment.
So ChartWatch was deployed on August 2020. But it was actually first silently deployed in November 2019. And I guess the reason for this large gap is because initially ChartWatch was supposed to go live in early 2020. But then the pandemic happened, it affected plans, the hospital's priorities shifted at that point, you know, they wanted to focus on containing the rising number of COVID cases in the hospital. And then, you know, ChartWatch kind of like took a bit of a backseat.
So yeah, it was first silently deployed in November 2019. And this is actually here a screenshot of what the ChartWatch repository looked like back then in November 2019. So the very first time we tried to deploy it. I guess a few interesting things that I want to highlight here. So first of all, there are two folders that start with 00. Interesting.
Chloe warned me that this might happen. Don't worry, they are testing. It's not a real fire. Chloe, I mentioned I could fill the time if that happened. I'm not sure how long the fire alarms will go off. But I did just want to remind people that you can ask questions if you're watching on YouTube Live. Or you can also use the Slido link, which I will show on the screen for us as well.
I see that there's two questions over there now. So I can read them out for you, Chloe, so that they can be top of mind as well. I think you might be covering this in just a second here. But one of the questions over there was, can you talk more about the development process for this? What came first? Were there steps dependent or parallel? Or did these changes happen simultaneously? And if the alarm is still going off, no worries, we can cover it later.
Okay, it's still going up. Well, I will use. Okay, awesome. I will use the time to also just let you know, I usually share this at the end of the event, but I saw somebody asked a question about it as well. So if you are interested in figuring out where to find upcoming events. I created a, this is actually a Quarto doc here that is published to our Connect instance. But it has a nice calendar where you can see everything that's happening in the next few months. So the meetups that happen on Tuesdays, but also data science hangouts that happen every Thursday at noon Eastern time.
And so this Thursday, we have Eric Nance joining us from Eli Lilly, as our featured leader for the week. So at the data science hangouts. I'm curious if anybody listening in has been to one, maybe comment in the chat. But there's a different data science leader that joins us every week. And just answers questions from you all. So maybe you have questions about how they're thinking about hiring or how they're structuring their team or about different technologies that they use. There's no presentations at the Thursday events. It's all just kind of open conversation among everybody and lots of questions.
Definitely recommend joining one of those two. And I see Libby says, Yay, Eric. Yes, I'm really excited for Eric to join us this Thursday as well. How are we doing Chloe on the alarm? I think it should be okay. If Eric comes back, I'll go back on mute. I saw there was a comment about like, is that smoke behind you? No, but it might be snow because it's snowing outside. So maybe that's what's behind me.
And I see I just mentioned the community events and I showed the link on the screen. But I also just copied it over to the chat if you want to check that out there too. But I will hand it back over to you, Chloe. Thanks.
Thank you. All right, where was I? Right. So I mean, it's funny if the alarm happened during the pause storytime period. Yes, I think you were going over. This is what the repository structure looked like when you first tried to deploy it. I was just finding out a few interesting things about that.
So just to recap, there are two folders. I start with 00. Interesting. At the very bottom, there is that error check.py file. So there is Python code here. And then also I have my .rproj file at the very bottom. So there is R code. And actually, if you look at each of these different subfolders, you'll find a mixture of R, a mixture of Python. I believe we also had some Java code at that point.
Oh. Sorry, I thought I heard the alarm. I think we're at almost at the end. I think there was some saying that it was still testing. Okay. Awesome. Bear with us here. Just a few minutes.
Okay, let's try. I think it should be over. I can't hear it.
Okay, let's see. We can take a few more questions from people, too. Any questions for me? Let's see. This is really scary. But I can chat for a little bit as well while we're waiting. How about any ideas for meetups or things that you'd really like to learn a little bit more about as well in the future? We're starting to think ahead for 2023. So I'd love to hear from you all.
Feel free to put some suggestions into the chat. I also can go ahead and make a poll over on Slido right now, too. I'm going to say any suggestions for meetups in 2023. What would you like to learn?
Okay. I just launched that poll over on Slido. Ooh, yes, Libby, I think supply chain topic would be great as well. Yes, thanks, Rebecca. We can all use a pause sometime. Everybody can stretch, get up, get some water.
Oh, Marlene, that's a great suggestion, too, how organizations are using new packages. I was also talking with a team recently who was talking about, like, creating their own company-wide Tidyverse packages. So, like, in a sense, Tidyverse, but packages that are important for their company and, like, company branding, too, which is cool.
Any other meetup suggestions? I know there was a ton of interest in the people analytics topic from last week, too. So I was just talking with a few people from the people analytics space about some future events for that as well. Oh, and yes, Jake, thank you. Yeah, suggestion on packaging app content with Golem as well. Yeah, on the topic of incorporating branding, I see, Marlene, you just put it into the chat. I was curious if people do have packages at their own, like, your own internal packages that include your company branding. Is that something common?
And feel free to put everything into the chat. Just a reminder, like, the chat is great for questions, but also to, like, share resources and links with each other as well. Chloe, how are we doing on the alarm?
I think it should be good, but I said that last time. Let's see what happens. Okay. So I'll just try yelling. See if you can hear me over the alarm. Yeah. So, actually, just want to address, like, there are some really great, great points in the chat. I really like what you guys are saying about branding.
So that's kind of, like, one of the things that we try to do with our, like, our ChartDB package. Yeah, definitely like that. Yeah, going back to how things were before. So when we first tried to deploy ChartWatch, we had a mixture of Python and R scripts and also some Java. And the way we were running everything was we had this complicated thing. Whoops.
Oh, sorry, there's a bit of lag in my slides being updated. Yeah, so, yeah, the way things are running is we had a mixture of Python and R scripts, and we had this cron job, which was calling different bash scripts, which was then calling different Python and R scripts. And then I guess where we were, the biggest red flag is that our test environment was our staging environment, which was our production environment.
So, for example, in ChartWatch, one of our outputs was an email which contained a list of patients with their risk group. And you can imagine, like, as I'm developing this, I would send that email to my email and, you know, make sure things look as they should be.
But because our test environment was our staging environment, was our production environment, when it came time to finally deploy, I would have to, like, comment out the line that had my email and then set it to be, you know, whatever test email we had. And presumably when we would move to production, I would then comment out that one line and set it to, like, the end user's email.
So, you know, like, if I'm very diligent, if I'm very careful, things will work out. But this method is very prone to errors.
So, with that being said, what are the things that you need for deployment?
Authentication and scheduling with Posit Connect
So, the first thing that we really need for deployment is authentication. So, all of our deployed applications run on RStudio Connect, which is actually now Posit Connect. So, one of the nice things about Posit Connect is that it interacts with the hospital's active directory. So, what this means is that users can authenticate using their hospital username and password. So, for the developer, you know, we don't need to keep track of an extra server username and password. And then for the end users, in order to access applications, they could just log in with their hospital credentials.
And a nice thing about being able to interact with active directory is that we can just use the existing groups to manage permissions. So, for example, if I have an application that I only want to give access to the SAA, so like the data science group, I could just use the existing active directory group to enable that.
Another thing that RStudio Connect, or rather Posit Connect, allows us to do is to schedule scripts. So, we run our applications from service accounts, which is basically a non-user account in which we can run automated services. And it's this combination of automatic scheduling and using a service account that really makes it so that deployments don't rely on one person. So, you know, this is how things were before we had this cron job, which would call different bash scripts, which would then call different Python or R scripts.
But now that we're using Posit Connect, we can leverage a lot of different things. Sorry, the screenshot is a bit small, so I listed all the different things, all the different tabs on the side. So, you know, we have the application information, we have the access tab, which is what we use to specify, for example, who can access a specific dashboard. We have the runtime settings, we have the scheduling settings. So, scheduling settings is what we use to schedule our scripts. And then there's the tags, really for organizational purposes. So, for example, all the ChartWatch scripts would get tagged as being ChartWatch environment variables and logs.
Monitoring and downtime
So, we're adding a few things for deployment. We have authentication, we have scheduling. Another thing that's useful for deployment is being able to know when there's a downtime. So, for example, here, this is a screenshot of one of our Slack channels where, you know, we received an alert, you know, ChartWatch is down, for example.
So, in order to enable this, we created Jarvis, which is an R package for helping us monitor our production applications. So, Jarvis is the IRONMAN's robot assistant. Yeah, that's where the package name came from. Yeah, so we use Jarvis for a few notifications. So, we can use it to send email alerts. So, in this example here, you know, Jarvis sent email letting me know that there's an issue with the COVID dashboard. As shown previously, we can use Jarvis to send Slack messages, like here. And we also use Jarvis to check the health of systems that we depend on. So, for example, if you want to check a specific database, right, I can check that my connection to the enterprise data warehouse still works. And I can also check that I can access different file systems with Jarvis.
So, on top of using Jarvis for different notifications, we also developed a few downtime protocols, which are pretty project specific. So, for example, with ChartWatch, if ever there's something that goes down, we would send this kind of email to the hospital, to all the hospital users. So, we have a specific email template that we use here. And this email template actually follows the one that IT uses. So, you know, it's color coded at the top where, you know, red is for unplanned downtime, blue is for planned downtime, and so on. There's a list of things that we need to specify. And this will go out to all the hospital users.
So, like, not all projects will require something like this. But something like ChartWatch, which is embedded into clinical care, does require something a bit more robust. Basically, like, when things go down with ChartWatch, it's not just the DSA team that needs to be notified. It's other people within the hospital.
Secure package management with Posit Package Manager
All right, so you have authentication, scheduling, downtime. I'd say another thing that you need for the deployment is a secure way to download internally developed packages. I think I need to expand on this a bit more.
So, to give a bit of context, around the holiday time last year, we started seeing these kinds of messages in our Slack channels. So, you know, like, not able to receive emails, you know, check log4j. Sorry, running late because I was on a meeting with IT because of log4j. If you use PyCharm, be careful. There's a log4j file in there. I found log4j files in my VS Code folder. There's something going on with log4j.
The log4j, for those who don't know, it's a Java-based logging utility. And in December 2021, there was the log4j flaw. And basically what happened is this could allow malicious users to access internal networks. And I guess just to highlight the severity of the log4j flaw, like, here are some headlines that were coming out around that time. So log4j is a pervasive vulnerability. Update your devices now. A hole in a popular piece of code is an open window for criminals. The log4j software flaw is endemic, according to the Cyber Safety Panel.
So really, this whole thing about log4j is to motivate the next point is that we need to limit who can access the hospital network. And this is where something like Posit Package Manager comes in. So Posit Package Manager, also formally RStudio Package Manager, is a repository management server. And what it allows us to do is download packages while being disconnected from the Internet.
So if I go back to renv, which I mentioned a few slides ago, remember we had our renv.lock file, which specifies the different packages and versions. So here it specifies how the package is being downloaded. So instead of being downloaded from, say, GitHub or from CRAN, we're downloading from RSPM, which stands for RStudio Package Manager, which is also Posit Package Manager.
So really, this whole thing about log4j is to motivate the next point is that we need to limit who can access the hospital network. And this is where something like Posit Package Manager comes in.
Implementation plan for hospital deployment
So a few more things that we've added for deployment. So we just added a secure way to download internally developed packages. One more thing I want to add is an implementation plan. So, so far, all the things that we've discussed previously, I guess you can apply to data science in general. For this implementation plan, it's probably a bit more focused on the health care setting and on deploying within the hospital.
So just to recap how ChartWatch is delivering predictions, they're going to different end users. So we have our emails that are going to charge nurses, the palliative care team. We're updating this electronic sign out front end tool that clinicians access. We're sending alerts whenever there's a patient that's flagged as high risk. This is going directly to clinician phones. And really, if you like, just looking at the way things are delivered, there's a lot of people involved. So you have like your church nurses, your palliative care team, you have I.T., you have residents and physicians.
And it's with that in mind that when we when we came up with what we put together, the implementation team, we wanted to make sure that people from all of these different end user groups are being represented and included in these implementation team meetings. Right. So we made sure we had people from general internal medicine, from the intensive care unit, from palliative care, from clinical informatics, from I.T. and then like from the data science team.
Just want to highlight that collaboration is really important here. And also the whole idea for ChartWatch actually came from a general internal medicine doctor. So they were the ones that are leading the project throughout the whole course and really making sure we were going from development all the way to deployment.
So one of the things that the implementation team was tasked with doing was coming up with an intervention. So after ChartWatch flags a patient as being high risk, what happens then? So the implementation team develop a clinical workflow. This is a clinical workflow looks like. I won't go into too much detail, but I will point out a few things. So, you know, when a patient is flagged as being high risk by ChartWatch, there are a few things that need to happen after that. So one of them is placing orders to increase vital sign measurements, reviewing recent labs, reviewing recent medications and recent imaging orders.
And really all the things that you find in this clinical workflow are all things that a clinician would normally do if they suspected a patient of deteriorating. So when developing this, this checklist, this clinical workflow, it was really important to make sure that it wasn't disrupting any existing processes. So you want to consider existing resources. You don't want to reimplement a workflow that already exists, for example. And also the alerting notifications all fit within existing processes.
So, for example, when it came to the timing of emails, some of the emails that we sent out is sent to the charge nurse and this email is sent out twice a day. And they're sent at a time that makes the most sense with how the shifts work. So once again, we don't want to disrupt how things are working currently.
And with regards to the clinical pathway, really ensuring that there are time targets. So, for example, after ChartWatch flags a patient as being high risk, one of the things that the clinicians must do is increase number of vital measurements within 24 hours of the alert being sent. So having this 24 hour time target makes it that we can then go into the electronic health record and then measure whether or not this is happening. And, of course, leaving room for clinical judgment.
And so when working on the implementation plan, we include a signed deployment period. So basically, this is where your entire pipeline is running, but the final outputs are silenced. This is useful for identifying bugs and addressing unexpected changes, such as change to how the Troponin lab test is done.
So the Troponin test is one that's commonly used to diagnose heart attacks and other heart related problems. So during the deployment of ChartWatch, like the way this is measured got changed. And this is something that we have to address. So we have to address that in our data extraction code and also in our data processing code.
Other changes that can happen can be due to deploying at the beginning of the pandemic. So, you know, during that time, there's lots of rooms that are being made available or unavailable based off, you know, the patients are testing positive for COVID. Patients are being moved to completely different units in the hospital just based on capacity. So this has to be addressed in our data extraction code.
Catching bugs during the silent deployment period
Another thing that the signed deployment period was useful for was for catching bugs. This is a really interesting bug that we encountered. I'm guessing that most people that are watching this talk are R users. So, you know, I can ask the following question, how are missing values represented in R? So I think most of you know it's an R missing values are represented as MA, which means unavailable.
Now, next question is, what's the chemical element for sodium? I think some of you can see where this is going. So the chemical element for sodium is also MA.
So we work in the hospital. In the hospital, there are blood tests that are done. And, you know, one of the things that gets measured in blood tests is sodium. So when we silently deployed ChartWatch, we noticed that there were no sodium tests that were being measured. This is an example. This is a graph that shows kind of all the electrolyte counts. So in blue is sodium. And if you notice here, the blue is set to zero. OK, like this doesn't seem right. Like what's going on?
So when you went and checked the code, what we found was that the R code that we were using to read some of the lab's data was interpreting all of the sodium tests as being MA not available. So yeah, this is what happened. The fix was actually pretty straightforward. It was just like updating one of the function parameters to not read MA as not available. It was just a very interesting bug that we ran into.
So when you went and checked the code, what we found was that the R code that we were using to read some of the lab's data was interpreting all of the sodium tests as being MA not available.
Going live with a phased approach
So once we were done with the sign period, it's time to go live. So actually for us, we had a pilot phase in which we first deployed to two general internal medicine teams only. And during that time, we had weekly meetings which were used to debug any issues we ran into, go over the patients that were flagged by ChartWatch, you know, check like, does it make sense or is there anything weird or unexpected? So that was the approach that we took for going live.
And then we really did in a phased approach to start off with two teams and deploy to the rest of the teams. And really throughout this whole process, so throughout the pilot phase, throughout deployment, throughout, you know, developing the clinical workflow and user engagement is important. This is not something that could have been possible with just a data science team.
So when it came to developing the clinical pathway, we really relied on the general internal medicine team, on the palliative care team, on the nurses we were working with to develop that pathway. And also when it came to developing education and training processes, that was, once again, a highly collaborative effort.
So yeah, so things need for deployment, implementation plan, super important.
Monitoring in production
One last thing that I will mention for deployment is something about monitoring. There's a question mark here because, you know, this could be a whole talk and we're already nearing the end of this talk. So it is a whole talk. There's plenty of other people that have spoken about monitoring. So I would redirect you to watch these amazing talks. Lots of great insights. With the time remaining, I will mention a few things specific to chart watch, though.
So a chart watch, one of the things that we want to look at is model performance. But that's a bit hard to do once the model has been deployed.
Hmm. Somehow we got kicked out there, but here you are. Am I still here? It was just for a second. Yep. OK. All good? Yep, we're good.
OK, so we just backtrack a bit. So with chart watch, one of the things that we want to look at is model performance. But that is difficult to do once the model has been deployed. So clinicians, one of the things that they asked for was a target positive predictive value of 0.33. So what this means is that for every three alarms, at least one of them should be correct. But that is kind of hard to measure once you are deployed and in production.
So what I mean by that is, let's say chart watch flags a patient as being high risk and sends out an alarm. And then that patient doesn't deteriorate like they don't transfer to the ICU. They don't die. Does that mean that the chart watch prediction was incorrect? Or does that mean that the chart watch prediction was correct? And then because clinicians received that alert, they were able to intervene and prevent that outcome from happening.
So, yeah, in deployment, we we get these cases where there are these false alarms, which we're not sure if they're actually false alarms or if they're averted outcomes just based off this successful intervention.
So, yeah, in deployment, we we get these cases where there are these false alarms, which we're not sure if they're actually false alarms or if they're averted outcomes just based off this successful intervention.
Yes. Another thing we measure in chart watch is the daily number of alerts. So on average, we're sending two alerts per day, which is a reasonable number that the clinicians are happy with. There are some days where there are no alerts and there are some days where there are lots of alerts. So, for example, I think there's like nine or eight alerts sometime in like April or May.
So, yeah, one of the things that we were really concerned with was make sure there weren't too many alerts. So we worked with the clinicians to ensure that we were minimizing alerts. So right now, the way things work is that, you know, a patient will receive an alert if chart watch classifies them as being high risk. We added some additional rules on top of that. And those rules we develop with the help of the end users. So, you know, we don't re-alert a patient unless it's been 48 hours since their previous alert. You know, this just ensure that we're not alerting for the same patient every hour, for example.
And we added this additional rule, which is that we don't alert a patient if they just came from the intensive care unit. Because if a patient is coming from the ICU, you know, like the clinicians already know that they may not be doing too well. And it's expected that, you know, their vital measurements, their labs might be looking a little off because they just came from intensive care.
The last thing I will say about monitoring is regarding clinical adherence. So how do we know that when a patient receives a chart watch alert, the clinician is actually doing something and they're not just ignoring that alert? So one of the things that we look at is the number of vitals that are measured after an alert is sent. So we do that because as part of the clinical workflow, one of the things that the clinicians have to do when they receive a chart watch alert is increase the number of vital measurements. This is something that we can easily measure just by looking at the electronic health care record, looking at the number of vitals that are sent after an alert has been sent.
I mean, it's not a perfect measure because the number of vitals can increase for reasons unrelated to chart watch, but it's a proxy and it gives us a good enough idea of clinical adherence.
Summary and closing
So this brings us towards the end of our talk. So let me just summarize some of the things that we talked about. So we looked at the different tools for development and deployment of an early warning system in the hospital. And so on the development side, the things that we found useful were database connections, reproducible environment facilitated through renv, package based development, the different development staging and production environments.
And then on the deployment side, we had authentication and scheduling, which were facilitated by Posit Connect. We had downtime protocols, a secure way to download internally developed packages with the help of Posit Package Manager. We had an implementation plan in which we're working with our different collaborators, our end users. And then we had monitoring at the very end. And, you know, all of this would not have been possible if it weren't for collaboration. I think it's something I keep saying throughout the talks that collaboration is really important. Developing and deploying chart watch in the hospital is not something that we could have done on our own. I was really by working with a great team and a lot of different people within the hospital that this was possible.
So this brings us to the end of what I prepared. And as I said before, the slides will be available. And at the end, I do have the list of all the papers, talks, blogs that I mentioned. You can all find them at the end. Awesome. Thank you so much, Chloe.
I really love the storytelling through the slides, too, and all the images. They're awesome. There are a lot of questions that are coming through both on Slido and through the chat. So I just want to remind people you can ask in the YouTube chat as well as the Slido link, which I'll just put on the screen again here in a second. There we go. Awesome. So, Chloe, I'd love to just dive into some of these questions, if that's OK with you. I'll jump over to Slido and see some of the most upvoted ones, too.
Q&A
One from the beginning, which we touched on for a second, is can you talk more about the development process? What came first? Were there steps dependent or parallel or did these changes happen simultaneously?
A lot of things were happening at the same time. So in the talk, I think I mostly focused on the implementation team and all the things that the implementation team was doing to get this from development to deployment. But there were different sub teams that were working on this. So there was the evaluation team, which is, you know, they were trying to address the question, like, how do we ensure that the ChartWatch deployment is successful? So they're the ones that are coming up with like, you know, we should measure vitals because you've got to measure clinical adherence in one way or another. And there was the modeling team, which is, you know, mostly the data science people and IT that are focused on actually building the model. So different teams working all at once. I see the way this all started is really from the general internal medicine doctor who came up with the idea first and came to the DSA team and said, like, hey, like, it would be cool if we had an early warning system. And then it's kind of like all snowballed into this.
So like the DSA team, or rather at that time we were the chart team, like we did have the chart team first, so we did have a data science team. I feel like the different groups, like the implementation team, is something that came as we progressed, as we started working towards developing the tool. So we had all of the building blocks, we were assembling it together, that kind of happened a bit later.
So on the topic of the team building, then, can you speak to any lessons learned from building four teams within DSA? What did it take for leadership or management to take on a big scale project like ChartWatch?
Yeah, that's a great question. So when we first started off, we did not have the four team structure. It was, we just had data scientists to do a bit of everything. We found that, you know, there were certain people who were a lot more knowledgeable in one aspect. So I think it just made sense to split it up into this four different teams. And then within each of the teams, there is a director for each of these teams. I think it makes the management and coordination a lot more streamlined. So as a data scientist, I can focus on doing data science work. I don't need to think about all the other unrelated things.
Absolutely. And I see Rebecca asked a question over on YouTube. What is going on under the hood for those alerts that you showed us? Rebecca just mentioned, how do you begin with either functions for emailing or database status?
Yeah, yeah. So I guess more specifically, when we're checking the database status, we are just checking that connection can be made. So, you know, I just try to connect using my usual connection function. If that doesn't work, then I know that something is going on. With regards to actually sending out the alerts, for sending Slack alerts,
we are working with a Slack API to send that. And then for sending out emails, there's something you can do. There's a bunch of R packages for that, actually. So the one we use is sendmailr. This is what we use to send out our emails.
Q&A session
I was going to go try and capture the link to put in the chat, but it's hard to do both of those at the same time. I see Ezra also asked over on YouTube, how early in a project do you typically shift to developing an R package?
Oh, that's a good question. That's something that we sometimes struggle with, because I guess you don't want to do it too early, because it might seem like you're kind of like over-engineering things. So I think for us specifically, for ChartWatch, So we first deployed in November 2019, and then everything was put on pause. And it was over the summer that we decided, okay, let's just build this all as an R package. So I'd say by that time, we had all of the requirements all mailed, and we all knew this is what the alerting schema would look like. These were the different groups that we would be interacting with. So I'd say wait until you know what the final requirements would look like before shifting to developing an R package.
I see Toyin just asked, how long did implementation planning take? Large stakeholder groups can be unwieldy. So the idea for ChartWatch, I believe it initially... The idea initially started back in 2018. And yeah, large stakeholder groups can be unwieldy. It's lots of schedule managing and trying to find a time that works for everyone. But I mean, I think the sort of things that we want to make clear is if you want this to actually be the point implemented, you need to be committed to working towards it. You need to be able to show up to our monthly meetings or our bi-weekly meetings.
So yeah, it was hard, but we managed to get it working. And I think a lot of credit goes to the support team who was able to make all of these meetings happen and ensure that the project coordination actually followed through with that.
Jumping back over to... Actually, somebody asked this, I think, over on LinkedIn. How does DSAA fit into the overall organizational structure? So we are directly part of the hospital. So I guess when we started off, we were a bit more research-based. But then I think it was in 2019 or 2020, we officially became part of the hospital. So advanced analytics is a priority for the hospital.
I know a little bit ago you were mentioning the idea for ChartWatch came from one of the clinicians. Mark asked, in what way are clinicians involved in the development process? How do you prioritize what you work on as a team?
Yeah, so clinicians are completely involved. They're involved from day one. Our project intake model is very clinician-driven. So it's not the DSAA group that comes up with the ideas. It's the end user that comes to us with an idea, and then we work together. So in this case, we have Dr. Amol Verma who came to us. So like, hey, I want to develop an early warning system. Let's work together into doing that. So yeah, clinicians are involved from day one.
Another question was, how did you succeed in gaining the confidence of decision makers in the application? I think for this is, also, we leverage the clinicians we were working with. So Amol was really useful in talking to the clinicians and telling them like, hey, the ChartWatch model is something that is reliable, that works. I think that also the education piece was useful in, education and training. So it was useful in getting the different end users accustomed and used to ChartWatch.
And also, when we first deployed, we did have regular meetings. So it was a good way to kind of get like a vibe check, make sure like, hey, how are things going? Are people still like receptive to that? So just checking in regularly and ensuring that, you know, everyone was happy with how things are going.
I see another question and I might need help with an acronym here. How do you deal with the medical device regulation? If applications, for example, ChartWatch fall under the category of SAMD? So I believe ChartWatch doesn't actually count as a medical device. So the way that we deployed, this was all under the umbrella of quality improvement. So I know that when we, we had to write a few like, we had to write a bunch of documentation. I always found their quality improvement. That is a really good question.
Oh, and thank you, Mark, for clarifying in the chat. SAMD equals software as a medical device.
A question that I had is if somebody maybe works in a hospital today and is inspired by the talk that you gave and is thinking like, I want to get started and I want to do this at our hospitals. Is there any advice that maybe wasn't mentioned in the presentation that you would give people for getting started or what you wish you knew ahead of time?
Reach out to other people that are doing this in the field. So I think just talk to other groups and figure out like what worked, but also what didn't work. That's really good for us because we know like, okay, like this didn't work. Let's not explore that. But like talk to the end users, right? Talk to the doctors and nurses. What are things that they like? What are things that they don't like? What are things that they want? Because at the end of the day, they are the ones who are going to be using this tool. So you don't want to waste your time building something that no one will use. So it's really talking to your end users will be really useful to a successful deployment.
So you don't want to waste your time building something that no one will use. So it's really talking to your end users will be really useful to a successful deployment.
I see there was a question earlier I had missed around Python support and I was curious if your team is using this tool with both R and Python. I think you mentioned a few Python users too. Yeah. So when we first, our very first deployment was a mixture of Python, R and Java. When we redeploy, we ended up actually cutting it down to, we went to like streamliner, but so the, the latest version of a chart watch is actually only our base. Yeah. But yeah, our team is, has a mixture of Python and R users.
I want to double check from the chat that there aren't questions that I have missed, but just to follow up from my question, Chloe, about like what you would have wished you knew and you said, reach out to other people. How did you go through that, that process of like identifying who else you should reach out to or other teams so you can learn what they did before?
I think I'm very lucky in that. Mohammed Mamdani was the VP who was, who led the project from beginning to end. He's very well connected. So he knew a lot of people, he knew a lot of people on like more on the application side of also on the research side. So he was able to reach out to them and set up these meetings and that's how we were able to have all these, these discussions that really helped to kind of where we are now. So that's the, I really just reach out to your network. Yeah.
I always like to say this too, and sometimes I forget, but if you do want to connect with people and you're listening and on the chat, if you want to share your LinkedIn, Oh, I realized I shared it from the posit account instead of my own, but if you want to share your LinkedIn to connect with each other, I know there's a lot of people here listening in who are doing similar things as well. So I'd love to just use the opportunity to be able to help connect people as well. If you're looking to get started and you don't know who to reach out to, feel free to connect with me too. And I can try and connect you in a way that you're comfortable with. You don't know who to reach out to feel free to connect with me too. And I can try and connect you with others in the space as well.
Let me just double check once more. I think we have gone through all the questions. Thank you so much, Chloe, for an amazing presentation. Really, really appreciate your time and answering all these questions. If people do want to get in touch with you is the best way LinkedIn or where? Yeah. Feel free to add me on LinkedIn specify that you're from the talk. Always love to meet people that, you know, have an interest in healthcare and data science. Also, I think it looks like I linked to our team website and also our team blog. So feel free to check that out. There's some cool things that we write about there.
That's great. Thank you all so much for joining us. And as I mentioned in the beginning, the recording will be available at the same YouTube live link, but I'm also going to go in right now and add timestamps from the Q and a, and then some of the helpful resources that Chloe mentioned too. I'll put in the details with the recording. Thank you all so much for joining. Thank you, Chloe. Have a great rest of the day. Thank you. Bye everybody.