Resources
Oct 27, 2022 |
1h 0min
|
348 views
|

Data Science Hangout | Patrick Tennant, MMHPI | Welcoming People into Conversations to Make Change

We were joined by Patrick Tennant, Director of Evaluation and Analytics at Meadows Mental Health Policy Institute.

► Subscribe to Our Channel Here: https://bit.ly/2TzgcOu

Follow Us Here:
Website: https://www.posit.com
LinkedIn: https://www.linkedin.com/company/rstudio
Twitter: https://twitter.com/rstudio

To join future data science hangouts, add to your calendar here: rstd.io/datasciencehangout (All are welcome! We'd love to see you!)

image: thumbnail.jpg

Transcript#

This transcript was generated automatically and may contain errors.

Hi friends, welcome back to the Data Science Hangout. If we haven't met yet, it's great to meet you. I'm Rachel, if you want to say hi in the chat, please do. The Data Science Hangout is an open space for the whole data science community to connect and chat about data science leadership, questions you're facing and what's going on in the world of data science. The sessions are recorded and they're shared to the RStudio YouTube. You'll notice right now they're a little bit quicker to go up there than to the RStudio Data Science Hangout site. But we'll get that going soon as well too. But you can always go back and rewatch or find helpful resources there.

We also have a LinkedIn group for the Hangout too, if this also helps you connect with each other. I know right now people aren't talking as much in there, but we'd love to have that be a place where you could continue discussions after each week as well. Together, we're all dedicated to making this a welcoming environment for everybody. So we love when everyone can participate and we can hear from you all, no matter your level of experience or area of work. So there are three ways that you can ask questions today. You can jump in by raising your hand on Zoom. You could put questions in the Zoom chat. And feel free to just put a little star next to it if you want me to read it. Or I could also call on you to introduce yourself. We also have a Slido link where you can ask questions anonymously. And I think Hannah or Tyler will share that in the chat in just a second.

I always frame this as questions, but if you have something to add to a specific topic or want to share your own perspective, we'd love to hear that too. So feel free to raise your hand for that as well. One other quick note, just wanted to mention that if you are currently hiring, we'd love to have you feel free to share your open roles in the chat as well. Thank you all so much for joining us here today. I'm so excited to be joined by my co-host, Patrick Tennant. Patrick, I see your title changed recently in the past month. You are now Senior Director of Data Science and Analytics at Meadows Mental Health Policy Institute. So congratulations on that new role as well. I'd love to have you introduce yourself. Tell us a little bit about your role, the organization, and maybe something you do in your free time outside of work.

Sure. Thank you, Rachel, for the kind introduction and for the congratulations. Yeah, that is brand new. As of this week, that just changed. Awesome. I wasn't even going to mention it, but I appreciate you noticing and bringing it up. So I am Patrick Tennant. As Rachel said, the Senior Director of Data Science and Analytics for the Meadows Mental Health Policy Institute. And let me read the mission statement of the Institute because the bosses will want me to get this one right. So the Meadows Mental Health Policy Institute provides independent, nonpartisan, data-driven, and trusted policy and program guidance that creates equitable systemic change so all Texans can obtain effective, efficient behavioral health care when and where they need it.

So that is a mouthful, but we, in effect, are a think tank that makes policy recommendations and helps localities implement policy changes around their mental health care system. We are Texas-focused. We, for most of the history of the Institute, have been exclusively Texas-focused, but in the past few years have become more national in scope and even international a little bit in that we are taking things that we see that are working here and adding them to national conversations. So we have expanded some. Though, as you heard in the mission statement, we are still pretty Texas-focused.

Really, the organization's only been around for 10 years or so, a little bit less than that technically. And so it's not an old group, but it's been very productive in Texas, made a lot of changes, and things in the mental health system here are really getting better in large part, not exclusively, but in large part due to work of the Meadows Institute. The 130 or so people that work there now is the largest it's ever been. We continue to grow and have grown pretty rapidly in the past few years, but the data team is only, there's six of us that are full-time, then we have a research assistant as well. So we're a pretty small chunk of the organization, but we work across the organization, touch a whole lot of different things.

Basically, anything that has a quantitative flair to it, they're going to ask the data team to weigh in on. So we work on sort of all levels of projects and with all different sort of subgroups or teams within the Institute. We on the data team are primarily researchers and academics by background. My boss, the Vice President of Population Health is a epidemiologist. It was a tenured professor before she joined Meadows a couple of years ago, and she has built out the data team, myself included, I joined about a year and a half ago. And we've been working with the data of the Institute for that period of time. Prior to that, it was a consulting firm that was on retainer that did all of the Institute's data work. And so we've been transitioning all of that in-house and building structures and processes around having all of that in-house, because it's a little bit different thing when you own it all yourself versus, or getting kind of discrete answers from an external partner.

So I'll stop there. That was a lot already. Rachel, your last question was about me personally. Yeah, just curious to get to know you a little bit. What's something fun you like to do outside of work? Sure. So I have a 15-month-old daughter. So outside of work time, most of it goes to her. We're also expecting another little one in January. So we're a little busy around here. But yes, it is. It is lots of fun. It is honestly not much free time, but as much joy as I can imagine.

Building the in-house data team

That's great. Well, thank you so much. We're so excited to have you here with us today. It sounds like the data team is growing quite a bit. And with your new role, I'm curious, what does the year ahead look like? Or what's something you're most excited about in the next year? So we are growing. We don't have a position posted currently, but I could certainly see one coming down the pipe. We are really, really kind of restructuring how we do all of our work. And we're organizing it a little bit differently than we have in the past. And that's, like I said, building those structures and processes has been a lot of the work that we've done sort of across our specific contracts or specific deliverable level projects. And so I'm really excited that we are going to have a kind of a more robust set of workflows and organizational processes in which people request data from us, request answers or analyses from us. And then also the way in which we sort of function as a team, right? So like I said, we built from the ground up a little bit. We've only recently started to really fully utilize Git for version control. So that's kind of where we started from. We were in Dropbox sharing scripts via email to each other when I first joined. And so just building from the ground up. But I think those steps are going to make us more effective at the work we have been doing, but also allow us to kind of expand and do some more exploratory and kind of neat, I don't want to call it cutting edge, but like closer to the cutting edge kind of work.

That's great. I see there's a few questions starting to come in on Slido and someone had a great question. It said, super interesting to hear about the transition from using consultants to an in-house team. I'd love to hear about the infrastructure build and how you did it. Sure. That could easily take an hour itself. I will try not to make it that long.

When we came on, when the data team in-house started proper, like I said, when my boss joined a couple of years ago, there were a few things or a few pieces that the external consultants kind of owned and managed for the Institute. One was a big hospital database. So it's all admissions and all encounters at ERs, hospitals in the state of Texas, and the external team owned and managed this, you know, and sort of like kept all of that on their side, but they would provide answers, provide discrete answers out of that to the content teams in Meadows. So we took all of that, brought it over, put it into AWS and built the kind of the structure for our analysis of it in-house totally differently than they had done it. So we had to like reinvent the wheel a little bit on that database. And that was a big part of the work. The other one was the prevalence estimation that we do. So if you want to know how many people in County X are dealing with major depression, for example, we have a proprietary estimate that we developed and that was owned by the consulting firm. We have now brought that over in-house as well.

Moving all of those retrospective data and then the integration of new data as it comes in each year was a sizable task. I think if there's a lesson learned that I could take from that, it would just be that we have a good ongoing relationship with that external consulting firm and that has been completely critical. We honestly couldn't have done it without that. If we had a, you know, like a cutoff or like an end date with them that was, you know, firm and they were going to answer no more questions to that point, we would be up a creek. And so the fact that we have been able to go back and forth with them and they can pull up old documents that, you know, maybe should have been shared, but weren't originally. And that kind of thing has been totally essential. So I hope that answers the question that wasn't too nitty-gritty, but basically if you can maintain that ongoing relationship in some way, even if it's just a small retainer in a few hours a month, I think it's probably worthwhile.

Basically if you can maintain that ongoing relationship in some way, even if it's just a small retainer in a few hours a month, I think it's probably worthwhile.

Yeah, that's really helpful. I see somebody else asked, knowing how much public sentiment affects what issues get attention, do you end up allying with news organizations to share data and compelling story ideas? Yes. I don't know that allying is the right word exactly, and to be clear, that's not my end of the operation. So I can't speak to it perfectly, but yes, we are certainly aware that we need public sentiment to be with us on these important issues. And so we have press releases and we work on stories and we put out information in a variety of ways, and I will say a growing variety of ways. We're really starting to get to that place where we're going to have nationally focused stories that are coming out, where we're sort of making the case for the policies that we believe the research and the data support.

Collaborative care and the data team's role

Are you able to give us an example of one of the projects that your team's working on? Sure. So at the institute level, one policy, one specific sort of mental health intervention that we really believe in works and that the research really supports, honestly, incredibly strongly, it's called collaborative care, which in very brief is this idea that your PCP, your family doctor, your primary care physician is your main point of contact for most of your health care. And they track things like your blood pressure, not because they're necessarily the one who would do open heart surgery on you, but because they're the ones who keep track of all of this information. So collaborative care is just the idea that we can bring mental health into that same sort of treatment space. So they'll track your blood pressure, but they'll also track your depression and anxiety symptoms. And then collaborative care says we build the system out from there. So the referral network starts from the PCP.

They are getting supported by a psychiatrist. So they know when is the appropriate time to refer to a psychiatrist, when is the appropriate time to refer to a social worker or a counselor. And then they also have somebody in house who's helping with those things and can do some of the sort of more straightforward behavioral interventions that are possible in like an outpatient PCP setting. So that's a very brief and rough description of the model, but that's the model. And generally, we think this is a good thing and should be supported. The Institute as a whole encourages this through, we're trying to get and have gotten Medicare and Medicaid uptake of it, as well as private funders to pay for these things, because in health care, that's what you have to do. You have to move the funders to move the treatment.

And then all the way down to the data team specifically, which I think is probably the point of the question. The data team is there to help say, you know, a given county in Texas says, okay, you know, we see this, we understand, we want this model to be implemented in our county clinics, right, in the health care that we as a county provide. But we don't know how many people are going to need each tier of care under collaborative care, right? How many people are just going to be sort of getting checked on by their PCP? How many will need the counselor? How many will need a psychiatrist? And there's a lot of sort of gradation between those two. So then the data team will come in and work with the data that that county has from their health care provision over the past X number of years, and work out estimates using that data, our sort of foundational prevalence estimates that I talked about a minute ago, county data from other counties that are comparable. So we'll do some matching on that and say in other counties that have implemented this, here's how you might see this go.

And so we're bringing together lots of data sources and analyzing them to, with the goal of getting to the answer of, okay, here's how you could plant a staff. Here's what you might need in terms of funding for your local hospital to be able to implement this well, and how, if you're going to build out this program, the kinds of uptake you can expect over time, and where you should, we would think you'd be in, say, five years.

Thank you. And thinking of, like, all the different problems that you're working on, how does a team prioritize? Do you have a process for that? Yeah, no, it's a challenge. So we do, we get a variety of requests. Some are contract level, that's kind of what I was just talking about. If we're in an agreement with a particular health care system or a state or local government, some of them come from legislators, right?

legislator wants to know something, or our CEO thinks the legislator wants to know something, we're going to generate that estimate too, right? That's going to be an important number or set of numbers to share. And so those two things typically bubble to the top. We have to meet our deliverables, and if there's a request that's going to go to some policymaker somewhere, we need to push on that, right? Those things, even if they have not been well planned in advance, which sometimes happens, those things need to get done so that they will be prioritized.

I think if there is a thing that drops to the bottom that does not get priority, it's our exploratory work. And that's, you know, that stinks. I'd love to spend more time doing that. We've got a really impressive team of people, and I think we could really move the needle in terms of methods and analytic answers. But we unfortunately don't have all the time that we want to do those things. So the priorities, to more directly answer the question, the priority list goes policymakers, contracts, everything else.

Thank you. I just wanted to mention, because I know I always say this in the very beginning and not everybody's on yet, but you can use Slido to ask questions, which Hannah just shared in the chat. But you can also raise your hand here on Zoom if you want to jump in and talk to us live, or put questions into the Zoom chat too, and I can call on you.

But another Slido question was, how has the focus on 988, mobile crisis teams or community-based stabilization, for example, impacted your organization and or your work on the data team?

And for people who aren't familiar with what that means, 988, would you be able to share that too?

Sure. 988 is a national number that is now up and operational that is sort of equivalent to 911, but specific for mental health needs, suicidality specifically. So if somebody is feeling suicidal and needs help immediately, they can call 988 instead of 911 and get that help. So whoever asked that question is well-informed, and that is absolutely a thing we're thinking about and working on.

It is all complicated, right? The different communities that we work with, and again, we're primarily Texas-focused, so the different communities have different levels of sort of uptake and buy-in of these things and of mobile crisis outreach teams, or MCOTs is what we call them. And we believe in the methods, the approach, the intervention I think is strong, but you can't force communities to buy into these things, right? So it's about providing the information, providing the guidance when there is interest, and supporting the implementation when the time comes. But in terms of how we work with that, it is about collecting and curating the data to show what we think it will continue to show, which is that these are positive programs. These programs work well and support people, and making a clear and compelling empirical case for that is sort of our task on the data team. So how does the implementation of a mobile crisis outreach team in a community impact that community? We're the ones who are going to be helping to calculate that. I've not worked too closely on that particular line of questioning, but in general, we're going to be the ones who are trying to quantify the value of these things, both in terms of real value, like improving people's lives, but also financial value or return on investment.

Calculating return on investment

That return on investment question comes up quite a bit, so I'm just curious, how do you actually calculate that? It is complicated. We have some economists laying around who help with these things. I'm a psychologist by background, so that is not my area. I actually trained as a family therapist originally, so I'm sort of a long way away from the data and have sort of slowly made my way over, but our economists bring in as many factors as we can come up with to try to understand that there's going to be a ramp-up cost, the building of the program that is typically going to be really expensive, and you're not going to be seeing the return immediately, so you're going to be at a loss most likely in the early days, but building that over time in terms of both population growth as well as changes in prevalence and need, once you factor all these things in and you take it out five, 10 next years, you can start to see the actual return on investment through reduced emergency room use, because emergency room use is really, really expensive. Improved taxpayer base, there are more people who are able to contribute to their communities if they're not in prison or in a psychiatric bed. There are also more people who are available to support that system, so as you start that flywheel, it picks up speed and peer support can jump in and take a lot of people some of the way, is how I should say that, so that you really end up kind of building a self-maintaining system, and the costs go down, but the return stays there.

Patrick's journey into data science

Thank you. I love getting to hear when people come from different backgrounds and end up in this data and analytics space. I'm just curious, what was your journey getting here? Yeah, it's a funny one. So, I started as a family therapist, like I said, practicing clinically, and my idea was that I was going to do that for the rest of my career. When I started graduate school, that was the thought. I realized pretty quickly that it is really challenging work. I was working in low-income clinics and working with child abuse cases and things like that, and it's really hard work, and my heart goes out to anybody who does it full-time, but I thought I'd like to have some diversity to my experience, right? I don't want to just be practicing clinically. I want to do other things, so I went to a PhD program in human development. It was sort of like a social psych-developmental psych hybrid, is how you could think about it, and I was in that program, still practicing clinically, but getting research training and getting data training. I started sort of learning at my advisor's knee on SAS, doing data work, and I was just sort of enthralled by it. So, I taught myself some SAS, and then I learned about R, and I picked it up, and I started running with it because it was more fun than SAS to me.

So, then my post-graduate school career, I worked in research and evaluation roles in academia, but I was always kind of the most data-focused person on the team, and then you start to, you start to polarize, right? You, as the most data-focused person, you get the data asks, and then you learn that, how to do that particular thing, and you polarize a little further, and then you're more data-focused than you were the last time, right? So, as projects come in, you just kind of polarize to one side, or that's what happened to me anyway, and at this point, my clinical experience is still useful. Like, I work for a mental health policy institute, so that is valuable to have a clinically licensed person working with the data. I can speak to what it is like to be in the room, and to sit with someone who is suicidally depressed, or with a family that is really, really struggling, and so that is valuable, but I don't practice clinically anymore. I just bring that in, you know, through the data, and through, you know, how I read the data, right? The numbers mean something a little bit different to me, I think, because of that experience, and so, yeah, so now I'm really almost exclusively data-focused, but I still have that sort of heart background of the family therapist.

The numbers mean something a little bit different to me, I think, because of that experience.

Yeah, that's such a good point. I imagine that's incredibly helpful to understanding the data. I see, Caitlin, you put a question to the Zoom chat. Do you want to jump in? Sure, I hope you guys can hear me okay. I'm in a cafe, but I was recently wondering what do you kind of envision for the future of mental health care and data science, because I always compare this kind of to bioinformatics, that there's kind of a whole discipline for biology and data science, and then human genomics, so I wonder what you kind of envision for the future of mental health and data science, and if you think behavioral sciences will ever almost kind of catch up to kind of genomics, and more of the physical life sciences?

The future of behavioral science and data

Thank you, Caitlin. That's an awesome question. I like that a lot, and I am hopeful that we're on that same trajectory. I don't know about catch up, just because, you know, they've got a head start, right, but I think there is definitely growth in that area for behavioral science, and that we can certainly move towards more sort of the biostats side of biology, right, have a more quantitative focus. We're pretty large in our focus here. We generally focus on populations rather than individuals, but at the individual level, I'll say one thing that I think is coming and that will change mental health and behavioral science broadly is computer adaptive testing for mental health concerns. So, you know, if you've taken the GRE, or I'm sure lots of other standardized tests do it too, but it's, as you answer a question, the computer adapts your next question based on whether you got it right or wrong. So, it's sort of targeting towards your ability, right, your level of knowledge on whatever that test is, and those, they're very, very effective at sort of honing in on where a person actually sits on a level of knowledge of that subject that they're testing on, but in mental health, we ask lots of survey questions, right? We ask people, the PHQ-9 is the standard measure for measuring depression in most populations, and so there are nine questions, but right now, they're static, right? Everybody gets the same nine questions every single time they respond to that, and the questions don't vary at all according to how you answered one or two or three.

If you bring computer adaptive testing into that space, you can make, one, reduce participant burden enormously, so you have, you don't have to ask as many questions, right? You can use AI to sort of generate models that will help to inform based on how somebody answers question one, whether they need eight more questions or whether they only really need two more questions, right? So, it reduces participant burden. It makes the output more precise, right? So, we don't have just sort of a single number, but we have a single number for all the people who fit into that really precise category of people answering those questions in that particular way or in that pattern of responses, and then it also allows us to, once you integrate computer adaptive testing, you're kind of inherently integrating that into, maybe not your EHR, but into your data world, right, into your data system. A lot of PHQ-9s are done on paper, right? So, you walk into your counselor or your psychiatrist's office, and you might fill out a form on paper and hand it back to them. They'll look at it and use that information, but once you go to computer adaptive testing, you have made the leap into sort of the digital collection of that information, which allows us to connect it to everything else, right, the ability to connect those responses over time within that person, but also for that, you know, whole community, for everyone being served there and to understand what is effective for each of these, you know, really precise subgroups of people in terms of how they respond to this. So, sorry, I'm a little passionate about computer adaptive testing. I don't know if you can tell, but I just, I think there's a whole lot of potential there, and that's just one, one, like, in the broader world you're kind of talking about, which is moving towards, you know, more advanced quantitative and scientific work on these things. Behavioral science is unfortunately behind on those things, but I do hope we're going in that direction. Does that help? Do you have a specific, like, thought or question around that? No, no, I love that answer, and that's really great to hear.

Why industries are slow to adopt data and analytics

Thank you. This is kind of related. So, on that, on that topic where you said, like, behavioral science is behind on, in that in general, why do you think, like, certain industries can take such a long time to get into, like, data and analytics?

That is, that is interesting. I'll speak to healthcare because it's a good example of an industry that's, that's lagged on that. Yeah. And it's also the only one I really, I know very well. So, in, in healthcare, the systems are largely, not, not exclusively, but, but largely outdated and kind of archaic, and they're slow to pick up new things. Part of the issue in my view is, is the regulation around them, right? There's a ton of regulation, HIPAA and high-tech and otherwise, around how these systems have to be built and maintained, and, and for lots of good reason, right? There's, it's not that, that regulation is completely unwarranted. It's information that should be protected and handled carefully. But there, there is, I used to work with an attorney who is a national expert in HIPAA, and he would say, you know, it's, it ends up most of the time just being an excuse, right? It's not that the law actually forbids this thing from happening, but it is a very, very good excuse to hide behind for, for pushing the boundaries or extending into, into new areas. And so, you know, I think that is probably right. That has been my experience where people will just say, we can't do that because of, because of HIPAA, because of high-tech. And I, I wish it was, it was clear that there is flexibility even within those rules and that those rules are, are, people are revising them, right? There's, there's new stuff coming out. There's going to be new policies released because we're in a different world now than we were when they were written. And so working, working towards a place where we are, you know, not, not afraid to make changes, but, but respecting the data as we need to while also kind of pushing boundaries and being willing to, to take some chances with new technologies. It seems like that's a balance we have not quite struck properly yet.

I used to work with an attorney who is a national expert in HIPAA, and he would say, you know, it's, it ends up most of the time just being an excuse, right? It's not that the law actually forbids this thing from happening, but it is a very, very good excuse to hide behind for, for pushing the boundaries or extending into, into new areas.

It kind of reminds me of like that idea of like not being able to submit to the FDA, but how that's changing and then how our idea around HIPAA is changing. I see Logan, you asked the question in the chat. Do you want to jump in? Sure. Hi, nice to meet you. I'm Logan. I was just curious, I don't know much about if Meadows works with rural health as well, because you talk like a lot about population,

but I know Texas is huge. So I don't know the dynamic there and also just your thoughts on how to better improve like data, data science for rural population health, because the data collection looks a lot different for those communities.

Certainly does. Yeah. Thank you. That's a great question, Logan. And it's good to meet you too. I think the biggest thing that comes to mind for me is, is that the telehealth transition, right? So telehealth as of two and a half years ago was, you know, existed, but, but was not widely utilized. And now it is, it is not only utilized even, even now it's not, it's not like gone away since COVID, but it's, it's sort of instantiated, right. It's built into policies. People have procedures around this. People have specific rooms for providing telehealth treatment. And so, and, and we have, we're, we have good evidence and we're getting better and better evidence that telehealth is effective and that works in behavioral healthcare. There's still some hesitation. And I, again, I'm, I'm a therapist. I do understand that hesitation of not being in the room with the person. I don't think that's misplaced, but I also don't think the evidence really shows that. The evidence that we have to date says telehealth works really well and that people are able to get good treatment remotely.

So in rural populations that, that solves some of the problem. In Texas, we certainly have some, some vast rural areas. We worked with one school district level system that was, it was serving, it was providing services for, I think it was 14,000 square miles out in West Texas. Like just, just massive, massive areas that like, you know, commuting isn't going to get this done. They're, they're really sparsely populated and so you need to bring in telehealth. And I think similar to what I was just saying about computer adaptive testing, telehealth, it kind of brings you along, right? It, it spurs data collection, it spurs the digitization of a lot of the information in a way that maybe hadn't been done before. And so I think it creates opportunities. I don't, I don't think we're, we're at the end of that. I think we're actually just at the start of that. But I, but I do think rural population health has started to, and is about to get a lot more really high caliber information about treatments and what's, what's needed and what works as well as, you know, how to set these systems up, how to implement them. Because it's not a, there are no one size fit all, fits all solutions out there. And so, so learning about how to set this up in different communities is also really important information. Thank you. Yeah.

Data sources for mental health surveillance

I see. There's a lot of great discussion happening in the chat as well. And Sam, I see you asked the question and you put a star for me to read it. So Sam said, when I worked in public health practice, we wish there were more mental health public surveillance data sets to use for our analysis or report development. What are your primary data sources and what do you wish was available? We have lots of wishes. We are working right now primarily with the NESARC, which I am not, if you, if you worked in public health, mental health, you will recognize that acronym, I think, but I am not remembering it off the top of my head right now, forgive me. But it's a large-scale psychedelia, psychiatric epidemiological study of people across the country or representative sample and asks sort of all of the mental health questions you can think to ask. Then they did really in-depth interviewing and in-person communication with some of the, some of the participants, some, I mean, like hours, sitting, sitting with these people for hours and hours and hours and sort of getting their history as well as developing the methods to track them over time. So that's our primary one.

NCSR and ECSA, the national comorbidity replication sample and the ecological catchment area sample are two others that we've used a lot. And then, like I said, we have this really, really rich data set in Texas and it's, it's Texas specific, but I think other states probably have similar things. That is all hospital admissions and encounters for the state. And it's not free and publicly available, but you, you can pay to get it. Or if you work with a state university system, you can get it for free. And so that information provides us really a treasure trove of, of data on both sort of treatments that are happening, but also we can track people over time, right? There, there's an identifier in there that allows us to connect people across encounters and say something about trends in, in their treatment and where they're going. It's not perfect by any means. And one of the big problems with it is that it's lagged right now. We're still working with 2020 data, which all obviously has lots of, lots of problems and we're sort of in a different world now than we were then. But, but even, even for some of its limitations, it's really, really useful dataset.

Integrating outcome data and EHRs

Great. I, I think there's a lot of suggestions in the chat too, and a lot of followup on that question. Marika, I see you had a followup. You want to jump in? Sure. So I'm just thinking a lot about the kind of the relationship between computer adaptive testing and how that relates to, you know, machine learning studies or recommendation systems. And so given the ability to kind of do that and to create really appropriate and accurate recommendations and the question flows in like PHQ-9 or some other testing, you probably need a lot of data and not just any kind of data, but true outcome data, which I think is pretty rare in the mental healthcare space. So like any recommendations on how to get that on like minimum viability, things like that.

Yeah. I, I've not really done all of that myself. I have thoughts about it, but I don't want to speak as if that's, you know, I've been down that road. I think you're absolutely right. And if, if we go towards more sort of open and less afraid EHR usage by healthcare systems, I think we have the opportunity to get that. So, so connecting, you know, the social worker who's talking to that person about their housing status and, and, you know, their food stability and things like that, connecting those kinds of interviews with their physical health follow-ups with their PCP as well as their, you know, changes in psychiatric medication regimens and things like that. That's all happening within the hospital system. So it doesn't seem to be impossible that we, that we get towards better outcome data.

I will say there's a, and I don't especially want to endorse one particular company, but I know there are companies out there that are, that are working on that cross system integration of data. And that's, that is their whole mission. And they've gotten some uptake. Enrique, if you want to reach out, I'd be happy to share the name specifically with you of the one I'm thinking of. But, but they have as their mission integrating the, you know, the social service and the city and county level data for that individual with their healthcare data with their sort of individual self-report on how, how things are going that you might get from say a PHQ-9 or a GAD-7 that you can tie all of those things together into a single system that is accessible to the person, right? So that individual has, has the ability to track all of their data from all these places in one, in one system that's accessible to the healthcare system and to the other providers. So that if, you know, somebody's housing status goes from stable to unstable that their healthcare system is going to know about that too. It's not just the community social worker. But then that is also accessible for that kind of long-term outcome review, like you're talking about. So if you have all of those pieces in one place, then you can make pretty, pretty comprehensive and strong claims about outcomes associated with say the CAT version of PHQ-9.

Language and cultural discordance in mental health

Thank you. I see somebody else asked on Slido, being located in Texas, does your team do any research on the impact of language and or cultural discordance in mental health treatment? So I have not personally done this. Yes, our team, language justice is the phrase that gets used. And so, yes, that is, that is a big, a big focus for us, but that's more programmatic than research for us. Again, we are, I think the easiest way to sort of conceptualize the work we do is as consultants, right? We're hired a lot of times for specific projects or to answer specific questions. And so I was in academia, my boss was in academia. So we have that kind of lens to us, but we don't get to just go out and explore, you know, this question because it sounds interesting to us or write a grant that would let us, let us do that. So, so no, we have not done that research. So I, I certainly hear you. And I happen to know from, from graduate school, there was a professor that I didn't work with, but was in my department that I knew, named Sue Young Kim, who did, did work on language brokering and the mental health impact of that on children, which was significant and important work. So if you're interested in that, I would go look up language brokering research.

The consulting process and balancing rigor with deadlines

Thank you. I know that you said the team is internal now, but you still act as, as consultants. So I'm curious, what does that process look like of being the consultant when a request comes in to you? What is the process? Or just like, how do you work with, like when a different request comes in, how do you treat this as like a consulting opportunity? Just trying to think like what we all can learn from that process too. Sure, sure. So typically what, what we'll have is, is a contract development process with a, it's either a locality or a healthcare system, sometimes foundations, but there's, there's a scope of work developed well in advance of any actual work happening on our end. But that, that lays out, okay, here, here are the three or five or seven questions that we're going to answer for you. And it's, it's not detailed to the point of the specifics of the data set or the analysis, but it does provide, here are the things that we're, we're going to, we're going to tell you, we're going to give you an answer to this particular question in the best and most rigorous way that we can. And so once, once that comes down to us, and that's, that is over my head, that is the executive team is sort of working out these contracts. But as those come down and, and, and we see the scope of work, we then divvy up within the team, okay, who, who has time and who knows about each of these particular areas? How are we going to address this as sort of like efficiently as we can, while also rigorously. And this is, this is a thing we struggle with is that like that tension between rigor and, and, and, and rigor and, you know, and actually providing an answer, rigor and actually getting something done, right? We, we very much have the inclination to, to push it everything and to make it all as airtight as we possibly can, but we also have deadlines and we also have to provide answers, right? We can't equivocate in the way that, that might be tempting.

Even, even if the information is imperfect, I think it's, it's, it's necessary for us to provide something. And this is actually, so this, this I've gone a little bit tangential, Rachel, forgive me. But no, that's why I was asking the question to get to this. I think it reminds me of, of, of open source, the sort of ethos which is, which is that we are, we're very transparent about all of the analyses we do. We want, we want to be as open as we possibly can, because we understand that, you know, one perfection is never going to be possible. But, but two, that we do have to get to some end state. We do have to make a recommendation and that might, you know, we reanalyze that same data next year. We might get to a different answer, but we need, we need some answer for that policymaker or for that client. We need to provide them some information. And so we do the best we can to make it as strong and rigorous as we can. But we also, the, the kind of the, the backbone of that is that we're transparent about it. We're going to say all the things that we know, all the ways we did this, the reasons we did those things, the way we did them. And, you know, long method sections make some people's eyes glaze over, but like, it feels important to me that, that we provide that and that we were clear with whoever we're providing that answer to about the way that we got there and, and do it in a way that is reproducible and shareable as in, you know, all things that RStudio trying to do, which is part, one of the many things I love about this community.

Even, even if the information is imperfect, I think it's, it's, it's necessary for us to provide something.

Computerized adaptive testing and outcome measures

That's amazing. I love that. Jordan, I think you had a question a bit earlier that I missed in the chat. Do you want to jump in? Sure. Maybe I'm imagining. Oh, you did. Okay. Yeah, it's probably less of a question more, just a, just a bit of a comment too on some of the computerized adaptive testing. It's a lot of the, a lot of the time, the data I look at actually outcome measures from clinical trials. So same things, just usually different. But, but there's a bit of a growing interest, I think, for a lot of specifically within CNS psychiatry and other sort of mental health fields that there's more interest in developing some of these, I guess what we call patient reported outcomes, then some of these clinical reported outcomes where the scoring process isn't necessarily burdened by a individual in their own clinical experience, but is a bit more standardized across thing across patients and patient populations. So just, you know, like increased, I guess, kind of some increased research and understanding of stuff like computerized assisted testing is, is really beneficial for making sure that when we're looking at these data sets that contain these sort of outcome measures, that they aren't so influenced across, you know, they aren't completely, or they're less influenced and less biased from sort of who's completed that assessment and that you're in your data are a bit more, have a bit more integrity between themselves. It's, it's great to get a large data set, but then, so hopefully all of the data is, is, is, is, is, is, is, is, is, is, is, is, is, is, is still very valid and very communicable between itself as well. So I'm, so I, you know, I, I guess if I want to turn that into a question, how a lot of data that you're looking at with some of the outcomes measures, do they come from a good like sort of variety of sources? And is there anything they have to do to sort of aggregate some of the sources together? Like if this one assessment is used in this one place, is there a way of rationally converting it or using

it as similar to another assessment that might still be a measure of depression or depressive symptoms, but isn't actually the same measure? Because I know that's a, that's a problem I run into a lot. Yeah, Jordan, thank you. That, that is an awesome question. And it sounds like we should be friends. I think generally what we're trying to do is, is to triangulate, right? To bring it all together. So as you were, as you were talking about in the problems with a clinician reported scale, you know, there are, there are not the same, but sort of parallel problems with a patient reported outcome scale, right? Like you, you get, you get that sort of inherent bias. And so, so really what we prefer I think would be to have both of those things available to us at, at any, you know, any given moment or for any, any particular analysis. So we think of that as, as triangulating and bringing together the data sources for, for a particular question, getting as many sources of information as we can.

But the other piece is, is that when we have say a, you know, static estimate of, of depression in some, some population, generating appropriate confidence intervals for that and how to extrapolate that to a new population is something we're, we're thinking about right now. And what's the most kind of, you know, scientifically appropriate or statistically appropriate way to do that. And we're, we're playing with some, some bootstrapping methods that I think are going to really help us do that better. And I can't explain it super well myself at the moment, but, but I think this is a technique we're going to start to utilize for small populations, right? Because you can't, you can't calculate a standard error in the same way if, if your, your sample is too small. And so we, we work with some small and pretty specific communities and subpopulations. And so, so to be able to get to a, an appropriate estimate for populations like that in other places, I, I think we're going to explore the, the, the bootstrap option.

Yeah. Yeah. That would be, that's, that sounds really interesting. And then I probably, hopefully that the, the measures are also sort of a bit more uniform, the dimensionalities and traits that they're measuring that a lot of times, a lot of the outcome measures are just reported as maybe just like a total score, some index, some normative value, but really you're, you're, you're measuring four domains here and then three domains here. So you're, you're not really making two equal. They're not that equal. They probably correlate. They probably look about the same, but they're still measuring some different traits within them. Yeah, exactly. The, the sort of, again, there's, there's that tension between like being as, as psychometrically rigorous as you can and actually getting things done. Right. Cause if we, if we we spend all, all day talking about the, the, you know, item response theory for a particular set of questions the, the policymakers will stop coming to us to further questions. Right. And so, so it's not that I can nerd about nerd out about that stuff all day long. And we have to remind ourselves of the times that we have to get to a particular answer. So that is that balance. Certainly. Yeah. They're, they're, they're mostly the same is usually what I, they're not the same, but they are. Thank you. Very good. Thank you.

Welcoming people into change

I see Colter was, was resonating with what you said about like archaic and outdated systems in government sector, public health as well. So just wondering if people are feeling that in their own organizations, what's like the best first step towards getting out of some of these like archaic and outdated systems. I think I was thinking about this particular question, but at a broader scale of like, how do, how do we get society out of archaic outdated systems and how do we move people forward? And actually this is, this is going to sound super cheesy, but I really, I mean it. Rachel, I think it's, I think it's this, like, I think it's what you're doing here. I think it is the, the inclusivity and the, the welcomingness of this that is, that is the answer to that. And so to, to bring that down to, you know, a sub society level, I think in your organization to, to have the people who are running those archaic and outdated systems welcomed into the conversation and, and not, you know, not brought in as in we're, we're going to knock your system out and do something way better. And we're going to, you know, sort of send you away, but to welcome them along with the people who want to make the change along with people who don't understand the conversation at all to get everybody there and feeling like they can contribute and have enough understanding to really be a part of that is, is a way, way more efficient and effective way to make change in my view than, than just trying to convince a single, you know, top level leader or trying to convince the, the gatekeeper on the archaic system that it needs to change.

I think you want, you want everyone, everyone there, everyone, a part of it or else they're going to be a lot of, a lot of barriers and a lot of roadblocks that I would think will, will hinder progress. So, you know, I don't have that power. If we work with the healthcare system, I don't have that power to congregate all of them and to change their system. And that's not our ask, right? That's not the thing that we were there to do, but I do think there is a, a way in which internally to that system that just creating a culture of inclusion and welcoming us around, around data and around tech systems is, is a way more effective, a way more effective path forward.

a culture of inclusion and welcoming us around, around data and around tech systems is, is a way more effective, a way more effective path forward.

That is amazing. I love that. I want to go share that everywhere right now. Yeah. You're saving the world. No, thank you. That's such a great point too. It's like bringing people into that conversation and making sure that, that they feel welcome into it as well.

Exploratory work and future directions

Let's see. I am going to go jump over to Slido and see, oh, Catherine, you had asked a question a bit earlier about some of the exploratory work. Do you want to jump in? Sure. This is a much less weighty question. Some of the previous ones is not my area at all, but you had mentioned earlier about maybe wanting some time to do some more exploratory work. And I'm just curious if you can elaborate, elaborate, like if your team was not weighed down with all the other priorities, what would you focus on? Thank you, Catherine. I love that question. It's like, like I said, we don't get to focus on as much as I want. So I'm happy to talk about it here. We, we have a few irons in the fire, I guess you could call them around that. One is predictive modeling at the population or ecological level. So, so where are needs going to be? Can we see into the future a little bit about, about what the community's needs will be, would be enormously valuable. So, so in terms of planning for services, building systems and structures, planning for services, building systems and structures it's not ever going to be precise down to an individual person or for an individual person over time. We're not going to get there and we understand that, but if we could get some foresight into what community X will, will need next year or in the next three years that, that could be incredibly valuable information.

We, we are not a team of machine learning experts but we're pushing at that and trying to train ourselves up on that so that we can hopefully provide some useful information with some foresight. So that's, that's a big one. The other one I'll say is, is sort of related to the bootstrapping thing, but we have, we have other related projects around, around simulation and sort of like Monte Carlo models using, using distributions to try to better, better quantify the risk associated with certain things. We are often tasked with providing a single number and we are inclined to provide ranges, but to make those ranges as, as strong and as sort of like well grounded as we can is a task that we, we're, we're working on. And so if we can do, use some simulation models or plug in, you know, reasonable distributions to, to generate ranges for our estimates, I think we're in a much better place. There's like, there's some really kind of basic probability theory behind all of that, but there's also some, some fairly advanced stuff that we're, that we're trying to do in that same vein. And that's, these are the things where it's like, we don't, we don't have the time to teach ourselves to have the time to do the work. And, and we wish we did, because we think there's, there's a lot of really high value opportunities there. But it is what it is. You get paid what you get paid to do. Yeah. Yeah. Thank you. I know media and, and legislature sure love those single numbers, don't they? Yes, they do. In every industry. They do. Thank you. Thank you, Gavin.

Community, jobs, and closing

Thank you everybody for all the questions too. I, I did just want to mention again that if you are hiring for any open roles now that you'd want to share in the chat, we do have a few community members whose teams have unfortunately recently had to lay off a few people. And we just wanted to be able to pull together any open roles that we know about. And I would be happy to like collect those and put it into the Hangout community on LinkedIn, but also with the recordings as well. I just wanted to call attention to that again.

I'm sorry. I think LinkedIn is dinging in my ear. There we go. Um, just to check, were there any questions that I may have missed earlier from the Zoom chat or on Slido? Feel free to raise your hand on Zoom here. Rachel, while we're waiting. Yeah, you go ahead. I was just gonna say while we're waiting for a question to pop in, uh, I would be totally happy to connect on LinkedIn. Um, and, and to do anything I can to help any of y'all, please feel free to reach out. I'm, I'm really have, uh, had a lot of people be generous with, with their time with me on, in the past and still through today. So, so please feel free to connect and I'm, I'm very, very happy to do what I can to help. Thank you very much. I'll share your, your LinkedIn in the chat too.

I see Brady, you just asked a question into the chat. You want to jump in? Or I can start by asking the question, um, what are some analysis problems that are particularly pronounced in the mental health domain, or maybe what are some thorny inference areas you would warn others about before wading in? So it is enormously important in, in mental health to, uh, to keep in, in mind always that, that we're talking about a sensitive topic for the people who are providing the data. Right. And so this is, I don't think exactly what you're asking about Brady, but I just want to say it again, uh, that having that sort of the, the heart behind the analysis, uh, is, is very important for us. It's something that we focus on a lot. We don't want to, uh, we don't want to purely quantify these things. We want to remember that there's, there's qualitative information in these people's, uh, in these people's lives too. And so it's, it's necessary to, to always keep that in mind in terms of the actual inference. Uh, we are, we're interested in change over time a lot of times, but, uh, as Jordan was alluding to earlier, uh, some of our measures of, of these things for, for the baseline and for the follow-up are, are not especially stable. Uh, so inference about unstable, about changes and inherently unstable measures is, uh, it's complicated, um, at best. And, and I think, uh, you can actually end up with a lot of overconfidence about the, the change or the, the impact of something, uh, if you, you know, if, if you're reliant as we often are on, on unstable measures. So that's, that's the thing I would say is that, uh, maybe computer, computerized adaptive testing can help. Uh, but, but we have kind of inherently flawed measures of a lot of the outcomes that we're interested in. Um, and so ascribing, you know, impact or change of a treatment or a program, uh, is, uh, is tricky and, uh, be, be aware of the quality and the validity of the measure you're using. That's what I'd say.

Thank you. I have one selfish question. Um, maybe not selfish, but a lot of times, um, I'll, I'll hear from people who are trying to build communities within their own organizations and just going on what you're, you're talking about how important it is to welcome people into these communities. I'm always interested in learning what other teams are doing or what challenges they're facing in this area. Um, and just curious what this looks like from your team's perspective. Do you have this internal data community? Yes, we do. Uh, we, we can certainly do it better and that's, that's a priority of mine, um, is to always be improving around that. Uh, but we have our core data team. We also have people who are, uh, working with data who are embedded within the, the content team to the subject matter expert teams. Um, and so there is, uh, there is sort of, uh, you know, a core group, but then also some tentacles. It doesn't sound very nice, but I'm thinking of an octopus, uh, out, out through the organization, uh, of other, of other data, uh, related folks. Uh, but really the, the, the main goal for us is to just have people feel confident, right? When we work with the subject matter expert teams, we want them to have confidence in that conversation to push back on us. If we say something that doesn't make sense or to, uh, to understand that, that they have expertise that is absolutely essential to this and that, that really weird, it seems like they're coming to us and they're dependent on us, but, but actually that is the inverse. Like we can't do anything useful without the subject matter expertise teams. Um, and so, so building confidence is, is actually my like number one priority. Um, and welcoming and includes inclusion is, is a part of that, but also getting people to see, uh, real world examples of that, of, of where these numbers might say this, but, but because you know about X, Y, and Z, you can tell us what this actually means. You can tell us how this is important or how it's not important, even though, you know, statistically we could make an argument that it is. That's extremely helpful. Thank you so much. Thank you for sharing all of your, your insights and experience. So that's, it's been a pleasure. I know we just hit the top of the hour here. Thank you for joining us and thank you all for the great questions as well. Yeah. Thank you guys very much. It's, uh, it has been fun. And like I said, I'm really happy to chat and, um, Rachel, again, this is, uh, it's an awesome community. So thank you for doing this. Thank you. We appreciate it. And we, um, we put Patrick's LinkedIn into the chat here, but I'll, I'll be sure to share that recording and the hangout group on LinkedIn too. Thanks all.

Bye.