image: thumbnail.jpg
Data Science Hangout | Lindsey Clark, Healthcare Bluebook | Measuring success of data science
Evolving and Leading a Data Science/AI First Organization.
One of the topics of discussion (33:46) was measuring the success of a data science team and calculating ROI.
How do you measure success for a data science team?
Lindsey shared a few tips from her experience at two different organizations:
One measure of success was the conversion rate of projects. How many projects come to us and what percent are actually converted to an actual deployment? Our target was 5% which may sound low, but the idea was that we want to fail fast. It can be like the drug discovery mentality - if things aren’t going to work, we want to know early on.
We also have targets around the number of specific models we wanted - what we called our data science portfolio library. They didn’t need to be really fancy machine learning, but a lot of them were business logic sort of ranking models.
Another measure was product value and financial targets around that. Could we pinpoint that our model went into the data product and it resulted in more clients contracting with us or greater revenue because we were able to upsell a data product due to another feature we added.
In my position now, we have recently been acquired so we are working on the success measures and what that looks like. We do utilize the Outcomes and Key Results (OKR) Strategy from the Measure What Matters book.
We plan OKRs quarterly and try to plan projects that are really no longer than three month increments in order to keep things going and satisfy the business.
Data Science ROI can of course be a difficult thing to measure. Data science can be a capital expense at times and sometimes you have to look at this team that’s going to do high risk projects as research. Things are going to fail and that’s okay. We’re willing to invest the money for our team and absorb that cost for what we potentially get out of the team in terms of deployed solutions, specific insights, and research that are valuable to the business.
Where to find more?
► Subscribe to Our Channel Here: https://bit.ly/2TzgcOu
► Data Science Hangout site: rstudio.com/data-science-hangout
► Add the Data Science Hangout to your calendar: rstd.io/datasciencehangout
Follow Us Here:
Website: https://www.rstudio.com
LinkedIn:https://www.linkedin.com/company/rstudio
Twitter: https://twitter.com/rstudio
Transcript#
This transcript was generated automatically and may contain errors.
It's the top of the hour, and it's super casual, so we can just get started. You may notice Rachel is not here today. So Rachel is usually the host of the Data Science Hangout. She is on vacation. I will try to do her justice, do the Hangout justice, but I'm excited to see you all.
If you are joining the Hangout for the first time, welcome. We're thrilled that you're here. The Hangout is just a super open space. Participate as much as you would like. We do record the sessions so we can upload them to our site and on YouTube so people can watch the recordings. Again, participate as much as you'd like. You can have your video on, have your video off, log out, connect again. Really the space is just for you. Conversation is super casual and open.
There are a few ways to participate. You can anonymously submit questions on the Slido that my colleague Hannah shared, or you can raise your hand and we'll get to you, or you can post a question or a comment in the chat and hop on video and audio. But anyway, my name is Rob. I work at RStudio. Again, Rachel is on vacation this week. She'll be back next week, and I am super happy to introduce Lindsay, our leader for today, Lindsay Clark.
Lindsay is the director of data science at, it is, I wanted to say Blue Book Health, but that's totally, I'm jumbling up the words. Help me out. Healthcare Blue Book. You got it. Healthcare Blue Book. Yeah. So maybe a quick introduction, who you are, what you do, and yeah.
Yeah, absolutely. Well, thank you so much, Robert. We'll get to it. Like Robert indicated, my name is Lindsay Clark. I'm director of data science at Healthcare Blue Book. Just want to say special thanks to Rachel and Robert for inviting me here today. I got to know the folks at RStudio mostly at my last job where we had all of our deployments in R. We only have a couple deployments in R at Blue Book, but we can talk a little bit more about that as the hour goes on.
Lindsay's background and path to data science
I'll give you a little bit of background about me and sort of how I came to Blue Book. We'll start with kind of present state. So I've been at Healthcare Blue Book for almost two and a half years. So if you've noticed, there's something been going on the past couple of years, which is called COVID. So I've essentially been at Blue Book during the entirety of the pandemic. So I started at Blue Book about a couple of months before kind of lockdowns for COVID happened. So I've been essentially remote my entire time there.
So I lead a team of two data scientists currently at Blue Book. We're a company of about 150 employees currently. So we're kind of billed as the Kelly Blue Book of healthcare. So if you think about that, we offer fair pricing and market pricing for different healthcare procedures within a particular market across the United States. If you think about it, we don't shop for healthcare the way that we shop for something like a washing machine or clothing. So really the mission of our company is to change the consumerism of healthcare and offer our clients, which are primarily employers, and we work primarily through brokers, if anyone's familiar with the healthcare space. But we offer kind of direct to employer benefits where people can actually shop for healthcare and save money if they go to a low cost, high quality facility in their area for what we call shoppable services.
So things like colonoscopies and mammograms, something that's planned or elective is considered shoppable, sort of trauma and things like that, car accidents, that's not considered shoppable. So we don't really cover things like that. We do have price information for our clients. And as of last year, we actually went through a merger and acquisition. We purchased a company called Quantros located in Greenville, South Carolina. So we do have kind of a quality division of sorts now, which is very similar to what I did at my last job. So we provide algorithmic support for quality models for providers and hospitals for different kinds of procedures and patient outcomes across the spectrum of healthcare. So that's really a little bit about Blue Book.
I'll give a little bit of information as to how I arrived here, and then we can kind of open it up to audience questions. So I'll try to make this short because my journey to getting to Blue Book was, I guess, a kind of a long one. I spent most of my life so far in academic research. That's beginning not to be true, but so I earned a PhD in 2005. And for 10 years after that, I mostly worked in academic research, primarily within basic science and drug discovery within type 2 diabetes. And I returned to Vanderbilt in 2011 to do a postdoctoral appointment, which was unusual to work for a while and then go back and do a postdoc. But I was really kind of having a hard time navigating my career. I didn't really know what I wanted to do specifically.
I had kind of this idea that I wanted to be a principal scientist in pharma, but it wasn't really working out. I had a few job interviews that didn't really pan out. So ended up returning to Vanderbilt for a postdoc, and it was really kind of a watershed time of my life. I started sort of figuring out that I really liked programming. I had gotten away from mathematical modeling as a graduate student and really wanted to return to that. Started teaching myself R, also started teaching myself Python. At the time, kind of 2014, 2015, data science was actually just so very new. And everyone kind of in the national market was talking about big data and companies were hiring data science teams.
So actually, I ended up sort of befriending, if you will, a startup company called Axial Healthcare, which was the first company I worked for. They were in an incubator at the time in Nashville. So the National Entrepreneur Center is the singular startup incubator in Nashville. And so I just got to know them. The chief science officer had actually come to Vanderbilt to do a career talk. And so I asked her if I could come and shadow the company periodically just to kind of see what they're doing. I wasn't really sure what I was going to get out of the experience.
But I started doing that. And interestingly, they gave me access to their database, which at the time was Google BigQuery. I didn't know SQL at all, had never written a line of SQL in my life. So I did kind of some freelance work for them. It wasn't really anything extensive. They asked me to do a few case studies and make a few charts. And I found it really compelling that my work was kind of friend-centered. So they were pitching this to their potential clients like the next day. And I found that really invigorating, just kind of the speed of which they were moving. And it seemed very impactful.
And so I ended up actually, like a few months later, it happened very quickly. They got a series B round of funding as a startup, and they were going to move offices. And I remember approaching them in the parking lot one night at like 1030. And I said, I don't want to be presumptuous, but I really like you guys. I would be interested in full-time work here if you're interested in that. And they just kind of shook their heads. There was like five people in the company at the time. And two weeks later, they offered me a full-time job as a data scientist. So that's really how I got into this field. It was really kind of self-taught for the most part.
It was really kind of self-taught for the most part. When I was in school, data science programs didn't exist. It kind of wasn't a thing.
When I was in school, data science programs didn't exist. It kind of wasn't a thing. So I've really kind of forayed my academic experience and research experience into a data science role. About nine months after I started at Axial, I was promoted to director. And really, for no other reason than I was a really good individual contributor. And they liked me as an employee, and they needed somebody to lead the team. So I was kind of there. And ended up building a team of about five data scientists at Axial.
And I have since rebranded to Wayspring. So if you notice some differences on my LinkedIn page, that company is now called Wayspring. But it was really a huge time of growth for me. I started thinking very differently about my role as a data scientist at a company. What is the value that I add? I really had kind of a mind shift in terms of how I viewed data science at a for-profit company and how I viewed research at a for-profit company. So while I was at Axial, I ended up going through a six-month leadership program, which was kind of life-changing for me. Really started getting super involved in my community. I now co-organize the National Data Science Meetup. I've been doing that for about six years now.
Also really just started doing a lot of volunteer work with local universities and also boot camps, just in terms of mock interviews and kind of helping students along the way. I think I personally struggled with my career early on. So I really get a lot of fulfillment from helping others within the community and sort of fostering data science within the Nashville market. So I do a ton of panels and speaking engagements and things like this. So I really enjoy that. I'm also an adjunct professor. That's, I guess, my side hustle. I've taught previously at Columbia State Community College and then at Lipscomb University within their data science program. And then as of last semester, I'm currently at Middle Tennessee State University teaching capstone research within their Data Science Institute undergraduate program.
Okay, well, that's a little bit about me and how I got here. I guess I kind of missed that I did leave Axial in 2019, in December of 2019. It was really just time to transition out of that company. And I started actually back in an individual contributor role at Blue Book. So if anyone wants to know more about that, I'm happy to talk about that experience. But I was a data scientist doing really just a couple of different projects for the company for about a year, and then have been subsequently promoted back to director at Blue Book with a team of two. Okay, well, I've talked long enough, it's 10-11 minutes. So is there anyone who would like to discuss anything, wants to know about healthcare, my path, data science strategy, which is very close to my heart now? Anything at all? I'll kind of open it up now.
Healthcare Bluebook's business model
Could I just press a little on clarifying the business model? So there is a question anonymously sort of about that, like, you know, how do y'all make money? But maybe paint a picture of like, how does a company usually procure, I don't know, healthcare solutions, and then what y'all offer that is different?
Yeah, so let me give for those of you who might not be familiar too much with our healthcare system. So when you sign up for health insurance through an employer, there's really kind of two different flavors of that. So the company could be at risk for their health insurance or not at risk. And what that means is, so small companies typically aren't at risk, meaning there's not enough employee base to actually have a meaningful pool of money from your premiums in order to pay out for healthcare services. So those are typically more expensive plans. Now your larger companies like your Amazon, your Walmarts, like really big, like fortune 500 1000 companies, those companies typically are at risk, meaning the role of the payer is usually administrative in nature only.
So basically, you pay your premiums, and those go into a pool of cash. And that cash is used for your office visits or your preventative care or any surgeries that are needed. So I get this question a lot, like how does Blue Book make money? So companies who are at risk, they typically want to preserve that pool of money in a lot of ways. And the reason is because they are essentially responsible for it. So it makes sense for their employees to actually go to a facility for their mammogram or their colonoscopy, that's quite a bit cheaper, but is actually similar quality to a more expensive facility, right. So if the pool of cash, you're only paying, let's say $200 for your colonoscopy, versus $1,000, then the company's actually made $800 or conserve $800, let's say. Now, that's a very basic example. But that's the reason employers want to purchase our products is because they save money and conserve healthcare dollars by using our product. So we pretty much exclusively contract with companies who are at risk for health insurance. Does that answer your question?
I think it understood. And the work that you're doing within Blue Book, specifically, you know, how is data science playing a role?
Okay, that's a great question. So we really kind of have, at least now after the acquisition, there's kind of two elements of our business. One is cost, the other is quality. So I'll start a little bit with the cost piece. So we do have an analytics team that is responsible for the strategy and algorithms behind what is a fair price, for example, within Nashville or New York City markets, or wherever across the United States, what should we expect people to pay for a colonoscopy?
Like what would you expect to pay for a washing machine, right? Like a fair price might be $500 or something like that. We don't really know that for something like a healthcare procedure.
So really, the goal of that team is to look at the cost of different procedures, kind of the spectrum and spread across colonoscopies that happen within that market and determine from that distribution, what is the fair price. So we have an analytics team that's responsible for that.
And then kind of separate from that, we have, we offer information on our website, we break it down into what we call kind of the stoplight colors, right? Green, yellow, red. So is this facility high cost, you're going to be paying more than the market rates? Is it yellow? It's kind of in the middle. Is it green? Is it below market rate?
That's a little bit where data science comes in. So it's actually another team that supports the model. But we have some effort around classifying facilities as green, yellow, or red. And a little bit more goes into it than cost. I can't give all the secret information. But we have classification models that actually do that on the cost side. So that's one example.
Predicting patient outcomes
An example on the quality side is around prediction of patient outcomes. So healthcare quality is a little bit of an abstract thing, I guess. You know, when you think about healthcare quality, there's no real accepted definition of what it means to have quality healthcare. There's a lot of guidance out there. I mean, this is something that's been around probably since the 60s.
It's mostly guided by what's called the Donabedian model, which is a framework of how we measure healthcare quality within the system that we have. So many companies, including ours, take the approach, and including CMS as well. The government also takes the approach of health outcomes as being a major indicator for quality. So did the person have a pressure ulcer? Did the person get sepsis while they were in an inpatient stay? You know, things like that.
So for specific patient outcomes, we support regression models surrounding really four main patient outcomes across many procedures. It's really very broad. At Axial, we had a large focus on drug addiction, mostly related to opioids. Blue Book really covers the spectrum. So we have four different patient outcomes that we model through regression models that basically says, essentially, do we expect through the regression models, would the person, based on their patient profile, their comorbidities, would they have expected to have a poor outcome in an inpatient stay versus did they actually have the outcome?
The idea behind that is if the person actually didn't have the outcome, but they were expected to have it, then that gap is where really quality lies, because the assumption is that the provider or the facility made some sort of intervention that would have prevented an otherwise poor outcome.
The idea behind that is if the person actually didn't have the outcome, but they were expected to have it, then that gap is where really quality lies, because the assumption is that the provider or the facility made some sort of intervention that would have prevented an otherwise poor outcome.
So those are the two examples that I can give where data science contributes to our business. I'll give one more example. So those two examples I gave, it's important to realize those are, that's information that goes to our products.
We also have an effort around internal optimization. So it's around engagement of clients. So as you can well imagine, some of the biggest challenges that we have as a company is actually getting people to use the website. So it's actually getting the consumer to engage with us, log on, search for their particular healthcare procedure. So that's more difficult than you might think.
So we have an ongoing effort to understand really kind of consumerism in terms of why do people use our website, who uses it, who doesn't use it, who interacts with the healthcare system and has Bluebook as an option, but doesn't go there and why don't they go there. And so really what I'm talking about is heavily related to retail, right? Why do people purchase certain things? So we have an ongoing effort to try to understand that data just in terms of who uses our product, why do they use it, how do they use it, and can we potentially intervene in some way to increase those numbers and actually save people more money.
Mindset shift from academia to business
And Phil made a comment or comment and have question. Phil, I'll give you a chance to chime in. Yeah, thanks. I'm going to save on video because I've got a bandwidth issue here. Thanks. Yeah, I was really interested in your introduction. You talked a little bit about a mindset change when you were in the transition between the organizations. I'm curious to understand kind of what that change was, what aspects changed.
Yeah, gosh, I could talk about this for the next five hours actually. So I'll try to shorten it a little bit. So you know, most of my experience really until that point has been in an academic laboratory and, you know, the incentives and the work styles in that scenario are very different than a for-profit business. You have to think a little bit more strategically about things like time to market, you know, adoptability, interpretability of the models. Like you don't really, I'm not saying you don't have to think about that in academia, but there was less of a focus there, let's say, as opposed to sort of a small startup business.
So I was really kind of designing projects that were maybe medium to long-term projects. I was really had a very like data-first mindset in terms of like I'm presented with a business problem and I'm immediately thinking, what fancy model can I build? And I really had to kind of shift how I think about that because there really is, you know, a lot to consider around just business structure in terms of growth opportunities, market opportunities that you have to think about in order to be a good data scientist and have a really high functioning data science team.
So I had to learn, and I made a lot of missteps honestly early on, but I really had to learn from a network of great mentors that I was able to build over time. Like how do I position myself, position my team strategically within a company to really let us be as effective as we can possibly be for the business? And that could mean a lot of things. I'm not trying to be too vague about it, but it could mean that I'm a little more deliberate around partnering with our account team, our sales team, our product team, sort of figuring out how do I work with these teams most effectively? What kinds of questions do I need to ask them? How do I speak the language of the business to get them to understand things like model performance and how does it bring risk to the business or not?
So that's really what I'm talking about when I say that I had to really kind of shift my mindset a lot. I started reading a lot of business books. I started listening to a lot of products, data product podcasts. I actually have this one book, I'm rereading it. So it's McKinsey and Company's Valuation, Measuring and Managing the Value of Companies. Like this is not a data science book by any means, but I think it's a good example and what I really encourage data scientists to think about is really taking some time to study the business and understand what is your C-suite thinking?
You know, if you feel misunderstood about kind of your team or what you bring to a business, then I think it's important to kind of start talking their language and sort of understanding where they're coming from. Something that's come up recently in terms of our own product strategy and business strategy as a data science team, I'm starting to think about, okay, the projects that we have currently, if we are in a growth time of the business, do any of these projects, can I specifically tie them to more revenue growth in the next six months, 12 months, whatever the time period might be, right?
I think a good data science leader sort of thinks about those things. I mean, maybe it's my opinion, but I've really kind of come to that conclusion over the years that no one really wants to geek out with me about the latest data science models except for my team, right? And so, you know, I have to kind of speak their language in a lot of ways, and I find it really interesting. I think, I think a lot about data science and bringing science to people to help people and what that means to a business and how do I communicate it. I just, I think it's something that's really misunderstood and even, you know, I don't get right a lot of times either. And so, I've just, I have found that to be a very compelling piece of my transition out of academia.
Open source tools and security in healthcare
Perfect. Thank you. Prabha has his hand raised. Over to you. Hi, Lindsay. Thanks for the introduction and talk. I have two questions, and your previous answer touched upon portion of that, my first question, which is what are your measures and metrics of success? You talked about revenue growth prior to, you know, if you go back maybe a year or further back, what were the previous measures of success that you had with your company before revenue growth crossed your mind or is part of your portfolio?
The second question that I have is you're in the healthcare insurance industry. It is my perception or knowledge that there is a boundary to in healthcare industry to which open source tools can be used. When I think of pharma companies for, you know, or there is some portion of healthcare industry where open source tools cannot be used because you need to have a liability or a tracking and also you need products like SAS or that can, you know, somebody takes the liability for it. So these are my two questions. I want to understand, you know, what were your previous metrics of success, first question. Second one is what is the boundary in healthcare industry for open source technologies to be utilized?
Okay, great question. So I'll start with your second question first. So you know, it is, I would say, a concern. Certainly privacy and data security is a concern. I would say at my last company and this company too, we are open source focused. So we actually have hybrid deployments currently in both Python and R for data science. On our development side, we also have things written in Python. So that's obviously an open source tool.
We currently have a hybrid also infrastructure between on prem and cloud. Bluebook's been around a little over a decade now. So really, at that time, building cloud infrastructure wasn't really what people did mostly because of security concerns. So we early on had clients who did not want their data to be in the cloud. And at the time, this isn't true now, but up until recently, Microsoft would not allow a specific time for data breach notifications to be had. So it was it was too long to actually satisfy our clients. So for reasons like that, you know, maybe we weren't able to utilize certain cloud infrastructure tools that we would have otherwise.
But I would say for us, it's been not so much a question of open source tools, but more of like on prem versus cloud platforms as to where we could potentially go. Now, Axial actually started in a cloud infrastructure. So it was a little bit different at that company. We didn't really have the same problem sets that we have now. We just we just have different problems than we have, I guess. But really, most of our security concerns have been around data warehousing and storage. And also, I would add to that data retention policies that we have had to fulfill for certain clients around that. So I would say I have personally not experienced security problems with using some open source tooling. It's more been around the platforms that we have.
Measuring success and ROI for data science
Okay, I'll go to your second question now. So measuring success and ROI for data science team. So I think this is a really interesting question. I'll give you some sense of how we I've done in the past, I do it now and then I'll sort of give my overall sense. So at Axial, we had a few different measures of success for the team, which were kind of interesting. I've never really seen anybody do this other than us. But we had one measure was conversion rate of projects. So we had a mindset, at least after the first couple years, it was we didn't really have
any measures for a couple years that I was there. And then I developed some for the team in conjunction with my current leader at the time.
So one of them was project conversion. So how many projects come to us and what percent is actually converted to an actual deployment. So our target was about 5%. And I know that sounds really low. But our idea was that we want to fail fast, we want to fail often, it's kind of like a drug discovery mentality. It's like if things aren't going to work, we want to know early on, and then we want to get rid of it.
So we had a 5% project conversion to deployment rate targets. We also had targets around the number of specific models that we wanted in what we called our data science portfolio library. So they didn't necessarily have to be like really fancy machine learning things. A lot of them were kind of business logic sort of ranking models. We had sort of data science portfolio targets. I can't remember an actual number. I don't. Yeah, I can't remember the actual number, but that was one of the measures.
And then secondly, was around like sort of product value. Could we pinpoint that our model went into the data products and it resulted in more clients contracting with us, or a greater revenue because we were able to kind of upsell a data product due to the fact that we had added another feature. So we had some financial targets around that. So that's kind of how we handled it at Axial.
Measuring success at Bluebook
Bluebook's a little bit different in that I'm not really sure. You know, honestly, up until this point, we've only had a formalized data science team since September. When I worked at Bluebook, before the acquisition, I was the single data science person. And I worked actually on the engineering IT side. Now since September, we do have a formalized data science team. So I'm currently working on sort of success measures and kind of what that looks like. So I don't have a lot of specificity for you in my current role.
But I can say that we do utilize the outcomes and key results strategy that is published in a book called Measure What Matters. I think the author is Dorr, I believe. If you've never read Measure What Matters, I would highly recommend it. I also posted on LinkedIn today, Katie Nuggets came out with part one top data science books ever published. Oh, that was really good. So I think somebody had mentioned in chat about book recommendations. I read books all the time, especially big business ones. So anyway, that was a side note.
So we use the OKR method of planning. And so essentially, we plan objectives and key results. And we do this by quarter, we've done this since I've been at Bluebook. So we have targets that are set every quarter. So projects really only last three months in duration. Now you might be asking yourself, well, what if a model bill takes longer? I think we'll cross that bridge when we come there. We actually haven't experienced that yet.
But we try to and I try to plan for both myself and my team to have projects that are really in no longer than three month increments in order to kind of keep things going and sort of satisfy the business.
Data science ROI
So I know that's not very specific in terms of ROI. I will kind of end this by saying I think that data science ROI is really just a difficult thing to measure at times. My point of view is that I believe data science is a little bit of a capital expense at times, it's a little bit of a luxury, especially for small businesses that are not specifically AI focused. And so I think sometimes you have to look at it as this is a team that's going to do high risk projects, they're going to do research, things are going to fail a lot. And that's okay.
Because we're willing to invest the you know, whatever amount it is $300,000 $400,000 a year in personnel salary for our small team. And be able to absorb that cost in terms of what we potentially get out of the team in terms of algorithmic deployed solutions, or specific insights and research that are valuable to the business.
My point of view is that I believe data science is a little bit of a capital expense at times, it's a little bit of a luxury, especially for small businesses that are not specifically AI focused.
Okay, all in that there any more questions around that? Thanks for that. Yeah. Very detailed answer. Absolutely. There's actually, I'll add on LinkedIn, there is a LinkedIn influencer, I guess his name is Vin Vishastha. And he is a data scientist consultant, and he publishes a newsletter on Substack and a lot of information around how to measure data science value. So I would recommend him also as a resource around that.
Handling ad hoc reporting requests
I see a question in the chat from AJ, AJ, I'll give you a chance to jump in and ask it live. If not, I can always ask it for you. Take that. I'll ask it. Being in data science, you must get a lot of reporting requests, is there a blueprint you follow to streamline things like data modeling, development to a finished report, when there are so many data sources and organizations today, like ERP systems, flat file, SharePoint, etc. Something that saves you time. Thanks. Okay, great question. Um, okay.
So do we get a lot of reporting requests? I'm actually kind of no, we did it Axial, we got tons of them. At Bluebook, we do have a dedicated client reporting analytics team. And we've been pretty successful, I think about making sure that people know that that is the reporting structure at Bluebook. So if, if clients need customized reports, or if there's questions about ongoing deployed reports that are typically either in like Power BI, or I think some of them are actually still like Excel based type stuff, then it really kind of goes to that team.
I will say that we do get a lot of ad hoc requests in terms of what I would what I would characterize as like pilot type work. And I typically fulfill that. However, I really do always advocate for sort of quick decisions in terms of like, do we do we know what we want the outcome of this pilot to be? If we don't, we have to define that. And if when we define it, we come back to the table and say, okay, it was a success or failure. If it was a failure, we move on. If it was a success, then we turn this into a more deliberate project. And we have a scalable deployment strategy for that pilot stuff.
So that's really a big piece of my job is to, I don't want to say protect my team. But it's a little bit like that. It's basically like kind of keeping everyone on track and saying, okay, you know, we, we have to really think very clearly about what we're doing in terms of kind of one off work, because you can drown in it very rapidly, in terms of like, that's all you're doing all the time. So that's part of my job is to kind of be a little bit of a gatekeeper there.
Let's see, given that there's so many data sources. Yeah, like I said, we don't really do reporting now. But we do have a lot of documentation that we support for the business in terms of methodology. So that's probably the most closely related thing to a report. We are documentation is currently just in like Pete static PDFs that get versions. We're also we're trying to think a little bit outside the box of how to do that a little more clearly, I brought up the idea of making TikTok videos of our methods. I don't know if that landed very well, I would love it. But you know, I just think, you know, clients, it can come across as kind of dry. I mean, let's be honest, sometimes data science and talking about an algorithm can be boring to a lot of people. And so I think anytime we can make it fun, accessible, think about different platforms in which we communicate methods and why we're doing things, I think is always good.
Can I ask what is the calculus in your head when you get an ad hoc request? Like you said, you typically say yes, I'm just curious what your thought process is. Yeah, so when I typically it comes in the form of some some kind of already ongoing projects. Typically, the calculus is in my head is usually to I don't want to say push back, but I always make sure that we're all on the same page in terms of what the objective of the request. And that can typically get me to where I need to be just in terms of getting people to think about because clients ask for things all the time. I'm sure that you all experience this as well in your own roles.
It's kind of a never ending battle for different requests, different numbers, this than the other. I think it's always important to kind of take a little bit of a deep breath and say, OK, do we understand why this person is asking this? Is there a deficiency potentially in our reporting structures or our product that's not meeting client needs? If so, we need to pencil that in and address it. Or is this someone who's trying to kind of get some free analytical work done, which happens, right? So that's kind of the calculus that goes on in my head.
I mean, certainly working for a small business, you know, you're never really going to get away from ad hoc work, pilot work. I've read blogs before that a lot of companies, it's like death by a thousand pilot projects because everything's always a pilot and never really becomes like an actual product. You know, I've seen that happen periodically, for sure.
So really, the calculus in my head is typically one of do we understand what the objective is? Does it fall in line with our current company objectives? Does it fall in line with our vision for what the data product actually does in the market that we sell to? So that's typically how I'm thinking.
Personality types and working at small companies
There is a list of anonymous questions that we should get to. First one, it was actually right out of the gate. So I'm very sorry that we're just getting to it now. I noticed that you have your MBTI on LinkedIn. Any tips for those with less common personality types in the workforce? I am an INTJ.
You know, look, I know that the science behind it is kind of weak, but I'm a big fan of personality tests, particularly leading people. I think it's really important to understand the person, how they work. It's something that was also sort of brought up in kind of leadership training. Look, I think there's room for all different personality types within data science or any field for that matter. I think the important thing is to really understand your own value proposition. I think this kind of speaks to kind of branding. I get a lot of career questions and do a lot of career talks and panels and things like that. And I'm a big proponent of branding.
I think it's important to get your GitHub together, you know, have your LinkedIn profile up to speed, put some time and curation there. So I think it's really important to kind of define for yourself, who are you? What do you offer to a data science team or to whomever kind of you work for? And I think there's room for all personality types. It's just, it's really up to you for you to define for yourself. How does that particular type, what benefits does it kind of play within the grand scheme of what you want to do or accomplish at a company?
Any tips for working at a smaller company? How do you handle underperforming colleagues when there isn't a lot of space for error? That's a great question. Okay, well, my tip for working at a small company is like things move quickly. And to say that I'm a data scientist or even a director of data science is probably inaccurate. I have to do some product development. I have to do people influencing. I have to do C-suite influencing. I have to do some marketing. I have to talk with clients. I do that often, actually. I'm brought into methods discussions. So I have to be an account manager at times.
So I think that's the key. Like, I think being successful in a data science role at a small company, or really any role for that matter, really requires you to think of yourself more broadly. I can make another book recommendation. It's called Range by David Epstein. Anybody read this? Any hands? Give me an emoji or something. I have half read it. I haven't finished it. Half read it. Okay, that's good enough.
This is a really good book that also heavily changed my perspective of who I am, just as a scientist and kind of what I offer. So I think, you know, my biggest sort of thing at working at a small company, my biggest piece of advice is you have to be a generalist. You kind of have to wear a lot of hats. You have to sometimes put on your engineering hat. You have to sometimes put on your engineering hat, your DBA hat, your data science hats.
And some people don't thrive in that environment, and that's okay.
There was a second part to that question. I'm sorry. I got sidelined by my book recommendation. It was around underperformance. Oh, underperformance. Nowhere to hide in a startup. Yeah, that's true.
You know, I think the biggest piece in working for a startup, and I think this is true across the board, but, you know, the thing is you really have to be comfortable with very open-ended work. And typically folks who are just starting out, who are pretty junior, they have a hard time with that. You know, someone gives me something to work on, and it's like super vague. You know, what am I going to do with that? I'm going to use my expertise to kind of figure it out. I'm going to consult Google. I mean, you know, I've heard me, I've seen memes about like the number one job skill is knowing how to Google properly. I think there's a lot of truth to that, right?
So it's really being able to take like very little information and provide something back to the business that is meaningful. And I think junior folks have a really hard time with that. I think it's a very, I think for some people, it's a very innate skill. I think for others, it's something they have to develop over time. And then I think a third group of people never develop it.
And so, you know, folks who are underperforming are typically folks who really can't handle work like that. And so the way that I approach that is typically through a training exercise. You know, I work with them, instead of telling them what to do, I say, what do you think the next step is? You know, can you write me a plan of what you're going to do in order to execute on this project? And can we kind of work through that together? There's not really a specific playbook per se for that.
But one truth I've learned is that when you don't really know what to do in a project, the best thing to do is EDA. Start with exploratory data analysis, understand the data. Like for example, you know, in our engagement work, really the first question I had was, well, how many people use our website? I mean, just a basic question that maybe hadn't really been explored very deeply.
But knowing those basic things about where we are now, and this is part of data science strategy or strategy with any team, like assessing where we are present day is always really that good first step when you really don't know what to do in a project. So that's how I sort of approach an underperformer.
I think it also, it can't be sort of said enough, just basic communication, sort of understanding where they are, what are their goals. In my experience, people who have underperformed have typically ended up leaving on their own. And there's usually a, again, in my experience, an understanding mutually that it's just not a great fit for whatever reason. So it kind of typically works itself out in that way, honestly.
Building a personal brand and speaking engagements
There's a question from Sloan. I'll give you the opportunity to ask if you'd like or feel comfortable. Sure. Nice to meet you, Lindsay. I am fairly early in my career. I've got a couple of years of experience as a data scientist, and I think I could probably do some work on my personal brand. And so I wanted to ask if you have any suggestions on really how to build a brand when you're just starting out.
Yeah. So that's a great question. I think there are a couple of things. And KD Nuggets, they actually published, I posted it on my LinkedIn. You can find it there or maybe just search their website. They gave a blog piece on how to build a portfolio. And I thought it was really good. They basically went through the different platforms that you could use. So what are the common ones? GitHub with your code and also probably Markdown to describe it. LinkedIn is a good platform. People sometimes use social media sites as platforms. So Facebook, Instagram, Twitter, even.
So yes, somebody posted the link. That's exactly right. So I think choosing your platform and your medium is something that's important to do kind of early on. Some people build their own website or use something like WordPress or something like that.
You know, really the biggest barrier I find with people starting out is they don't really have any content, right? To me, the number one best way is to actually volunteer for a meetup speaking engagement and get somebody to record you speaking and teaching a group of people something about your work. It doesn't have to be advanced. I have lots of videos on my LinkedIn of where I've had speaking engagements and really I just started out. I agreed to a meetup one day and they recorded it and I'm like, well, I'll put this on my LinkedIn. And it really just kind of snowballed from there.
So it's always amazing to me how many people starting out don't really take those opportunities that are so easy to get in terms of speaking engagements and being in front of a crowd. I know that it can be intimidating, but really a lot of meetups and the ones in Nashville are the same. They're very much geared toward people who are new. They're very grassroots kind of efforts. And so they're very open and welcoming to anybody with any sort of skill level coming and talking about a topic or teaching the group about something. That's one of the easiest ways to start building content for your brand which is typically video or it can be text.
So it's always amazing to me how many people starting out don't really take those opportunities that are so easy to get in terms of speaking engagements and being in front of a crowd.
I'd put some time thinking about your bio and LinkedIn and sort of that. Running a blog is the next best way. I have a small blog that I probably need to pay more attention to but really just putting something out there just in terms of like your point of view on a particular topic. It really could be anything. It could be data science strategy. It could be leadership. It could be from the point of view of someone who's very junior and how they're feeling about how they're being led. I've not really seen that perspective a lot and I think that would be really impactful.
So those are a few different ways. I would also encourage you guys to also post on LinkedIn. The more that you can kind of post things and share content then the easier it gets. It just kind of exercises that muscle a little bit and people will like your content or they'll share it and it really just kind of gets the ball rolling. So those are my biggest pieces of advice. Just choose your platforms and then start building content and find kind of low barrier ways to do that.
Thank you. I have not heard the meetup piece of advice before so that's a good nugget that I'll definitely hopefully use in the future. Yeah, I'm sure everyone's located different places and most every city, even smaller cities, even Knoxville, we have meetups here. So even if you're not in a place where there is a large tech meetup sort of scene, you can always find like Philadelphia or San Francisco or maybe Austin, Texas. Like they all have meetups that I'm sure you could reach out to organizers and I know from experience as an organizer, they're always very hungry to find presenters and so they'll pretty much always say yes. So it's very low barrier to entry.
Thank you. There's a similar, somewhat similar question that was submitted anonymously. How do you get started with speaking engagements? Yeah, meetups to me are the best way, the number one way to get started. And then I know in Nashville, we have an annual analytics summit that it's mostly local people that come to and so you can apply, you can submit an abstract for a talk. You can submit an abstract for a talk.
So just finding those kinds of different platforms where you can give a talk or you really can't discount and I know it's hard, it's hard starting out but if you really take the time to like volunteer yourself, people are always looking for help. And so it really just takes kind of like, tamping down the imposter syndrome and putting yourself out there and saying, I want to do this. I mean, you'd be surprised at how many wins you can really get by just raising your hand and saying, I'm available. I would love this opportunity. Do you have any opportunities? Yeah, the answer might be no sometimes and that's okay. You just move on to the next thing.
Healthcare data, compliance, and RStudio at Bluebook
There's a question. Any thoughts on sites like GoodRx which seems similar but for shopping for prescriptions? Yeah, so we currently do have a prescription product, an Rx product that's relatively new at Blue Book. So, I mean, my thoughts on GoodRx, I've used GoodRx so I liked it because I saved on my prescription. I don't know that I have any thoughts on it other than I have heard through the grapevine from pharmacy friends that they don't like it because I think administratively it's actually very difficult.
But, you know, I think anytime you can use something to empower a patient and I think that's really why I like Blue Book. I think it's fair to say that our healthcare system is very broken. It's 20% of our entire GDP in the United States, which is nuts. Another book recommendation I can make is called An American Sickness. It's a book about the history of our healthcare system and how healthcare insurance actually came to be. And it really breaks down administratively how different entities within the healthcare landscape work. It's really, really interesting read. It's also frustrating because I actually read it right before I came to Blue Book. And one thing I really like about Blue Book is that we are really devoted to helping the patients. Our products and services are very patient-centric and go directly to the patients themselves. So I personally like that about the company.
I think I like GoodRx because, again, it's really devoted to helping patients save money. I don't know that I would consider GoodRx a direct competitor of ours necessarily. I think we offer just a little bit something different than GoodRx. But I like it. That's my opinion.
Question about regulatory things. When dealing with healthcare patient data, how do you ensure the backend infrastructure is compliant to use AWS or some other vendors? So our cloud pieces are on Azure. And we do go through what's called HITRUST and SOC certification. I actually don't know the... I think we've passed one of them and not the other so far. I don't know. It takes a really long time to get. So HITRUST and SOC are really the two biggest kind of compliance sort of checkboxes, I guess you might say, that a company like ours who's considered a third-party associate can kind of go through.
A lot of people don't know this, but HIPAA does not provide any sort of guidance as to what specific security controls that you have to have in place. It just says that it has to be secure. And that's it. So a lot of people don't realize that. So HITRUST and SOC certifications were really created to kind of address some of those. Can we provide specificity around what needs to be in place in order to really say that we're secure? So we go that route. I mean, certainly both AWS and Azure have products in place that they sort of bill as HIPAA compliant, which really means that it has encryption or other types of controls in place that would benefit us in terms of having a data breach. So those are the types of things we do, and that's not different than any other healthcare IT company.
I just want to be mindful of time. Do you have a few minutes or you have a hard time? I have until three o'clock. Very good.
So I asked this question with caution, so it's anonymous question and you don't have to answer it. I say that because the Hangout is not supposed to be about RStudio products at all, but there is an anonymous question. How is Bluebook using RStudio? What tools are you utilizing? Feel free to be as general or a closed mouth as you'd like. Sure. So I'm a big fan of R.
Right now we do have hybrid deployment and data science at Bluebook. So we have things in both Python and R. Of course, Python and R are not the only games in town, but I think that's mostly what people use. So our regression models for quality are currently scripted in R using the RMS package. We have kind of a, I don't know what you would really call it. To me, it's not what people would consider a true deployment because a lot of the stuff that we do is really batch in nature, right? And I think this is not talked enough about in the data science community that sometimes you really don't need like, you know, API calls and sort of, you know, there's no reason to build in kind of functionality in which a user would be selecting data and you would be doing predictions on the fly.
Like we don't have that. We just don't have the business case for it. So we currently do have batch scripts that run on Spark for our regression models. And then we have deployments in Python for other types of algorithms. So what I mean by that is we have some basic statistics stuff. We have fuzzy matching for some of our data cleaning efforts that are scripted in Python.
I support those myself. They run mostly through cron jobs on Linux servers. So that's kind of the landscape of deployment for the most part. I mean, it's more nuanced than that, but I would say that's most of what we do.
You know, I think we use the fully paid kind of RStudio at Axial and we really liked it. And it was, again, sort of the same thing. We had batch scripts that ran once a month and they were kicked off manually. It wasn't like a scheduled job or anything like that.
You know, I'm not as hardcore maybe as some people. Like, I think if the deployment scheme works and you have really appropriate QA and controls and things like that, I'm a big fan of logging as well. If those things are in place, I think it's fine.
You know, again, like it doesn't have to be like this full-fledged sort of totally hands-off, entirely 100% automated scripting kind of modality. But that's how we use R currently at Bluebook and I don't really see that changing.
Working across functions and building business acumen
Very good. So I want to end on a topic that has come up a lot at these Hangouts. And I thought of the question when you gave a previous answer. So you wake up one day and you say, I have to work with other functions. I'm a data scientist. These other functions are not data science teams. You know, I'm a data scientist. Data science teams need to work with product or engineering or the C-suite. Like, how do you go about doing that? Like, you just wake up and start doing it.
It's a great question. I would say, well, no, you don't really just start up and wake up and start doing it. I would say there's really been two things I've done to sort of be that person. And one of them is find mentorship.
I think that is an underrated thing. I think a lot of, I've mentored some college students and honestly, I feel like I've not really been very useful to them because I think college students and folks starting out really need a different flavor of mentorship. I think once you've been working for a few years at least, maybe up to five years, then you really start thinking, wow, I really need somebody I can lean on to talk about my problems essentially and give me advice.
And so really through my leadership training and then really just my involvement in the community and talking to a few people or several people, I mean, it's really become apparent to me that having really strong mentorship is really key to that. So people can give you advice in terms of what mistakes that they've made in the past, how they've approached talking to the C-suite or whatever the case may be.
I think one thing I've really learned in terms of leaders, it's not just C-suite, it's management that's above you. I think oftentimes they are more willing and wanting to talk to you and hear your ideas than you might think. I think there's sometimes a perception that they sort of live in this very, they're very mature and they've got all the answers and that's really almost never the case. I mean, I think in my experience, your leadership really does want to hear from you. And I think sometimes they don't really provide obvious ways for you to do that.
And so I think it's really kind of an exercise and kind of taking it upon yourself to find those ways that you can interact with them and talk to them in kind of meaningful and sort of low stress environments. It could be taking them to lunch one day. I mean, I know we work in a mostly remote environment now, so that's a little bit harder, but just finding ways that you can connect with them I think is important. And your mentors can help you sort of figure out what those ways are.
The second thing that I've done is I read. I read a lot of books. Yes, I'm one of those people who actually reads physical books still. I listen to a lot of podcasts. Actually, one of my favorite podcasts is Pivot. If anyone knows Scott Galloway, he's an economist at NYU. I'm a huge Scott Galloway fan, like probably unhealthy, but I love Scott Galloway and I listen to all of his podcasts and looks like Robert's with me in the fan club.
And listening to just like business oriented things, I think has really helped me, again, talk about data science very differently. Instead of talking about the algorithm and I've interviewed a lot of data scientists over time, right? And not that this is necessarily a red flag, but 95% of the data science interviews that I have asks this person to tell me about a project and they immediately go into the R library or what package they use or the algorithm. I actually don't want to hear about that.
I want to hear about what was the problem you were trying to solve? Why did you think a data science solution was appropriate? Did you consider another solution? Can you talk about it in terms of what's written in McKinsey's book as opposed to very scientific terms, right? That's typically what I'm looking for in very experienced high performers is that they talk about data science work very differently.
I want to hear about what was the problem you were trying to solve? Why did you think a data science solution was appropriate? Did you consider another solution? Can you talk about it in terms of what's written in McKinsey's book as opposed to very scientific terms, right?
So those are really the two things that I've done over time to kind of get me to this point and I'm still not there. I think it'll be a lifelong journey, quite frankly, but I think once you really start to understand the landscape of data science and how it fits into different size businesses, then I think there just gets to be this kind of turning point in your life. And I just personally, it's resonated with me. I've really enjoyed the prospect of making my science very real to a consumer. And I love that.
Practical advice for getting started
Yeah, before we go, I'm going to hold you to a tangible piece of advice. So like, what is the, if you could think of it, the crawl, walk, run for like someone who's just starting out in their career to actually like get exposure with data science? Leadership or other functions, like, is it a weekly email or five minutes stand up once a week?
Yeah, it's a great question. A couple of things I've done that have been successful are asked to go to meetings. So if you know that there's a standing leadership meeting, some of them, they might not want everybody in, right? But if there's some sort of like product focused meeting or, I mean, you just get to know what happens at your company, right? So I think that's a really good way. It's a very low barrier way to get some of that exposure. I think anytime you can get exposure to what is discussed on other teams, it is a win.
It does take time. It takes time out of your day, but it doesn't really affect the meeting. It doesn't affect the flow of business. And so just ask to sit in on stuff, right? Yeah, you might be told no for certain reasons. That's okay. But I think a lot of people will appreciate your willingness to really want to learn other aspects of the business. I mean, I have found that always to be true. That just me saying, hey, I want to learn more about this. Can I sit in on one of your meetings? Can I maybe sit in on a client call? I mean, I've always been told yes, 100% of the time. So that's a really easy way to do it.
I think going to local meetups or networking events that are geared toward leader professionals, I think is another way to gain that exposure. Really just talking to people who do this kind of stuff. I have had less success with what's called informational interviews. So one of the career pieces of advice that I got early on was do an informational interview. Find somebody who has the job you want and reach out to them and say, hey, I would love to learn more about what you do. I've done that a handful of times. It's typically the people I've talked to have really kind of hated their jobs. So it never really worked out for me that good. But I do think it's still a good way to go.
Like if you can, again, invite somebody for coffee, say, I would love to meet you and learn more about what you do. I mean, really, that's how I got my first startup job at Axial is I just reached out to them and say, like, I would love to come visit your company and just like be a little fly on the wall for the day. Are you willing to do that? And they said, yes. And so it completely snowballed into a professional networking relationship.
So my biggest piece of advice is it really doesn't matter what you do. Just do something, right? Just take that first step. It doesn't have to be the right step. It doesn't have to be the right person. But do something to kind of kickstart that and do it regularly. That's my biggest advice.
So my biggest piece of advice is it really doesn't matter what you do. Just do something, right? Just take that first step. It doesn't have to be the right step. It doesn't have to be the right person. But do something to kind of kickstart that and do it regularly.
Very good, awesome stuff. So how can people get in touch with you if they would like to? Yeah, so the biggest way to get in touch with me is on LinkedIn. I'm on LinkedIn all the time. Have my phone app. I do just a lot of LinkedIn stuff. I learned a lot from LinkedIn. I follow influencers. I keep up with data science news that way. So I'm pleased, happy to connect with any of you on LinkedIn.
I think my email is listed on there, but you can always message me on LinkedIn. I have the paid LinkedIn version or whatever. So you can drop me a note if you'd like my email or whatever. But that's absolutely the best way to get in touch with me.
Very good. Well, thank you so much for the time. I don't know if you had an opportunity to look at all the comments in the chat. I did not see them. Yeah, everyone thought you were great. You really were. So thanks everyone for attending and sticking around. And Lindsay, I'm sure we'll be in touch next week. Rachel will be back. So thank God for that. All righty, y'all. Have a great day. Thank you. Thanks, everybody.