Resources
Feb 14, 2025 |
57 min
|
528 views

Data-driven people analytics | Josh VanderLeest | Data Science Hangout

To join future data science hangouts, add it to your calendar here: https://pos.it/dsh - All are welcome! We'd love to see you!

We were recently joined by Josh VanderLeest, Manager of Data Analytics in People Analytics at Progressive Insurance, to chat about how he got into analytics, the uses of people analytics data, ethical concerns when using AI with employee data, and a bit of career advice, too.

This one is a must-listen if you're interested in learning about actual examples of applied people analytics. Josh gives examples of actual projects he's been involved in that aim to improve the experiences of employees, such as onboarding... and we all know that the onboarding experience can be make-or-break!

Resources mentioned in the video and zoom chat:
🔗 Handbook of Regression Modeling in People Analytics by Keith McNulty → https://peopleanalytics-regression-book.org/
🔗 SIOP → https://www.siop.org/

If you didn’t join live, one great discussion you missed from the zoom chat was about ways to remember to document your accomplishments for your "brag sheet". Let us know below if you’d like to hear more about this topic! Many of us struggle to record our wins. Libby has been testing out using a google form to submit all of her "brags" easily to a google sheet and keep track of them that way.

â–º Subscribe to Our Channel Here: https://bit.ly/2TzgcOu

Follow Us Here:
Website: https://www.posit.co
Hangout: https://pos.it/dsh
LinkedIn: https://www.linkedin.com/company/posit-software
Bluesky: https://bsky.app/profile/posit.co

Thanks for hanging out with us! 💛

image: thumbnail.jpg

Transcript#

This transcript was generated automatically and may contain errors.

Welcome back to the Data Science Hangout, everyone. If we haven't met, my name is Libby. I'm a Community Manager with Posit. I help host the Data Science Hangout and foster our Hangout community. I am based in San Antonio, Texas. And if you're not familiar with Posit, Posit builds enterprise solutions and open source tools for people who do data science with R and Python. And we are also the company formerly called RStudio. So if you've used RStudio, you've used Posit. I am joined by the creator of the of the Hangout and my lovely co-host, Rachel. Rachel, would you like to introduce yourself?

Hi, everybody. I'm Rachel, as Libby said. And I'm actually usually, well, I'm usually in Boston, but I'm in Minneapolis this week for our company-wide workweek. So Libby will be our main host today, but I'll be helping out behind the scenes here.

Yeah. The Hangout is our open space where we hear what's going on in the world of data science across all of our different industries. We chat about data science leadership. We connect with other people that are facing similar things to us. And we get together every Thursday, same time, same place, with very few exceptions. So every once in a while, a Thursday, we'll, you know, we'll have to take a week off, but almost every Thursday.

If you are watching a recording on YouTube and you want to join us in the future, we would really love to have you here live because you get to ask questions that way. So there will be details in the description box below on how to add it to your calendar. Just as a call out to anybody who is adding it to their calendar right now, make sure it adds for the right time. It's 12 p.m. Eastern time. Sometimes if you add it when you're traveling or your computer isn't set to be in the right time zone, it can add for the wrong time. So double check that for sure.

Alrighty. I wanted to thank everybody for making this a friendly and welcoming place. You have all made it that way, and we are committed to keeping it that way. If you have feedback about your experience today, whether it's good or bad, we want to hear from you, and there's going to be a survey that makes that really easy. It'll pop up when you close. Your Zoom call should pop up in your browser. So please, please, please fill that out and let us know if we did well, poorly, whatever you've got. You can also reach out to us on LinkedIn. We are both really active there. I'm also really active on Blue Sky. I am the only Libby here, and I think on both of them.

At the Hangout, we love hearing from you. The Hangout is a community discussion. So this is not a presentation, a slide deck. This is not, you know, a conference talk. This is a community discussion. We want to hear from you, no matter your level of experience, your title, your industry, what language you work in, or whether or not you even use a coding language. And we also really encourage you to connect with each other in the chat. The chat is your place to do whatever you want, have a party there. We usually do. Please introduce yourself, include your role, where you are, what you do, something that you do for fun. And then also, if you are hiring for a role or looking for a role, this is the time to share that. Let's connect everybody in our community to opportunities that are good for them. Please remember to share your LinkedIn profile or a website where somebody can find you, because the Zoom chat is ethereal. It's going to go away, right? Or ephemeral. It's going to go away. So make sure people know where to find you.

There are three ways to jump in and ask a question today. Because this is a community-led discussion, we will not have questions to ask our wonderful co-host guest here today unless we all jump in and ask them together. So you can raise your hand on Zoom. We can call on you to jump in. You can put questions in the Zoom chat. Please feel free to put an asterisk somewhere in your question in Zoom chat if you are not able to ask it. Maybe you're somewhere loud, like a cafe, or you don't have a mic today. There's also going to be a Slido link where you can ask a question anonymously, and we would love to have you do that as well. We always get wonderful anonymous questions. I'm also going to do a quick shout-out because I'm pretty sure that registration for PositConf 2025 is open. Are talks still open? Can you still submit, Rachel? I forget.

Yes, until February 14th. So it was just recently extended for all of us procrastinators out there. Yes, I'm pretty sure that the one time I submitted, I did submit in that like week extension period. I really needed it as well.

With that, I am so excited to be joined by our co-host today, Josh VanderLiest, a manager of data analytics and people analytics at Progressive Insurance. Josh, it's so great to have you. We would love to have you introduce yourself and tell us a little bit about what you do for fun.

Josh's background and journey into analytics

Sure, yeah. What I do for fun? Well, yeah. So hi, everyone. I'm Josh. I'm based in Cleveland, Ohio. Work for Progressive Insurance. Been here for seven and a half, coming on eight years now. I'm in the people analytics space, but sometimes that means data engineering, data science, analytics, consulting, all sorts of things that I can get into. Something I do for fun, you might, if you've never been to Cleveland or surrounding area, you might not know that we have one of the best park systems in a national park nearby. So I like to hike, find new waterfalls and new things to see and do. So that's something I like to do for fun. And probably many of you play in data science-y stuff and explore AI and figure out what the hype around DeepSeek is and all those interesting things too. So that's a little bit about me.

So I went to undergrad in Michigan and initially I was focused just on psychology. I wanted to be a therapist or psychologist, something around there, like an internship around that space. And turns out I don't like that at all. So I took a very different route and I was like, well, I think I like stats. I like cool charts and things. And so I got more and more into the statistics, cognitive psychology. I started working for a consulting firm called Center for Social Research. And so we were just doing really any kind of community-based research, understanding effects of the watershed, doing surveying for local communities on effective learning, just kind of all sorts of things. And I stumbled on something called, and this is so wordy, industrial organizational psychology. So if you know people analytics, hopefully you know IO psych for short. IO psychology, while kind of small, is growing rapidly. If you look it up, you usually see like number one career in the social sciences and things like that. So it's a growing space. If you haven't seen it before, give IO psychology a Google, you'll probably end up on PSYOP website, which is society for IO psychology.

So I ended up going to graduate school for IO psychology. And while I was there, I continued to really like consulting for companies. That's how I ended up in Ohio. I moved to University of Akron for my PhD in IO psych. And I didn't love the theory. I really liked the practical stuff. I liked writing code. I liked creating impactful reports that people actually used. And I ended up leaving after getting my master's. So I didn't go all the way through to PhD, maybe someday, but master's. And I ended up finding the people in, well, it was a data analyst role at Progressive. So I wasn't so sure if it was necessarily exactly people analytics. So kind of starting out, it was, I don't know if I'm just in data analysis doing insurance stuff, but turns out it really was, back then we called it human capital analytics, but I thought that was just a terrible name. So we did end up changing it to people analytics. And starting out, it was, and I think some of us can relate, it was around just report requests, like, hey, I want to know what turnover is in IT or, hey, we need this staff for this report.

But slowly kind of grew and developed. When I joined the team, there were four, five of us, and then now there are 18. And I started as an analyst, and now I sort of carved out a niche more in the data science side of things, research and analysis. I lead a team of three, but a majority of my day is individual contributor work, developing, reporting solutions, applications, using R, Python, and all that kind of stuff. So, okay, that was a long-winded answer, but that's how I ended up here. And I'm just so grateful that I am still in this space. I love what I do. I get to do all sorts of things. I think that's what's fun about people analytics is it touches so many different areas of the business. Even at Progressive Insurance, where we have 70,000 employees, I get to do everything, all people data, surveying, recruiting, hiring, employee retention, I get to do it all. So really still super engaged in my work and look forward to the next day.

I think that's what's fun about people analytics is it touches so many different areas of the business. Even at Progressive Insurance, where we have 70,000 employees, I get to do everything, all people data, surveying, recruiting, hiring, employee retention, I get to do it all.

Thank you so much. I think that you are not alone in mastering out of a PhD program in this group. There are a lot of people here who would identify with that. And actually, many of the people that I went to grad school with had a psychology background, right, and ended up studying data analytics because they realized that they really liked statistics and maybe didn't like all the other stuff. So your story sounds so familiar to me from a lot of my friends.

What people analytics teams work on

I'm wondering if you could give us a little bit of an example of a problem that your team works on so that we have some context and we can ask more informed questions. Like what problems do you tackle? You mentioned like turnover. It's probably a big one. What else?

This is not exactly answering the question, but I'll get there. In a larger company, you know, 70,000 people at Progressive, but other companies like us, you usually have multiple functions within people analytics. Your team might have a different name. It might be human capital. It might be employee development, whatever. We have specific functions. So I say I work in people analytics, but I'm one of four teams. So we're all small teams, but we have business intelligence, business intelligence developers. So these are the people really focused on creating data solutions, sometimes building apps. So, right, we have data coming from all these different source tools. How do we get them into Snowflake or whatever database you're all using? How do we get that into Snowflake and how do we get it available to customers? So we have that BI team. I'm not that good because I don't want to do that. We have data governance, definitions, making sure people have what they need, making sure the right people get to what they need, and then reporting solutions.

This is our largest team, still not me. They are creating reports that any HR related function might need. So, you know, we have maybe three, 400 HR consultants out in the field supporting IT, CRM, contact centers, claims, right? All these different kind of pretty unique functions rolling up through one company. We support that kind of HR reporting, might it be employee resource group participation or performance, disciplinary activity, who's, you know, avoiding work and all sorts of stuff around that, you know, hiring practices, throughput, efficiency, things like that. So that's one, two, three reporting solutions and then me. I'm that last one, which is, we call ourselves research and analysis. So think data science, we do all employee surveying, and then support kind of, here's kind of a, think of the examples high level and I'll give some real examples. AI solutions and trying to not be too wordy or, you know, trying to make it practical. Research for decision-making. Hey, we want to change the way we give people PTO. How can you help us predict the impact of that? Employee surveying, legal support when it's needed, DEI, data-driven DEI, is it working? How do we do it? How is it effective? How do we integrate it? And then a whole bunch of other smaller things where it makes sense, research into different areas.

An example of a project, surveying is a fun one. So with the rise of generative AI, suddenly it's become a whole lot easier to summarize huge amounts of comments. So, right, we just had a survey go out to everyone about engagement and culture. How can I summarize, call it a hundred thousand comments for our CEO to get some kind of understanding of, oh, this is what people are talking about. And here's how it looks different from past years. So summarizing and reporting on surveys using traditional analytics, Quarto, Shiny apps but then also using more of the hip, not sure if it's useful or not yet generative AI to summarize large swaths of comments. And then, yeah, the turnover is a good example. We have a project trying to better use hiring resume data to understand where people might fit best at Progressive. And then also try to understand, hey, who's most at risk of being unhappy or turning over earlier in their career and kind of improving the onboarding.

experience for folks. Awesome. So it's sort of like attrition, I guess, is, is the HR word.

Attrition retention. Yeah. It's, it's always, it's obviously really challenging. It looks a lot different where you are in the business. I don't think it's a surprise that call centers tend to have higher attrition. It's a tough job and the challenges there look a lot different than when I look at attrition for our it, we have five, 6,000 programmers at progressive. So their needs, their attrition patterns look a lot different too. So it's tough figuring out the right level to think about attrition as well. Right. I can relate to that. I used to be a call center employee at an insurance agency. It was very big. I know those problems.

I have a follow-up to the, the summarization part, which is how did you summarize surveys from tens of thousands of employees before LLMs? What, what did that look like? What, what is the transition from like the before state to now we use LLMs?

NLP approaches before LLMs

Yeah. Yeah. So I should also talk about the tech stack we use. I didn't mention that. That would be great. Yeah. So the, the survey comments, I mean, great question. So we still do some of it. Before LLMs, we did, we're still doing what we would call NLP, natural language processing, but it was at different levels of sophistication.

Simple as word counts. What are the words mentioned in these surveys for each, each, each question? So simple as word counts, and sometimes that's useful. We would do bigrams, trigrams, which just means like, what are two words that are being mentioned next to each other the most often? It was a lot of like data wrangling to get the data in some kind of interesting cleaned up way. And then think about single word, bigram, trigram. What was, I think it's called tidy text, the R package. Is that what it's called by Julie? Julia Silge. Silge, the amazing Silge.

What's that text, tidy text? Is that what it's called? Am I making that up? Yeah. Tidy text. Still a really useful approach and not something we've just like thrown away.

And so then you have word counts, you have sentiment that you can just pull out of sentences or words based on huge data sets of like, this word tends to be negative or tends to be positive. We still have that available. Folks will still use that, prefer the, hey, I see these types of words are mentioned most. And then the one other thing we would do was structural topic modeling. So there's an amazing R package called STM.

I still truly think it's one of the best approaches to summarizing big amount of comments. If you haven't gotten to the LLM space yet, still maybe even better than LLMs, but very intimidating to start working with. So it was a combination of tidy text from Silge and team and then STM to do structural topic modeling. Structural topic modeling is similar to LDA. If you've been in that space before, it's sort of just take all the texts, take all the comments and look at in a vector space, what are the words that tend to be mentioned near each other? So people that are talking about mentioned the word flexible and work remote and pick up my kid from school. Like those are all similar and words that are mentioned in similar spaces. And so STM is really good at picking up those relationships and creating, hey, we think you have 80 topics in this corpus and here are the top words mentioned for each of those 80 topics. And by the way, for every single of those a hundred thousand comments, here's the likelihood we think it's in each of those 80.

I still truly think it's one of the best approaches to summarizing big amount of comments. If you haven't gotten to the LLM space yet, still maybe even better than LLMs, but very intimidating to start working with.

And so I'm pretty sure Silge also has a big, beautiful blog post on STM. Yeah. Topic modeling. So we still do that sometimes, but to be honest, I find large language models are pretty good at coming up with those 80 topics and very good at the feature engineering. Hey, I have this comment, please come up with the right topics that are represented in it. So that was the before. And then the after is throw everything away and use LLMs for every step.

But where it makes sense, there's obviously the challenge of, hey, every API call is costing progressive money. How do we do this responsibly and efficiently? And most importantly, as accurate as before, if not better. Right. Yeah. All good considerations.

Resources for people analytics

I have a great question from our attendees. Javier, would you like to ask?

Hey, thanks. And Josh, just want to say, I love your hairstyle. It's great.

Wait for him to get on camera. So that makes sense.

Um, yeah, just, just my, my question was pretty basic. I'm in, you know, our focused data scientists, machine learning engineer. Um, if you had a resource or resources where we could learn more about people analytics and kind of, you know, IO psych or anything like that, that we could apply at work. If you have any recommendations now, or maybe later, it would be great to see those.

Yeah, I've got a few, actually, I should have like come up with a list before we even started. So my favorite resource, actually, this is, this is it. My favorite resource. Um, thank you, Javier is, and I'm not being able to keep up with the chat, but thank you everyone for all your engagement. You are amazing. Here's my favorite resource that I send to pretty much everyone that, that talks about people analytics. It's called people analytics dash. Oh boy. Losing the chat again. Regression. Why does this chat go like that? Regression book.org. Um, the author is a, um, is Keith McNulty. And, uh, if you're a LinkedIn person, I know we aren't all LinkedIn people, but, um, he's great on there. Keith McNulty. He's one of my favorite on LinkedIn. So yeah, I did know of this book. So, okay. Honestly, that's where I would start. And what I appreciate about him is he's not, he's pretty agnostic about it approach. You know, I, I, you don't have to use our, you don't have to use Python. He's got examples everywhere. I think in the world of co-pilots and people helping, you know, AI helping us write code, it's a little less and less important, which paradigm or which programming language you use. So, um, big, big fan of this book.

Um, and this is usually where I start. The other thing is like, if you're really into people analytics, that's the place to start. But honestly, I usually am just sending folks the CyOp website too. So CyOp.org. Um, I know it's not exactly people analytics, but it's people analytics adjacent and I went to school for it. So, you know, I'm biased. Um, what I like about CyOp is they, and they, they have, um, meetups similar to the deposit ones coming up, um, where it's industry research and consulting. And I love seeing where those do or do not intersect. Um, because sometimes what's being studied has nothing to do with what we do in our day jobs and vice versa. Um, and also where consulting firms are headed, um, when you can get over the hype and actually see what's useful. Awesome. Thank you. And thank you, Javi, for the wonderful question.

Most impactful people analytics project

We had a Slido question, so an anonymous question, and it was, can you describe one of the most impactful workplace changes that you've been involved in that came from analyzing people analytics data?

I can't. Don't feel too much pressure with that. You can go with that, whichever direction you would like. This is, this is good practice for my next, uh, next job interview. Um, okay. I, I actually have one. Um, when I started years ago, I looked at how progressive does people development and onboarding. And, um, I know it's recorded and who knows, we'll hear it, but I say this to everyone. Um, there was work to do. Um, and to me, the biggest piece that was missing is, okay, when we're in person, I can see you onboarding is fine. Cause you know who to talk to when we all went virtual, we really lost that part of onboarding where it's making connections, finding people like you and just practically like where, where's that SharePoint you mentioned and where, like all big companies, we have 12 different places to find the same information and 11 of the same information and 11 of them are out of date.

So it, all this comes back to like, we had to rethink the onboarding process at progressive. It no longer worked to just like, here's your computer. Enjoy. Like I'll give you a project in a week. And so, um, you know, I'm in people analytics. I'm not, I'm not on the ground, like the HR rep, making sure things are going well. I'm a couple steps removed, but what I saw was we don't have a good grasp of what's going well and what's going poorly. I, I can give recommendations based on IO research, based on you know, best practices. Um, we don't have a view at, at the aggregate level and individual level of what's going well for people and what can we fix? We can try to guess.

Um, and so I think a more recent of like people in analytics insight that, that made a difference was, um, and this is one of my proudest accomplishments of last year. We introduced our very first onboarding survey program where we would check, we check in with you now several times through your first year and just ask what you ask, if you have what you need and how can we help you? And I know it sounds lame, like really another survey, but I promise you this one is great. And, um, what we've seen come from it is our leader seeing, Oh, this thing is consistently being mentioned as a pain point. And it makes them really easy to put interventions and say, let's change the way we do this. Let's come up with a buddy system for people in their first year. Let's find them a cohort where they can meet with them every week.

And so what I think is really useful about the way we've approached this is it can't be a one size fits all at Progressive because our jobs are so distinct for our function. Like I said, but like we have people going to body shops and people getting their cars and going places. We have some people taking calls, we have sales reps, we have product managers, and like the onboarding experience is so unique and different that, um, I love seeing, Hey, we saw trends in this pocket of people and we put in this new intervention just for our marketing team, right? We, you know, just, just for this group that really had this difference because of a process unique to them or, you know, software they were using.

Let me turn my light back on. This must mean it's 1230 and I need to take lunch. Uh-oh. Is that your alarm?

That is my, that was my like less intense alarm of, Hey, stop staring at your computer. My light turns off and I have to go get lunch. I have those like throughout the whole day of like, Hey, stop working and take breaks.

Um, okay. So I think that's my favorite from last year is implementing an onboarding program. Um, with the survey that's focused on finding pockets of not just what's working wrong, but also what's working right. And where can we apply it other places? Um, we do a lot of that where like we let our business call it our executive leadership. They run their, their organizations fairly independently and like figure out what's working, what doesn't. And so one of my roles, when I get the chance is connecting and pointing out like, Hey, this is working in this organization. We could try it here too.

So yeah, that, that was, um, implementing something, helping folks get insights and then changing processes. I guess I didn't give a specific example, but there were a bunch of those like doing the buddy system, uh, changing how we're delivering equipment to our file owners, putting more easier, like plug this in here. This is how the networking cables work to folks that maybe aren't as familiar with hardware. That can be a huge pain point on your first day and a pretty bad experience when you're already very nervous to start a new job. So just really like simplification of the things that people are complaining about in different like surveying platforms.

That's great. I think you're doing good work. Um, onboarding for me is one of the most important parts of any job. And I feel like personally for me, it has a really big impact. I see Blondie nodding. Yep. Yep. Uh, really big impact on how long I stay and what the quality of my work life is. Maybe everybody can kind of chime in, in the chat, their opinion on that too. I would love to read it. I can't read the chat right now, but I can go read it later.

Um, Sarah had a great question in the chat that I think ties into this a little bit. Sarah, are you available to ask? I am. Yes. Um, I have a question around, um, the application of the insights from your, um, analysis specifically with HR attrition and identifying high-risk, uh, employees that might be leaving within a year or whatever that criteria is. Do you find yourself either on that project or others working closely with the stakeholders that are then developing the action process intervention to, uh, to respond to that information?

Sarah, this question sounds like you've, you've been there before.

I'm there right now. Yes. So what are you running into?

Uh, well, it's something that comes up over and over again. I mean, I think that's really, um, kind of one of the challenges in the field with the uptake of models and, and transitioning it out of a proof of concept or something that's running in the background to having impact and value. And so I know, uh, as a individual working in the healthcare system that the the similar equivalent is a high risk for readmission. And there's a lot that, uh, a lot of barriers to overcome with a very general outcome. And I'm just curious what your experience is and, and what that looks like in terms of collaboration with stakeholders for that piece.

We have the same, same challenge. Um, so we live at the, we, like my team, we're in corporate HR. Um, Oh, I sent an image in the chat, like way long ago. And it finally just sent, it's just something about IO psych and the different branches that I liked recently. Um, so we live in corporate HR. So I think of ourselves as like enterprise wide and we have that same challenge of, well, I can create a really sophisticated, validated attrition model, but I'm not the one in CRM on the, on the floor, like doing the interventions. We have counterparts out in the business, right? So like I'm in people analytics enterprise wide, but we have call it like, I don't think they call themselves as like field people analytics. Like they're smaller little groups that we work with to do analysis specific to each of these functions. And I don't know if that's true of other organizations. We're pretty federated, right? Like we're pretty like, Hey, you can do what you want and we'll just help you. Other ones. It's really like central and it's like one group and that they do everything. So we're more federated. So I run into this exact issue is, Oh, these three groups came up with their own turnover model and it doesn't seem like anyone's really validating them. How do we ensure they're of high caliber?

So, sorry, I'm going to answer like from my perspective, I'm probably missing exactly what you're asking. So one is I see, I see it as a, for Progressive and our company, but it's probably different depending, you know, like know your stakeholders in a different way. Um, there's only so much I can do. So I think ourselves a little bit as give them the right tools because they may not have the technical expertise that we do. So for example, we've created a suite of here are like, here's the tool kit we suggest you use to do attrition modeling. If you're not just going to use what we create, right? Cause in one world we could create the, the best, most predictive attrition model and just make it available through like an API, right. That you could just hit and here you have your predictions and you're good to go. So like there's a world where we do that a little bit, but I think usually a team out in the field kind of wants to own it themselves. Maybe they have some business metrics specific to them. They want to use. So I haven't figured like out, figure out the perfect way of doing this for now. It's a toolkit of here are the things you should be thinking about. Here's the way we recommend modeling attrition.

Because I think that's one, a lot of people are not don't currently really understand the right way to do it. And if you don't, you might be doing a really bad job and not realize it. So for example, it's easy to use something like logistic regression control for something that actually has a big time component to it. And now you've just modeled time rather than, than actually worrying about the event. So things such as here's how you can use survival analysis. Here's how you do time dependent and time independent covariates. And suddenly you have a really predictive, really solid model. And then the final component of like, here's how you can validate a model, right? Just simple data science concepts like holdout sets, training, test data, K fold, things like that. So I think that was maybe more technical than your question, which is more like, how do you partner with like HR that actually want to go and make interventions or like go and use this data?

That response was great. Thank you. I really appreciate it. And the technical pieces important to one suggestion I did give for an HR attrition model at our center that wasn't just sitting in that proof of concept stage was, yeah, if your barrier is that your stakeholders aren't sure what to do, consider we could look at doing maybe cluster analysis, see if we can find some meaningful groups and think about interventions from there. But yeah, thank you so much. That was, that was perfect response. Appreciate it.

Validating models and interventions

Thanks. Yeah, it's tricky. I think, you know, one way we're doing it now is use the, none of these models are gonna be perfect. There's so much individual difference that you don't have data to back up. And we also don't want to be big brother-ish and like, Hey, we saw you weren't sending enough emails or whatever. Like we don't have that data, but like you could get to that extreme of like super watching people, trusting your model and go doing a big thing. In our case, even simple things seem to be helping. So use a somewhat simple survival analysis model, predict your high risk folks, and then just have HR set up a check-in with them. Have, do a, we like skip levels. So, you know, instead of just me meeting with my boss, I'll meet with my boss's boss, things like that. So Progressive really focuses on like transparent leadership. Meaning I feel like I can actually trust the CEO and like talk to them. And I could just, you know, set up time with the CHRO and he would actually accept. And he really has, and we do meet.

And so even like that skip level, we found using that as an intervention tool, using the model that I collaborated with, with claims, for example, has been effective at reducing some of our high risk folks. But the tricky part there is how do you keep it simple enough to actually validate? Is that actually helping or no? And that's to me, like that final step of validating a model and validating intervention effectiveness, it's really hard to get by it. Like, why should I care? I used it. It's probably working. And I'm like, oh, come on, we don't know if it really is or not. So like getting by it on that last step of model validation is really hard outside of the data science community. It's just not something people can really resonate with. And to be honest, I still don't always have a great case that I can like simply explain to non-technical folks. That's usually where I think I'm actually pretty good at my job of like connecting and translating technical IT components to HR partners and customers of mine.

validating a model and validating intervention effectiveness, it's really hard to get by it. Like, why should I care? I used it. It's probably working. And I'm like, oh, come on, we don't know if it really is or not.

Yeah, that's been a big topic lately. Actually, I've been talking to a lot of people about selling things that data scientists feel are really important, like reproducibility to leadership that maybe that doesn't resonate with as much as far as a priority. So that's a great topic. I think that we can talk about that a little bit in the chat, too. I wanted to get to we have 20 minutes left. Somehow 40 minutes have gone by super fast. Yeah, we have some really great questions to get to. So my goal is to maybe squeeze in as many as we can. And a couple of them are along the same lines, which is great.

Ethical concerns with AI and employee data

I wanted to ask because we had just talked about using this employee data for interventions. And there's a Slido question that is, you know, what are your thoughts on ethical concerns when using AI on this employee data? So like how do companies ensure that fairness and transparency when using AI models with human data?

I love it. Yeah, so to be honest. The the AI space is something newer to the people analytics team. So I'm I'm on the responsible AI cross-functional team at Progressive, which is honestly a ton of fun. So they they have really been forward thinking on like making sure folks are doing this responsibly. So we have and Libby, you're going to have to remind me the question after I say this, but we have a cross-functional team that even has an HR perspective where we review every single AI model we deploy at Progressive. Every single one we have listed, we rate it for risk and rate it for how it's being used. And is it responsible and ethical? And we better always say yes. So at a higher level, not just employee, we have this documented and reviewed by technical experts. I don't know what I am in this group, probably not the technical expert, but the HR representative. But, you know, obviously, I know that space pretty well, too. And then our legal team, we have, you know, our IP attorney in every call, too. So at a high level, that's how we think about it across all models, right? Employees have employee data has extra considerations, but at a high level, we have a responsible AI committee that reviews every single model.

The other thing to get to, again, before I actually answer it, is there's this really important distinction happening between models that a company deploys and a model your vendor deploys. More and more we're getting, and not we, just Progressive, but IO and People Analytics, more and more vendors are trying to squeeze their way in and do the AI for you. And they'll say, don't worry about the responsible part. We got this. It's responsible. I would say if they can't tell you an explainable AI model, don't do business with them. And I know it's not always your decision, but at least push back and say, we need to understand how your model is validated, run, where it's hosted. Is it in the US? Is it following regulations? How do you make decisions? And even if it's a black box, at least in the employee space, you can still do the very traditional adverse impact on the decisions it makes.

I would say if they can't tell you an explainable AI model, don't do business with them.

I think that's a really tough. I don't know why, but it seems like, oh, it's machine learning. We can't validate this thing. We think it's working. We pretty sure it's not biased. You can always do the very simple 80% rule and a quick P value on the decisions the system is making. So now to, I guess that partly answered it. How do we responsibly deploy AI with employee data? One is, yeah, that's tough. I'm not convinced we need AI for everything, which is a weird thing for me to say, because that seems to be all I do lately. But I think a lot of us are jumping way over the good data, easy to get data, well-defined to the AI, and then we're just missing all this stuff. And then you end up with a model that's poorly predicted. You can't explain it and it's not being validated. So you take steps back and say, do you have well-documented data of high quality? And then do you, are you just doing reporting analytics? And then maybe you have that last step of like, okay, let's use AI to drive decisions. And still it's like, I bet you could just use a linear model and get 96% of the way there and have it much more explainable and you'll feel better about it. Okay. That was a lot. Responsibly use AI. You can always look at the decisions it's making and you better always have a human in the loop, especially with generative AI.

AI in the hiring process

Right. Yeah. I agree. Awesome. All right. Let's see how many more we can fit in here. Um, I saw you had a question about the hiring part specifically. Are you available to ask?

All right. The question was hiring right now is pretty tricky. So with LLMs or NLP to filter through in the hiring process, how good is that process actually at finding a good talent fit in the company? And I think this is a, this is a great question for our wonderful hiring market right now. How good are our systems that identifying good candidates, good fits.

Yeah. I think you mentioned, um, sort of comparing the hiring resume of people to fits in their jobs. And so I think this question might be more towards like, how do you apply that? Are you applying that to the hiring process itself where you're saying, okay, we have these new resumes and how do we know if we're actually doing a good job with finding fit?

Yeah, I think that's correct because I think it's finding a good fit for a company. Let's do this again. Um, I think you say LLM or, you know, larger and more, uh, I think it could very tricky. Uh, do they really do the good job because they've, they've trained pretty much bodies or everything in it, but we have a specific people for that position. You, you, you see.

Yeah. I, I think I follow and you might need to ask a couple of followups, but I'm not convinced it's all that useful yet. When you think about the data that you've actually sent in, when you apply for a job, it's a resume. I know what you've done, types of experience, how long you were there, et cetera. It doesn't really give me that much information. And, you know, there was this era in IO psych where we were all in on bio data, which is essentially that like everything you could pull up, pull out about a person, it only gets you that far. And it really doesn't carve out enough to predict like, Oh, that person's going to do great in the job. So I'm still actually a kind of call it traditionalist in the, like, how do we find the right candidate where I still think it's resume screen just for men, qual types of things. And then really having the right assessments

throughout the hiring funnel, which is also the worst experience taking like seven assessments. So I, I still think a little bit more traditional assessments that measure the right cognitive abilities, personality fit and job fit that don't need fancy AI. Maybe there's a little bit of it, maybe some interview phone screening. So we don't have such to have to have such a huge recruiting force to manage it. But having the right assessments to me is important. That's tough. And you're going to work with vendors to do that kind of thing. But usually you have a say in let's not just make it a good assessment, but also you could make this thing give a realistic job preview. So our JP realistic job preview, I think is really undervalued in a recruiting process.

Okay. Don't just give me a personality assessment. Give me a half an hour experience of actually doing that job. And it rates me along the way. Now I know, Oh, that job is not what I thought it was going to be. I don't think I want to do that. And now you have some data to know if I'd be good at that job anyways. So I am all in on a valid predictive assessment. That's a pretty good experience for the candidate and gives them a realistic job preview along the way. So it's kind of like we can serve them back during that process as well. So LLMs in the hiring, boy, I don't know.

So I am all in on a valid predictive assessment. That's a pretty good experience for the candidate and gives them a realistic job preview along the way.

For me, where we've looked into that is more like using LLMs for feature engineering. Hey, we want to look at candidates that have this type of experience because now they're at Progressive and we can send them, Hey, we think that this might be a good next step for you. So that's where I think they're more useful for now is feature engineering, pull out types of experience, lengths of experience, those kinds of things for more formative and development, less around selection decisions. I would not use LLMs in selection decisions or anything like that. I don't trust those types of things yet.

Yeah, I think you're not the only one. I think that's a really good chance to have. There are a lot of discussions out there about how do we make good ethical decisions and keep computers out of the loop as far as not having any oversight on direct decisions. I think that's a really great stance. Okay, we have 10 minutes left. So let's get a couple more questions in here. I saw one that Sunday asked and it's actually really close to one of our Slido questions that was anonymous. Sunday, are you available to ask?

Sure. I'm talking about hairstyles.

Joining the club Sunday.

Yeah, great talk so far. Yeah, listening to you talk about onboarding, I was just wondering how much is A-B testing playing to that? Try to see if you're going to try something new and try it on different groups of people and seeing how that plays out. And then maybe I'll just throw something extra in since I have the opportunity right now. You're talking about dealing with time varying, you know, variables that vary with time. And looking at things like survival analysis, do you get to do more fancy stuff like GEs or linear mixed models ever?

A-B testing in people analytics

Okay. Thank you. Thanks, Sunday. And again, I have the memory of a goldfish, so you probably have to remind me. But the first one is around A-B testing. I only have so much influence. So I have this challenge where sometimes my HR partners will be like, hey, we implemented a thing. Let us know if that worked. I'm like, oh, you told me before and we could have done A-B test, right? So like, sometimes it's just like, okay, right? So often there are just process changes and they're happening so often, there's just no way to do an A-B test. So practically, not usually. Not usually doing A-B tests just because I don't have that influence yet, but I am trying.

Progressive senior leadership highly values data. And I can even tell them they're wrong. If I have data to prove it, they will believe me and change things, which I think is amazing. There are places we do A-B testing. It's usually smaller and maybe less risky. So I'll give a very silly example. We do these inclusion quarterlies. We call it inclusion quarterly. It's just about different kinds of things. I think the last one it's a progressive wide thing. And it's last one was about like accessibility. How do you create like Tableau reports that are accessible, whatever, a lot of different types of things all throughout the day. And so the question was, how can we encourage participation?

And so we did an A-B test around our approach to encourage people to go to the event and get engaged. So in this case, our experimental condition, we put a block on everyone's calendars. Not that they had to go to the event, but we put a block there so that no one could put meetings on that day. So then they're able to actually attend versus our control was we just let them do what they wanted. So in that case, it's not as maybe impactful as like, let's look how this huge onboarding experience affected half the folks. But it was a smaller scale thing. We're doing that throughout the year. Smaller things we're able to, especially with different like experiences that we can highly control. The other thing that dawned on me on the onboarding experience is that.

Yeah. So real quick, but I'm assuming that if you could, you'd do more A-B testing that is possible. Would I do more if I was able to? If you could, you would do more. You think there's some value there? Definitely valuable. I think companies in general undervalue the existing research in this space. We already know what works and what doesn't in onboarding, in development, in performance event. We know a lot of the stuff, big companies are just stubborn and think we need to do the research ourselves. So I honestly think A-B testing is not always worth it because I know the answer, just they're not listening. So it's more about giving a compelling story and effectively looking at external research before you would bother doing A-B testing that space. So I think it's a balance.

And the other thing I want to mention was sometimes practically speaking, it's just not possible. So you have to really think about the right way to do A-B testing. For example, if I just did a pure random sample, okay, now some people on this team are experiencing the thing and some aren't, and everyone talks to each other. So there's a lot of that bleed if you don't think about A-B testing in the right fundamental hierarchical way, making sure you're doing it to folks that are equal, that probably aren't actually interacting and still doing that responsibly where you're not affecting some group negatively. This was a really fascinating question, Sunday. Thank you so much for asking it because doing experimentation on people has a lot of ethical components to it and a lot of ethical questions and goodness, yeah, you wouldn't want to give somebody an intervention that affects them more negatively than they otherwise would have been affected.

So I kind of like your example because it had a little bit lower stakes and was still really important. But yeah, fantastic question, Sunday. Thank you so much. And that kind of helped us cover a Slido question, which was just like how, what is the role of experimentation or randomized experimentation play in people analytics? So thank you both for those questions. Fantastic questions on Slido today, y'all. Hop in in the chat if you feel comfortable. These are wonderful questions. I wanted to ask, since we really just have a few minutes left, if you had any career advice for us, career advice that maybe you've received that was really great or that you like to give out because it's really meaningful to you.

Career advice

I have a boring career. I started at Progressive in one department and never left because I like it too much. So I have smaller things, not some big like here's how to rethink moving your career. I'll give you some small things. One is for your own year in, year out performance, keep a brag sheet. Keep a brag log. I probably have a link to what I'm talking about. If I Google this brag, I don't have it in my Chrome history. Shoot. But that's basically the concept is throughout the year you're writing code, you're working with people, document what worked really well so that when you want that promotion, you have your data-driven documentation of why you deserve it.

I think some of the most brilliant people do such a terrible job advocating for themselves. Brag about yourself. So document every year, set a calendar invite once a month, once a week where you give yourself 15 minutes to write down, here's what I did really well this week. It could be small, it could be big, effective PR or pull requests. You found code that was wrong. You implemented the new onboarding program, whatever. Brag sheet. So that's a small one, might not be career-changing, but I found that for folks on my team that are way too humble, a brag sheet has helped them.

I think some of the most brilliant people do such a terrible job advocating for themselves. Brag about yourself.

This is the piece of advice, and I said this in the chat, that I would most like to implement for myself, but I never do. I always forget. If anybody in the chat wants to chime in with a great way to remember to do this or to incorporate this sort of brag document into their work process, I think we would all love to hear it. And I think that it's one of those things where I would almost need to build myself a Google form or an app or something, so at the end of every day I could submit data to myself.

Put my day, half an hour recurring, every Friday at 10 a.m. and just label it. Ooh, on your calendar. Yeah, that's a good one. That's what I do. It doesn't have to be fancy, and I just write a couple things in an ugly formatted OneNote or whatever you use at your company, and then just by the end of the year, boom, you've got it ready. Nice. Luke, what do you do?

I just create a folder in my email, and every time I get some kind of reply that's, hey, wow, this was awesome, this was incredible, I go straight to that folder, and I have it available for me when the time comes. I love it. I think actually I could probably do some form of that also with screenshots. Sometimes I'm moving too quickly, and I'm like, I just need to screenshot this and move on. I could make a folder in my pictures folder or downloads or something that's screenshots of great feedback. These are great ideas.

Thank you, everybody, and thank you so much, Josh, for joining us and imparting all of your people analytics wisdom with us. We had so many questions left over that we could not ask them all. Maybe you and I can get together after this, and we can rapid-fire get some answers. I'll post them on LinkedIn or something maybe. Sure, I'd be happy to. This was a lot of fun. Thank you, everyone, for the participation. It was an absolute honor. Thank you.

Absolutely. Thank you, everybody, for spending your lunchtime with us or your breakfast with us. We hope that you enjoyed it. When you close this, there's going to be a survey to kind of give us feedback or ask us questions. Please, please, please do that. Your feedback means so much to us, and we will see you again next week on Thursday. Have a wonderful week.