image: thumbnail.jpg
People Analytics at AT&T with Liz Esarove | Data Science Hangout
Liz walked us through several use cases for people analytics, including reducing employee attrition and forecasting future attrition rates and workforce changes. She also discussed how her team uses text analytics to analyze employee comments from surveys. Liz emphasized the importance of ethical considerations when analyzing employee data, such as avoiding the use of race and gender in models to prevent potential discrimination. She also highlighted the value of continuous learning in the rapidly evolving field of data science.
Resources mentioned in the chat:
š [People Analytics Regression Book](https://peopleanalytics-regression-book.org/index.html)
š [Fundamentals of People Analytics with applications in R](https://www.routledge.com/The-Fundamentals-of-People-Analytics-with-Applications-in-R/Starbuck/p/book/9781032312327)
š [Predictive HR Analytics: Mastering the HR Metric](https://www.koganpage.com/product/predictive-hr-analytics-9780749482484)
š [Posit Conf 2024 Announcement](https://posit.co/blog/posit-conf-2024-announcement/)
š [Data Science Hangout Recordings and Upcoming Events](https://posit.co/data-science-hangout/)
āŗ Subscribe to Our Channel Here: https://bit.ly/2TzgcOu
Follow Us Here:
Website: https://www.posit.co
LinkedIn: https://www.linkedin.com/company/posit-software
To join future data science hangouts, add to your calendar here: https://pos.it/dsh (All are welcome! We'd love to see you!)
Thanks for hanging out with us! š
Transcript#
This transcript was generated automatically and may contain errors.
Hi everybody, welcome back to the Data Science Hangout. I'm Rachel, I lead Customer Marketing at Posit. I'm excited to have you joining us here today. I actually wanted to remind everybody that registration is now open for Posit conference, which is going to be in Seattle in August. And so Curtis will share a link here in the chat, but just wanted to remind everybody because I think the early bird pricing ends this week. I also put together a like post and a PDF on how to convince your boss for budget approval to go to the conference. So I'll be sure to try and go grab that and put it into the chat too. But again, thanks for joining us today.
The Hangout is our open space to hear what's going on in the world of data across different industries, chat about data science leadership, and connect with others facing similar things as you. So we get together here every Thursday at the same time, same place. So if you're watching this and want to join us in the future, if you're watching on YouTube at some point, there's details to add it to your calendar below. And just make sure that it adds it for 12 eastern time so that you can join us live. I know a few people had some issues with the time zone before. But is this anybody's first Data Science Hangout? I'd love to have you say hello in the chat just so we can all welcome you in and say hi too. But we are all dedicated to keeping this a friendly and welcoming space for everyone. So we love hearing from you no matter your years of experience, titles, industry, or languages that you work in. Totally okay to just listen in here. Maybe you're having lunch, out for a walk, but you can also be a part of the party that happens in the Zoom chat too.
There's also three ways you can jump in and ask questions or also provide your own perspective on certain topics. So first, you could raise your hand on Zoom and I'll just call on you. Second, you could put questions into the Zoom chat. And feel free to put a little star next to it if it's something that you want me to read out loud instead. Maybe you're in a coffee shop or something. But I'll call on you to introduce yourself and add some context. And then third, we have a Slido link where you can ask questions anonymously. And I see Curtis was right on top of it and shared it there in the chat. I guess one more note to share, we do have a LinkedIn group. If you wanted to join that, it sometimes makes it a little bit easier to find each other after the fact. But with all that, I am so happy to be joined by my co-host today, Elizabeth Esero, Lead Advanced Analytics and People Analytics at AT&T. And Liz, I'd love to have you kick us off here and share with us a little bit about your role, introduce yourself, but also share something you like to do outside of work too.
Okay. Well, thank you, Rachel. And thanks for inviting me to be here this week. My role is I'm a data scientist on the People Analytics team at AT&T. And our focus in People Analytics is that we use data and insights gained from that data to help make business decisions regarding employees, work, and business objectives to try and improve the environment, improve business outcomes for the company. And something that I enjoy doing outside of work, I am an avid photographer and mostly nature photography and love visiting the local gardens or various areas around the Southeastern US and taking pictures, landscapes, wildlife, whatever I can find that's interesting. It's a lot of fun and very relaxing for me.
I love it. Well, thank you so much for taking the time to join us today, Liz. As we get some questions coming in from everybody here, I'd love to just learn a little bit more about your journey into the field of People Analytics. I know I got to hear some of that when you spoke at a meetup a few years ago, but I think it'd be really interesting for everybody here too.
Liz's journey into people analytics
Sure. I originally started working in the banking industry where I built a lot of my data analysis skills and joined an HR team in Bell South, which eventually became merged with AT&T a little over 20 years ago and was primarily hired to do data analytics. They were building their their staffing organization. Back then, data analysis in HR was fairly new, at least at that company. Since then, I've moved into various roles and spent some time working as a project manager serving as a liaison between HR and IT as they were building tools for us so that our HR professionals could develop reports. From there, I eventually decided, or as I was learning, more and more companies were getting involved in doing predictive analytics in HR and how it could be applied. I decided to go back to school and get my master's degree in data science and have graduated about six or seven years ago and have been applying those skills at work ever since then.
Awesome. How did you make the decision to then go back and get your master's? I had reached a point in my career where I was ready for a new challenge and was looking for what did I want to do next. At the same time, I like to stay informed of what's going on in the industry in general. I read a lot of news articles and attended a few conferences where I saw more and more companies were discussing the need to apply predictive analytics. I read a book by Eric Siegel where he presented a case study of a company that applied it to help reduce employee attrition in a business unit that had a very high attrition rate for employees, those leaving the company, that is. I was fascinated with the concept, did a little more research, started checking into programs, and decided on a master's degree program through Northwestern University, which fit my schedule perfectly, and worked on the degree in the evenings while I was working full-time during the day.
Wow, that's impressive. I see a few Northwestern alums, it seems, in the chat as well. Abigail, if you had a follow-up question here on the master's program, you want to jump in? Yeah, so you mentioned scheduling, that it's around your schedule. What else were you, like, what criteria were you looking at when you were, when you knew you wanted a master's and you were trying to pick? I was, at the time, there were not a lot of programs available and I found a few universities, I think only four maybe, that were offering an online program that I could do remotely. And the local Atlanta universities did not have programs like that at the time. I believe Georgia Tech now does have one. So, one, I needed something with the flexibility so that I could do the classes in the evening while working full-time, and that was, narrowed it down tremendously in terms of which universities offered those types of programs. And from there, just researching, checking into what types of classes were included in each, I knew I wanted some hands-on experience, programming, and just looking at class offerings and considering the different university programs from there and just narrowed it down that way.
Now, I think there's a lot more courses available through organizations like Coursera and Udemy, and those were not available back then, but there's a much wider range of opportunities for people who want to learn data science. And there was, at the time, I think that was in 2013, 2014 is when I started my degree.
People analytics use cases
Thank you. I saw in the chat a few comments about the Posit.com background I have here. I'll put it in the chat if anybody wants it and wants to help promote the conference, feel free. But Liz, I think to maybe put us all in your shoes and kind of understand some of the problems you're working on a daily basis, would you be able to walk us through an example use case of analytics on the People Analytics team? Sure. Many cases, we work closely with business unit leaders, and we also have HR business partners who serve as the liaison between HR and each of our business units. So they'll typically come to us with a business question that needs to be solved, or we may identify on our own a business problem and do the research and analysis ourselves. But some of the cases where we apply analytics would be reducing employee attrition, trying to find ways that for teams that have a high attrition rate, find out what are the aspects of the job or the work environment, or even the external labor market that could be affecting or making our employees more likely to leave the company, and figure out from there, what can we change to help improve things and reduce that attrition rate? Because it is very expensive for companies to train employees and replace them on a regular basis.
And another area, I've done a lot of forecasting. So if you're thinking about getting into People Analytics, time series forecasting is a very good skill to learn. And I've done forecasting to try and determine what our future attrition rates would be, estimating how our workforce will change in the future in terms of headcount, looking at skills and demand, and trying to figure out what changes do we need to make to the workforce as the company is making changes to our product portfolio and making future plans. And we do a little bit of forecasting two or three years into the future to try and give them some numbers to help estimate what's going on there. And another for employee surveys, there's a lot of need to analyze comments that employees enter in those surveys. So we've done a lot of text analytics there as well.
ROI and cost savings
Nice. Thank you. I see a few questions coming in here in the chat. And Saul, I think you had a question around like maybe ROI of People Analytics. Do you want to jump in there? Yeah. Can you guys hear me? Yes, I can. Yes. Great. Thanks, Liz. Appreciate the insights. I think, yeah, I'm curious as far as, you know, I work in finance. A lot of our analytics has to be around ROI and dollar cost. How do you and team go around like identifying those numbers? So when you say it's expensive to hire and train, you know, how detailed do you get? What's that process look like? And then, you know, when you see maybe a metric associated with attrition go down, like, you know, turnover retention is down. How do you then go back to your stakeholders and show those cost savings? Thank you.
Oh, good question. We can get into a lot of detail in terms of determining how much it costs to hire and train a new employee. And it varies from job to job. So we need to look at each type of job and how long they spend in training. How long does it take them once they finish training? How long does it take them to get up to speed to a more experienced employee? And in terms of filling the job, we also look at the cost of advertising that job, interviewing candidates, time spent interviewing those candidates and so forth. And all of that can be added together to determine what is the cost to hire and replace a typical employee for a specific type of job. From there, once we determine how much we've been able to reduce our attrition rate, we can go back to that cost of hire number and estimate a cost savings by reducing the attrition. And that's how we figure out the ROI for some of our projects.
I see Rob has a follow-up question, I think, on this piece too. And Rob, I'd be really curious to hear what this SCR is as well, if you want to jump in. Sure thing. Can you hear me? I can. Can you see me? I can. All right. We're functioning on a Thursday. Yeah. I am trained in IO psychology, PhD, and then I've dabbled in data science as well, but have recently come into this data storytelling realm. And I've stumbled across the McKinsey way of doing this called Situation Complication Resolution, and was just wondering if you had something similar as far as how do you get the executives engaged in what you're doing for the organization and really kind of drive home the impact that you're having?
Well, that's a good question. And it varies from project to project. Sometimes if the executives come to us asking us to help them with data analysis to solve a problem, it's much easier to get their support and buy in on those projects. In other cases, we've gone to them and said, here, we've done this analysis and we've determined, and I'm not familiar with the SCR template that McKinsey has specifically, but I think it's kind of similar to what we do. We describe the situation, what business problem were we trying to solve, what were some of the key insights that we found that could help solve that problem and make some recommendations. And then
it's up to the business leaders to decide which of those solutions they want to implement.
Excellent. Thank you. Thank you. I see Kyle asked a question a bit earlier. I see stars next to it. So as a reminder, if it's something you want me to read out loud instead for you, just put a little asterisk next to it. But Liz, since you've been in data science for 10 plus years, what do you use or recommend to stay up to date with the latest tools or methods? Do you find any particular tool or method recommend to stay up to date with the latest tools or methods? Do you find any particular challenges? Challenging staying up to date with all the R packages. Tidyverse did not exist back when I was working on my master's program. So I've been trying to learn that and incorporate those packages into my work. And that makes it much easier. And I'm writing far less lines of code on my projects now.
But as far as how I stay up to date, YouTube videos are often very helpful. I keep up with a lot of, I have a large collection of books now on R programming to help me stay up to date there. These days with people analytics, I've gotten to the point where I have LinkedIn connections that share news on what's happening in people analytics and the latest trends there and try to stay up to date that way with what's going on in our industry in general as well. And those are, I think just a few of them. Attending webinars, attending events like the data science hangout where somebody will mention something new that they're trying and I may do a little research on the web on that as well.
Survey data and attrition analysis
Great, thank you. Thanks for the hangout shout out as well. Abigail asked, are the surveys you mentioned anonymous? Are they all text or do you also analyze in-person exit interviews? Curious about the data used in analyzing factors affecting attrition. We do have exit interviews that are answered separately from some of our corporate surveys. We have a regular survey that we do to just measure what's the general attitude of our employees. And it consists of a combination of questions that have a agree to disagree Likert scale. And there's usually maybe one or two questions that just have an open-ended, please tell us about whatever the executives in the company decide is key point.
It just allows the employees to type in whatever comments they want and from there we try and extract some of the common themes that show up in those comments. And as far as analyzing other items related to attrition, we may look at unemployment rates, anything that we can find in the labor market from US census data that we think might be helpful for that specific project. As well as looking, talking with our business unit, developing a good relationship with the business leaders for the area of concern helps tremendously because then we can get insights from them on things that they are observing or that they feel might be a factor in causing attrition. And we work on finding what data can we get that backs that up and then do the analysis and try and determine does the data really show that what you believe is happening is truly happening.
And that often helps. Sometimes we may dispel some myths and sometimes the data fully supports what they suspected. What's one of those myths? Can you share that with us?
Probably not. Okay, no worries. Yeah, there's many times where people think something may be causing employees to leave but we just can't find data to support it.
Networking across a large organization
Yeah. Greg, I see you had a question a bit earlier here. I noticed my virtual background keeps making my arm disappear here. Greg, do you want to jump in? Yeah, absolutely. Thanks for being here today. I actually also used to work for AT&T. I came on board as part of the DirecTV acquisition years ago. And then like another person mentioned, I'm now in finance working with Schwab. But one of the things that I found a struggle when we were acquired by AT&T, you know, our company was around 15,000. And when we joined AT&T, we just immediately jumped up to like 300,000 employees as part of AT&T. So keeping up with the larger data and analytics community was a really tough task, like knowing who is doing what and where data was and how to access things. And I was wondering if you have any advice on how your team kind of networks across a large organization like that?
A lot of it is through knowing the right contacts. And our AVP does a really good job at marketing the team, I guess you could say, talking to other leaders within HR and even within the business units to let them know our team does exist and we're available to help them. As far as finding data for our HR professionals, we have a centralized system that we encourage everybody to use to get employee data to answer most of their common questions that they have. And that way everybody has one version of the truth. Everybody's numbers are matching from one report to the next and so forth.
But then, of course, there's other systems such as our finance department has their own data sets that are more focused on the financial metrics of the company and budget metrics. And each business unit has different performance metrics that they measure for employees. And it's oftentimes trying to find the right contact person in order to figure out who is the right person to talk to to get access to that data and learn more about what's in that database. But I guess you could say a lot of networking is involved is probably the best way to go about it. Awesome. I appreciate that. Thank you.
HR analytics community of practice
And on this topic, Liz, maybe it'd be helpful to also chat a little bit about the data, the HR analytics and data science community of practice. I noticed that on your LinkedIn. I was just curious to know a little bit more about that group and how often you meet as well.
We just started the group in January and we have not had our first meeting yet. But we've created a virtual environment in Viva Engage. And each week I've been sharing different news tips with the members of the community. When we started it, the idea was we wanted it to be a community where people who knew nothing about analytics and had no experience could just start from the very beginning and understand what is it that we do in analytics and how do we apply this to HR problems and solving those types of problems.
So I started from the very beginning about sharing articles, sharing YouTube videos. I just do a little research to try and find something that's pertinent and relevant that HR people can relate to and share something new each week in short videos. Everybody typically has very busy schedules, so I'm trying to also keep in mind that they may not have time for a learning course that takes several hours, but spending 30 minutes watching a video to learn something new each week, they seem to appreciate that and it helps keep them up to date.
So we've been gradually increasing and getting into more complex topics including things such as how do you present your results of your analysis project to an executive, doing exploratory data analysis, typical what are the key analytic skills that people might want to learn from the very beginner to more experienced and even to more advanced and where could they learn those skills.
We are planning in the second quarter of this year to start doing some meetings and I will be contacting some of our community members to get some feedback from them on whether they would prefer an open question answer format similar to this or would they want formal presentations or maybe we'll alternate between those. But it's been, I guess we're in the very early stages and in talking to other people that have started communities, I've learned most of the participants are kind of shy, so we haven't gotten to that point where there's a lot of back and forth conversation going on yet, but I do get emails and comments from people telling me they appreciate what's been shared and they're learning something new, so I do like that fact, as long as they're learning something new.
But we have, what's been challenging is we have members that are across the board in terms of their analytic skills, so I try and alternate between sharing something more advanced and sharing something for those that are at the beginner entry level. I love that. Well, I'm excited to hear more about the group as you go as well. Is that just in AT&T or is that external? It's just within AT&T. Okay.
Data ethics in hiring and promotion
Joy, I see you just put a question into the chat. Do you want to jump in here next?
Hi, sure, yeah. First time caller. Thanks for joining us. Nice to meet you. Nice to meet everyone as well. I was just curious, you didn't mention this, but I would be curious whether your team is involved in using data for hiring or promotion decisions, and especially how you finesse potential conflicts between ideas of fit or the right quote-unquote type of person versus the sort of benefits of diversity or the potential issues of bias. One of the things I think is so great about data is we can use it to expand stakeholders' thinking, but I know that sometimes it's a little tricky.
Yeah, and our team does not get into the details of making recommendations about individual employees, but we have done some diversity analysis in general, and I'm trying to think. I personally have not worked on a project related specifically to promotions or hiring, but we do have some other members of our team that have been doing analysis related to those, and so I may not be able to provide you much in the way of details on those, but it is a careful balance, but at the same time, you still have to. We use data to try and help support those decisions, but at the same time, there are the individual hiring manager and the folks in our talent acquisition team get more involved in making those decisions when it comes to the individual employees in each individual job opening. Great, thank you so much. Thank you.
Yeah, and if anybody listening in today wants to put into the chat their own thoughts there too, it sounds like there's some interest in connecting with other people analytics folks, so I was trying to at the same time go grab the link of a group from one of our other Hangout attendees, but I will get that one again.
Large language models and text analysis
I see Ethan had a question a little bit earlier. Liz, have LLMs changed your, and when I say, I try and be better about acronyms, have large language models changed your approach to text analysis? If so, how have you incorporated them into text analysis workflows?
Yes, it has. We have a large language model in AT&T that's protected behind our firewall so that our employees do not have a risk of sharing any proprietary data with any external LLM tools, and I've found recently there was a survey that was done after an event. They wanted to get some feedback from the participants and had some open-ended questions in there, and I chose to use our internal LLM model and gave it all of the comments from the employees and asked it to give me some summaries, and by interacting with the LLM and asking it different questions, I was able to summarize those comments much faster than I would have using some of the traditional methods that I was using in R, so now that's become my go-to tool in analyzing comments.
traditional methods that I was using in R, so now that's become my go-to tool in analyzing comments.
Abigail, I think you had a follow-up on this question. Is it, you said, is it OpenAI and Azure, or is it something like Bedrock, or is it open source? Do you know?
It's, I don't know if it's open source tool, but it's one where our data and artificial intelligence team, which is a separate team from mine, they implemented it, and it is an internal tool that's available only to AT&T employees. Okay, cool. Thank you.
Confidentiality in employee surveys
Thank you. Nolan, I see you asked a question here. Want to go next?
Yeah, sure. You know, Liz, I was curious how maybe you all are balancing like confidentiality levels within employee surveys and then integrating that into your models or any analytics. I know at times it's hard to make that balance and do like, you know, collect data at the individual level and then have access to that while also balancing those confidentiality thresholds. Any thoughts on that?
There, we have a team of IO psychologists, industrial organizational psychologists, that manage a lot of our surveys, and some of those surveys are done through vendors, and it depends on the situation. Sometimes they are completely anonymous where we have no ability to get that data, and sometimes we might be able to get that data
in order to match things up at an aggregate level. We never report at an individual employee's information of what they responded to in a survey, but we can report it at an aggregate level for an entire department once it meets a minimum threshold in terms of responses.
Great, thank you. I see a follow-up on the text analysis discussion. James Yu, do you want to jump in here?
Hi Elizabeth, this is James. I'm over at Guess supporting data science and analytics, and one of the questions I had was for the sentiment analysis that you've been doing based on text data, and I understand there's a lot of data mining that's involved within the customer surveys. My question is, as you're creating this sentiment label, if you will, to classify to different levels of the sentiment in which you're seeing for these employees, how do you train your data to label someone as seeing someone who is dissatisfied or understanding that someone who is content with their work? That's got to be very difficult to understand context, so I'm wondering how does your team resolve it to the level of context of which the employee is speaking to, and how do you label it properly?
What we found in our questions is it's very hard due to the way our questions are worded. Oftentimes we're asking people what can we do to improve, and therefore we're not getting that some of the traditional methods of examining sentiment does not allow us to get really good sentiment from that. In other words, when employees are making suggestions for improvement, they could consist of a combination of positive and negative sentiment, and so we've kind of have not been using sentiment analysis specifically in the traditional sense of analyzing all comments and saying, is this one positive or is this one negative? We focus more on looking at what are some of the common themes or suggestions that we're getting out of that. What are some of the common concerns that we are seeing among our employees instead of looking at are they positive or negative?
Now, on the other hand, we have some surveys are done by vendors, and I don't have knowledge of how they go about analyzing their data to determine the sentiment. They keep that since it's proprietary to them, and it was developed in within their company. They don't share that much. Gotcha. Okay, thank you.
Forecasting models and analysis customers
The lots of great questions coming in here. Let's see. Emma had a question a bit earlier on what are your forecasting models of choice for time series, and I wasn't sure if we covered that yet. I don't think we have. I have found many times exponential smoothing works very well for many of the cases and in cases where I've tested different types of algorithms. That one is often turned out to be the most accurate, but I will choose a few different methods to test and see which one works out well.
We don't have a lot of seasonality in our time series. At least I haven't run into a case yet with that, but it's something that I do check for when I first start any new project for a specific business unit and see if that's something I need to consider in doing my model. So I haven't had much occasion to go into the more complex time series models yet.
Thank you. I see Jared just put a question in the chat. Who is the analysis customer at AT&T, the employee or the HR business partner hiring managers? Our analysis customers could be HR executives or business unit leaders who are trying to get insights about the workforce in their specific business unit.
The future of people analytics
I guess something else that I had wanted to ask you. Let me jump over to some of my questions here. I have a lot of questions coming in all different places, but how do you see the role of people analytics evolving in the future?
I am seeing more and more companies are interested in using analytics and are hiring analytics roles. Richard Rose now maintains a website where he posts open people analytics jobs, and that has grown tremendously over the past few months. So I'm seeing more and more companies are realizing the benefit of analyzing their employee data and using that data to help support business decisions. Whereas traditionally, many, many years ago, HR was doing a lot of decisions based on gut instinct or specific scenarios from different business leaders. I think we're moving more and more away from those gut instinct decisions, and more and more companies are incorporating data analysis.
And in our case, we eventually grew to the point, I started doing data analysis 20 years ago, but eventually grew to the point where now we have a central team within AT&T that is focused on our people analytics projects. And we have various team members that focus on reports, and some that focus on data science and machine learning. I think we're going to get more and more efficient and faster at being able to produce those insights to different business leaders. And more, I think, just more advanced analytics and more sophisticated analytics, let me put it that way. I can see that growing and becoming more commonplace among companies.
Whereas traditionally, many, many years ago, HR was doing a lot of decisions based on gut instinct or specific scenarios from different business leaders. I think we're moving more and more away from those gut instinct decisions, and more and more companies are incorporating data analysis.
Advice for teams starting out in people analytics
Yeah, I definitely notice a ton of interest on people analytics posts on LinkedIn, or when we put your meetup recording up on YouTube, there's a lot of views on it. And so it seems like there's a lot of teams that are just starting to get into people analytics. And I was wondering what advice do you have for those organizations that are really just starting to explore, implement people analytics?
I think if you're just starting a team, developing good relationships with your other leaders in HR to find out what are their big concerns, what are their pain points where we could apply analytics to help solve problems would be a big part of that. And that will help you decide what direction you want to take in building your team if you're just getting started. And from there, if you can find the HR leaders, if they are in touch with the business units, they can also tell you what are the pain points of the different business units. And from there, you can decide how large a team you need or what do you need people that specialize in different areas and having a good relationship with each of those different business units.
Since our company is quite large and complex, we found it's very helpful to have people that specialize in serving as a liaison with the different teams and different organizations within our company. And that helps direct them to the appropriate team. But at the same time, they can help get familiar with what are the common problems that this particular business unit is known for experiencing? And are we seeing the same things coming up over and over again in their requests? And from there, can we be proactive in trying to develop other tools to help them answer their questions or solve their problems? And that may be a good place to get started.
If you are on LinkedIn or any other social media where you can network with other people analytics leaders, just within the past two or three years, there have been a lot of people analytics meetup groups developing in different cities. And from there, that's been a tremendous help to be able to go to those meetings and talk to people in other organizations, other companies, to discuss common pain points that we're all experiencing, common solutions that different people have developed. And you get a lot of ideas from those meetings as well.
AI, hiring, and tools
That's great. Thank you. I noticed there were two questions, maybe two questions over on Slido, I'm going to try and combine a little bit. But the anonymous question was, there's a lot of worry about how AI can affect job prospects. And to rephrase the question a little bit, any thoughts on the forecasting for how advancements in artificial intelligence affect hiring?
Going back to what we said earlier in relation to large language models, when those large language models were introduced within AT&T, there were a lot of discussions of what would be the benefit of that and how that would affect people's jobs. I think what we've ended up concluding so far is it does help those who use those tools, it's helping them become more efficient. It's serving as a great resource to get answers to our questions much faster than we could on our own when we were searching Google or whatever other tools. And there's the common phrase that a lot of people are saying is your job won't be replaced by an AI tool by itself. It has trouble making decisions, especially the human aspect of those decisions and ethics and all of the things that factor into the types of decisions that are made in human resources. But people using AI will become more efficient, more effective, and better able to respond to those questions.
I know among our people analytics team, a lot of our people that are doing coding and developing reports have found these tools extremely helpful in getting answers very quickly and much faster than before. And it's enabled all of us to work much faster.
And there's the common phrase that a lot of people are saying is your job won't be replaced by an AI tool by itself. It has trouble making decisions, especially the human aspect of those decisions and ethics and all of the things that factor into the types of decisions that are made in human resources. But people using AI will become more efficient, more effective, and better able to respond to those questions.
Thank you. I see Abigail asked a question, I'll read it out for you, is what are tools, packages, or processes do you personally use most often? Or do you use other languages? And I think that's the end of it actually. Primarily I use R right now. When I first completed my master's program, I alternated between SAS and Python and R and eventually just got to the point where I found it was easier to focus on one language instead of trying to translate from one to another. And so I'm exclusively an R programmer these days. We have a few other team members that work with Python.
As far as R packages, I love Tidyverse. Some of the packages related specifically to time series forecasting like Feast and Fable. And there is Text Analytics. Oh, I can't remember the package names specifically. I'd have to pull them all up to remember them. But at this point I've built my own personal library of R code. And so for different types of projects, and depending on what the project is, I can usually pull up the code that I've used from a previous one, copy and paste, and modify it as needed.
Career advocacy and what makes people analytics unique
Awesome, thank you. I asked this question kind of as a joke in the chat, but I was saying how do we get ourselves to show up in the data as somebody needing a raise? But more broadly, just thinking like, do you have any advice to share with all of us as somebody coming from human resources and people analytics space of how to advocate for ourselves when it comes time for career progressions?
Ah, very good question. In general, I think prepare yourself for that conversation with your supervisor. Do some research. These days there's a lot of information available on different websites to find out what is the average salary for different types of jobs. So you can see whether or not you fall within the expected range in the market. And you can present that data to your supervisor as well. And show your accomplishments. Show the benefit to the company in terms of how that helped the company in your accomplishments.
So instead of just saying, I build a report that does this and provides these insights, think more in terms of how did that help solve a business problem? Or did you save time, become more efficient in order to help our company progress in terms of being able to get more work done, be more productive, and things like that. So do your research, be prepared for that conversation, and bring all of that in and schedule some uninterrupted time with your supervisor to discuss it.
Great, thank you. Yannan asked a question a bit earlier. What is the big difference between people analytics and other kinds of general data analytics? Any subjective data related to people analytics is very dynamic. Is there any challenge to derive a solid insight from that analysis?
If yet, what is it? A lot thrown into that one question. So let's start with, what is the big difference between people analytics and other kinds of general data analytics?
The biggest difference is when we are analyzing employee data. So we have to constantly keep in mind the ethics of what we do with that data. And in our team, we work closely with our legal department. We also have a privacy team that can get involved in our projects and has given us guidelines as to when are we doing things that, even though there may not be a law against it, we feel it could be considered unethical or we just feel this could be an invasion of privacy.
So it's important to think about those things when analyzing employee data. For example, when we're looking at attrition, you wouldn't want to put race and gender in a model because that could be leading to a potential situation where you're discriminating against those employees. So we're constantly thinking about that. We're also constantly thinking about testing our models for bias to see, is it harming any specific group? And I think that's one area that's very different from other types of predictive analytics or data science projects where you're not analyzing employee data. That's one of the key differences there is we constantly think in terms of, is this appropriate to put into our model? And we feel very comfortable consulting with others in order to help us make those decisions when we need to.
The biggest difference is when we are analyzing employee data. So we have to constantly keep in mind the ethics of what we do with that data. And in our team, we work closely with our legal department.
Thank you. So many great questions coming in here in the chat. So I'm sorry if we don't get to every single question today. But I did forget to say this at the beginning, and I want to just add that if anybody that's listening in is hiring for any roles, feel free to share that in the chat. Share the links to those roles. We don't see that as spammy at all. Love to be able to connect people there too.
Joseph asked a question. I see the asterisk, so I'll read it. But how do you keep track of that library of past code? And Joseph mentioned, would love to hear what other people in the chat use for this as well. It's a very simplistic system right now. I have a series of folder. I have one folder on my drive, and I've tried to put things into different categories according to the type of project it is. And from there, that allows me to easily go back and get that code. At the same time, I still have files from each analytics projects that I completed, which has all of the input files and everything else. So, in a separate folder, I just copy the R code only and none of the other files. And that allows me to have a quick reference where I can say, okay, I know I did a model that was using survival analysis in the past for this type of project. I can go to my folder and copy my survival analysis code for the next project and just replace it.
Thank you. Thanks for all the thoughts there in the chat, too. I see Bill had a question. Are people returning to the office, and in what percentage remote or on-site?
People are returning to the office, but I am not directly involved in those projects where they're looking at what the percentages are, so I can't give you much in the way of details on that.
Okay. I see that I missed part of his question, but it was around, like, if anyone was doing, like, a satisfaction assessment from both the worker and management side on that.
Not that I know of, but I have seen reports and articles that are coming from independent consulting firms where they've done nationwide surveys from employees to measure their satisfaction with regard to returning to the office or continuing to work from home.
We are fully remote at Posit, but I have been trying to go to the office at least, like, two days a week now. Let's see. We have a few more minutes here, but a question I always ask our leaders is, is there a piece of career advice, Liz, that you've either received or that you've given that you'd like to share with all of us?
Career advice and continuous learning
Best career advice I've received is continuous learning. Always be willing to keep learning your skills, keep your skills up to date, and as I've seen various trends, I've even volunteered to go take classes and learn those skills on my own before being asked or told that this is the direction we're going to take. Maybe that's the best advice I can give. Necessarily, when you work in data science, you've constantly got to keep your skills up to date because things are changing, and they're changing fast in this field.
Best career advice I've received is continuous learning. Always be willing to keep learning your skills, keep your skills up to date, and as I've seen various trends, I've even volunteered to go take classes and learn those skills on my own before being asked or told that this is the direction we're going to take.
When you decided to go back and get your master's, did you have a conversation with your leadership at the time, or was that just something that was, like, all outside of work, or did AT&T support that? Yes, I did have a conversation with my supervisor and leadership above my supervisor as well because AT&T has a tuition reimbursement program, and so I gave them the justifications of how my degree would benefit the company and applied and told them I would like to also have it included in the tuition reimbursement program as well, but it gave them all of the benefits of that program. Even once I got to the point where I narrowed it down and selected Northwestern, I then gave them information about, here's all of the information of what I will learn in this specific master's program, and that had to go up several levels in the company in order to get approval for it. That's awesome to have that program. Yes, it is very nice, so they paid for part of my tuition expenses.
Leveraging LLMs for text analysis
That's great. I'm going through to just see some of the questions I missed earlier, so feel free to just copy and paste your question again into the chat here so I see it, or raise your hand here on Zoom, but I saw, Bernie, there was a question I had missed from you a bit earlier, and I might need you to jump in and add some more context on leveraging embeddings. Do you want to jump in?
Sure, yeah, I was curious and interested in how this was leveraging the LLM in her text analysis for the surveys. I was curious if you were like, I guess, kind of your approach to it, were you sending responses into the, I guess, the kernel or whatever it is to get your embeddings back and then kind of put them in categories based on cosine similarity, or was it more like kind of a prompt interface where you're asking it to summarize some group of text and spit it back that way?
More of a prompt interface. I had all of the comments without any identifying data. It was just a list of all the comments that were received from the surveying responses, and in our internal LLM tool, I was able to upload a document that had all of those comments in there, and then I just started interacting with the tool and told them, this is what's in my attachment, and please examine all of these comments and give me a summary of the common themes that were appearing in these comments, and we went from there. It became a two-way conversation with the LLM. As I learned better how to interact with it and tell it what I wanted, then I was getting better responses from it, but it never gave me cosine similarities or any math values or anything like that.
Well, thank you. Yeah, that sounds like a huge help. I'm sure it takes hours of time reading all those comments too. Yes, it does. Thanks, Rachel. Thanks, Liz. Absolutely.
Recommended books and resources
Liz, before we go here, do you have any specific books or resources that you'd recommend to us here? Sure. There are a few books that have been published that focus specifically on coding, and I think some of them also cover Python coding, and have specific examples related to people analytics projects, and one is from Keith McNulty, Handbook of Regression Modeling and People Analytics. Let me see if I can copy this link. He has made it available online. Awesome.
Thank you for sharing that. We have Keith McNulty fans. I follow him on LinkedIn. He often shares good tips, and let me see. There are a few books. Craig Starbuck recently published a book on the fundamentals of people analytics with applications in R, and there's also a book on predictive HR analytics from Dr. Martin Edwards and Kirsten Edwards.
Great. Thank you. Let's see who can post the link to that one fastest.
Bias and MLOps in people analytics
Yeah. I see I had missed a question from you a bit earlier, Rob, on working with an MLOps team. Do you want to jump in? Sure thing. Yes. As people analytics grows, I've seen that there's been more involvement with machine learning operations teams as far as deploying apps, but what advice would you have as far as how do we make sure that we're really taking into account any sort of bias that could occur with the data that's being used to train the models and keep that in mind throughout the whole pipeline?
As far as the employee-based models are, one thing is our privacy team does require that we do testing for bias, especially with regard to race and gender in the results of the models, but we also consider what information are we putting into those models in the beginning and could any of that really relate to possible bias in the outcome. And the other requirement is the privacy team or the ethics team, whichever way you want to refer to them, they often require us to give exact details of here's all of the data that was put into the model. Basically any machine learning models that are using employee data is reviewed by their team and analyzed from a legal perspective as well as ethics and privacy in terms of how we want to work with that. Right now, a lot of our models have been focused on just gaining insights and helping the business make decisions and we're working towards putting those models into production so that we're getting predictions on a regular basis and so that's the direction we're heading to in the future. Excellent, thank you.
Thank you all so much for all the great questions and huge thank you to you Liz for joining us and sharing your experience today. We really appreciate it. Thank you.