image: thumbnail.jpg
Why regression still matters | Keith McNulty | Data Science Hangout
We were recently joined by Keith McNulty, Analytics Leader at McKinsey & Company, to chat about his unique career path into data science, the importance of regression and network analysis, strategies for self-learning and balancing work, and navigating the evolving landscape of AI.
In this Hangout, Keith talks about navigating the evolving landscape of AI, including trustworthiness, risks, and its impact on the data science profession. Keith highlights that AI is rapidly changing and discusses the balance between acquiring technical AI development skills and understanding the trustworthiness and inherent risks of large language models. He suggests prioritizing knowledge of AI's reliability, as these challenges are likely to endure, and recommends identifying trusted sources to sift through the hype.
The discussion also covers AI's capacity to identify complex patterns in large datasets, which can be beyond human capabilities. However, it also acknowledges AI's tendency to be convincingly incorrect. A significant point raised by Keith is his concern that data scientist could be pushed more towards software engineering, and he asked practitioners to consider if they are still engaging in the "science" aspect of data science.
Resources mentioned in the video and zoom chat:
🔗 Keith McNulty on LinkedIn → https://www.linkedin.com/in/keith-mcnulty/
🔗 Keith's Regression Book (Handbook of Regression and People Analytics) → https://peopleanalytics-regression-book.org/
🔗 Keith's ONA/Graph Book (Handbook of Graphs and Networks in People Analytics) → https://ona-book.org/
🔗 PositConf Registration (including virtual option) → https://posit.co/conference/
🔗 "Storytelling with Data" book → https://www.storytellingwithdata.com/
If you didn’t join live, one great discussion you missed from the zoom chat was about the evolving role of data scientists and the importance of "putting the science back in data science". Let us know below if you’d like to hear more about this topic!
â–º Subscribe to Our Channel Here: https://bit.ly/2TzgcOu
Follow Us Here:
Website: https://www.posit.co
Hangout: https://pos.it/dsh
LinkedIn: https://www.linkedin.com/company/posit-software
Bluesky: https://bsky.app/profile/posit.co
Thanks for hanging out with us! 💛
00:00 Introduction
00:46 "Will you let us tell us a little bit about yourself, what you do, what you like to do for fun?"
03:10 "Were you that active (on social media) before you were in data science?"
05:33 "I was just wondering if you could maybe talk a little bit about your most favorite resources that you used while retraining yourself as a data scientist?"
10:12 "You wanna give a quick pitch for both of (your books) so we have some context?"
14:52 "What do you think about the possibility of extracting latent character traits about people from text?"
19:51 "How do you balance self learning with working time?"
25:16 "What kind of questions are you using networks to ask in the people analytics space?"
30:09 "What would you say is the most important AI related skill to learn for the future and to stay ahead in the workplace?"
35:59 "I'm interested in how talent is measured among white collar workers. Could you share some of the metrics you use and how you address the limitations of more straightforward measures like time in the office?"
40:26 "So I was wondering if that (early) role, that engagement role, helped you bridge this switch from academia to industry, or, how did that role help you choose the skill you wanted to build."
43:54 "I just wanted to get your perspective on imposter syndrome, especially for people who are making some sort of transition in their career into data science."
47:00 "Where do you source your daily math problems?"
48:19 "Do you have any thoughts on policy for AI in the current setting and and also, specifically thinking about there's a lot of people in my company that are hesitant because of the environmental impact of AI."
51:32 "What did we not talk about that you wish we had talked about?"
Transcript#
This transcript was generated automatically and may contain errors.
Hey there, welcome to the POSIT Data Science Hangout. I'm Libby Herron, and this is a recording of our weekly community call that happens every Thursday at 12 p.m. U.S. Eastern Time. If you are not joining us live, you miss out on the amazing chat that's going on. So find the link in the description where you can add our call to your calendar and come hang out with the most supportive, friendly, and funny data community you'll ever experience.
Can't wait to see you there. I'm just really super excited to introduce our featured leader today, Keith McNulty, Analytics Leader at McKinsey & Company. Keith, welcome. Will you let us tell us a little bit about yourself, what you do, what you like to do for fun?
Yeah, thanks, Libby, and thank you guys for inviting me. It's a lot of fun to spend an hour speaking with such a diverse and interesting community here. Yeah, my name is Keith McNulty. I kind of navigated a path into the data science world in what's probably a little unusual a way. I think nowadays people are coming into data science having done some sort of data science-related qualification, whereas I started out as a mathematician. I then went into management consulting. I then moved into psychometrics, which is a fascinating field which I still am very involved in, and the study of employee behavior and how it affects the workplace. And then about maybe seven years ago, I caught the data science bug and decided to retrain myself as a data scientist, built a team at my employer, which is now a fairly large team that kind of focuses on the applications of analytics to solving questions related to people and talent.
And in terms of what I like to do for fun, I mean, I just love engaging with the open source community. It's how one of the reasons why I've managed to have a lot of success in my current career is because of the contributions of a lot of open source individuals and the fact that I've been able to take advantage of that and my team has been able to take advantage of that. So I believe a lot in giving back. So I spend a lot of time developing materials and publishing open source books and things like that to make sure that things that I've learned are passed on appropriately to other people so they can take advantage of that as well. And I think, you know, for me, one of the things that attracts me to working in this space is the amazing open source community and the way people like help each other. So that's something that, you know, I'm happy to talk more about today, but it's definitely the thing that attracts me to working in this field.
I mean, same. The people are the biggest differentiator for me, no matter where I go or what I'm doing. I first learned about you via LinkedIn. You're very active on LinkedIn. Were you that active before you were in data science? Were you like already out there spreading knowledge, writing books, doing all the things that you do? That sort of like, oh, there's a big community here and there's a place where I can connect.
I certainly wasn't very active before I got involved in data science. I think part of that is because my involvement in data science kind of coincided with when LinkedIn became more of a social network. But certainly, you know, there was this very exciting period on LinkedIn, probably between 2016 and 2018, where people were discovering a lot of this tooling for the first time, lots of people. And there was a huge amount of sharing of kind of open source knowledge. And a lot of that happened on Twitter as well during that period. And so, you know, one of the reasons, one of the things I try to do is keep that going. I think, unfortunately, LinkedIn has become a bit more salesy and a little bit more, you know, it's kind of its culture has changed a little bit since that time. But I certainly personally try to keep that spirit going and making sure that like what I share there is useful to people and, you know, that it's something they can take away with them and potentially use themselves.
Well, it's definitely changed, for sure, since I started using it, I think the first time I was forced to use LinkedIn was in undergrad, I was forced to create a LinkedIn account, someone made me use it. And I thought, what is this thing? I'm never going to use this ever again. And now I'm on it all the time. And I have found it to be a social network. And if you curate your feed the right way, it can be pretty nice.
Well, I already have some questions popping up in Slido, which I feel like never happens this early. So let's go ahead and hop over to Slido. I will sacrifice some of my selfish questions I wanted to ask Keith. I bet somebody is going to ask them in Slido, so that's OK.
Tony, you asked a question and I don't see an asterisk. Tony DiPesa, are you able to answer that? And did I say your last name right?
Close. DiPesa. DiPesa. Sorry, Tony. No, no worries. In Italian, it's probably different, but that's the Americanized version.
So, hi, Keith. Thanks for sharing your information today. I was just wondering if you could maybe talk a little bit about your most favorite resources that you used while retraining yourself as a data scientist. Things that seem to be the most helpful or most enjoyable, engaging.
It's a great question, Tony, and I'm not sure my answer would necessarily align with how people would retrain themselves today. And the reason I say that is because I retrained in 2016, and there was a significant limit to the amount of public information that was available at that time in terms of example code, how you might get something done if you want to achieve a particular task with your code. And what I found was that the learning during that period for me was in a huge amount of trial and error.
So I'd write code and I'd just watch it fail. And then I would dig into Stack Overflow and some of those kind of resources that date back quite a while to try and work out how it failed. And sometimes I wouldn't be able to find the answer, and I would just keep changing my code until eventually it worked. And then I'd realize why it worked. And that type of process was a huge part of the learning for me to get underneath why your code works.
And of course, the resources that got me started are the resources that a lot of people still rely on today. I started out in R. I'm a big user of both R and Python now, but I started out in R. And of course, Hadley and Garrett's book was huge at that time. You know, their early editions of that were out to get me started with the basics of using the language. And there was a little bit of online resource, but that only took me so far. It was really having my own data set and trying to do my own tasks with my own data that really kind of forced me to learn. And I learned almost entirely through trial and error.
On learning to code and AI tools
And I think there's an element of that that we still need to have today. So I think that if programmers are not learning to program through trial and error, then they miss out on a lot of learning. In particular, one of the things that I see is people are using Copilot to auto-complete their code lines. And then their code lines don't work, and they don't know why they don't work. And so they have to go back and dig into it. And that process is very valuable, right? It's how you actually learn what's going on under the hood.
And then what I found is later on, after I'd gone through that initial kind of hump, and I'd, you know, worked out how to get my code working, I always say there's this period of success, where you kind of, you've passed the initial hump of success. And that point is when you type a line of code and you expect it to work. So there's a period you go through, you type a line of code and you say, I kind of know this is going to fail, because I don't understand it well enough yet. But then there comes a point where you say, I type a line of code and I'm surprised if it fails, right? When you hit that point, that's kind of your first step up point in learning to code, I think.
When you hit that point, that's kind of your first step up point in learning to code, I think.
And then once I passed that point and got further past it, I then started to, you know, use a lot more advanced resources. A lot of, for example, Hadley's more advanced books on R. And started to really get under the hood of that language before I moved on to Python. So it was, it happened in various stages, right? But I think that the message I want to get across is the actual resources are not helpful if you're not actually applying them to your own context and data sets, because that's how you really learn, I think.
I really like that answer, because it's the answer that I've given a lot of people. And it makes me feel a little bit more justified, because I get asked a lot, should I learn this or should I learn that? Should I learn R or should I learn Python? You use both, what do you think? And my answer is always, you should go do both of them. You should go do whatever you find out there and try it, and physically type the things and do it. And you will figure out from there what tools best suit your needs, what brings you joy, what community you find around the stuff that you're doing. Go do the thing.
Keith's books on regression and network analysis
And Keith, we're talking in the chat about how your book is actually the resource that a lot of people are now using to get into data science. So this is sort of a full circle moment. I'm going to put the link to his book, well, one of his books in the chat really quickly. And I figured that I would go ahead and just ask you to talk a little bit about your books. I put the regression one in there, but you have another one. You want to give a quick pitch for both of those so we have some context?
Yeah, sure. There's two areas I think that I learned very quickly were critical to the space that I work in, which is the understanding of people and talent and how they operate within organizations, but obviously have much broader applications. So one area is regression, and the other area is organizational networks or network analysis and graph theory in general.
So regression is something that I think people have lost sight of a little. And one of the reasons why I wrote a book on regression was to try and get that back into the forefront of people who are particularly interested in doing data science to explain things. So we ended up getting into this situation, I think, in 2015 to 2018, where a lot of the new kind of Python scikit-learn toolkit came on board. There was a huge amount of machine learning algorithms that weren't very explainable coming onto the... And Python and scikit-learn make them very easy to use. So we got into this situation where data scientists were jumping on and just running algorithms and getting results from them. But a lot of people were having problems understanding those results, and in particular, understanding that those results were predicting something, but they weren't necessarily explaining something.
Now, in a lot of fields, like my field in econometrics and sociology and a whole bunch of other fields, the explanation of what's going on is more important than the prediction. And so regression is just an incredible Swiss army knife for doing that type of analysis, for being able to look inside a kind of phenomenon and understand what's actually going on. How do I explain how one thing leads to another? And so one of the reasons I wrote the book was to get it back into the kind of consciousness of data scientists and say, there's a whole toolkit here, which you're kind of forgetting about, which has a lot of applicability in kind of explaining the problem you're dealing with.
And so regression is just an incredible Swiss army knife for doing that type of analysis, for being able to look inside a kind of phenomenon and understand what's actually going on.
So, and part of the book doesn't just cover the code and how to execute it. A large proportion of the book covers like when you get the results, what do you do with them? Like, what do they mean? How do you explain them to others? How do you like make yourself a compelling data scientist by being able to really well explain the outputs of your models, right? So that's kind of one of the books. And I'm actually working on a second edition of that, which is going to be fairly expanded. So I've got a busy year ahead of me. Oh, that's awesome. On that one, yeah.
The other one is kind of more niche, which is around network analysis and graph theory. So, you know, it's a huge area. I'm fascinated by the fact that data can be structured in ways that are different from rows and columns. So the idea behind a graph is that, you know, your kind of entities are stored in nodes and your relationships between them are stored in edges. And it's just a beautiful data structure to work with. And it solves a lot of problems that like typical tidy tabular data does not solve. And so the role of that book is like to get people who are interested in those types of data structures and have applicability in their work or their studies like fluent in how to use them in R and Python. I even do a little bit of introduction to graph databases, which is kind of the real burgeoning technology in this space now. So that was a joy to write that book, but it's a little bit more niche. I don't get the same readership on that book as I get on regression books.
As someone who comes from operations research side of things and mathematical optimization, same. I feel like whenever I talk about things like graphs and nodes and edges and stuff, people's eyes sort of like glaze over. But it's such a powerful tool to solve so many types of things. And you mentioned when I talked with you last that the results of those types of models can sometimes be a lot more intuitive to people to understand and explain, which I think is great.
And as we're talking about psychometrics, we're talking about people data. I would love it if Arsenis could go ahead and unmute and ask his question because I thought it was interesting. Thank you. Arsenis, are you available? There he is. Hi, Libby. Hey, thank you so much for joining us, Keith. And I'm going to cheat a little bit because I'm going to double up here and ask my question that I put in Slido and one more about something you just asked.
So the question I put in Slido was, given your background with psychometrics and data science, what do you think about the possibility of extracting
latent character traits about people from text?
The reason I'm asking that is this is something that I'm thinking about. There's a project that I'm thinking about that would be helpful for employers looking at individuals from recommendation letters, all that. So if I gave you 10 letters about me, is it possible to extract character traits rather than just my skills?
And if it's OK, I'm going to ask, do you have a title for the book on graph analysis? I'm presently getting into a project where I'm doing network analysis on skills. I work for Indeed. And so this is kind of my domain. So, yeah, those two questions.
So I'll answer the second one first, which is basically any technical book I've written starts with handbook, the word handbook. So it's handbook of graphs and networks. And that's deliberate because I want the books to be very practical. So I want you to be able to bring them up, open up the chapter and be able to get going with them.
So the first question is a more kind of complex question, of course. I mean, in theory, the answer is yes, because text, the way that people express themselves and the way that they write words and the order in which they write those words and the types of words they use are indicative of their background and personality. And but the problem is that you can't really identify that if you don't have an appropriate model of personality that you're trying to map it against.
So you can't really say to a model, here's a bunch of text. Tell me what this person is like, right? The model is unlikely to give you anything that's useful or structured from that. But it is absolutely possible to say, here are a bunch of constructs of people's personalities. Here are the you know, how we're defining those constructs. Can you kind of, you know, use can you use the language in this data source to map individuals to these constructs?
The I'm also very interested in the role of AI in this space. So I'm like not a massive AI convert, right? I approach AI very, very carefully because of kind of my not my under the hood knowledge of how it operates. But areas where I think. AI excels is where it can identify patterns that are kind of beyond the capacity of the human brain to identify, right? Because the amount of data is just, you know, too large to handle. So, for example, we all a lot of us know situations where there's large company surveys where, like, tens of thousands of people have written text, and it's just not possible for one person to read through those and synthesize them, right? So this is where kind of AI has a really big or large language models, in particular, really big role to play.
And I would, you know, there's things that we should consider. For example, one way of approaching your problem is to say, here's, you know, here's a bunch of letters that have been written by an individual. Now, I want you to use those letters to kind of extrapolate how this individual would respond to the following situation. And kind of see what that tells you, right? So I think AI gives us kind of more ways of looking at this problem than we have previously. Previously, we would just have to rely on, like, breaking this text up into chunks and using statistical models to relate it to, you know, to constructs that we're interested in. Now we have, like, methods through which we can extrapolate to new situations and ask, you know, models to determine, based on this input, how might somebody behave in this particular situation or respond in this particular situation? So those are things I haven't done a lot of work in, but I think are definitely worth a lot of exploring with this new technology that we have.
Book resources and availability
Amazing. Thanks for the questions, Arsenis. And there are so many links in the chat right now. But I will say, Keith, both of your books are available openly, right? Yes, they're completely online. They're completely online. If you would like a physical copy of them, you can absolutely find it. But there are links. One is, like, ona-book.org. So look for those links there. To check those out right now, you could go scroll around and look.
Balancing self-learning with work
All right. So we have so many questions on Slido. Will we fit them all in today? I am not sure. But there was one. Let me see. There was an anonymous one that I really liked. Let me see if I can find it. Aha! Here it is. It was an anonymous question that was about, how do you balance self-learning with working time? And it says, I know you do a math problem every morning, but do you also plan structured study times? Tell us a little bit about your math problem every morning. And I also see you do them on plane rides. I love it so much.
The math problem every morning kind of started by accident. I started to do a little bit of math teaching about three years ago. And I just enjoyed the kind of cut and thrust of working with the student to kind of solve. Some of these problems were very hard, right? And I enjoyed the cut and thrust of working with the student to try and solve it. And I kind of enjoyed the euphoria of solving it. It kind of brought me back to what it was like when I was a student again myself. And then I thought, you know, if I did this every morning, what a confidence boost that would be, you know, to like get started every morning having said, look, there's a hard problem, which I've already solved, you know. And so I got into that habit. It doesn't work out every morning. You know, sometimes I have to go 10 mornings in a row before I finally solved the problem. But it is a really good kind of confidence boost and exercise for the brain, I think. But a lot of people disagree with me and say, the first thing they need in the morning is a cup of coffee.
But it is a really good kind of confidence boost and exercise for the brain, I think.
But so that reminds me of the other part of your question. I got caught up in the math problem part. The other one is just how do you balance this sort of like intentional learning time with your work? Yeah. I've got better and better at this, I think. Because first of all, like, if you're in work that really interests you, then work time is learning time, I think. Because you're doing problems and you're cracking things that require you to stretch your brain and do new stuff.
And one of the things that I've always been kind of very conscientious about is, if as part of solving something in my day-to-day work, I've come across something that's reproducible, you know, like, you know, it's a method or even a few lines of code where I thought, you know, I can see how I could use this for a lot of other things, then I'm very deliberate about grasping that and making sure that I've recorded it. And, you know, it's kind of part of my toolkit. And if I think it's very generalizable in the sense that, like, I'm, by sharing it, I'm not sort of giving away any proprietary information or anything like that, then I'll also share it with the broader community and say, like, look at what I did today. I think this is super useful. Maybe you guys would find this useful too.
So that's something I've always deliberately done. In the last maybe two or three years, my job became extremely busy. So it became harder and harder to actually do real technical work. Because one of the things you get in more senior positions is you get dragged into a lot of meetings and a lot of things that, like, take you away from hands-on keyboard work. And during that period, one of the things I deliberately did is kind of I tried my best that it wasn't always successful to kind of designate my Fridays as coding day. And I would make, like, a lot of people feel bad about interrupting my coding day. And so that created an environment where I did get a lot of time on Fridays to actually do technical work. And I would, like, hold it off until Friday if I wanted to do it.
So if there's any of you kind of on the call who are kind of struggling with that balance, because I know it's a really common problem, you know, like, where do you find time to do technical work versus have to sit in meetings and, you know, where you can't get hands-on keyboard? One tip is, like, designate a block of time for it. And another tip is, like, don't be afraid to make your colleagues feel guilty about interrupting that time.
Yeah. Actually, Rachel just mentioned in the chat that she and I are both trying no meetings Wednesdays. So that was yesterday for me. And it was the first time I'd actually achieved it. It was the first week since I started trying it that I actually got declined all the meetings, moved everything around. And I got so much coding done. I did so many productive things. I caught up on so many admin tasks. Highly recommend. It's not always easy. But also, I will say, I took a pre-calculus course a couple of years ago through, I think, Arizona State University on edX or something, because it's free. Gosh, go do that. It was so much fun. I enjoyed that pre-cal class so much. I highly recommend, like, getting some math in your day. That high that you mentioned of, like, solving a problem is so good. It's also it's kind of like squashing a bug in your code.
Okay. Very similar. It's so similar. Thank you so much, anonymous question asker. That was a great question. I wish you had put your name on there. DM me and say, hey, I asked that question. So I can thank you.
Okay. We have so many more. Let's go find some more great questions on Slido. Let's see. Let's see. Nicole had a question. Nicole, will you be? Nicole Burke, if you're available. You want to unmute and ask.
Hi, Keith. It's really nice to meet you. So I am a big fan of your network work. I am a psych PhD who also did a lot of network, like, analysis in my doctoral work. So, yeah, I just kind of wanted to ask you, like, what kind of questions are you, like, most excited about, like, using networks to kind of, like, ask in the people analytics space? And do you kind of see, like, networks still, like, taking off? Like, I know in that space, I know it's kind of, like, slow adopting or slow moving. So, yeah, I would just love to kind of hear your thoughts about that.
Network analysis in people analytics
Yeah, thanks, Nicole. And whenever I hear that somebody's actually using stuff that I put together, it always makes me really happy. And so thanks for telling me that. The, I guess, in terms of whether people are using network analysis and that type of approach really depends on what I found is it really, really depends on the data that they have available to them. So, one of the big blockers to people using a network based approach to their analytics is the fact that they don't have data in any sort of form that you can put in a graph.
And I actually dedicate a whole chapter in my book to, like, how do you transform rectangular data into graph data for the purposes of solving a problem? Because that's the biggest blocker that I've seen. You know, like, if you go into a random organization, you show them a graph, a way of working with things in graph, they'll say, that's great. But none of my data looks like that.
But that said, if you can get your data in that form, there's immense number of really impactful problems you can solve with them. And some of them are even problems that you wouldn't think that you would be able to solve with graphs, right? So, a couple of examples, a couple of, you know, a few years ago, I was faced with this problem of, like, a really open problem of, you know, if we, if we were to reorganize our organization with different organizational units, you know, than the ones we have today, what kind of organizational units would be most optimal to ensure that people collaborate with each other better?
And I basically created a massive graph full of a bunch of employees, got data on, like, their current collaboration, you know, populated the edges with that, used various community detection algorithms to try and identify, like, where are the hotspots of collaboration? And very quickly, it was obvious that there was one way of organizing that was particularly, you know, conducive to high levels of collaboration. And to be able to put an analytic point of view behind that is, like, a really powerful, you know, because most people view organizational, you know, network analysis and organizational behavior as a very, like, touchy-feely thing that that's not very data driven. So, to be able to actually drive a data-based approach to that is huge. And to be able to put good visualizations behind it, and this is along the lines of what Libby was saying earlier around making things intuitive. You know, if you can visualize a bunch of dots with those dots coming together more in a particular framework than in others, you're actually, people can actually see the dynamics of what's going on in front of their eyes. And that can be hugely impactful to them, you know, believing what you're putting in front of them.
So, and more recently, you know, we've been using graphs to solve questions of time and how people are using their time to be able to make inferences of what people are doing with their time. You know, and even then, you don't have direct data on that, but the inferences you can make from the connections in your graph can make you, like, very, very certain that, like, a certain type of activity is being done by people at a certain point in time. And that helps you kind of analyze, you know, a lot of things about, you know, efficiency, workforce efficiency, and a bunch of other topics. And so all of these things are very possible. But one of the things I will say is we've done a lot of work on engineering our data well to be able to go into graphs. Without that, it's much harder to kind of get started on that topic, I think.
Important AI skills for the future
What a good question, Nicole. Thank you so much for asking. All right, we have one that has risen to the top here. This is a question from Ashley Wilson. Ashley, if you are available to unmute and ask your question, that would be great. There she is.
Hey, Dr. McNulty, how are you? Good, thank you, Ashley. And thank you for picking up my question. I think I had written, yes, as a self-taught data analytics person and computer programmer. What would you say is the most important AI-related skill to learn for the future and to stay ahead in the workplace? I know you've written about, you know, things such as AI agents, but what do you see as more important, like, whether it's, like, for me to learn if I wanted to get a good job or really impact the business, whether it would be specifically knowing the techniques or helping organizations know how to implement AI, something like that?
Yeah, thanks. That's a really, really good question.
And first thing I would say is, like, AI is moving so quickly and at such an early stage that anything I say now could be very dated in a year or two, right? And I think that's important to call out.
But I think there's a balance here, right? One part is, like, learning the technology behind AI, right? So there's a lot of very rapidly moving technology. As these models develop, you've got, like, things like tool calling. You've got to learn things like data validation. There's a whole set of tooling that's involved in building an AI agent, for example, that you could think about skilling yourself up with, right?
But on the other hand, there's the whole issue of the trustworthiness of AI and the likelihood that the response you get from a large language model is usable without a lot of risk to your organization or to the work that you're doing. And that's much less of a technical, at least it's much less of a coding problem and much more of a kind of how knowledgeable are you about how these models work and what the benefits and risks of using them are.
I would probably say that if you are not in a situation where you have to actively build AI workflows or AI technology right now, I would limit the amount of time that you're spending learning how to, like, code agents and things like that. Because if you're not doing it right now, you don't really have a use case to use it And by the time you do have a use case, it could have changed a lot between, you know, between now and then.
So for that side of the things, I would be saying, you know, stay in touch with the latest developments, but you don't necessarily have to have fluent hands-on keyboard skills for that unless you're directly working on something where you're building something related to it, right?
On the other side of things, I think these issues of how trustworthy are the results of large language models, when they're safe, when they're not safe, those issues will persist and they'll persist for a long time, possibly forever.
And so being knowledgeable about that and being able to, like, advise appropriately and, you know, to be able to deal with super AI enthusiasts, to be able to, like, pull them back and say, hold on, there's a few things you need to think about here before you go gung-ho on this. That skill is very useful in all contexts, I think. So that's one I would index on heavily. It's like, stay in touch with what the experts are saying. Look for a set of people that you trust in terms of what they're saying about this, because there's a lot of hype out there, which you have to be very careful of.
Try to find a few people you can trust, follow those people, read about it, make sure you're staying up to date on what we're discovering about these models. And that is, like, something I would work on, you know, now and make sure you're staying fluent in it, whereas the technical and coding aspects of it, that depends on your situation and whether you need it right now or not.
On the other side of things, I think these issues of how trustworthy are the results of large language models, when they're safe, when they're not safe, those issues will persist and they'll persist for a long time, possibly forever.
Questioning AI outputs
Awesome. Great advice. Great question, Ashley. Thank you for asking.
And I will chime in and say, with the AI stuff, just question everything. It's, like, the biggest advice I can give you. I wrote something a couple days ago, and I decided to hand it to Claude and be like, Claude, what do you think of this script? A script that I had gotten, ChatGPT's help writing. I didn't write all of it myself. I got kind of assistance. Gave it to Claude. I said, is this going to do what I think it's going to do? And Claude said, no, it's not going to work for all these reasons. And it listed, like, eight big reasons that were, like, these are never going to work. And I'm, like, I think Claude's wrong. I think all these are wrong. Ran my code, tested it. Claude was wrong. I came back to Claude and went, you're wrong. You have to make a point. My code's totally fine. And it's, like, oh, I'm sorry. You're right. It is fine. Here's all the reason why it's great and it's going to work. I'm, like, okay. So question everything.
All right. Thank you so much for that question. Let's go find another fantastic question.
Topological data analysis
I feel like we had another one that I really liked that was anonymous. And it was about math. There we go. I did an anonymous question that says, interested in topological data analysis, but intimidated. Are there any resources or packages you would recommend exploring? I am not sure if you're in the topology mathematical space, Keith.
Oh, that one. So, I mean, I did study topology when I studied mathematics. But as it applies to data science, not hugely knowledgeable on that topic.
I do hear topology used a lot in terms of networks. And I think that that tends to be a description of, like, how your network is designed and, like, what are the determinants of what defines a node and how you're connecting your nodes. Topology in general has a kind of a meaning of, like, how are you shaping things in the way that you're working. But outside of that, those general comments, I wouldn't want to pretend I'm an expert on that topic by any means.
Well, I don't either. If there's anybody else in the chat who happens to know a little bit more about topological analysis, leave some resources. For whoever asked that wonderful anonymous question, thank you, anonymous asker.
Measuring talent among white collar workers
It's from Trey Hendricks. And it says, I'm interested in how talent is measured among white collar workers, since it seems challenging to operationalize. It does sound challenging to operationalize. Could you share some of the metrics you use and how you address the limitations of more straightforward measures like time in the office?
Yeah, thanks. It's a really interesting question. And it obviously takes the time up of a huge number of what we would probably call psychometricians nowadays, whose kind of job it is to determine ways of measuring constructs related to success in the workplace. And certainly for topics like time in the office, that comes down to almost entirely like what data you have available to you, right? It's quite possible that in a lot of organizations, there's just no way of measuring that if there's no systems in place to be able to capture it. Whereas in other organizations, you might have like badge in, badge out systems. And that might give you fairly accurate views of people's time in the office. But you're also dealing with questions of privacy about whether you should even be using that data or not. And so there's a lot of complexities around that question.
I would break it into kind of two components, which is there are a set of constructs which we kind of organizations impose on that relate to what they regard as success of their employees. And I'm specifically referring to white collar employees with this. And those are usually designed by like psychometricians and talent teams, and they'll often be part of development structures or evaluation forms, all of those things. And those things tend to be developed on a theoretical basis using research. It's usually qualitative research with your employee base.
But then you have this question, which is more interesting for me as a kind of quantitative individual, which is what data can you get to kind of validate those constructs? Right. So one of your constructs might be that you're successful in a particular job over a particular time period. And there's several constructs, several measures which you could use to validate that construct for individuals. One is what's their performance rating? But what if your performance rating system is rubbish and everyone gets the middle performance rating and nobody gets anything else, which is quite common in many environments? You also have promotion, right? But what if you're in an environment where promotion is automatic after a certain period of time? Then that's not a great differentiator of success.
So a lot of what my work involves is how do we take data that could be indicators of a certain construct and use it in some sort of intelligent way to try and answer some of the questions. And often there's a lot of data that we'll throw out because it's just not useful from an analytic perspective. And there's other data that turns out to be very related. And one of the things, going back to network analysis, to be able to understand like colleagues networks and how they interact with other people, that turns out in many, many use cases to be a very, very valuable indicator of a lot of things with white collar workers, right? If they have large networks or if they're very central to their networks, those sorts of indicators I found are often very valuable. But it's a hard question. It depends on the context. There's a lot of subjectivity to it.
But the job of like a quantitative individual like me is to try and match those constructs to some sort of data indicators.
Yeah. And to use your subject matter expertise and all of your education and stuff to sort of bring that nuance to the forefront and say, this isn't an easy answer. This is a tough answer. We have imperfect measures, all of that stuff.
From academia to industry
Thank you so much for the question, Trey. And Cecilia had a question that really ties on to this one. Speaking of the sort of subject matter expertise, Cecilia, would you like to hop in here and ask?
Yes. Hi. So my question comes from the fact that I talked to you on LinkedIn and I saw the path that you had and I found it very intriguing because from a PhD in math and then the first role you had in industry was an engagement manager role and then into technical, more technical data science. So I was wondering if that role, that engagement role, helped you bridge this switch from academia to industry? Or like how did that role help you maybe like choose the skill you wanted to build? Thank you.
Yeah. Thank you, Cecilia. I don't get to talk about this part of my career very much. So it's great that you brought it up. It was a real baptism of fire to move from academia into industry. Right. And I think anybody who's done it probably kind of can relate to this, particularly as a mathematician and a pure mathematician at that. You know, the way I describe it is it's a very enclosed and protected environment. And because of the kind of intellectual kind of nature of that environment, there's very little investment in how you communicate as an individual. So a lot of the attitude is, you know, if you don't understand what I'm saying, that's because you're not as smart as I am. Right. And that's kind of a protective envelope that a lot of like, particularly in mathematics, there's some that kind of prevents you from having to work hard at how you communicate because you've got that sort of comfort blanket.
But when you move into industry, it's like, you know, sometimes more than half of getting to the right outcome is how you communicate your approach and how you solve your problem. Because if you can't, if people are not bought into the solution, they won't go along with it. And all that work is wasted. Right. They have to believe what you're saying. And that was just I had to learn that skill because that was not something that was in any way taught to me in my academic career. And that took maybe a year or two to really build my confidence there. And it was incredibly enriching because I don't think I'd be the person I am today if I hadn't have gone through that. You know, certainly I wouldn't be a good communicator by any means.
And and I think that's I've taken that through with me because it's taught me the value of communication. Right. I think if there's a danger that if I'd gone straight from academia into into data science, I might have still had that mindset of, look, you don't understand my code. That's because you're not as smart as I am.
But I know now that writing the code is one thing, explaining the outcome of the code and being able to convince people that you know what you're doing and that they can trust you because you have good, good knowledge and a good foundation to be able to listen to their questions and understand what it is that's concerning them, to be able to give you give tailored responses to that. All of those things I learned in those first three years of my career, I think, which have served me well in the long run.
But the bottom line is I love the technical work. Right. And I was always going to come back to something where I was really deeply using my brain to kind of solve interesting technical problems. I just have that additional toolkit now, I think, because of those early years of my career.
But when you move into industry, it's like, you know, sometimes more than half of getting to the right outcome is how you communicate your approach and how you solve your problem.
Imposter syndrome and career transitions
Thank you for the fantastic questions, Cecilia. Everybody really liked that one. And thanks. Thanks, Keith.
I had a question. Let's see if I can find it again. That is a good follow up to this one. It was Siraj. If you happen to be able to unmute and ask your question, I would love that. We only have about 10 minutes left. Let's see if we can fit into just a couple more questions for Keith.
Sorry, Libby. I didn't find Siraj fast enough. I was just checking. Oh, that's OK. I think I found Siraj. Thank you.
Hey, guys. Sorry, my mic is on mute. Thanks, Libby. I'll make it quick. I just wanted to get your perspective, Dr. Keith, on imposter syndrome, especially for people who are making some sort of transition in their career into data science. I myself used to work in marketing before. And I'm kind of making a career change into data science, although I did some work in data analysis and a little bit of modeling before. So it's a question that a lot of people have. And I just wanted to get your perspective, since you also made a transition of sorts and also maybe you've managed teams where people have started off as a new data scientist in the field. So yeah, just happy to hear your thoughts.
Yeah, thanks, Siraj. I mean, imposter syndrome is a really big thing in many, many fields, right? Anytime you kind of change your environment and you often have to join a bunch of people who you perceive to have substantially greater knowledge than you in the field is a massive opportunity for you to feel kind of imposter syndrome. And it's perfectly normal, I think, to feel it. And I felt it myself on many, many occasions.
I honestly believe that you overcome imposter syndrome through, first of all, you have to be in the right field, right? If you are not feeling comfortable in the field you're working in, if you don't have
that passion for it, if you're not interested in it, it's likely that that imposter syndrome will continue because the incentive is not there for you to become more knowledgeable and to become more fluent at it because you don't enjoy it, right?
So that's the first thing I would say. If you are lucky enough to be working in a field that you love, you get over your imposter syndrome through good hard work and collaboration with your colleagues and learning, right? Where you go through enough experiences that you get that pattern recognition so that you can say, you know, I've seen this before. I know what to do. And over time, you start to have success from that. And that's the confidence building, which helps you get over imposter syndrome.
So I mentioned previously, like, this period when I was learning to code and where, like, I got to a point where I realized, hold on. Now, when I'm typing my lines of code, I actually expect them to work. That's, like, a huge step up for me. That was, like, step one of me overcoming my imposter syndrome, right? Because then I could get on calls with my colleague and I could, like, live code and not be worried about embarrassing myself, right?
So there's various ways in which you can overcome it. But the first thing I would say is you're probably not going to overcome it if you're not working in a field that you enjoy. If you are working in a field you enjoy, it's hard work and learning. And, you know, putting yourself out there and making the mistakes that help you learn is what helps you overcome imposter syndrome, I think.
I just said the same thing in the chat. The evidence is what leads to confidence, but you can't get the evidence part until you actually do the stuff, which requires failing a lot. Suraj, thank you so much for that question. That was wonderful.
The evidence is what leads to confidence, but you can't get the evidence part until you actually do the stuff, which requires failing a lot.
Can we fit in a couple of more questions in the eight minutes that we have left, which is really more like six minutes? Let's take a look at Slido here and see what we've got. I saw a really fast one you probably answered really quickly. Kenneth had asked, where do you source your daily math problems?
I'm very interested in, like, when people leave school at the age of 18 before they start college, they are making a big gap from kind of general maths into much more specialized math. And particularly hard math problems that bridge that gap are really, really interesting, right? Because you need a broad knowledge of math to solve them. And they're not so specialized that you think, I don't have the knowledge to solve it, right? So that kind of period of transition from high school to the start of college is kind of where I source a lot of the math problems. I often take them from, like, exam papers and things like that.
Exam papers. All right. Interesting. I bet you could find a lot of those online. Thanks for asking, Kenneth.
What else do we have here in our Slido that we can hop in here and maybe answer quickly? John had asked about actually putting a metric on productivity. I feel like we talked a little bit about that, about how difficult that was.
Liz, you were asking about AI governance. Would you like to unmute and ask that one? Elizabeth Eisenhower, by the way, is who we're asking to unmute here.
Yes. Can you hear me? Yes. Hi. Thank you. Yeah, I was just curious if you, because you have a lot of experience in organizational structures and things like that, if you have any thoughts on policy for AI in the current setting and also specifically thinking about, there's a lot of people in my company that are hesitant because of the environmental impact of AI. And so if you have anything specific on that.
AI governance and organizational risk
Yeah. I mean, AI governance is like a minefield at the moment because we're so early on. There's a combination of things happening, right? One is that there's constant flow of new things coming on the market. We've got this great new model. It does X, Y, and Z. At the same time, I think organizations are feeling a lot of urgency to do something with AI because they're looking around and saying, oh, look, everybody's talking about AI. So we have to do something or we'll be left behind.
And that can create a challenging environment for professionals because I'm a huge believer, right, that you do something because there's an obvious benefit from doing it. Like you can see straight away, I'm taking on this project because if I get this right, there's this clear benefit from it. But AI can lead to a little bit of backwards logic, which is where leaders will say to people, find a way of using AI because everyone else is using it, right? And you should never with technology be told to find a way to use something. It should be obvious, right? It should be very clear to you what the use is.
So that puts a lot of professionals in awkward positions because they're kind of looking at this technology and saying, what are the ways that I can use it? And my first thing is, I think, when it comes to governance, I do think that organizations need to have a good view on, like, where is the value, right? Don't do 20 different things, all of which you're not sure the value of. Try to narrow it down very early on to two or three things that you think would be massively valuable and balance that with risk, right? Because AI can give you a lot of value, but there's a significant amount of risk involved with it as well.
And the question is, like, if something was going to save you, like, several millions of dollars in cost, but there's a, like, 1% chance that something could go wrong that leads to a lawsuit that could cost you, like, several million dollars, right? You've got to kind of do that. You've got to work that balance out. So, you know, I'm not involved in AI governance myself. It's a bit of a minefield. But I do think that you need a good business head to think about, like, what is the genuine value and not waste too much time on speculative stuff and make sure you've got people who really understand the risk and can evaluate the risk. Thank you.
Thanks, Liz, for that amazing question. And I wanted to sort of hop in here. We have a few minutes left. I wanted to, one, ask Keith, what did we talk – what did we not talk about that you wish we had talked about? And, two, remind everybody, there's a new Hangout next week. It's Juneteenth. So just putting that out there. Join us the week after next. All right, Keith, what did we miss?
The blurring line between data science and software engineering
I think just the one topic which I want to get across to people, which relates to something that we spoke about earlier. But, you know, I am – I have a little bit of a concern about, like, what data science is becoming in the kind of world of large language models, right? And I see a lot of people who are data scientists, but what they're actually doing is they're just – they're really doing software engineering, right? They're interacting with language models and trying to get them to do stuff for them. But if you look at the code that they're writing, it's software code. It's not data science code.
For me, like, data science is working with data, understanding the patterns in that data, using those patterns to drive insights, and taking those insights to your business or your organization to, like, have impact. And I just see that gray area coming up now where a lot of people kind of have the data science label, but really everything they're doing is software engineering. So – and that's fine, right? Some people might prefer the software engineering, and that's cool. They should go and do software engineering.
So one of the things for you guys to think about in your career is, like, as we move towards this idea where people are asking data scientists to interact with large language models, how much of the data science part of your job are you giving up in that? And are you actually moving more towards a software engineering path in the work you're doing? And are you happy with that? And if you're not happy with that, how do you get yourself back onto the kind of data science track where you're – where I think you said it, Libby, when we connected earlier, you know, the science word, right? If you're wrangling large language models for software purposes all the time, you're not actually doing any science, right? So that's something for you. I'll leave that as a thought, a career thought for you guys to think about, I think.
If you're wrangling large language models for software purposes all the time, you're not actually doing any science, right?
Wonderful. I appreciate it. And I agree. I – Keith and I have been talking about, like, hey, you know, the word science and scientist is in data science and data scientist. Wouldn't it be great if we could do some science in our roles? If you hire me to do that, I would love to do the science part. I think it's okay to ask that question and ask yourself what you want and whether or not you're happy in the direction you're going. Read a job description and ask yourself if it's actually a software engineering job, because sometimes half of it is, and you're like, do I really want to do that? I don't know.
All right. Well, this has been a fantastic conversation with Keith. Keith, thank you so much for joining us. This was wonderful. Everybody go follow Keith on LinkedIn so you can stay in touch. Thank you so much for having me, guys. Really, it really was fun.
Okay, everybody, if you want to save the chat, there are three dots at the top right of the chat window where you can click that and save chat so you don't lose resources people have mentioned. We are, like I said, not having a Hangout next week. Next week is Juneteenth. That's a U.S. holiday for anybody who is not in the U.S. And Rachel posted in the Slack, like, hey, we're testing out some new registration for Zoom, so go click that link in RSVP to help us test our registration, please and thank you. Have a wonderful Thursday, everybody. Have a wonderful week, and I will see you in two weeks.
Libby, can I just add in real quick? Yeah. Anybody has any issues with the new Zoom registration to please let me know, and I'm going to put my email here in the chat. But as the Hangout continues to grow, Libby and I want to do our best to keep it the safe and welcoming space that it is, and so the format is going to be completely the same, but with Zoom events, you have your own personal Zoom URL is how it works. So if you all could help me with testing that out, that would be amazing. If you could RSVP for the Hangout on June 26th. I'm already getting some great feedback there in the chat, which I'll go in and grab. But thank you so much, everybody. Nice to see you today. Thank you, Keith. Thank you, guys. Take care. Thanks. Bye.