image: thumbnail.jpg
The changing landscape of data science | Kanchana Padmanabhan | Data Science Hangout
We were recently joined by Kanchana Padmanabhan, Director of Data and AI at Homebase, to chat about data science team structures, the role of math in understanding LLMs, building effective hackathons, and communicating model insights to stakeholders.
In this Hangout, we explore the importance of understanding the probabilistic nature of LLMs and how that understanding should influence how data scientists approach their work. We also discussed how to structure a hackathon to encourage learning, centering on the customer problem, and collaboration between technical teams and business stakeholders.
Resources mentioned in the video and zoom chat:
🔗 List of R Conferences for 2025 → https://rworks.dev/posts/r-conferences-2025/
🔗 Posit Conference Call for Talks → https://posit.co/blog/speak-at-posit-conf-2025/
🔗 Julia Silge's workflow demo on model cards → https://www.linkedin.com/posts/posit-software_join-us-for-a-live-workflow-demo-on-creating-activity-7287998741557522432-jseQ?utm_source=share&utm_medium=member_desktop
🔗 Shiny Assistant Gallery → https://gallery.shinyapps.io/assistant
🔗 Data Science Hangout Playlist → https://www.youtube.com/playlist?list=PL9HYL-VRX0oTu3bUoyYknD-vpR7Uq6bsR
🔗 Add Posit Team End-to-End Workflows to calendar → https://evt.to/aoimiohuw
🔗 Making of a Manager Book → https://www.amazon.com/Making-Manager-What-Everyone-Looks/dp/0735219567
If you didn’t join live, one great discussion you missed from the zoom chat was about how to gain domain knowledge for a new industry, where attendees shared their experiences and advice. Let us know below if you’d like to hear more about this topic!
â–º Subscribe to Our Channel Here: https://bit.ly/2TzgcOu
Follow Us Here:
Website: https://www.posit.co
Hangout: https://pos.it/dsh
LinkedIn: https://www.linkedin.com/company/posit-software
Bluesky: https://bsky.app/profile/posit.co
Thanks for hanging out with us! 💛
Transcript#
This transcript was generated automatically and may contain errors.
Welcome back to Data Science Hangout, everybody. If we haven't met before, I'm Libby. I am a community manager here at Posit, helping to enrich our beautiful, wonderful Data Science Hangout community. I am also a Posit Academy mentor, where I help people learn R and Python to do better work with data in their everyday job. We are so happy to have you joining us today, and if you have not been here before and you're not familiar with our format, it's an open space to hear what's going on in the world of data across all different industries.
This is where we chat about data science leadership, we connect with other people in our spaces who are facing similar things as we are, we learn about other industries, and we get together every Thursday, almost every Thursday, same time, same place here on Zoom. So we hope that you have this added to your calendar. If you are watching this recording sometime in the future and you want to join us live, there's going to be details below in the description box on how to add this to your calendar. Thank you so much to everybody who has made this the friendly and welcoming space that it is today, that it has been over the last few years. We are all dedicated to keeping it that way, so if you have any feedback about your experience that you'd like to share with us anonymously, good, bad, whatever, maybe even suggestions for topics that we could cover or people that we could have on, we are going to share a Google Doc or a Google Form in the chat, and that will allow you to give us your anonymous feedback, but you could also find Rachel and I on LinkedIn and let us know there or leave comments on any of our posts, of course.
We learned, Libby, that we can automatically launch a survey right at the end of the Hangout, too, and then that way we can know which specific Hangout it was associated with as well. So thank you to everybody who shared feedback in that Google Form before, but now you'll see when you exit out of the Zoom, there'll be a pop-up survey there, too. Oh, fantastic. Okay, so if you still have the link to the survey form and you want to use it, I think it's probably fine. We'll still find that feedback, but now I think that the Zoom survey pops up in a browser tab from what I've heard, so look for that once you close it. I see Ted saying, yeah, that's right. Okay, cool.
So we really encourage you to connect with other people in the Hangout. This is a place for us all to get together and the chat is your space to have a party, have fun. We really recommend that you introduce yourself. What do you do? Where are you? What do you like to do for fun? Leave a link to your LinkedIn so other people can find you after our chat goes away. This is a very ethereal thing. Or a link to your website or any other way to contact you. We really, really encourage you to share any open roles or if you're looking for a role. So if you have an open role, post it, a little bit about it, where it's located, some of the requirements, maybe if it requires a PhD or something, that's good to know. And then if you're looking for something, again, all of those details will help us in the community connect you with opportunities.
There are three ways today to jump in and ask questions or share your experience. So we are going to be having a group discussion and that doesn't happen without everybody asking questions. So you can put your question in the chat. If you can't talk today, if you don't have a mic, or maybe you're in a very loud place, you can just put an asterisk somewhere in your question, we'll ask it for you. You can ask anonymously on Slido. There will be a Slido link in the chat that Rachel or Isabella will put in there. There we go, Isabella just put it in. That will allow you to ask anonymous questions, but you can also raise your hand here on Zoom and we will call on you to jump in, maybe if you have a follow-up or something to the current conversation.
All right. With that, I would love to go ahead and introduce our amazing co-host today. Although I feel like Rachel, you haven't introduced yourself. Rachel is my co-host today. You probably all know her. She started the Hangout years ago. Rachel, can you introduce yourself? Yeah, absolutely. Hi, everybody. I'm Rachel Dempsey. I lead customer marketing here at Posit. And I don't know if anyone heard this in the beginning, but Libby and I are kind of in this transition period here for the Hangout. And I'm so excited to have Libby coming on board full-time. And so while we've been doing like 50-50 hosting the past few weeks, you'll see like starting to slowly transition over into more of Libby being the host here. I will still be here as well, but I'll be hanging out in the background and chatting with you here in the Zoom chat.
I'm handing over my party in the chat license to Rachel. I don't know how I feel about it. I will try and keep it up. Yeah. I miss the chat immensely.
Okay. Any other housekeeping stuff, Rachel? I know that registration for PositConf 2025 is open and also call for talks is open. So if you are looking to submit a talk, like it's time to start figuring that out. Absolutely. Yeah. I just put that into the chat right now, but I was interested if anybody is thinking about submitting a talk for the PositConf conference in Atlanta this year. I find the Hangout is a great place for us to all cheer each other on and get feedback and ideas for talks too. So I'll share some more information about the conference in the chat too.
Introducing Kanchana Padmanabhan
Cool. Awesome. Well, I'm excited to introduce our amazing featured leader today, Kanchana Padmanabhan, Director of Data and AI at Homebase. Kanchana, could you tell us a little bit about yourself, what you do, maybe what Homebase is, and a little bit about what you like to do outside of work? Sure. Thank you, Libby and Rachel for having me. I think the participant counted 121 and my nervousness is slowly increasing. So if I'm a bit nervous, forgive me. So hi, I'm Kanchana. As Libby said, I'm heading Data and AI at a startup called Homebase. Homebase is a company that builds software for small businesses. So we build scheduling, payroll, hiring, timesheets, and a lot of other features. We also have lots of integrations with POS vendors like Clover, as an example. We also have a lot of financial products for employees of small businesses. So that's the plethora of things we do.
My background is basically data. I've been in the data space since I graduated many years ago from NC State. And then I've been through different industries. I started in social media when Twitter was still called Twitter, when we used to ingest all the firehose and build products off of that. Then I went to retail for a bit and supply chain, then healthcare, and then I'm here.
Yet, I think, what else can I say about myself is, yeah, my, I think I said that to Libby when I started and Rachel is that they had some questions around data science and how it's evolving and changing. And I basically said that I have a belief system. It seems to be updating over the years that I've been in this space. And so, you know, so it's a constantly evolving thing. So I may share opinions now, which are really opinions, which what I think about right now. And I may get more information in three months that may alter what I, you know, what I believe. But I think that's true for all of us. I think we're all still learning how the space is changing. Absolutely. I think anybody who decides this is the way that things are, and the way that they always will be, is probably not telling the truth and doesn't know what they're talking about. So it's really refreshing to hear somebody say, like, we're all dealing with evolving and changing opinions of things.
Kanchana, is there something you like to do outside of work? Oh, yes. I absolutely love music. So I learned to play the keyboard. And I also learned vocals. So I try to practice both. And I also have two kids. So with that plus kids, oh, I also teach, I should have mentioned that I teach part-time at the university. I actually teach at all three universities in downtown. So I teach at the University of Toronto, the Queens University, as well as the TMU. I teach at the business schools in University of Toronto, which is Rotman and Smith School of Business in their data science. Amazing. What are the topics that you teach? Many. Most recently, I've been teaching deep learning. But prior to that, I've taught big data tools. I've taught machine learning. I've taught data visualization once. And introduction to Python many years ago. So amazing. I don't know how you find the time to do that and work full time and have two kids and learn music. But that's amazing.
Data science problems at Homebase
Well, I was wondering if you could give us a high level view of the problems that your team solves so that as we are developing questions for you, we can understand the types of problems that you work on in data science and Homebase. So we work on both what we call customer facing or external ML features. So just to give context, so my other teams I own are data engineering, data platform, data science and ML platform. So all of those teams are in my area. So I'll talk specifically about the data science part of my job. So the problems that we solve are both for customers as well as for our internal customers, as I like to call it. So we have customer facing features. We have ML features and scheduling, for example, predictive scheduling, like using the data, like building out an optimization engine that can optimize a schedule with various constraints, as an example. We also build tools and models that can kind of gather availabilities of employees, which sounds like a very simple thing. It's a very complex thing to gather information from employees to kind of use to schedule their hours.
We also have timesheets and around how to optimize their timesheets and how to optimize their hourly billing and how that works. We also build models that are lots of risk models. So we have a lot of financial products, so we have an entire space around risk modeling that we build. We also build more for our internal customers, so we build recommendation engines for our marketing team, so they could target appropriately. We build propensity models. Yeah, so a mix of lots of different types. I recommend predictive models. More recently, LLMs. I guess everybody's using LLMs now, so LLM type models. So just a mix of different types of models.
Defining data roles
Awesome, thank you. And I already saw a question come in from Kylie. Kylie, would you like to ask? Sure, I actually have two there. So one, I guess I'll, well, I'll just ask both of them and you can answer them in whichever order. So the first was that you were describing the different teams that you sort of are in charge of or work with, and you said data engineer, and then data science, and all this stuff. And I was actually wondering, to you, what are the difference between those things? What is a data engineer versus a data scientist versus an analyst versus a software engineer? Okay, it's a good question. Probably different companies define them slightly differently. So I'll just talk about how we define it or how I've seen it defined in my previous jobs.
So for a data engineer, it's at least, I have two data people, so data platform and data engineering. So they basically work on making sure that all of our data within our product, as well as external vendor data, all of that is collated, brought into one system, and then organized and ready to be used either for analytics, for reporting, you know, we also have data products. So just making sure the data is ready and available for these different use cases. The platform team takes care of the infrastructure, the ingestion, and also sending data externally. The data engineering team is responsible for all the transformations, like as the data comes in, the monitoring, the validations. And so that's kind of roughly the split. There's some overlap. Some people can do a bit of both. So it's not like a strict of like, you don't touch another system. It's a bit more fluid on my team.
For data science versus data platform, then my data science team is primarily involved in, I would like to think building data-driven solutions. So it could just be as simple as figuring out a bunch of rules that need to go in to solve a business problem, or it could be all the way to predictive models, hooking up an LLM, anything in that spectrum. And the ML platform team are the ones that provide the tooling. So making sure that real-time pipelines can be built in a repeatable way, or batch inference can happen in a repeatable way, or the data scientists can build their prototype and they can package it up in either a wheel or some way, and then can be deployed very easily and gotten into production. So the time that it takes from prototype to production keeps getting shorter. At least that's the goal is to build all the tooling that can enable that.
So between analytics and data science, I feel is always a bit of a gray area because in the company that I'm in right now, and the one company that I was in previously as well, analytics was very heavily involved with business. So they were involved in a lot of business reporting. They were involved in a lot of looking at metrics with the business to kind of pull data that would support any of their campaigns and any of the targeting that they did, or any of the business objectives. So for example, if you think of, oh, a certain 1D signup is going down, then they'll pull all the data related to that. They'll figure out why it's happening. And then potentially from the data, some insights could happen. Like, oh, you're seeing some correlation between this particular incident and something that happened there. So they're pulling it. And once it comes in the realm of potentially needing slightly more sophisticated models, potentially, like if not that
analysts don't build models here, they build some simpler ones. And let's say they think of, oh, we want to start targeting more specifically using our data, which is not just let's gather some insights, but let's massage and kind of analyze all our data and figure out who to target. Then it becomes a bit more of you need a larger system for that. And the data scientists come in to like figure out how to build out that system. Again, it's not, my analysts sometimes do build simpler models. We support them. Like sometimes if they have questions, we work with them. So it's not like they don't, but it's just the scale of the problem. Like when you need to work with more data and when you want to build predictive models, when you want to do something with recommendation, when you want to take in lots of different inputs and process it, then we come in.
But like I said, it's very specific to this company as well as some previous places I've been.
And also the title data scientist is very, it's one of those, like what is it, like who is a data scientist? It's entirely dependent on how, you know, what your company has given you as well. So now we also have applied AI scientists. So it's like, yeah, so that I won't go with the title, but just roughly what my teams do and what the analytics team that we have do.
Yeah. I hope that answered the question. Yeah, yeah, yeah. Definitely did. Well, and I was just having a conversation with a colleague of mine about this. So I just thought I'd, I'd pose it to you and be like, well, what are your definitions? Yeah. If everybody has different ones, please feel free to add them in the chat. I saw a couple of people add them.
And I think Kylie, that answered your other question too, about like, what do you model? Cause that, okay, great. Fantastic. Thank you, Kylie, for the amazing question. I see Abigail, you had a question earlier in the chat. Are you available to ask it live?
Yes. Hi, thanks. So when I started doing data science, I was, it was a lot like, here's a linear regression, you know, like the data stuff was easier and now it's LLMs where there's just like a lot more theory and a lot more math that you could possibly understand if you wanted to. How do you figure out like how much math or how much theory you, is like the right level for you to learn? Like if you're using a particular model versus like if what you need to get into is like the engineering or the topic knowledge or something like that, like, do you have any like useful heuristics for other people trying to figure out like, is the thing I need more math?
How much math do you need for LLMs?
That's a really good question. And I feel like it kind of came up when Libby and we're also chatting a little bit of, so, okay. I have to talk a little bit about how I built my team, but maybe I'll start answering your question and then we'll get there. So how much math do you need? I think, I think it's important. I think that you need enough math to understand that these LLMs are basically probabilistic machines and that, that they are not going to, they're not sentient or they're not going to understand what you're saying. And I think that's a really important thing. Like, I think, I think I've seen a lot of people not even get, I mean, not getting there. So I think if, if, if that level of understanding that I think it's a great start, because I was in a demo just before this and it's kind of like very interesting company, but they call the LLM a name and refer to it as she, and says she has knowledge and she, and it's just, you know, I can't get beyond that to actually enjoy the demo. Cause I'm like, I'm so stuck in that bit of terminology.
So I think if you have the understanding of just, just one level of like how these transformers work, right? Not, not every math, not how they are trained, not even like the exact algorithm, but just understanding what they do from the perspective of, oh, it's just, it's a weighted average, right? You're learning the weights and then you're producing some vectors and then these vectors are thrown into an N dimensional space. And then you're predicting based on what's close to each other. Like that's kind of what you're doing. So I think we have that level of understanding, you at least know, and you're not surprised by the answers the model is producing. I think, I think that that's the most important thing. You're not shocked that they didn't understand something you put in, you're able to like reason out that.
I think that you need enough math to understand that these LLMs are basically probabilistic machines and that, that they are not going to, they're not sentient or they're not going to understand what you're saying.
For my own, I think company, I think one of the things I've had to do is two things. One is that my team is a mix of what you call data scientists. You know what I would call more traditional, like the ones we were like, we built predictive models. We understood data, we did exploratory analysis. We built a certain type of models and a few people who are now, who have done research in LLMs, who do understand the deep parts of the math behind it. So once there's a cross-pollination, like together they're learning, you know, some of them come from research, they're not very used to how to work in a product environment, how to build product features, but they know how, they understand the math and the research and then vice versa, right? These folks don't understand the depth of LLMs, but they do understand how to produce good models and output things. So it's a cross-pollination.
I've also had to like set up what I'm calling the center of excellence, because with ChatGPT, I think you technically don't even need a data scientist to run these prompts. Like everybody in my company is running stuff using prompts and building assistance as they call it for different purposes. So I've really had to set up the center of excellence around like teaching and teaching how these things work, even being technical, talking about evaluations, talking about hallucinations, what they could mean. And so it's become a constant like teaching, providing guardrails, you know, making sure people are thinking about the right things.
Because math can be learned by people who want to, and who are also probably, it's easier, but there's also a whole plethora of people using these tools that are not going to learn the math. So it's really making it consumable for them in a way that, you know, taking the probabilistic idea, but again, teaching them in a simpler way that they would still have that intuition rather than knowing the math. So that's been a big part of what I focused on for the half of last year. And I think I'll continue having to do that, do more. And we're doing that more specifically with cohorts of different groups from business and sales and all of that as well now. So thank you.
Structuring a learning-centered hackathon
Amazing. Your answers are so thorough. I love them so much. You're like covering all the follow-ups that I would possibly ask, but I do have a follow-up because you mentioned having a center of excellence. I'm wondering, let us know in the chat, if you have a center of excellence as well, I'm curious because that's been growing. But you also just recently in the past several months, set up, helped organize, however you want to say it, a hackathon, but it was a special hackathon. It wasn't just like, we're going to get together and bang out a problem really quickly. You really centered the learning and expanding everybody's horizons around the topic. So could you talk a little bit about how you did that? Because I think other people would love to do this too.
Yes. So this happened in November. So we typically have two hackathons a year, and I kind of hijacked this one for this purpose. So what I really wanted was there was a growing interest in the company with LLMs and different people were excited to try things. We had set up an internal community called the AI Builders Community. It's a Slack channel where people who are building with this could come and ask questions and we would support. But the one thing I realized was, I think, as Abigail mentioned, that there was no context to how these models were working. And so people would try things and be sometimes frustrated with not getting quite the output they want. And the other thing that I was noticing was people, I think it's happened to us many times, is we get so enamored by the technology, we forget what we're solving. So it's like at the end of the day, we're solving user problems, we're solving some problem for ourselves or we're solving some problem for our customers. And so you want to center it around that when you're solving, even with an LLM, because it doesn't matter if it's an LLM or XGBoost or the other, it doesn't matter, as long as it's solving a real problem.
So what we did was we kind of did a few things. So one, we said that part of the hackathon would be learning. And one learning was obviously the technical side of things. We went through how LLMs work, we went through how to evaluate them, how to think about evaluations. I also had one section on security and what these APIs are, what data should you throw in, what should you not, and how should that work. And then another piece of it was that I also invited my head of design and her group to talk about customer centricity. What does it mean? How to do a mind mapping, how to think about a customer problem and how to center yourselves on the customer problem. So we had these two sides of like, let's think about the customer, let's understand this technology and let's put it together for a couple of days of hack to come up with something interesting and useful.
And one other piece that we did was we gave each team a mentor, an executive mentor, like from our VP, CTO, VP level and up, so that they could kind of see what these teams are building and they could guide the team from the perspective of like, you know, will this solution be useful? Like what are some things to think about? Because we think about the technicalities like hallucination, but there's also an individual or a human on the other side who's going to use it. So kind of centering them around what to think about, what's the business problem, like what benefits should we see? So we also gave mentors. Also at the end of the hackathon, I organized these office hours with my principal and staff engineers so that people could come and just vet their solutions. They could just tell them what they built. So one, the engineers have a sense of like, what's going to come downstream? Like what's going to come two months from now? Because many of these solutions are quite productizable, so what's going to come downstream? But they could also give them some insights into, hey, what are some caveats that could happen in production? Like what are some things to think about? Like latency is a big issue with LLMs, like we all know that. So if you're expecting a system that needs a fast response, then latency is something you have to consider. So we had an office hours with a bunch of our senior engineers across these different domains, data, product, infrastructure, and so they could give their advice. And so what they had to present was their solution, but they also had to present their learnings and the feedback they got from this engineering group of like, what does it take to take this feature into production? So yeah, that was kind of, I think I covered most of the pieces, so.
Yeah, that's amazing. How many days did this last? I'm wondering.
It was actually five days. So we had our training sessions between Monday and Tuesday afternoon, and then the teams had like about three days, so like two half days and two days to hack. And between that they had office hours they could go whenever they were ready. So it was like spread out. And then the Friday afternoon was our show and tell or the hack presentation.
Amazing. I would love to hear from anybody in the chat if they have hackathons at their kind of place where they work and do they look like this? Do they look different? Did you enjoy them? I'm really curious because I have never participated in a hackathon myself. I came very close to volunteering to mentor for one for like undergraduate students, but I just didn't have the time. But I haven't experienced them in a professional organization before, so I'm super curious.
Challenges and ROI of LLM projects
Kind of on this same topic, I saw Rungua, you had a question about challenges for LLM projects. Are you available to ask? Yeah, I think, how to say, lots of companies, we talk about the large language model and the general AI, and lots of companies spend a lot of money on that. But I think maybe many companies are not getting much from profit from the AI.
And I think, could you explain what the challenge and how to be able to convert it to a profit RRI, you see? And what do you think about the agencies? Because now the agencies can be able to replace some of the, maybe most of the data science work. So what do you view about that? Sorry, too long. No, no, no, no. It's a good question. The short answer, I think we learn along the way. That's the short answer. But I can tell you what we think about and how we, you know, we think about.
So for LLMs, I think there's two parts. So I'll talk about one part very briefly, because I think Ron has questions more around the cost. One is just, you know, evaluations and, you know, actually putting your evaluation set together, actually getting your metrics in place, doing qualitative and quantitative validations, kind of knowing where the boundaries of your, you know, of your, you know, failures are like, where are you going to fail? You know, having to actually collect manual label data. People think label data has gone away. It hasn't. Like you still have to evaluate the model, even though you don't need the millions of data points to train it, you still need some set to like validate it. So that has to happen. So without that, that's a big part of the support we're giving and the coaching we're giving is just that just because it works for a couple of examples, doesn't mean this works. And you still have to validate it with the same rigor you would any other machine learning model.
I think from a cost perspective, I think I was definitely nervous when everybody got excited about LLMs. My first instinct was also that, oh my God, cost is going to go through the roof. And what I realized was I was the person who needed to solve for it. And so what we do is, yes, these OpenAI endpoints, you know, as you scale or any other like Anthropic or any other endpoints that you use can get expensive over time, definitely. But I think what they help you do is kind of evaluate your use case really fast, right? Because one of these things we don't know about these LLM type features is you don't know, especially if you're putting out a customer feature, you don't know whether customers will like it in the first place. Like you don't know how they will interact with it. You don't know whether it's going to be good enough. And so the key thing with AI, and I think it's always been true with ML, but I think with LLMs it's also true is the iterations are important. Like how fast can you ship a version and how fast can you get feedback from it? And these LLM endpoints from these other companies really enable that. They make it really easy to do that. Like you can put an API in and get some feedback really quickly. But I think one of the ways that I have built out or we continue to build out these kinds of endpoints is the product, well,
the product feature, we're the ML platform team. We build, you know, just very common patterns. We build an API. So we kind of encapsulate. So the product doesn't hit OpenAI directly. We have an encapsulation layer. They hit it through us. We hit OpenAI. And I've already started, we've started doing experiments on small language models. Like we start training on our own data and then getting more specific models in-house and seeing if, you know, like basically getting, get ready for the point when we'll have to like potentially remove these OpenAI type endpoints and then maybe replace them with smaller, maybe use case-specific, context-specific models. That's experiment that's ongoing right now. Will we be super successful? Can these models fully replace them?
It's definitely a question to be answered. I don't have an exact answer right now, but my thought process is like making sure that the product is not affected. So making sure that everything's encapsulated and set up in my side. And yes, it's going to be expensive in the short term, hopefully until like we figured out what features to build. But the longer term strategy is always going to be building models with our own data. One, it can have a lot of context of our company, but also it'll probably be more effective. And of course, a bigger part of our discussions always doesn't even need an LLM. Can we replace it with a much simpler model? And for many use cases, it could very well be the case. Like we can probably, an LLM is a short answer, but it doesn't have to be the long-term strategy. So we may not even go with an LLM. Six months down, we may replace it with a bunch of simpler models. But that's definitely a learning that I'm having as well. So if I'm going to hang out one year from now, I may have more firmer answers on how that exploration is going.
Thank you. Yeah, no, that's really, really helpful. I think a lot of people are sort of having the same questions and the same experiences as we go through the LLM hype cycle, I will call it. I don't know where we are on the hype cycle just yet. I think we might be like edging towards the downward slope, right? Into the trough of disillusionment or whatever they call it. I wanted to ask because I know, I remember looking at your LinkedIn and thinking, wow, Kantana has been in the management space in data science for a really long time. You came into industry from academia and you very quickly went into management. How did you know it was time to be a manager? Did you always want to be a manager? How did that happen?
Path into management
I think I always tell my boss, I'm one of those happy managers. I wanted, it's a calling, I enjoy it. It's what I thought I wanted to do. I think even when I was in grad school, we were a team of 15 PhD students in my group, 15 men, me alone. It was just naturally something that I picked up, supporting people with their projects, just collaborating, making sure things were going on track because research can get sometimes very lonely and you're on your own trying to figure something out. Just making sure the people are on track and that they were getting the support and getting organized. I think it was instinctive to make sure that not just my work was going well, but everybody was moving along. It was just instinctive for me to do that. When I went into work, I found myself in a similar space of always caring about the bigger picture, always caring about what's going on holistically on the team. I was never like, assign me a ticket, I'll do it. I was never that person. I was always like, okay, I'll do my ticket. Also, what's that person doing? What's that person doing? How are we fitting all the pieces together? What output, impact are we creating as a team?
I don't have a perfect answer. When I decided to transition, I think it was just an instinct. It felt natural. When my manager asked me at some point, he had to decide. He was like, hey, I have this position. Do you want to be a manager? Do you want to be an IC? You could be both. It's your choice. I was like, yeah, management seems more my calling. Although the one thing I would say is management, and I feel a lot of people may think that management will take you away from technical stuff. I never stay very far away from technical stuff in general. I generally, I love keeping up with things. I love making sure that I understand what my teams are building. I love being in the architecture discussions. I love understanding how these systems are built.
I have opinions. I think I have the balance. I don't think it's taken me away. I'm doing things I love. I do love to manage people and help them grow. It's just instinctive to make sure that other people are also progressing the same way as I am. At the same time, I also have time to make sure that I get into the technical stuff and I have a sense of what's going on.
Although the one thing I would say is management, and I feel a lot of people may think that management will take you away from technical stuff. I never stay very far away from technical stuff in general.
I love it. I bet teaching on the side helps you stay engaged in the technical side and how things work a little bit. It definitely does. I think the teaching helps both ways. From an industry, I can take my experiences and share them in a way that, oh, all of these are great, but this is what's happening in the industry, or at least from my experience, and vice versa. I've always also had these MyTax-type programs and collaborations with academia. I've always worked with professors, PhD students, publishing, and working with them. That also definitely keeps me in the loop with things.
Thank you so much for answering that. I would say, as a sort of echo to that, is that I am also doing what it is most natural for me to do. The community building stuff, the connecting people to each other. I've been doing it since grad school. I also tutored other people all through undergrad and grad school. Teaching became just a very natural thing for me to do and mentoring people. I really highly encourage anybody out there, if there's something that just comes naturally to you and you feel really called to do it, find a way to do it in whatever industry or you're in.
Model explainability vs. accuracy
I saw there was a question from Zach, and it has an asterisk, so I will ask it, but there's a little bit of discussion about it in the chat. I think it's a great question. He says, when building a predictive model, how do you find the right balance between ensuring the model is explainable versus how accurate it is? Do you often see situations where an AI or machine learning model is less ideal than a simpler model? I'm assuming because a simpler model is probably more explainable.
Yeah, it's a good, it's always a good question. I think the general rule of thumb, at least we've, we've tried and it's not perfect, is, and also explainability is interesting because like it's, it's like, you know, explainability itself, almost a lot of explainable models, unless you're using a linear regression model itself is another model that is, is kind of running a certain scenario and putting together what it's gathering. So for the models that were more, I think, risk-based, I mean, things that were determining if someone would, we have a product called Cashouts. So we have models that, you know, that internally we have risk assessment of like, you know, when people are drawing cashouts, you obviously want to be, there's a responsibility angle to it, right? Like even if the business may not think about it as a data scientist, we have a code of ethics. We write up for data science, we have a conference page that says, we have a code of ethics that says do no harm, like make sure that you're not biased and you know, all of that. So we, we try to make those models as much as possible explainable. And, and the one thing we do is we don't reject anybody, if you know what I'm saying, like everybody gets a minimum of something, right? Like there's no, like the model will not make a decision in the negative. And we make sure that when there's a negative decision, that it, a minimum is given and there is a review. There's something else happens beyond that. It's, it's never a model making a decision. So we, we try to balance out like trying models that are a bit more explainable and making sure that there are business rules around not just blindly saying no to somebody in some critical use cases.
But in other use cases, you know, what we try to do, at least in the recommendation type use cases is we still use more complex models, but we try to vet the output to kind of make sure like to the best of our ability, there's no bias. And if there's nothing else we're missing, like just try segmenting in different ways and seeing if a particular recommendation is given to a particular cohort. And if there is, you know, some inherent, you know, demographic or other types of bias that exists. So it's, so in those use cases, we try to use more complex, but we do some post analysis to make sure in the, in a financial thing, we, we make sure that we are not, the model is not making a negative, like if there's a negative recommendation, then it's a review. It's a minimum. It's just other business rules kick in and it's not, you know, it's not blindly saying no to somebody. So, right. Yeah. It is very challenging to make those decisions. Yeah. It's been in the news a lot lately, right? Like models making decisions that negatively impact people and not having any oversight into them. So avoiding that is a huge deal.
Advocating for better analytics tools
All right. I have a Slido question that I think is wonderful. Thank you, Slido person. Any success stories slash advice on advocating for better analytics tools and better analytics infrastructure. And this person is saying that there's someone who is new to a larger company, like a thousand plus people company where maybe less than 1% of people use code first analytics tools. Okay. Let me think about, have you ever been in a place where this is a problem or is this something that you haven't experienced yet? I think I've definitely had to, not from an analytics tooling perspective, but I've definitely, even this year, had to get buy-in for revamping our entire data architecture because it was in a place where our pipelines are failing and, you know, stakeholders are not getting the data on time. And so it, so the discussion became like, oh, we need to re-architect things. We need to like reorganize our data. We want to move to using more data bricks. We, you know, Redshift was being used as a PostgreSQL. We do not need that. We need to have like proper tools for the proper things. So a lot of it has to really do with just putting into the context of business value. I know it sounds cliched, but that's really what helps is talking about like how many hours are, is my team spending on these types of problems? We have a dedicated person every week on-call and literally on-call just means they're helping fix some of these things. And it's not even a great productive on-call. So it's going through the motions of, and then with the MLEI event, like it became more important that our data infrastructure is in place. And so it was really making the case around like time spent, how much money is a data engineer making? How much money are we spending on this? How many failures are we having? How many times have we delayed things for marketing? And in turn delays ads and ads, if you know, if everybody works on it, and I'm assuming many people do, any delay in sending data actually causes a ripple effect. It actually really affects things. So it's really putting the case together with all these things and saying, oh, by the way, this is a proposed solution and this is how easily you can onboard. This is how long it'll take.
And one thing that's really helped me with, especially new analytics tools, because I am trying some of these new LLM based tools. We did a POC very recently, not successful, but we definitely did a POC. It's just more of scoping it out as a POC, like scoping it out as, you know, we can try it. It'll just cost you maybe 10,000, 20,000, whatever it is. You can scope out a POC and then go from there. And if it's useful, and if you can have the right people testing it out in that phase, that also helps you get buy-in, which is kind of what I'm trying to do with a newer BI tool. We have Looker originally and it's become a mess over the years, like, and it's very painful to use. But so we're trying to move to another BI tool and that's what we're trying to do. Like basically get a POC going and then having our business users, our power users, and, you know, some key people start to test it and, you know, if they see value, it becomes easier. So, so not a perfect answer, but just stuff that I've done or tried in this.
I think that's a great answer. Gather the information, do the legwork for the research to make it easy for them to say yes and have the answers to some of their questions ready to go, right? Like I feel like that's standard prep work that everybody should do when they're pitching anything. I appreciate that. I love it. Okay, we have a billion good questions. Let's find another fantastic question. Let's see. Hey, Rachel has a question. Rachel, do you want to ask yours?
Communicating model outputs to stakeholders
Sure. So when you're talking about building models, I'm thinking about conversations I've had with customers the past few months when they build a model for a team to make a decision who haven't used the model to make a decision before. And I was wondering, like, how do you communicate what that model is to them and explain it to them? And like, what tools do you use for communication of that? Well, that's a great question. For me, that starts with my definition of done that I have for my team, which is like, unless somebody's using it, your model is not done. I think a lot of these are two things. One is that understanding that model building is an iterative process in the sense that you slowly get requirements from our, like, no end user is going to come and tell you, build me this thing. It's always going to be a vague problem that you're trying to, like, distill down into something technical. So the first step is, like, can you distill it down into something technical and then explain to them what outputs they're going to get? Like, what are you going to get out of this model? How will you use it downstream? Like, understanding how these outputs are going to be plugged in, whether it's into a Google ad system,
whether it's into, like, an in-product messaging system that's going to recommend things at certain points, like, whatever it is, like, kind of understanding that. And then what I usually recommend to my team is let's put a quick POC, like, let's put one iteration of this model together. It can be slightly rudimentary. We can still have metrics and everything, but let's get our baseline model ready to go and let's get an end-to-end going. Let's actually hook it up and let's actually see what it's going because, I mean, it's like, all of you will agree that machine learning models are never finished. Like, they're living and you get newer versions out and at some point you stop updating them for whatever reason, but eventually.
So, I think it's really getting that first iteration and I focus a lot on getting V1s going, which are, like, more end-to-end and not just us iterating on a problem by ourselves. And I think that generally helps. And I think the caveat and the thing, the learning that I've had to work with the team say that, hey, these things are iterative. Like, yes, the first version might not look ideal, but we'll try to solve for the subset of use cases, right? This one subset of things will be solved. Yes, a lot of other things you asked are not solved for, but we can get to it over time, right? It's going to be built on top. But I first need to know how you're going to use this and where do we need to hook it up? What's the engineering? What's the effort required for you to use it? And so, if they're not willing to engage at that point, then you kind of know they're probably not as serious and then maybe you can abandon your project sooner. But if they're willing to go the next length and say, okay, yeah, let's try an experiment, then you know they're a bit more serious and then it helps us to continue working with them.
The one other buy-in, at least since I've, a lot of times I've worked on product companies, so this helps, is get a buy-in for like what I call R&D experimentation where I say, I'm going to spend a bunch of money. Like, let's say we're getting cash out or anything. We're going to spend a bunch of money, but this is for learning. It's a learning budget, right? We're going to run some experiments. Yes, some of them can fail, but I'll cap it to say 10K or 5K, whatever, something that's manageable for them that they can accept. But this is budget that, let's assume this is gone. This is just money you've paid out, but this will give us learnings that can help us build the next version. And then our stakeholders also feel a little bit easier about running an experiment so they don't feel it's on their head if something fails, so.
some of them can fail, but I'll cap it to say 10K or 5K, whatever, something that's manageable for them that they can accept. But this is budget that, let's assume this is gone. This is just money you've paid out, but this will give us learnings that can help us build the next version.
Career development and domain knowledge
Thank you. Thank you, Kanchana. All right, we have reached the 50-minute mark. Somehow 50 minutes have gone by. I don't know how that happened. It was very quick, but I would love to try to dedicate at least part of this last 10 minutes to some sort of career development type questions because I see some coming in on Slido and also in the chat. So if you're up for that, I would love to tackle that. Sherwin, if you are available, I saw that you had asked a career development type question. Would you like to ask that?
So hi, I'm available. So I want to know from Kanchana, how can we develop our domain knowledge? And this is especially in context of analytics, because you might have technical skills in analytics, but it's difficult to build domain-related knowledge if you haven't worked in that domain before. So let's say if you haven't worked as an accountant, you would not know much about finance. And if you end up as a financial analyst, you are going to run into a lot of problems because you just have technical skills, but you don't have the business logic to back that up. So how can someone increase his domain knowledge if he's in college or grad school or anywhere?
Oh, that's a good question. And I actually don't know if I have a great answer for that, to be honest, because I think, of course, there's resources available that you can probably learn from. But I think the way you learn is assuming that your college grad wanting to start, you go in and then you spend a lot of time with the experts who are there. That's the thing that I do with my data science team. My analytics team is very embedded, so they're already embedded into different domains. So they just naturally are in the meetings and they learn. But even for my data scientists, I make sure that they are sitting in all these meetings with the team, even if it's not specifically a data science problem that's being discussed, like even if they're just discussing a particular product area and what's going on and what are their goals that they want to achieve. OK, they want to get this many more customers. And this is some pain points that they're seeing. I get them to just go and attend these different meetings and sit in. And most stakeholders have been good about it because sometimes our stakeholders may not know how to define a problem for us. We may see a problem. We may see something that can be solved with data ourselves and we can propose things.
So I don't know if anybody else has a good answer about like learning it from university itself. I mean, one of the domains I learned was doing research. And I went through my PhD. I just started with not knowing very much about computational biology and knowing a lot. By the end, I finished my PhD. But that was a very specific path that I took.
But yeah, definitely, once you're in, definitely spend, just be attached to the hip, as I call it, with any and all domain people you can. Most of them are usually very helpful and they want you to learn because it makes their life easier. So they're not hesitant. But I don't know if anybody else has suggestions for his question. Yeah, there are some in the chat.
All right. I think that my suggestion is even if you need to get a job that's not analytics, that's in that domain or an internship that's not analytics in that domain, you will learn so much about that domain that you can then apply your analytics skill set to.
Yeah, and I think that those skill sets and those that domain knowledge can transfer over to other industries as well. Like if you are in supply chain, you went about optimization, you might end up at home base doing optimization for scheduling, which is just another type of optimization, right? Yeah, and coffee chats, actually. I'm not, I generally don't turn on, I mean, I try to have many coffee chats if people are asking me for it. So as long as you don't say, hi, I want to have a coffee chat with you, give me a little bit more. I think I'm willing to invest sometimes. Yeah, make friends and also join professional organizations for that domain. They probably exist, right? You can go make friends in those areas.
Learning from managers
All right. Kylie, you asked another question that I feel like is definitely personal development. And if you are still here, you can ask that. Do you remember what it was? Was that my manager's question? Yes. Did you have good managers in the past, or did you learn from bad managers? Yeah, because basically you're coming from academia, right? And academia is not a very great example of good management. It's people who have fallen upwards because that's the way it works and shouldn't be managing people. And so I'm just curious about sort of how you manage and how you've learned to manage, whether it was from a good example or a lot of bad examples.
Oh, I love that. Love that question. And that academia thing hit home so badly. It's like, oh, my God. Yeah. So the story is that when I finished grad school, my advisor, as all advisors, had a certain pattern to it. And so when I finished, I was a very nervous person, if you can call it, like very nervous about work and then finishing things on time and answering emails the minute it gets into my inbox and having no delay and like being available on the weekend, even though I was not on call and didn't need to be. And I was like in a very, not in a great shape, to be honest. And then I had my very first manager, Eddie Kim, who literally came to me one day and said, OK, you're going to leave your laptop at work on a Friday. He came to me, he said, you're going to leave your laptop at work. And I said, what's going on? And he said, like, I've seen you reply to things and doing things like, why are you doing that? You're doing two things wrong there. One, why are you doing it? And two, you're setting a bad example for the rest of the team. Like, why are you responding to something and you're setting the expectation that everybody else needs to when it's not necessary? You're kind of creating a weird peer pressure situation that you don't have to. So and he's also like, you know, even if I send out an email and it ends up in an odd hour, clearly I'm not expecting a response. If I did, I would mark it as urgent, like, you know, like I would do whatever. And he made me do that. And I suddenly realized, oh, my God, like this, this is a way to live. Like, I can do this. Like, this is a way that people can be managed. And it's not like I have to be on my toes the whole time. And it really, really like switch something in me where not that, you know, not that I would have gone and done that to some other person, but it's like, I don't think I was even mentally there. Like, like, he turned my life around. He just flipped a switch for me when I'm like, oh, I can breathe. It's okay. Like, like, yes, I can work through these things. Like every piece of work has a deadline I can work through until the deadline. And so I think I would say him, he made a big difference.
he turned my life around. He just flipped a switch for me when I'm like, oh, I can breathe. It's okay. Like, like, yes, I can work through these things. Like every piece of work has a deadline I can work through until the deadline.
And he was, had this right balance of like, like being like being flat, like his organization was very flat, like everybody could talk to him and anybody else. And he didn't treat anybody any differently because of their titles or how senior they were, how junior. And at the same time, he knew the right point to like, set up his like, but everybody knew he was the leader. Like, you know, like, you know, you know that he's the leader, you're not going to disrespect him. And, and he had that balance. And I kept trying to figure out how he did it. And maybe I imbibed some of it myself. So, and have I had bad managers after a hundred percent? Like I have, like, like, like I've had people who expected a certain type of behavior, but I have generally fought against it. And I've always like, like, like always at least at the very least try to protect my team, like, cause you know, and, and make sure that my team still was sane and healthy and, and, you know, manage my own relation with my manager, but try to keep my team protected and safe and, and, and, you know, give them the same thing that my manager gave. So, so a bit of both, but my first experience was with a excellent manager. And so he, he, he really, really genuinely changed my life. So.
Thank you. That's wonderful to hear. I, I think that a lot of us here in the chat have probably been that person who was answering emails too quickly and setting a bad precedent. And then later ended up really, really regretting the fact that they had set the instant precedent on email responses. So thank you. Yes. I had a manager once who said, Hey Libby, there are no analytical emergencies. Like you can, you can answer that email tomorrow, or you can, you can do that tomorrow.
Tech stack and closing
Well, we only have two minutes left, so there is not a lot of time left to ask anything very, very deep. I feel like you answered what I normally would ask, which is a career advice question with that one. So thank you. I did want to ask really quickly, just because it got asked, I think in Slido, and that is just about tech stack. Is there any way you can talk about like what types of tools your team uses or in what languages they use?
Oh yeah, absolutely. So the one rule of thumb I try to follow is keep my stack very boring and very standard. And, and, and because I've been in a place earlier when it was not, and we had Hadoop failures all over the place. So right now, oh my God, I've been on call for those things and it gives me nightmares. But right now our tech stack is Databricks for almost all things ML and data. We do have Redshift for our warehousing, trying to move a bit away from it. We use Airflow for all of our pipelines. We have DBT that helps, you know, that helps do a bit of self-serve with analytics because they can build their models and build their metrics a bit more easy in SQL and they don't have to do it in Python. We have Looker as our dashboarding, would not recommend it. So it's, it's one of those eventually, hopefully I will work through. But what else? SQL, of course, Python, SQL are the, are the fundamentals. And yeah, because Databricks, we use Spark for a lot of our processing.
Yeah. We also have other tools in the company. Like we have integrations with Salesforce and we have integrations with things like Hightouch and, and some mark, marketing ops type tools. But but the main awesome or tech stack is this.
Thank you. Well, thank you everybody. And as housekeeping, I will remind everybody that Rachel put in the chat, Julia Silge's workflow demo on the 29th of this month is going to be on model cards for transparent, responsible reporting. Might be a great follow-up to this conversation if anybody is interested. Thank you so much for joining us. We hope you have a wonderful day. We will see you next week, same time, same place. And thank you so much, Kintana, for joining us. This conversation was amazing. This was great fun. Thank you so much. Thank you for all the questions. Everybody had great questions. All right. Have a good day, everybody. See you. Bye. Thank you, everyone. Bye.
