image: thumbnail.jpg
Data Science in CPG and the Zero-to-One journey | Joel Ash @ Kraft Heinz | Data Science Hangout
Joel provides a behind-the-scenes look at how Kraft Heinz leverages data to optimize promotional spending, which exceeds a billion dollars annually. He discusses the complexities of navigating relationships with retail partners like Target and Walmart, emphasizing the need to balance cost reduction with maintaining strong customer relationships.
With some internet issues, we were only able to learn from Joel the first half of the Hangout. We switched to an *open leader* format with the community for the rest of the conversation. Thank you to those who jumped in as well!
Discussion topics with Joel included:
⨠Decoding the CPG landscape: Joel explains the nuances of the CPG industry, highlighting the distinction between their customers (retailers) and end consumers. He provides specific examples, such as the market dynamics of cream cheese, to illustrate the challenges of predicting demand and optimizing promotional strategies.
⨠Embracing the "zero to one" journey: Joel introduces the concept of "zero to one," referencing the challenges of building data science capabilities from the ground up within established organizations. He discusses the importance of integrating data-driven insights into legacy systems and processes, often requiring adaptability and a focus on interpretable models.
⨠The battle against cannibalization in CPG: Joel explains the concept of "cannibalization" in the CPG context, where the introduction of new products or promotions can negatively impact sales of existing products. He discusses the strategies Kraft Heinz employs to minimize cannibalization and maximize the overall return on promotional investments.
Resources mentioned:
š Ego is the Enemy by Ryan Holiday
https://egoistheenemy.com/
āŗ Subscribe to Our Channel Here: https://bit.ly/2TzgcOu
Follow Us Here:
Website: https://www.posit.co
LinkedIn: (https://www.linkedin.com/company/posit-software)
Twitter: (https://x.com/posit_pbc)
To join future data science hangouts, add to your calendar here: pos.it/dsh (All are welcome! We'd love to see you!)
Thanks for hanging out with us! š
Transcript#
This transcript was generated automatically and may contain errors.
Welcome back to the Data Science Hangout, everybody. If we haven't met before, I'm Rachel, I lead Customer Marketing at Posit. And so I like to let people know here that Posit is the company formerly called RStudio. Just in case we haven't shared that yet with you. We build enterprise solutions and open source tools for people who do data science with both R and Python.
And Libby, I'll have you introduce yourself too. Awesome. I am a Community Manager here working with Posit for the Data Science Hangout, but I also mentor in Posit Academy. So I help professionals learn with R and Python to help them do better work with data in their everyday jobs.
We're so happy to have you joining us today. The Hangout is our open space to hear what's going on in the world of data across all different industries to chat about data science leadership and connect with others who are facing similar things as you. And we get together here every Thursday at the same time, same place. Just not if it's a holiday. So heads up with Thanksgiving coming up in the month, we won't be here. But if you're watching this as a recording and want to join us in the future, there will be details below to add it to your own calendar.
And thank you so much to those who have helped to make this the friendly and welcoming space that it is today. We're all dedicated to keeping it that way. So if you ever have feedback for me or Libby about your experience that you'd like to share anonymously, good or bad, or maybe suggestions for topics to dive deeper on, I'm going to share this Google form in the chat right now, which is just a feedback form.
And I will also say that this feedback form and hearing feedback from attendees is what inspired me initially to reach out to Joel. So Joel and I had been connected for a while, but somebody specifically asked to hear from someone from the consumer packaged goods industry. And that's what got us reconnected again in the past few months. But you can always reach out to me directly on LinkedIn as well. And Libby, I'll turn it over to you for the next part.
We would love to hear from you today. So no matter your years of experience, your title, the industry you work in, the languages you use, we would really love you to introduce yourself in the chat. Say hi, include your role, where you're based, something you do for fun. Any roles that you are looking for or also hiring for, please, please, please share open roles on your teams. And then there are going to be three ways to jump in today. You can raise your hand on Zoom and we'll call you. You can unmute and ask a question. You could put a question in the Zoom chat. And if you can't ask it yourself, maybe your mic doesn't work or you're at a cafe, feel free to put an asterisk next to it, and we will read it for you. And then there's also the Slido link, which Isabella just put in the chat, because she's awesome. You can ask questions anonymously there. So as we go, we will also be checking Slido.
So with all that, I'm so excited to be joined by our co-host today, Joel Ashe, Senior Data Science Manager at Kraft Heinz. And Joel, to get us started here, would you be able to share a little bit about your role and some of the work that you do, as well as something you like to do for fun?
Joel's role and background
Oh, I'll start with what I like to do for fun. I'm a hobby collector. So my current one is woodworking. So if you're on, if you end up connecting with me on LinkedIn and ask, I will send you all sorts of fun pictures of things that I like to make. It's a nice split from coding, all the conceptual technical things we do to get out there and make things physically. So I enjoy that a lot.
So my job. Like I said, Joel, I work with Kraft in our RGM area. So revenue growth management. So if you think of any time you go to like a Target, Walmart, any grocery store, and you see a sale for one of our products, you know, we'll just use the Blue Box Kraft Mac and Cheats. You give like a 50% off. So sale promotion. We actually pay for that. So we pay the company, the customer, those are our customers, Target, Walmart, and so forth. So they don't lose out on their revenue and profits, and then it's then sold to consumers. So differentiation in terms there, since you generally think of our customers as the actual end result. That's not the case. It's the middleman is who we actually sell to.
So as you can imagine, it's kind of expensive to run these because anytime something is purchased on a promotion, we pay for it. We end up paying in excess of a billion dollars a year in promotions domestically. So we are tasked with optimizing that, finding the opportunities for better pricing, more optimal pricing, reducing the costs, since as you can imagine, much like marketing, it's not always a net profit generator, it can be quite just expensive out of the gate. So we will kind of want to reduce those while also maintaining customer relationships. Since companies like Target, they want to maintain traffic, foot traffic, and a lot of these are our loss leaders. So it's a balance of finding optimal math solutions, while also recognizing the limitations of how far we can push that, since it is kind of part and parcel with the business.
Promotional optimization in practice
So could you dive a little bit deeper into one of those projects and promotions and maybe share a little bit about like a specific product we might be familiar with?
So one thing we're working on now is trying to expand our understanding of what the actual trends are domestically so that we can minimize the expenses. I'm trying to think of the right way to say this. Minimize the expenses of a promotion such as cream cheese, like Philadelphia cream cheese is ours. It's 80% of the market. So 80% of the cream cheese in the United States is Kraft's product. Why would we even bother promoting it when we have that kind of market capture, right? Well, we don't actually have a lot of people domestically that buy cream cheese. It's about 4% of the markets. So you can see that little split, right? We don't have a lot of people that purchase our goods, but of the people that purchase cream cheese, we are the bulk of that.
So we are looking at ways both with kind of marketing results and with our promotional results to try to identify what the right opportunities are to put those on sale at the right time throughout the year so that we both minimize the expenses that can be associated with it. So making sure we don't discount too highly so that people get it for free. And then also enticing new customers to come into the market so that we can expand our market share, our household captures. So that's a very high level look.
The modeling can be anywhere from quite simple to a little bit more complex, although I would say that we, I try to push my people to not go on solutions like XGB to be direct about it because we still have our customers. Now I'm going to say this is a different way. Our business stakeholders, our customers as a data scientist still need to understand what's happening in our models. It's largely a recommendation engine. So getting them to understand where it's coming from, where it's kind of the risks are and the benefits of our models so that they can then negotiate those contracts as best as possible with our customers.
Interpretability is important, right? It's number one for us. It's truly number one. There are definitely areas in the business where we don't really care. Just do the best operations like flow control in a, in a, in a factory. Nobody cares about interpretability as long as you get the right answer. But our, my space is principally business recommendations. And so it's, it's number one, two and three, I guess, in terms of importance.
But our, my space is principally business recommendations. And so it's, it's number one, two and three, I guess, in terms of importance.
Zero to one
Well, I have an interpretability question for you. In your, your sort of bio or description about what you wanted to talk about. You mentioned zero to one and I have no idea what that is. And you knew this question was coming for me. What is zero to one?
I love that. Yeah. So there are some books on this that relate to startup cultures, starting from nothing and making something. There are a couple of different areas that you can think about with this and how it relates. Like if you were to work in an insurance company, zero to one happened decades ago. You know, the data tends to be incredibly tightly controlled, very, very, well, relativistically clean, relatively clean, right? There are actuaries, there are patterns, there are processes for how you can optimize and kind of what the walls and constraints are for your models with respect to what features you can use, you know, governance, that sort of thing.
Some companies, this is not the only one that benefits like this, where data science is a newly discovered talent, more or less. And this has been going on at Kraft for quite some time, about five years. So not to claim that it's brand new, but you have to integrate more complicated processes into systems that were never originally designed for, right? So if you're looking at Goliath monostack platforms and you're trying to say, hey, we need to inject all these models into this old Java system, you can get a lot of pushback on that. So it's learning how to be dynamic and how to create and how to create, I guess, value capture and build new systems in a way that's also hospitable and can be ingested into something that was never originally designed for it.
This also relates to business processes where you can be working with, you know, salespeople, marketing, other business folks where they have always had universal control over a process and then to inject math, which may be seen as witchcraft to some, into their decision processes in a way that might be a little adversarial or controversial for what they might originally think, but, you know, helping them along to kind of get them into a more optimal flow state. So it is very, very tricky. And it comes down largely to one interpretability, as we talked about with our models and understanding that the needs of the customer, which is largely kind of a product mindset. So in this case, the needs of the business are critical. So you cannot just simply go off, build a great model and show people how amazing it is and win them over. It's a lot of bureaucracy on top.
Book recommendations
I was just Googling. Have you read the book? There's a book called Zero to One by Peter Thiel. I actually have it on my reading list coming up. Oh, nice. Yeah. I have not had an opportunity to, but I've heard good things.
Do you have any books, other books that you maybe have read that you might recommend? Think about that one. Yeah. There's actually one that I just finished that I would highly recommend. It's called Ego is the Enemy by Ryan Holiday. I think it is incredibly valuable to really, really grasp and understand that that process of failure, to be quite honest. Failure is a skill. It is a necessary skill in data science. We do it every day when we're testing models, but at a very, very grand scale. You could do an entire project for a year that gets cut down. There's all sorts of things that can happen.
And learning the kind of core tenements of what that book is sharing actually reminds me of a quote from Star Trek, because I'm a nerd, I'm going to do this, from Picard, that you can do everything right and still fail, right? That is just life. That's not to say we always do everything right. That is to accept the reality of the world we work in, is that failure is happening. What's important is to kind of center yourself, learn, and proceed. So highly recommend it. There's some cool quotes in there. And yeah, I just, I actually just finished that up last weekend.
Measuring promotional effectiveness
Thank you, Isabella, for sharing the link to that book in the chat too. I see a ton of great questions coming in right now. Mark, I saw yours first. You want to jump in here?
Yeah, sure. Just a question about kind of the methods that you guys are using. Is it more along the lines of, I guess, like, are you looking at wholesale marketing effectiveness across craft as a whole or like within individual campaigns? I know there's like media mix models and marketing mix models for looking at like the effectiveness across, you know, across the board, but I'm wondering if there is something different for individual campaigns beyond, or that you would be using to measure those individually.
So you nailed it already. So that's great. You know what the core methods are. In terms of what we're doing uniquely, one of the limitations that we have is that we don't control like promotions and marketing activity, right? Like when we have an ad, it's like everybody needs to see this. And then our results, what we end up finding out from these companies is at scale, what was good? Like, did it work? How many units did we sell? Blah, blah, blah. So the best way that this ends up happening is we have to pay for the data from customers that are doing this themselves. I'm going to be a little resistant about what examples I'm allowed to share, because I'm not actually sure contractually what we're allowed to do. But you can think of like if Instacart were to do a promotion, they can do it uniquely
to an area, to a couple stores, and we can then go around and purchase that data and find out what happened. But we can't direct those. There's trade laws that are applied to that. It's kind of an interesting experience that there's a lot of information that we're just simply not privy to in CPG, but we still have to find ways to like divine out what the actual results were, which largely comes from paying other people to give us the right information to do those analysis.
But you're right. Other than what those, the methods, the marketing mix and that is exactly right. Okay. Super interesting. Thanks. Catherine, I see you had a question in the chat as well. Do you want to jump in here next?
It's kind of centered around a little bit quiet. Sorry. There you go. I need to buy a new microphone. My question kind of centers around if you and your team are focused on sales and finances around like what recovery do you get from promotions and things like that? What does the process look like for defining how badly you're allowed to miss and how frequently you're allowed to miss on those projections before you start getting in trouble? And before they start questioning, why are we even having machine learning people do this if we can just, you know, mail out a coupon to everyone and get the same results?
That's a great, yeah, I love how you're saying this. So there's a couple of things that we're being tasked with. Sorry, that was my cat. I was afraid that she was going to turn that off. We love cats, but they like to get in the way. Currently the expectations are simply that our predictions are better, more accurate than what the current processes are. That's it, right?
When we're doing our retros on whether or not it was valuable, that's actually a secondary process. So we're, I have one of my teams is more insulated from, did you predict it right? Because it's just, did you, are we assessing value correctly? And that's how we separate this out. So our accuracy, what we just do is like a WMA calculation of, did you, were you able to correctly predict how much a promotion would, would yield in terms of unit sales in a couple of months? And then our targets just are improving upon that through time.
So there's no like hard and fast rule around that with respect to how much dollar capture we're going to work. We're trying to go after, but in my other pod, it is largely an analysis process. You know, you sold a million units, the expectations are you had to do at least 800,000. Some of that is cannibalization, therefore we're going to give you a net 805. It's not positive and then it becomes fine.
On our team there as to whether or not we should be, I'm thinking the right way, should be hammered for like the actual, whether or not it's valuable because those decisions come then from our customer management teams, right? How many promotions do we have to run in order to continue to keep target happy? Ours is just largely about, it's more likely than not that you should do these 10 rather than these 15. And then it's up to them to go farther than that. I think that kind of got to the point of what you're asking. Let me know if that was, if I kind of missed it.
Yeah, it does. I think, you know, the answer being providing suggestions to the other teams that have the other pieces of the information does kind of protect you from, we said that this was going to get you $6 million and it only got you five and a half and like you were counting on that margin. Yeah. Awesome.
Arsenis, I saw that you had a question too, would you like to ask that? Yes, thank you so much, Libby. And thank you, Joel, for being here today and talking with us. My question actually has turned into two and it's around this topic of interpretability. I'll try and make this really concise. And I ask this because I've actually spent a lot of my time in my career kind of straddling the worlds of models that are built on frequentist statistics and inference, you know, things like regression and all that, and machine learning, things like extreme gradient boosting and all of these kinds of fun models. And so I actually have had some success with interpretable machine learning methods like partial dependence plots and things of that nature.
But I know that there is some resistance still from, you know, from certain types of audiences. So my question is, what has been your experience with those types of methods? If you've tried them, could you tell us about that? Yeah. When I was in insurance, partial dependence was what we did when I talked to actuaries. When my CFO wanted to know why we're making a drastic change, we had a conversation about golf.
It is wholly dependent on us to find the method of communication that results in a person understanding. That is key. And if you'd say, you know what, you're just gonna have to trust me on this, and you have built a relationship with the decision makers where that is a thing that they can accept, you do whatever modeling you need to do in order to get to the end state, and then next you'd be, yeah, go for it, right? Go for it because it's going to work, and you can prove that it's going to work. But there is no singular answer because it is never the same conversation.
It is wholly dependent on us to find the method of communication that results in a person understanding. That is key.
Our communication needs are to be very, very dynamic and flexible, and this is why I kind of prefer the management and really the technical product leadership aspect of this because to me that's really the code to crack is to get a person who perhaps never really cared of anything past geometry can understand the complexities of what these models are doing and why it really matters to them. And it's understanding them personally a lot of times, which can take a tremendous amount of time, but you do what you need to do, and you keep practicing and trying different ways until they hit the aha moment to get the message across. That's all there is to it. I wish there was some specific pathway, but every time you talk with somebody, it's going to be different.
Model interpretability vs. predictive power
Thank you. Yeah. So, yeah, the quick add-on to that that I had was in situations ā so, there have been a few research papers that have shown that with, you know, decent regression, you actually get almost as good accuracy or predictive quality as you can with some more advanced machine ā or more complex and obscure machine learning models, but it's not always the case. So, have you ever run into a situation ā I'm sorry ā oh, have you ever run into a situation in which the accuracy or the predictive quality of a machine learning model was so appealing that you wanted to try and push it but couldn't, and then how did you make that decision?
Oh. I think we lost Joel for a second. Joel's back. Okay, Joel's back. You are. Hey, Joel. Sorry about that. I saw you stop moving, and I thought that was definitely me. Okay. With more advanced machine learning, there's papers that are saying it's more valuable. I think that was kind of the key. So, yeah, the question is, when you do find that there is better predictive quality with these more ā these machine learning models that use more randomness, and you've got a decent interpretable model, how do you make the decision to try to kind of push for the predictive power or just kind of fall back and say, okay, this is good enough with like a regression model or something like that?
We might need to ā maybe we'll jump to see if other people have thoughts they want to add here while Joel's over to a different Wi-Fi, I think. Anybody have thoughts on this? Jacob, I see you had questions about specific modeling as well. Yeah, I was actually going to change my question, but I mean, I work in the CPG industry too with Schwann's Shared Services, so we're the guys that make the Red Baron's pizzas and the Mrs. Edwards cream pies and all the BB Go products that you see being advertised everywhere. In my experience, whenever I've done projects for consumer brands or HR, the regression assumptions are almost always completely violated, and it's very difficult to implement a regression model and have it make sense. So, unfortunately, a lot of the times we do have to kind of go towards a tree-based approach, whether it be a random forest XGBoost or light GBM, that just seems to lend itself better to explaining what effect each variable is having on the response, and that tends to go better, in my experience at least, with a business unit.
So just as a quick aside, I had a project a few years ago where I was just predicting employee churn, and due to the severe multicollinearity of most of the variables, I wasn't able to really use much of a regression approach. So we just went with an XGBoost, you know, model, and were able to find some variables that were really driving the response of people quitting. And as a result, we presented it to the stakeholders, and it helped them reduce turnover. So that's just a quick example, but yeah, Arsenius, I hope that kind of addresses your question a little bit. I don't know if you heard that or not, I don't know what's going on, so.
Community discussion on NPS and stakeholder relationships
No, thank you. I see Joel's back as well, at least I see Joel back in the chat. How about now? Can you hear us okay? Okay. We may jump to more community chat if we need to. I did see a note somewhere, somebody shared that there are solar storms happening that could affect Wi-Fi. I wonder if that has any impact.
Let me jump over to another question and see if we have one that we can kind of discuss as a community. Let's see. Alan, do you want to ask your question there about the stakeholder relationships? Yeah, maybe other folks have a perspective on it if Joel isn't able to weigh in on it. I'm curious if I'm understanding right, like, there's a focus that we're primarily talking about here about relationships with sellers and how do promotions there have an effect, and I'm curious about the interaction between that kind of work and work at the consumer-facing side of things, stuff that measures brand loyalty and NPS scores and all that kind of stuff. I'm just generally curious if there's a relationship there, if those kinds of marketing teams interact or if their missions are really separate or have some kind of collaboration. Be curious if anybody's, since we've got other folks in the business around, if they've got some perspective on that.
Anybody have thoughts on this? Mark, I see you unmuted. Yeah. I was going to say, I used to work in patient experience measurement for a hospital system down in Houston, and we always, I guess like the corollary would be patient admissions and choosing to use the hospital versus other hospitals in the city, and kind of like the core thing that we saw was NPS was kind of a weird indicator to use in terms of as like a predictor because in the case where patients have poor experiences, they report a bad NPS, they respond to the survey, report a poor NPS score, and then they stop coming to your outpatient center or your recurring care centers or whatnot, but then you have this patient who had generally negative views dropping out of the system. So your patient population is decreasing, but your NPS scores are going up because you're only retaining the people who had good experiences. So it was kind of a weird thing to use as a leading indicator, more so than like a measurement, like a temperature check on its own, at least that we found in that industry.
That's interesting to hear because in medical device, we're really interested in NPS, but there's also an interesting disconnect between, you know, NPS measures physician sentiment largely because we don't have a direct, you know, like end recipient relationship with folks who like receive devices in procedures. And so it's interesting to think about like what's perspective of those folks on us as a company versus physician customers. And then there's all kinds of weird health system interactions as well that get wrapped up in there. It's hard stuff to measure.
So, Joel, I think that you may still be stuck in your Wi-Fi problem, but if you want to
type in the chat and share any thoughts or turn off video and see if that helps.
But we might shift gears a little bit until we have Joel back and Libby and I have gotten good at rolling with going with the flow here.
But a question that Libby and I had wanted to ask the community here is we often share about different pieces of career advice that have been in the Hangout, either something that's been memorable to the future leader that someone's given them as advice or that they've given others. And Libby, you mentioned a few people had shared with you advice that stood out to them that was shared here. And I was just thinking it might be fun to share a little bit of that. And I'd love to hear that from you all.
Yeah, I would like to hear it too, because I remember somebody messaged me or they posted on LinkedIn about something that they had heard at the Hangout. They were like, that has stayed with me and I've continued to think about that. And I was so excited about it, but I don't remember who said it or what it was. And I didn't write it down. And the search function on LinkedIn is not the best. Has anybody ever tried to find something on LinkedIn that they had seen before and it's just impossible?
So if you have something like that that you can think of, absolutely, either put it in the chat or raise your hand. And I saw Chirag had your hand up just a second ago. Do you have something that you would like to talk about?
Yeah, I just wanted to add to what Alan and what Mark was saying in terms of like, so I worked in the hospital area as well in the Boston area. So one of the major health system and I did a paper a couple of years ago, but what we found it's easier to retain patients than to attract anyone. I mean, everyone knows that, but we had a similar situations where we open up our own specialty pharmacy in-house or health system. And then what we found when we're trying to attract more customers where we were basically comparing services to CVS Specialty or Onco360 or Rio or something along those like specialty pharmacies. And when we compared our services, we because we had more control, we just went all in on them. We did like the we went with the adherence method where we would call them just to make sure the medications were taking on time.
We also did follow ups if they had a doctor's visit, they'll have the prescription ready for them in the clinics. And we just went in all in on all the services we could offer them to attract it. And yeah, that helped. So just wanted to add that.
Thank you for sharing. Anybody else want to weigh in on that specific topic? It seems like there are a lot of people interested in in that area.
I think there's a difference, especially when you're talking about health, because navigating the US health care system is so complicated that unless you have a specifically negative experience, it's a lot easier for you just to stay where you are. They have your paperwork, they have all of your information. And so like unless you were like, I need to leave, like I do not want to go back here. It's you have like inertia in terms of staying with that where you may not see, yes, I'm going to recommend this hospital to a friend also because you don't really want to say like, I hope you you don't get sick. But if you do go here, like that's also kind of a weird human conversation to be having. So it's very different to talk about health than it is to talk about a lot of the other industries when you're talking about things like retention and attraction of, you know, who is your customer base?
Agree, but you can also entice new customers or patients or whoever with like additional services, more personalized recommendations or services, and I feel like that's where you can attract more. Oh, absolutely. I'm also in health insurance.
So I'm just trying to bring it back around to, you know, Joel as well, like, you know, enticing a new customer to buy cream cheese is very different than enticing them to come to your hospital. Yeah, it's more like when you need us, we hope you pick us. Right. But we hope you don't need us.
Cannibalization and new product launches in CPG
I was actually this is a question I was going to ask Joel, but I could ask everybody here, especially people working in CPG or consumer packaged goods. But when we were talking about cream cheese as an example and impacting the growth of people buying that product, I was wondering how they influence that. And like, for example, I really like to drink Aperol spritz and I feel like the creation of that being a drink had to have had like a huge impact on sales of Aperol. And so I'm wondering, is it like sometimes about making new recipes or trying to like attach to new holidays or how that works?
Jenna, I see that you you popped up on camera and unmuted. Well, I work I work at Brown Foreman as a I was a data scientist and an AI developer a lot specifically for Gen AI. But so our products are like Jack Daniels, Woodford Reserve and stuff like that. And if you're not super familiar with our entire portfolio, we don't have like a ton of different flavors. We have for Jack Daniels, we have like honey and a couple other things. But we do a lot of like cannibalization studies before we launch new products.
And we also look at kind of competitors because we do have a lot of competitors that have a lot more kind of different flavors of things coming out. And so there's a lot of things that we look into before doing that. And sometimes it's a big focus on cannibalization. And even like recently, we had a new ready to drink product that we launched. So we wanted to understand launching this new ready to drink product, what impact did that have on the parent brand? And so some of it we try to do before we even think about launching new products. Sometimes we do after we've launched a new product and even looking at competitor sales that I think it really depends on the company and the brand and their strategies. But sometimes kind of looking at the reverse of that has been really valuable as well. So seeing like if we launch this new product, what is the likelihood that it might take sales away from another product in our brand? And what are the kind of impacts of that as well?
Jenna, Luke and I both thought you were going to reach for a bottle and bring up like. Oh, I was reaching for my charger. I was just going to show us both. That would have been great.
Catherine, does that answer your question on the cannibalization in that context? Yeah, I think Sol's answer cleared it up. In my mind, it was like. I don't know where you take like pieces of recipes that people like and smash them together and maybe it does or doesn't work, but hit salt answer makes way more sense than what I made up in my head.
Awesome, any any other thoughts on this before we we switch gears? Let's see. Well, Joel, if you come back to us, let us know. Let's see, I can see Joel keep trying to join, so maybe we will get him back. If not, we will definitely have him back another week because there's so much more that I think that we can all learn from his experience to.
Liv, do you have a question that you wanted to ask everybody? I don't know if I have a question so much as like I I'm still thinking about your recipe question because it makes me think about a YouTube show that I love called Tasting History. And if anybody watches it, you'll know it's fantastic. But like the host makes recipes from history, whether that's like ancient Rome or like right now he's working on World War Two recipes. And so many of them were actually really driven by like radio shows and things where personalities like Betty Crocker, who is a fictional personality, were kind of like made up to to sell things. So that's what I'm thinking about. Not that I'm going to go into all the details about that, but I'm just thinking about it.
not so much a question, although I would love to know, I put this in the chat. I don't know if anybody saw it, but are there things that we would want as a community of the Hangout community that we don't have that we would like to benefit from? So be thinking about that and put that in the chat or send it to me in a direct message or something on LinkedIn or things that you would like to talk about. Hang on, Betty Crocker wasn't real. No, but Betty Crocker was a fictional kind of conglomerate of a whole bunch of people. And there was a radio show that was voiced by one person, but like, she wasn't Betty Crocker. She was an actress.
Sol, did you want to chat about maybe a little bit about cannibalization just so we can talk about it on video and not have it just be in the chat where people on YouTube can't benefit from it? And Sol, I see you have an idea for a Discord with different channels as well. So while we're talking about it, it might be good to hear your feedback on that too.
I don't know if Sol can, oh, can't get on camera today. OK, let me see if I can find Sol's explanation of cannibalism. The chat moves so fast here. OK, so Sol said that in CPG terms, they'll try things out, but most things are seasonal. So there'd have to be a large push for a new flavor to be a new hero skew, like a hot or a spicy flavor. And sometimes those are more successful than expected. And there's conversations about keeping it. So I guess maybe your product's sales kind of take over the sales that you would have had for other flavors. Like if you have something that's really popular, then you're eating into your own sales of your previous flavors. Like if you have something that's really popular, then you're eating into your own sales of your previous flavors. Does that sound right? Yeah. It sounded to me like you're not actually attracting a new customer. The customer is still buying the same quantity of your product. It's just a different flavor. A different flavor. All right. That makes perfect sense. That was my understanding.
I've actually kind of thought of that in terms of community events, too. So if we have a data science hangout, but then we also have like a meetup or a workflow demo that month, like at what point does that become too many events where somebody is leaving one because they want to go to the other one? So it's almost like instead of thinking of it as a product, the event is the product. But Ralph, I see you had a question kind of on, I think it's on this topic, but you called it steel effects. And I wanted to see if you want to jump in and weigh in on this, too.
Yeah. So the background, I worked at General Motors CPG Industry. That was my initial corporate role. And it is on Betty Crocker Drive in Minneapolis, if you didn't see that in the chat. She isn't real, but she still has a street named after her. It's just the exit off the freeway. And yeah, that was the term. It was called steel, which is basically like, hey, there's like 30 varieties of Cheerios. If you add another variety of Cheerios, how much of those sales are going to be incremental new customers and how many of them are going to be steel, which is basically just taken off of a core brand. And that was never, that was closer, I think, to witchcraft and dartboards than a science. So I was always curious how other companies were trying to calculate that out.
Experimentation and learning from outcomes
Absolutely. And Libby, I see you just called out a question in the chat. Do you want to point to that one? Yeah. Amrit, did you want to ask that? Do you want to unmute? If not, I can ask it. Yeah. So I'm here. Hey. Hey. Yeah. So I'm a relatively new data scientist and I am kind of diving into the world of experimentation and all of the, yeah, so efforts that we do as data scientists, right? So I was just wondering, this is for like experienced folks here. So I'm just wondering how many of your experiments in a project have led to successful outcomes versus not?
All experiments are successful. Just whether or not you get the answer you were looking for is the question. But if you've asked the question and you're like, man, I really hope the outcome is X and you find out that it's not, that's really valuable. And it's really valuable to write that down because then the next person who comes behind you doesn't have the same question and spends their time looking into that. And I come from like a science background and publishing, and there's this hesitance to publish anything that doesn't have a significant P value. And so you end up with all of these different research labs doing the same experiments over and over again that nobody's telling each other about because you don't want to publish your results because it's not, oh, I discovered this significant correlation where like it is important to do the experiment and to learn what that experiment tells you regardless of what the outcome is.
All experiments are successful. Just whether or not you get the answer you were looking for is the question.
And then in a business context, you do have to balance like, do I need this experiment to be able to like drive business outcomes and how much overhead do I have to just run experiments and see what the outcome is? But I feel like all experiments are successful.
Yeah, there's an apocryphal or, you know, semi-apocryphal story about Edison and the light bulb. And he talks about it, took him, I can't remember if it's a thousand or ten thousand tries to find a really stable working, you know, model of the light bulb that would be a workable product. And he was talking to someone, you know, at like a cocktail party or something of the equivalent. They said, well, do you feel bad that you wasted those, you know, for argument's sake, let's say nine hundred and ninety nine other things under designs? He said, no, none of those are wastes because I learned nine hundred ninety nine things to never do again.
And I'm just I'm just backing up what what the person ahead of me just said. But it's a little bit more of a of a story, you know, a little fireside chat kind of story. But but but that was, you know, his his thing was I learned nine hundred ninety nine things that don't work and that may not make money.
So if success, if the measure of the success is profit, then I would argue that maybe actually a lot of experiments don't work from a profit point of view, but from a knowledge acquisition, they always work.
Steve Wozniak, I saw him at a talk at at at my company recording with talk at my company and he said kind of the same. I'm actually copying him. He's somebody asked him, well, tell me about your failures, you know, things you designed, you know, in your days before Apple and through Apple. What did you design? That was tell us about your failures and looked right at the audience and said. I never failed. Just some of the things I did didn't make money. And I think that's those are just those that's my way of putting some story around what what the person ahead of me just said to think about a little bit.
I never failed. Just some of the things I did didn't make money.
I'm going to steal that. And Mike, I see you have your hand raised and Jenna, I know you had a comment here, too.
No, I will also endorse what Catherine said and Chris as well. But I think that the key thing is that if you're in business, be careful what you're experimenting with, right? Playing with stuff to see if it'll work or pushing things and pushing some buttons and see what I get. You know, yes, you'll learn stuff, right? But if sometimes if you're just experimenting for the sake of it in business terms, sometimes you can pick the wrong experiments, you know, the wrong experiment at the wrong time. And yes, you learn stuff. But what you learn is that you're going to get parts of your anatomy handed to you because you didn't do something that was moving the business forward. So it's, you know, it's like picking your battles as a parent. I hate that phrase, pick your battles.
Well, while we were talking about things that we make, hobbies in the Hangout, I have this album right on my desk right here. So I had to share it.
Oh, Veronica, I want to jump in next. Oh, yeah. I just want to just elaborate a little bit more on the experiment. Just remember when you're testing or experimenting, if you don't get the result that you're looking for in most cases, and especially most business cases, you just be prepared to provide alternatives and alternative methods that are going to help you get to a more better solution or to a more profitable. I work in products, so to a more profitable product, so alternatives.
I could give you one example. My prior company that I worked for, it was an MVP product that we were working on, took a lot of money invested, a lot of developers invested. I did a test and some of the information we were displaying on the product wasn't very beneficial. So and also it wasn't getting adapted very rapidly. So I had to create a report that provide alternative ways, kind of like change management. So users can start adapting it while we make additional changes, but use what's currently available within the product. So just remember that on the business side of things, if the experiment doesn't come back to the result that the business is looking for, remember to provide alternatives is going to help get to that next, the next step.
So how would you, does this answer change for, let's say just machine learning model development, where we are iterating through different data processing steps versus like trying out different models. So would this answer change there saying that none of the experiments are a failure?
Hannah, you had some thoughts on this question, especially the one right before. So I just want to see if you want to jump in here.
So I kind of mentioned, I think what other people have already said. So just to kind of echo what was already shared was really great. And sometimes, you know, the, the modeling and everything that we do goes well, and we find out the answers that we want to know, but for the business, they might've gotten in there with some very specific expectations, like just generically launching a new product is going to lead to this big halo effect on a parent brand, for example, and we might not find that that's the case. But in order to make it so that the work we did is valuable, sometimes it takes even more than one discussion or presentation. And what, you know, what can you add to your results for context that allows them to do something with that? And it might be like, well, we didn't see a halo effect, but we also did not see cannibalization, which is a good thing. And you can bring in, you know, other strategic studies and other information that's valuable that maybe isn't work that you did, but that's already been done in that area just for additional context.
Yeah, I find sometimes spending just as much time, if not more, on how do we present our results, how do we explain things to people so it makes sense and they can do something with it than just doing the analysis and here's the results for our modeling. Like sometimes it takes a lot of work and how do you communicate that out? And these days, that's what I spend most of my time doing. I don't get to do as much fun coding and stuff anymore, but I do a lot of presentations and discussions and stuff like that. So I think that part of the work is also just as important.
I think what you do probably adds more value. I find it a lot harder to do sometimes, but it's important. How do you explain that in a way that makes sense and what can we learn from that? We even had a lot of times a lower adoption rate for the work that we were doing and they would just go to external companies because they might be more likely to maybe think they're getting better results or something like that. But sometimes you just learn like, well, if we had this other kind of data, maybe we could do a better job with this. So even just pointing out things that might be missing that would be helpful and that at least gives them something to go off of, not just like we didn't see the sales that we wanted. We could add in there. I think Veronica mentioned alternatives. So in that line as well, I think that's great as well. Sometimes it just takes thinking outside the box a little bit. But it's super helpful just to think about all the ways that you can explain why it's still helpful, even if you don't see what you wanted to see.
Being rooted in the business domain
Oh, yeah, sure. I think I was just emphasizing that how important it is, I think, for experimentation and exploratory analysis and so on to be really rooted in the domain that you're in, the business domain that you're working in so that you have some kind of clear expectation for what you're going to find and what you'll do, what kinds of decisions will be made one way versus another way based on what you find. The tension that I've experienced are when folks are kind of distant from a business perspective. I think this is why analysts and data scientists need to be really rooted and embedded in the business that they work with.
Often folks who are who are not have this idea that, well, we'll just explore and an insight will be emergent. And you can find interesting things that way, but you can't get direction on what you do, what's valuable to your stakeholders, to your customers. And so I think that even when you're experimenting and exploring, I think being rooted in like what's the work about, what do we want to find, what do we want to answer and what will we do differently based on it is just is just super important.
And so I think that even when you're experimenting and exploring, I think being rooted in like what's the work about, what do we want to find, what do we want to answer and what will we do differently based on it is just is just super important.
And when folks ask for like, well, I need reports to do all of the following things, then you can really say, like, well, what are you going to do? Then we can be really focused and we can find what you're looking for, successful or not. We at least know if we found it. So I just think that's super, super important.
Totally. Mike, I see your hands up. Back to me again. One example of a thing that really irritates me as the lead of a group or an SME is when someone will take an example like, hey, I'd like to I'd like to use Python to recreate all the dplyr functions because I'd like to learn dplyr, I'd like to learn Python. And it's like, OK, that's valid to a point, but you could do all today in R using dplyr. So what are we doing here? What are you actually doing? Are you learning Python just for the sake of it? Or and once you're done, what is going to happen with the experiment that you've just done here? Is this just for you to learn something which is valid, but if you're doing something for the business to move the business forward, that experiment may not be the best use of your time.
I love this unexpected direction we went here with experimentation.
I know we have just a few minutes left here, so rather than jumping into a new discussion, I just want to say thank you so much to everybody here for just kind of jumping in on our chaos mode here. I would love to have Joel join us back again when we figure out our Wi-Fi issues so we can give him the session we wanted to have with him and get to learn from his experience. But thank you all for for jumping in here and still making this a great hangout. Always happy to be here. Thank you all so much. Have a great rest of the day. Bye, everybody. Bye, everyone.