Resources
Mar 28, 2024 |
58 min
|
293 views

Kshira Saagar @ DoorDash/Wolt | Data Science Hangout

We were recently joined by Kshira Saagar, Sr Director of Data Science & Analytics, International at Wolt & DoorDash to chat about getting Analytics & Data Science teams a seat at the table - by ensuring that the work done by the DS & Analytics folks impact real-world outcomes and there is credibility & trust built on all sides of an organization to treat Analytics as a True Partner.

Speaker bio: Kshira Saagar is currently the Senior Director of International Data & Analytics for Wolt & DoorDash, and has spent 32.1% of his life helping key decision-makers and leaders make smarter decisions using data, and believes that every organization can become truly data-driven.

He has been consecutively recognised among the Top 10 Analytics Leaders in Australia, for 2019,'20, '21 & '22. At every place he has worked, working and will work - he likes to go back to the fundamentals, asking people to ask more questions of data and being a tough taskmaster on getting actionable outcomes from data, on all possible occasions. Outside work, Kshira spends a lot of time on advancing data literacy initiatives for high school and undergrad students.

________________________

ā–ŗ Subscribe to Our Channel Here: https://bit.ly/2TzgcOu

Follow Us Here:
Website: https://www.posit.co
LinkedIn: https://www.linkedin.com/company/posit-software

To join future data science hangouts, add to your calendar here: https://pos.it/dsh (All are welcome! We'd love to see you!)

Thanks for hanging out with us! šŸ’›

image: thumbnail.jpg

Transcript#

This transcript was generated automatically and may contain errors.

Hi, everybody. Welcome back to the Data Science Hangout. I'm Rachel. I lead Customer Marketing at Posit. So excited to have you all joining us today.

The Hangout is our open space to hear what's going on in the world of data across different industries, chat about data science leadership, and connect with others who are facing similar things as you. So we get together here every Thursday at the same time, same place.

So if you are watching this recording on YouTube at some time in the future and want to join us live, there's details to add it to your calendar below. I'll just want to say this in the recording too, to make sure that it does add it for you at 12 Eastern time so that you can join us live.

Is this anybody's first Data Science Hangout today? Say hi in the chat so we can all welcome you in and say hello too. We're all dedicated to keeping this a friendly and welcoming space for everyone and love hearing from you no matter your years of experience, titles, industry, or languages that you work in.

It is totally okay to just listen in here too, and you can be a part of the party that happens in the Zoom chat here, but there's also three ways that you can jump in and ask questions or provide your own perspective. So you can raise your hand on Zoom. I'll keep an eye out and calling you to jump in. You can put questions in the Zoom chat. And if it's something you want me to read instead, you can just put a little star next to it. And then third, we have a Slido link where you can ask questions anonymously, and Isabella will share that Slido link in the chat here in just a second.

One more thing. Oh, I guess I have two more things. We do have a LinkedIn group for the Hangouts if you would like to join. I know those groups aren't always the best for discussion, but it can make it easier to find each other after the fact and connect.

And the other quick note I wanted to share is registration just opened for the POSIT conference in August, and I know there's super early bird pricing for I think just three weeks. So I want to make sure that I let everybody know that here. Jeremy and my team made this fun virtual background that talks about it too, but I will share that link. Oh, Isabella already beat me to it in the chat as well.

Well, with all that, welcome again. I am so excited to be joined by my co-host today, Shira Sagar, Senior Director of Data Science and Analytics International at WOLT and DoorDash. And Shira, to kick us off here, would you be able to introduce yourself and share a little bit about your role and also let us know something you like to do outside of work too?

Well, thank you so much for having me, Rachel. I'm really excited to be able to come in and share what little I know. The first thing I'll say before I get off my mark is I tend to speak very fast. If you don't get it, if nobody here gets it, I'll really try to pause. I tend to speak 245 words a minute, which somebody has measured it for me, thankfully. But if you feel like I'm going too fast, I'm always happy to stop so that everything I say doesn't sound like gobbledygook. The point is there's no filter between my mind and my mouth, so I tend to say a lot of things and then think, should I have said that? But yeah, hopefully it stays in a safe space.

Everyone who I haven't met, my name is Shira. I take care of analytics and data science for WOLT, which is a part of DoorDash International. It's an international wing of DoorDash. I'm sure all of you know who DoorDash are and what we do. And for those of you who don't, DoorDash is the largest food delivery provider and grocery delivery provider in the US. And so they have an international wing in Europe, in Canada, Australia, New Zealand, in other markets. And so I take care of the analytics and data science team for those markets. So in a nutshell, that's what I do.

But to be very practical, what does that mean? What we do is the team and I, we act as addition enablers to the point where the way analytics is structured at DoorDash and also at WOLT is how we are not consulted after the fact, which has always been my bugbear. It's not the team which has come and told after everything's done, by the way, we finished doing this thing. Can you actually have a look and tell us if it's successful? But the other way around there, the way the teams are set up from scratch is we are part of the conversation on, do you want to build something new? Let's have a conversation. That's the right thing to build. Do you want to take a big decision? Let's have a discussion of the right decision. So the big team for us is being true partners and having a seat at the decision making table. And so that's something that's different for us here.

And that's typically my calling card for, when I try to explain to people what I do, that's different to other analytics I've done before and why I really enjoyed it. What I do here is you get a seat at the table, which is quite hard to get in other places. So that's me. And that's what my team does. And I'm happy to probably share more as we chat about it.

What I do here is you get a seat at the table, which is quite hard to get in other places.

What do I like to do outside work? I used to love running. I used to do a lot of marathons and half marathons. I haven't done that in a while, but that's what I like to do outside work. And it brings my mind clear. I think of it as control C and control delete of my brain. And when I run, I try to process, throw everything out. So that's really helpful. So that's what I do outside work.

Awesome. I feel like the past few weeks, we've had a lot of marathon runners. It's like, we need to do some sort of a road race here.

Getting analytics a seat at the table

I love that you just mentioned making sure that analytics has a seat at the table, right from the start there. Because I know when we were chatting before, that was something I was really interested in learning more about. Could you tell us a little bit more about really what that means and how you make that happen?

Sure. So in essence, what that means is three things, right? One is we don't, we make sure that everything that we build as a business, not just the product team or the ops team or whichever team, everything we've built as a business all ladders up to what the business eventually wants to achieve. So we are the truth seekers that make sure that anything that's being built, we keep everybody honest and accountable that these are the right things to build for the business. So you can use big words like strategy or vision and stuff like that. But at the end of the day for me, it's numbers that all ladder up to what the business needs to achieve. So we are the truth seekers that help achieve that. So that's number one.

Number two, what that also means is we are the team that keeps people accountable in the fact that if we can build something, we can, we can all think it's a great idea, but somebody has to run an experiment to figure out if it's the right idea, if it actually works. The thing with having ideas is anybody can have an idea, but does it actually work and measuring it and keeping everyone honest and accountable and in an unbiased manner, that's the other thing. So apart from truth seeking, we also make sure that we experiment a lot.

But the third and most important part that we do is the fact that we try to bring teams together and connect the dots. So the problem with having a much bigger business is everybody's focused on their part of the business. And this word that gets thrown around with the silo. So nobody connects the dots. I like to think of my role and my team's role as being the people who help connect the dots. So my favorite character growing up from Sherlock Holmes was not Sherlock Holmes himself, who seemed to always solve the problems. For those of you know, Sherlock's brother, Moria, Mycroft Holmes was the person who could actually connect all the dots and sit in a room and see everything that's going on and understand why something actually happened. So being able to know the second and third are the things and connecting dots. So those are three things that we do. And that's what we mean by being true partners is we are very helpful to the business in actually making sure what we do is the right thing for us to do. That's what we do.

Use cases and experimentation

Thank you. So I'm thinking of myself using DoorDash and was wondering if you could give us an example of something that your team is impacting across the business or maybe an example use case.

Sure, I can quite a few things I come to mind. One thing I can talk about is, for example, if you see when you shop for something on DoorDash, you have a carousel that shows you what you need to see instead of if you let people decide what needs to be shown on the carousel, the challenge is there is a very delicate balance between completely personalizing it and going the I don't want to name big brands that are not ours, going through a music subscription platform route where you saw things that slowly become completely irrelevant because it's super personalized. I have never seen it before, or you can make it super manual and everybody sets up what restaurants you need to see and we need to strike a delicate balance between that. So every time you open the app and you see a list of restaurants there or a list of stores that you want to shop from, the team has actually thought about a very a solution that can solve for both. It can augment both the personalized side of things and also provide the operators the right set of inputs and the right set of parameters. So we come up with something called the heuristics, which is if you show people X number of carousels, Y number of times and X number of restaurants, there are X percent probability they cannot. So they know what they need to show. And and if they want to push their own restaurants, if account managers want to make their restaurant pop up, they know that that doesn't need to show up, whatever. There's a number of times that need to show up. So we come up with these heuristics and magic numbers that we provide people that can then be plugged into the personalized platform as inputs from the operator side. So we bridge the gap of being not completely manual, not completely personalized, but a good balance, of course. That's an example of what you see from the output of the team.

Thank you for that. I see Arsena's just put a question to the chat here. Do you want to jump in?

Yeah, thanks, Rachel. So sure. Thanks for for your time today and just have a really good question for you here. I'm wondering if you could talk about some of the the challenges that that you all try to deal with that DoorDash with data and if you could talk about some of the methods that maybe you use, like modeling or deep learning or some of those things. So, yeah, that's pretty much it.

Sure. So we the team works on all aspects of the business. So we have three parts of the business. So we have the consumers, people like you and I who shop on DoorDash, then we have the merchants who are the restaurants and the retail partners who sell on DoorDash, and we have the Dasher partners who are the people who actually deliver. So we tackle and touch lives of all of these individual people in different ways, be it what they buy, where they buy from and how they get delivered.

What kind of things do we try to tackle with data? Like I said, in the sum essence of what we try to do is instead of what we realized is a blanket approach does not work. So if you want to come up with a pricing mechanism, having the same pricing mechanism across all countries does not work like that's we all know and we all agree that having a one is to an approach makes no sense because having the same pricing for all markets is not good. And then somebody can come up with an amazing approach saying maybe we should personalize pricing for everybody. That's what the end of pricing is. And trust me when I say that that also does not work because extreme personalization also does not work because it does not work for the edge cases. What we need, we've realized at DoorDash is a balance for having personalization, but also at the same time having the business context built in because when you let personalization solve for a problem, the objective function of personalization could be say conversion or whatever it is, which basically means that you're not solving for immediate business needs, which could be getting more new customers or making a particular restaurant's work or making some particular markets grow faster. So if you want to have multiple optimization features, what do you realize is the better team comes in is we try to find the balance between both.

So like I said, the kind of problems we try to solve is how do you take something that's extremely personalized and apply the human element to it by providing the business the right levers to tweak. So we come up with a lever. So to your question, a lot of what we do is we do a lot of experimentation created work, which gives us the relationship between an input variable and a set of a lot of input variables instead of output variables. So if, say, conversion is your output variable, what is the relationship to all the input variables? And we do multiple studies, short term, long term and long term holdouts to understand if one person improvement in conversion happens, what does it mean long term for retention, et cetera, et cetera, et cetera. If one extra impression happens here, what does it mean in terms of the merchants revenue, et cetera? So we come up with those relationships. And that's what at the end of the day we try to do is distill all the findings into those so that eventually we can plug it back into our relatively amazing machine learning systems that can then go away and give humans human operator the inputs to do what they need to do. So that's that's what I think. That's why I look at my teams and some output of what we do.

Team structure: data scientists vs. machine learning engineers

Svetlana, I see you had a follow up question about personalization, do you want to jump in or was that covered there?

Yeah, sure, thank you. Yeah, that's very interesting about like what you mentioned about personalization. I'm curious, like what's the what the roles are? So you have like data scientists versus analysts versus machine learning engineers. So like what's the balance between those roles and who's building, who is deploying this?

Yeah, that's a great question. So we have a separate machine learning, machine learning department. So we call them DSML or machine learning engineers. So we have separate set of machine learning engineers and they focus on building the software at scale. So these are these are applications that are going to be touching literally millions and millions of lives in any given day. So we want to build it at scale and for it to work at scale. So we have some amazing human beings who know what they're building at scale. So the machine learning engineers and people who can do the MLO. So that's one side of it.

Before we get into that stage, we have the data scientists, the analytics side of data scientists who get involved, trying to understand what are the right features to think about, what are the right things we should optimize for and what kind of business metrics can it move? Because we eventually want to be able to plug it back, like I said, into the whole conversation of, yes, we can build this amazing algorithm, but what will it move for the business? And so we start from there and then we think of it as us running the first leg of the relay, building it, seeing what works and then handing it off to the machine learning engineers who can then build it at scale and then optimize it and make it better. So that's how we like to think about it. It's a partnership for how we do it.

Hey, Alan, I see you had a question in the chat, too. Want to jump in here?

Yeah, thanks. I'm really curious, you mentioned when you're sort of introducing yourself that you've been in places where I think maybe I read this in places where analytics was not at the table, like as a partner. I'm curious if coming to DoorDash, was it already there? Was that already established or did you have to work to make that partnership really

effective? I'm just curious what that kind of journey was like. How do you become really visible as a partner?

So the honest answer is when I came to DoorDash, it was already established. The people who had established it had made sure that that was part of the secret for why DoorDash grew as it grew and how we would find out what it could do. What we are trying to do, though, is now that DoorDash has acquired Volt and I've now moved from DoorDash to Volt, we're trying to bring the same culture here. So for me, it's an ability of seeing what the future can look like and trying to bring the future again to companies that we don't have it in its truest sense.

And to your point, what I've seen in the past, I've worked in other companies where you could have a role, you could have the title, but you're often consulted after the fact, where you can't effectively change things. You can only measure things. So you're mostly the measurer. I was mostly the measurer rather than the influencer. So you can measure things and anybody can measure it. But having a say and saying, yep, don't do this or do this or do this because this makes sense. I think that gives you a totally different responsibility and skin in the game. And that makes it all the more fun.

That's what I've seen. But yeah, so to answer your question, already at DoorDash, not at Volt. And so at Volt, I'm trying to build that culture.

That's great. That environment where you can experiment and have business buying for doing that and then bringing things into action sounds really cool. Sounds like a neat environment.

Thanks.

Experimentation at scale

I know you mentioned some of the experimentation that you're doing, but I remember from our our pre-call, you mentioned like there's millions or maybe it was billions of experiments happening at one time. Could you just share a little bit more about that and like that, that huge number that's happening?

Yeah, so at any given point in time, my understanding is we have three to five thousand different experiments going on at any point in time. And so if you're accumulating some data for multiple weeks, multiple months, you're only talking about tens of thousands of experiments. And so the reason is not to experiment ourselves and increment ourselves into irrelevance. It's about trying to understand truly what works and what doesn't. And there is a culture, like I said, it's not just us doing it.

As the saying goes, you can't clap with one hand. The other side of the party also needs to come to the dance and also dance. So the business comes down and says, we think there's an amazing idea, but we want to be really sure this is the right thing to put our money on. Can we experiment with this? We experimented. We see what the increment is. We say we get X percent more. Let's put let's go all in or it doesn't work. Let's try something else. And so I think that's a fun part to be in. And so the team sees value and the businesses value.

So that's that's what it is. So what we do have is, like I said, the machine learning models are at scale. The last that I remember from a publicly released document was we have around a billion different predictions made at any given point on a given day because we have all the personalization systems, all the dashes who deliver food, all the merchants who show up on the everything is personalized at scale. We have a machine learning platform that are set. So that's the kind of scale that we are talking about.

And so the reason is not to experiment ourselves and increment ourselves into irrelevance. It's about trying to understand truly what works and what doesn't.

Spatial analysis and routing

Very impressive. I am I see quite a few questions coming into the chat here. Let's see. Joseph, you have one on spatial analysis. Do you want to jump in? Yeah, I was just curious about like how, you know, you apply things like isochrones or nearest neighbor analysis to find like, all right, we have an order in this neighborhood, the restaurants here, like where is your nearest dasher and like kind of what that looks like at enterprise scale?

Yeah, so, yeah, we probably could talk to that, but the answer to the question is I probably can't point to just one particular special analysis we do or one particular thing we do. Isochrone is definitely one thing we do to try to understand how do we timebox it and because we pay for time and not for like the dashers are paid based on how much they spend with us and the time is money. So we do a lot of time related special analysis.

We don't analyze per se because we have a system. The system takes care of all the routing and optimization. Yes, we contribute to it. We try to figure out what how can we make better. But we don't spend a lot of time analyzing things like who's the nearest available driver. We more worry about how do we make the supply of a particular market more balanced, knowing what we know and how do we mobilize them better and how do we mobilize them faster. If we mobilize them better and if you have enough number of people ready to deliver, the system is smart enough to figure out who to pick for the next order. So that's not something we need to solve for. We need to solve for the second and third order.

Data standardization in the industry

Thank you.

Bill, I think you sometimes like me to ask the question, so if you want to ask it, jump in, but I see you wrote one would assume that DoorDash, Uber Eats and others in the space would need to consume the same type of data from places like McDonald's or Kroger's and local restaurants. Is there a standard for the way data is structured in this industry, the data like item descriptions, price or location?

So, unfortunately or fortunately, dusting has never been standardized. So as anybody who's worked in retail, any kind of retail knows that because all of this is manually created, if you imagine how a restaurant sets up a menu, it's a human being at a restaurant setting up a menu for their own restaurant. So there is no standard way that you can get it sorted out. Even the biggest chains don't agree on how the data is going to look like. So we do a lot of work in-house to come up with our own taxonomies. There's a lot of effort involved in how do we taxonomize this? How do we understand what is actually a starter? Is the side dish versus the main dish? What is the actual price point? We do a lot of cleaning up work on the backend.

I guess it can be solved by better data entry mechanisms, but none of the industries have a standard format in which it's shared and I don't think there has been a case for trying to standardize it because like I said at the end of the day, these are human beings at the end of the day who are running their own small mom-and-pop shops trying to set up their own business and it's very hard for them to add another friction point to tell them how to enter data in a particular way. So all they're trying to do is copy their menu from the real world into the digital world and there's only so much they can do with that.

Zombie retention and experiment attribution

This is a new term I hadn't heard of, Mark, zombie retention. Do you want to jump in and ask that?

Yeah, sure. I was, so I put a question in the chat about like, how do you, when you're, you mentioned earlier modeling or like running experiments on retention and I have kind of a twofold question, if that's all right, one was about like zombie retention, which is the case of somebody churns and like in a subscription model, you know, somebody churns, they don't necessarily come back, but when it's sort of like voluntary leaving and coming back like DoorDash, I imagine there's a case where somebody churns for like a long time and then makes a return. Like, I don't know, a year or six months or whatever later is the first part is like, how do you handle that if, if you do, and then the, the, the second was like, it's kind of relatedly, how do you handle like a problem of anonymous versus like logged in customers and keeping track of who's who, from like an experiment, like in an attribution level.

Sure. Talking about the people who get reactivated, we call them reactivated people, the fact that the way we look at reactive people is we understand that not everyone has the need to use the platform the same way that every other person. So some people are going to come back once in a while, go away, then open up the app again, come back again. So we do understand there is a need and potential for people to come that way. So we don't treat them any differently to how we treat others. What we do want to understand is why does it happen more? So we have a threshold for how much it can happen. If it happens beyond that level, then you want to understand why, or if it goes beyond that level, you want to understand why, but we understand that it is an acceptable threshold of people who will do that. And that's the marketing behavior. People are going to open up once a year or once in six months or so that they're going to do that. But if it gets anywhere outside that threshold is when we start looking into it.

And so we look into why, what happens, what market did we do something that somebody else do something. So that's what we look at.

In terms of the other question on logged in, this is logged out. Most of our experimentation is on logged in, you can't shop on a pad, like you can't buy without logging in. So more often than not, anything related to conversion related stuff is logged in. But if you're just about the app usage and things on the app feature, the logged out people mostly, unless they have some kind of other identifier that they can tie them to, they probably won't be considered in the export, because the experiment won't load for those people. If they're not logged in, it gets featured based on who they are. It's each, each person is in a bucket, so they get a lot of things that they need to see based on the logged in status.

Okay, cool. Thank you.

DoorDash vs Wolt data science workflows

I think Arsenis, you had another question in the chat. Want to jump in?

Sure. So yeah, this was just about your experience going from DoorDash to Walt. I'm kind of curious about the data science workflows in between them, whether you noticed any differences and you could, and if you could maybe talk about that a little, So in terms of the workflow, I think given we are similar industry, same, almost same company, similar industries, the approaches to solving the problem are the same. It's just that where the problem, where we got involved and how we started solving the problem was different. So like I said, the workflow at DoorDash is more around, we think about the problem, everyone comes together. I should say, if it's the right thing to solve, and then we go away trying to understand how do we solve it? How do you build for it? What do we do? So there's a different workflow, which is, there's a deep dive first. Then it goes into running a lot of smaller experiments. Then after the experiments are done is when we actually start building the feature, then we'll roll the feature out, et cetera, et cetera. At Walt, it's slightly different where we first build a feature. Then we say, okay, this is the right feature to build. Then we come back and see, did it work or not? And then we do a deep dive, why did it not work? What would you have done differently? So I think then it becomes something of, and there's the sunk cost fallacy with doing things like that, because once you spend time doing something, you probably want to always try to find out if there is some way we can make this work rather than saying if it's the right thing we should build. So that's, there's a difference in the workflows here, but other than that, we use similar systems, similar tooling. So not a lot is different. I would say in terms of how we get the data from where it needs to be to what it needs to be.

I hope that answers your question.

Yep. Would it be okay to ask a follow-up here? Yeah, of course. Yeah. So, so that's an interesting thing because, so in one instance, DoorDash you were talking about, it's more, you kind of start from the question and then build to the tool. Whereas it sounds like with Walt, it's going kind of the other way in the other direction. Like we're going to build these tools and all that, and then we're going to see how well they work to solve whatever question you have.

So my, my question around that is how, how do how do the teams differ in their approaches to building models and kind of answering questions? And which one would you say, well, you know, it's, I'm not going to ask like which one works better, but what would you say are the advantages and disadvantages of either from, from that perspective?

So I probably won't try to compare it too much. What I would say is the benefit of the DoorDash approach is not having to waste time and effort on things that don't work, but the loss of that effort is a lot of ideas that actually might not look good on paper might end up doing pretty good because we can never say that we know everything. So if you knew everything and we knew, figured out how things would work, we would not come up with amazing ideas. So I think there is pros and cons of both techniques. So trying to find that balance is what you're trying to do. Like Walt builds a lot of amazing, what we call bold, great products. We are really proud of what we build and really in that customer experience. So some of these things are not quantifiable in a way, so it would be hard to justify. So that would be the pro and con. The pros is not wasting time on things. The con would be not getting time to do actual things that it all doesn't look good on paper. So that's, those are the challenges that we're trying to face and seeing how we balance a bit of both.

Advice for small data science teams

So something I wanted to ask is, I know sometimes featured leaders might be from huge companies. So if people are from a smaller team, they might not right away see how some of it might relate. But I was wondering if you have any advice for somebody who's maybe on it, maybe the first data scientist or on a small team of data scientists for making an impact on the business.

So this is something I often get asked. I tell them if the work that we do as a team analytics team or data team or data science team, that's not direct. We don't know how it directly changes how the business works. It could be a smallest metric to the most important metric. Then it becomes more of a passion project. I've seen a lot of times where people come up with a great idea and then try to fit the idea into the solution rather than saying, what does the company need to go and then try to put it together. So I think that's a really important thing to keep in mind when you're trying to build a business. So that's my recommendation could be, it could be a data scientist or you did a sense either. Typically what I've seen work and that's when we get credibility and a seat at the table because you're actually helping you be part of running the business and not about just measuring the business. That's how I feel about that.

Typically what I've seen work and that's when we get credibility and a seat at the table because you're actually helping you be part of running the business and not about just measuring the business.

So that, when you mentioned before, like understanding what the business like objectives are or their OKRs, objectives, objectives, and so on. That's right. Yeah. So some businesses run with OKRs. I didn't want to generalize it because not every business runs that way. Some businesses run with OKRs. So if you have OKRs, then making sure that everything you work hits the, what are the capital OKRs, you call them a top OKR. If your business has three things that they want to achieve, which are the missions and are we tying up to the mission exactly, and can you quantify it? If it's just sales and revenue, it's all that we care about. we care about in that, whatever is the business, this key thing that it exists for, are we trying as a team trying to build something to help influence that is probably what is how I would measure the team success and also eventually like the function of it.

Definitely. AJ, I see you have a question here. Want to jump in?

Yes. So I had a question like in Doordash, maybe when you were working. So how do you identify default customers or fraud customers who try for a cash back and what techniques do you use? And do you flag them based on say a previous customer data and the trends that you have seen?

Yep. So the team, I probably can't talk to the exact techniques as such, but the

team does look into chargebacks and trying to find out fraudulent refunds and people exhibiting fraud behavior.

It's a combination of tooling. So we have standardized tools that can then flag this. So we don't directly do it. What we do is we feed the tool. It's a graph database. At the end of the day, we flag, we provide the features and the relationship entities that we need into this tool. And then it can then keep flagging for us. And there's a bit of manual review. Given the scale and size at which we operate, we can't possibly manually do all of this. So we come up with the rules.

And what the team works on is actively is, are these the right rules to have? And are we losing a lot of revenue from these rules? Or are we making up, so what is the true positive and true negative of these rules is what the team tries to understand. So instead of identifying consumers, we try to identify policies, come up with policies, which is people who look like this, do like this, maybe stop their transaction. So that's the policy that they would come up with. It's easy to input into a system. Like I said, so the system is smart. The system needs more and more policies. So we can come up with those policies based on what we see.

And then somebody will then make a claim or push us saying, this is being too restrictive. It's actually limiting us in this market. There are more customers who might look like it, but are not potentially fraud. Then we go revisit it. So the team spends a lot of time coming up with the policies rather than trying to come up with individual customers.

Tools and platforms across the organization

Would you be able to share a little bit with us about the different tools that you use across the organization and maybe how that differs by teams as well?

So very simple. I think it takes 10 seconds. So we use a lot of SQL at the end of the day. It's all about the data we have. And then there is a lot of the analytical work spent on time on trying to understand why something happened to spend on R and Python. So we have our own R and Python. Different people, people are very free to use whatever notebooks they want or work pages they want. So it's a combination of R and Python that tries to understand from the SQL data, what's happening.

When you start building models, we have our own, so we can build models. We can deploy models and deploy stuff. We have our own machine learning platform in-house, which can take a Python package and then make it into the end point. And then that can start feeding anybody who queries it with a recommendation output to visualize. We have our own, we have multiple visualization platforms, as you can imagine, as any big business does.

And the beauty is everybody who has at doordash.com or at volt.com email pretty much has access to almost half of all the images, all the dashboards can actually play with almost all the workbenches and also tweak the queries. So we've built it intentionally in a way where nobody has to feel limited because they don't work in the analytics team. Everybody can get a query as long as they know how to run a query and see what they need to see, and they can do it. They can't probably save it. They can't get notebooks and stuff like that, but they are not limited by getting some information and then working with it.

Open sourcing and the DoorDash engineering blog

I, um, I believe DoorDash has your own, their own blog to like open source work. And it was wondering, yeah, I was wondering a little bit about that and if it had always been that way where you're able to share the work that you do or how you made it that way.

So I think it's good on a multiple fronts. One front is for the team to talk for the great work they do. Um, and two, it also pushes them to be more, um, how do I put it? What we've always gotten from the blogs are people who interested in those kinds of things, come to us and say a bread is productive, can be better. So there's no better way to get feedback on what you do than by sharing it more and more widely rather than keeping it closed.

So there's no better way to get feedback on what you do than by sharing it more and more widely rather than keeping it closed.

So if you want to know more about our Sibyl platform, which is a machine learning platform, want to know more about our data science platform or how we run experimentation or an experimentation platform, which is also completely built from ground up, which is called Curie, you can go to Curie and write DoorDash and you can get a blog and that talks a lot about it. And we've had people reach out and provide product feature ideas and also recommendations and improvements on how we can experiment from that. And so that's the other thing. So one is to share, but the other one is also to learn from people whom we would not have had an opportunity to talk to otherwise. So that's how, that's what happened. I think once it becomes organic on its own, it's just, and more and more people want to contribute to it. It becomes a thing of its own.

Surprising insights from data: delivery timing across cultures

Alan, I see you had another question. Want to jump in here?

Yeah, I have to hop back over from, from checking out the blog.

I'm really curious if, if you can share basically a story of, of where you see, you saw something in the data, you think you have an explanation, you go and explore it, and the answer just turns out to be really different than what you expect, maybe just because people are unpredictable, like, you know, I'm curious what that process is like. And, and if there are those moments where you go like, wow, this is really surprising, or this is giving us some new insight into customers in a way that we just didn't expect, anything like that, that, that you can share.

It was more often than not. So I can talk about a very unique use case, right. And it's something I often bring up. So in the U S and all of you in the, in the most fear in the U S you'd realize that if something is late, you get pissed off. You're like, don't be late, be on time or be early. If a food is supposed to come at 35 minutes, you want it to be on 34 minutes, 36. It's fine, 36, but you don't have to be 40 minutes. You really hate it.

So what we try to do is all our algorithms or models are set to deliver things earlier than what we promised. We're trying to keep our promise or at least try to deliver it. That's that's inside. That's a common thing. What we don't realize is the world is a very diverse place and each culture is very unique. So when you try to build something similar, roll it out in Japan, we realized that people really were pissed off, not by lateness, but by early. So the earlier we came in, the more consumers are coming back.

We were quite surprised because, um, consumers are really pissed off by because they wanted us to be on time. A slightly later, but never early because of who these people are and how they were shopping. They were shopping for us and they were coming home and they didn't want to put it outside, but that's the kind of thing when you assume that something is true and then try to apply it 29, 30 countries, then you get caught out. And so then we start bringing in the lens of, are we doing the right thing by this country? And so going to that level of country level detail, then it's very interesting.

Hypothesis testing and bias in experimentation

Uh, James, I see you had a question you just put in the chat on hypothesis testing.

Hi, this is James. You, um, I had a quick question regarding hypothesis testing in the digital world. As we know, there are a lot of, uh, compliance standards to anonymize data. And, um, one of the crippling effect is you cannot see the cohorts correctly, whether they have bias or not. So having that said, how have you combated this issue and what are some metrics that we can use to perhaps, you know, build a better sampling size that is, you know, possibly stratified and more representative of a population.

Fair enough. Because like I mentioned, maybe I quickly ran through it because we only roll out experiments and experimental features to people who are logged in, we are able to identify who these people are in some way. Uh, and all our features, it doesn't matter. We don't know who their age or name or gender, any of this stuff. We just know who, what kind of behaviors they exhibit. And so we are able to identify them as an individual unit block. Uh, and then you're able to feel the experience. And so I'm sorry if I'm missing, misunderstanding the question. Uh, that's how we feel it when you want to remove bias and how we run them.

If it's say things like I, if it's like the, what do we call them? Uh, don't track me features or things like where you can't track there. Those kinds of audiences are naturally excluded from the experimental sample because we don't know anything about them, or we might get erroneous data because we can't track them properly. Uh, but in other ways we do have bias reduction techniques, or we have bias identification techniques, um, before the whole experiment is automated. And so that's how we do it. But if I've misunderstood your question or misrepresented it, please correct me. And I'm happy to find it.

Yeah, no, I mean, you hit the nail on the forehead. Um, in a given circumstance where you have deterministic, we identify the user. It's, it's a lot easier. Obviously you have behavioral metrics you have, um, to your point, demographics are able to build the proper cohorts and you're able to build the proper sample.

Um, having that said, um, in the digital world where, you know, advertising and experimentation is required and you're, you're often, you know, it's really difficult when you're in this, this vacuum where you're in the silo where you do not know exactly if, you know, these potential users are being targeted in, let's say a demographic region that is highly skewed towards a propensity to purchase at a much higher rate. And we have no idea, right? Um, so that's, that's where the, um, the difficulty that I'm kind of basing is, is there, is there advanced statistical statistical methods that we can do to combat these types of issues?

Um, I don't want to be the representative for this, but, um, what I can do, James, like maybe connect to people who spend a lot of time thinking and focusing on this on experimental research groups within our business. Um, I, if you're interested, do you want to talk more? I can definitely connect you to someone who can then help you with that. But, um, that's something maybe it'll help you with your, um, with your questioning, line of questioning.

Collaboration between data science and the business

I'm always really interested to hear about the collaboration between data science teams and the business as well. And I'm thinking about all these like experiments and predictions. How do you come up with the ideas for the experiments and what does that collaboration look like with the business?

Um, so it, like I said, the, we are driven by the business to actually do things. So the fundamental reason why we exist and what we do is pretty clear for us. The reason we exist as a team and what we do is to make sure that we are investing in the right things. We're putting money in the right places and we're going after the right target. So to know that, um, the only way to learn it is by experimenting.

The one thing I really love is how our CEO often talks about it is we read these experiments, we read these documents where people talk about things that they say, oh, we experimented this and we failed and we feel bad about it. And then his comment would be, if you didn't expect, that's the whole point of experiments. It's to fail and to learn things. If you didn't fail, you would never learn. So it's that culture is built in, in the business.

It's to fail and to learn things. If you didn't fail, you would never learn.

And so what happens is the business comes to us and says, we want to build this new, go to a new market or build this new product or build this new feature or whatever it is, um, let's do it in an incremental fashion and let's measure along the way and make sure if it's the right thing to do, uh, if it's useless, let's throw it out the window. If it's great, let's evolve it. And so that definitely happens. So that's what that, that different, that, that, that happens, right? Like I said, it can't, you can't clap with one hand. So it's not just us championing how we do things or evolving out to things. It's the business also wanting to do things in a better way. And so that's, that's where this comes from.

Tools for data presentation and decision-making

The question was, um, what role do Microsoft products? So Excel copilot power BI play in your data presentation and the review of your results by internal customers?

Um, so we do use, um, I don't think we have a lot of Microsoft products as such. Um, yes, obviously we do use a lot. There's nobody who does not use Excel, but we don't use a lot of the other things. So all our presentations are on Google slides. We do a lot of the document culture. So everything is written down. So anything that you need to make a decision on, be it an experimental feature or be it a business decision, everything is written down and somebody comments on the talk and says if it's the right thing to do and if it's the right. So it's all a lot of Google docs is what we work on.

Um, and we use, uh, Looker from Google, which is the visualization platform to share dashboards and stuff like that. But dashboards are best dashboards. You know, the decisions are made on a document, uh, be it investing money in a place. We're shutting down a feature or being that pickup feature. Everyone writes up an experimental doc, um, talk about what the feedback has been or what the output has been. And then people ask questions there. And then the decision is made there in the document on what that's what we do.

Dealing with ambiguous data signals

Um, so the question was, how do you and your team deal with ambiguous data signals where things are not certain?

Yeah. Um, I'd like to understand more by more on what those ambiguous data signals could be solved. Um, because it could mean anything.

Yeah. So like, I think there's two things. There's external ambiguity and internal. So external would be like, uh, you know, let's say credit scores are going down, but income is going up, right? Like, so I assume we're doing the internal, right. We've had a lot of situations where one data set has 80% coverage, right.

Fair enough. That makes sense. No worries. So internal, I can talk about the internal data points. We will absolutely make sure that we understand. So data quality and lineage is a massive thing for us. We spend a lot of time and effort on understanding how we track something, where it is tracked from, what is the coverage for it? Uh, we have monitors for it. So we know at any point in time, what is the quality of our data? Um, and because like I said, we're extremely obsessed with metrics. Uh, even if something moves by a couple of basis points, you know, immediately, if it's a moment due to us, due to the market or due to the data being wrong. And so if, if anything looks good to be true, the first question is data is wrong. So because that has been told so many times, that doesn't happen anytime anymore. It hasn't happened in a long, long time. So we have tight control over the data quality of everything that we fight through internet.

On external data, we think that we don't have control over example, market share data that we get, or app download data from download partners and stuff like that. The only way we provide credibility to it is by trying to triangulate it with something else, which is one other external data and also our in-house internal data, try to find a proxy for it and then seeing if that all makes sense. So we never trust anything until we can triangulate it. And we know for sure that makes sense.

So for example, the same external party would also be tracking us and providing a version of our data. And so we'd buy that too and say, we know what, what is true for us. We know what they are saying, and then apply that pinch of salt on the other external data and try to take it. So there's always going to be an element of uncertainty with that. So we don't make big decisions on external data, but we always, it's always good to understand. So we try to triangulate it. So that's what we do.

Accounting for model influence on outcomes

Um, I see George asked a question in the chat and there's a star there. So I'll ask it, but it was, how do you account for the influence of the models on the outcomes you see? For example, did the user purchase or sign up for DoorDash because of the recommendation or marketing?

So I'm probably not sure exactly. I understand if you're trying to ask me, what is the impact of every model that we build and do we ladder up to it? Uh, some models, yes, we do understand it. So if it's a personalization algorithm, it's a recommendation algorithm. We know, for example, we showed a restaurant to a person consumer and she clicked on it and she bought it. We know exactly bang on because we track it. Or we showed a delivery order to a delivery person. He picked it because it was most profitable for him. And then he delivered it. Bang on. We know exactly who that was. So there's not a need for a lot of attribution there. We know exactly who did what.

And therefore we try to understand if we do this model out, what is the impact to the business? In cases like marketing, where we make marketing spend decisions. Um, obviously we don't have that level of knowledge. So we do use aggregated models. So for example, we have market mix models that end up understanding. What is the decision? What is the overall impact of the business short term, long term from a particular marketing vision that we decide to spend more money on Facebook versus Google and stuff like that. So that's the level of attribution we can do. We don't tend to go too deep, not because we can't, but because we find that that level of granularity is not as beneficial as, um, given how fast we move and how sometimes we have to make irrational decisions in the world of marketing and branding, where not everything that you see is not what you can measure. So you just have to take something with a pinch of salt.

Career advice: being vocal about your work

So you're thinking about your own career journey. Uh, is there a piece of advice that you've either received from somebody or a piece of advice that you've given that you'd be open to sharing with us?

Um, so the one thing that I've, I've heard from, and so this is completely probably tangential to what we're talking about is if we don't talk about the work we do, nobody knows what it is. And so I've taken that. So a lot of times we do some amazing, this is something I tell my team, you do some amazing work, but you don't want to talk to anyone about it because you assume people automatically figure out about your work, come understand what it is, and then appreciate you for it. That doesn't happen. And so at scale, especially if you have a great idea, what we do believe in is you have to be vocal about your idea and talk about why that makes sense, what is the great project you've done, why, how is it impacting the business if we don't do it?

if we don't talk about the work we do, nobody knows what it is.

Um, and you just worry that nobody appreciates what you do or things don't happen for yourself that, um, passive, uh, credibility never happens. It needs to, you need to take an active foot in it. It's what I've been told. I was advised. I've taken it to heart and that's what I tell my team. So we create forums for the team. So the teams can come share their ideas, the things that they've done. We push our teams to be vocal about things that they find are wrong. If they see something, we ask them to say something and fix it. And so there is a lot of effort on, um, being vocal about the work and being in the data analytics space where everyone likes to be with their stuff. Not be bothered. Tell me what you need to do. I'll do and walk away. We push people to be a bit uncomfortable and actually speak out and make themselves heard. So I think that's something that I've learned and I pass on to my teams.

So one thing is we do create, we do make sure that the work that the team does is incredibly technical and complicated. So instead of boring the team with boring everyone about, we did the CB test and we saw this, it was like, no. So if you do this, you get X back and trying to translate that, translate math into English is what I do. And I try to teach the team, but how do you translate your math into English? So we teach people on the world that they do. So I do that and I try to help my team do that at scale, uh, which helps them. Then that transition helps us understand the business problems in the business this lens, and rather than understanding it completely from a technical or a data lens, which stops us from being limited by what we can do, and then opens up the possibility to what all we could potentially do and solve.

Community announcements and R4DS

Thank you so much. I love, I love hearing that advice and all the different career advice that's been shared in the hangouts. I know John Harmon, you mentioned you have to drop off in a second. So I did just want to call attention to what you put into the chat about the R4DS Slack community. Um, and feel free to just jump in if you're still here too.

Do you want to introduce it and are you still here? Sure. Yes. All right. Um, thank you. Um, I'm here for the, uh, R4DS online learning community. We, uh, have a Slack community where people can get help with all of their data science questions and we have a whole bunch of book clubs that read various, um, data science related books. And, uh, the, uh, nonprofit that we are part of announced last week, all of, like out of the blue that they are dissolving and we aren't going to be able to accept donations for a while after March 15th, so, um, we're doing a drive right now to try to do what we can and set us up to find our new home. Uh, and so if you can, I put the link in the chat. Um, we would love everyone's help. So thank you very much for letting me hop in.

Absolutely. Sorry for just throwing you on the spot. I saw you were about to jump off, so I wanted to take the opportunity, but I know a lot of people who joined the Hangouts find that Slack community so helpful. Um, so I just wanted to give a few minutes to that too. Thank you very much.

Miko, jump over to see if I have missed any questions in the chat. I know Isabella and our team had a question about how you keep up to date, um, with the latest trends. Isabella, want to jump in here?

Yeah, sure. Thanks. Sorry. I'm just turning on my camera. Um, thanks. Yeah. I was just wondering, uh, like with so many techniques and so many things going on, especially right now with AI and everything, like how do you keep up to date with like the latest models, techniques, methods, anything like that?

So the one thing I take as a true advantage of working with amazing people is everyone's trying to push themselves constantly. So people take an active interest in one particular topic and then they take it upon themselves to either run a forum or bring a topic or work with the academies. For example, I can talk about the experimentation topic. The extremely obsessive experimentation here. And there's only so much you can do with experimentation. So after a point, you need to understand how to improve it with better variable reduction techniques. How can you improve it? Better sampling techniques. So what the team does is they go away, work with, um, professors from academia who also are also searching on the topic, bring them in, help them talk to us about a particular area that everyone's solving and talk about. So what the team does is they take extreme interest in these topics. And so each person has a passion project. They work on it.

So causal inference is a massive thing here because everyone likes to know why something happens. We all live for that. And so there is a group of people who always, every fortnight bring an interesting causal inference topic, either themselves or from outside. And so you can then subscribe to it. And if you're interested in it, you can go. So the answer to it is the team takes an active effort and it's all community team runs it for their own benefit and others benefit. And I'm always a fly on the wall on most of these things. And that's how I learn about things that happen.

Book recommendations

That's fascinating. Thank you. Because on a related note, I know, um, we often share books with each other here in the hangouts. And I, and Libby had put together a whole list of book recommendations. Are there certain books that come to mind for you that you think we should check out?

Um, at the funny thing that I often recommend one book completely invisible women book is what I typically recommend. Um, and the reason I recommend invisible women is it's talks about, um, it talks about something that's real in our lives and uses data as an example of how we often miss the other 50% of people in our, in our world and how the world is not designed for them. And I often use it as an example of how we make wrong decisions for the majority or the wrong decision for the assumed majority, and therefore don't make that addition for the business. So that's a book I would recommend. So it's not a book. It's a book about women empowerment that everybody should read, but it's also about how data you can use data to actually make a case for making the world a better.

Community and collaboration within DoorDash and Wolt

Well, sure. I, I know you also, you had answered in Isabella's question about how everybody shares resources, um, and keeps up to date, but I would love to just learn a little bit more about how the team also helps each other solve. Problems like what is community look like within DoorDash and does that extend from DoorDash to Wolt as well?

Yeah. Yeah. So they're definitely, so the beauty is if you ask somebody a question, you don't get an one word answer. More often than not, you get, this is how I looked at it. This is how I solved it. These are the five things you can look at it. So you get the whole thing. So you don't feel like. So the people know when they ask for help, they will get help. And I think you've definitely built a great community of folks here and be it within DoorDash or DoorDash to Wolt. So if somebody from Wolt reaches a thing, they're trying to think of building this thing, what do you do? And you won't get a one word or a one sentence answer. You'd pretty much get like a six paragraph answer about everything. Somebody's filled all the documentary.

So we, we love people for that flag that they take an effort and take an interest in actually trying to help. Does it happen instantly? Maybe not, but when it does happen, you do get all the help you need. So that's been good. And so people do fall back on each other to peer reviews, to review their work, to have critics of the things that they do, or also get a third set of eyes that look into it. So that's always been the case. And so there is no sense of who can review whose work and stuff like that. So, which has also been great.

Keeping momentum across many projects

That's great. I think we have time for one more question and, Fares, I see you just asked one in the chat, want to jump in? Well, sorry, my question is a bit general, but I was just curious, when you're so busy with so many projects, how do you keep up with each of them with their separate, sort of universe that's going on for each of these projects to keep the momentum going?

So one thing we do is we have, we have artifacts that people can share. People do share once every six to eight weeks, what are they working on? So, like I said, this is a forum for every team to come and every small team of eight to 10 people to come share what they're working on. So that really helps anyone else who really wants to take an interest in it to come read. As leaders in the business, we all read almost all the teams once in six to eight weeks, sort of things that they work on and then get to ask questions and engage in it. So our approach to that is to help people put together as artifacts. And then those artifacts help us keep interested and involved in what is happening, because we don't really need to know on a week to week or even a daily basis, some of these topics by fortnightly or once every two or three fortnights is good enough timeframe for us to actually answer these questions.

Thank you. Thank you so much, Shira, for taking the time to join us today and sharing your insights and experience. And thank you all for the great questions. I saw Abigail shared in the chat here, and I just wanted to call out to another community event. It doesn't have anything to do with the Data Science Hangout, but I've seen a lot about it on LinkedIn. But tonight is the Data Mishaps Night. And so it sounds like a really fun time. They feature a lineup of data mistake stories with a focus on the human aspect of data work and lessons learned. So I think Isabella just re-shared that in there, too. So if anyone wants to check that out, it sounds like a fun time. But thank you all for joining. Have a great rest of the day.