image: thumbnail.jpg
Dara Straussman @ Stripe | Partnership-model when forecasting | Data Science Hangout
We were able to get a glimpse into their financial modeling methodology at Stripe after a question from Alan at 12:59
Question:
How do you get buy-in from business stakeholders who historically would have been setting targets (like salespeople and their leaders) that the forecasting you're doing seems like a realistic enough picture of the world?
Answer:
It's an ongoing evolution for how we do this interaction because it's very important to get the input from stakeholders.
There might be a big sales deal coming down the pipeline that no statistical model will know about. There might be some product launch plans that we're just not going to be able to learn from history. So we find that interaction very important.
When we started doing the – we call it the statistical forecasting and the adjustments — it was all done in spreadsheets by finance folks. We did really have to prove our value and show how a statistical model will actually help.
So our current system:
1. We have the statistical model that's all done by data science.
2. We do backtesting and reports to be able to build confidence in each level of the hierarchy so that people can understand where do they expect accuracy, where do they expect less accuracy.
3. We have a whole process with the finance and strategy team, where they make adjustments to the forecast and are pretty clear about where they are doing adjustments and how it's affecting the total.
4. Then we track both - the statistical versus the adjusted forecast. What we've learned over time is that making adjustments in certain ways or certain places is very important and very effective. We sort of encouraged the finance and strategy team to focus their efforts there. Whereas in other places, the statistical model is quite accurate, and it's best to spend effort elsewhere.
So that's the baseline of how we think about it– again, ever evolving.
We also have an embedded model so we sit with finance and strategy. We work really closely with them and really understand their problems.
It's very much a partnership-type model rather than a "throw a model over the wall" type model.
► Subscribe to Our Channel Here: https://bit.ly/2TzgcOu
Follow Us Here:
Website: https://www.posit.co
LinkedIn: https://www.linkedin.com/company/posit-software
Twitter: https://twitter.com/posit_pbc
To join future data science hangouts, add to your calendar here: pos.it/dsh (All are welcome! We'd love to see you!)
Transcript#
This transcript was generated automatically and may contain errors.
Welcome to the Data Science Hangout. Hope everybody's having a great week. I'm Rachel. If we haven't met yet, let us know if it's your first time joining in the chat so we can all welcome you in.
This is our open space to chat about data science leadership, questions you're facing, and getting to hear about what's going on in the world of data across different industries. Every week we feature a different data science leader as my co-host here to help lead our discussion and answer questions from you all, and together we're all dedicated to making this a welcoming environment for everybody. So we love to hear from everyone, no matter your level of experience or background or area of work.
It is totally okay to just listen in. There's always three ways you can ask questions too, and also provide your own perspective. So one, you can always jump in by raising your hand on Zoom. Two, you could put questions into the Zoom chat. I know you've all heard me say this a lot of times now, but feel free to put a little star next to your question in the Zoom chat if you wanted me to read it out loud, if you're in a coffee shop or something, or if your dog's barking. But we also have a Slido link where you can ask questions anonymously. And I think our team just shared that in the chat right now. We'll reshare it after for people who have joined after the first few minutes too.
We do share the recordings of each session up to the pause at YouTube. So you can always go back and rewatch or share with a friend or coworker. If you hear anything loud in the background, there's some snow falling off this side of the building right now. Hopefully you won't hear that too much. But thank you so much, Dara, for joining us as the co-host today. Dara is a data science manager at Stripe, and I'd love to have you introduce yourself and maybe share a little bit about your role and also something you like to do outside of work.
Sure, sure. So thank you for the intro. My name's Dara. I, as Rachel said, I'm a data science manager at Stripe currently. I've been at Stripe for about five years now. My current role is leading a team around finance and operations data science. So we are very into forecasting. We do a lot of work on hierarchical forecasting, financial forecasting at Stripe, spend a lot of effort there. We think a lot about metrics and overall company metrics and analytics and how we can understand our business performance. And then we work in the operations space as well. So we are working on some various machine learning projects there as well as capacity planning and forecasting on that side too. So happy to talk about, you know, any of that work in more detail.
Let's see, one thing I like to do outside of work, I used to be kind of an endurance sports person, you know, long distance triathlons and running, but I now have a 15 month old daughter who keeps me pretty busy and spend most of my time just parenting.
Forecasting at Stripe
Well, thank you so much for joining us as we kind of wait for questions to come in from everybody in the audience here. I'm curious if you could kind of give us a little insight into some of the problems that your team is trying to help solve or things that you're working on. So you can put us in your, your frame of reference.
Cool. Yeah. Let me talk about a couple of things. So, um, at Stripe, we're interested in forecasting as many businesses are in understanding how much payment volume are we going to have next year? How many active users will we have? What will our uptime be? There's a number of forecasting problems that come, uh, they come into play that, you know, are a business advantage. If we are good at them, we can set the budget, we can set our targets, we can direct our efforts appropriately. Um, and so my team is, spends a lot of time thinking about how we can produce the best forecast for Stripe.
Um, I really, as a side note, I like forecasting as a data science area, because it's not one that you always learn about in a machine learning course or spend a lot of time thinking about in, um, your, you know, more formal education. I didn't at least. Um, but now in, you know, sort of on the business side, it's an extremely practical, uh, business problem to be able to solve at the same time. It's very interesting statistically. And there's a lot of work that we do, um, to, you know, sort of experiment and find the best methods for doing forecasting.
Um, so we do what's called hierarchical forecasting. So this means that we want to have a forecast for Stripe overall. We want to know, you know, what's our total payment volume going to be for the whole company. But we also want to know what is it going to be for each region? What is it going to be for each business type? What's it going to be for each cohort? There's all these dimensions that flow down. So you can kind of think of this as a hierarchy, right? Where all of the dimensions, um, then need to add up to the total, right? We have to have a coherent hierarchy. That's how we talk about it.
Uh, and so we use, uh, this method where we can actually choose any forecasting method we want. Um, for each node of the hierarchy, you know, we use a lot of classical forecasting methods like ARIMA and exponential smoothing, um, we also experiment with other types of methods. Um, and we use those to make estimates and then we apply sort of a meta model on top of it, hierarchical reconciliation, um, to basically tweak each forecast as little as possible so that they all still add up to the total forecast. And that then, you know, we apply, um, some additional adjustments on top of that, and that becomes the budget for the company.
Target setting and non-financial metrics
Thank you. I, I see Mark had a follow-up question on that. Mark, do you want to jump in and ask that?
Yeah, sure. Let me turn my video. Um, you know, I, I'm asking, um, just curious about your, um, and you kind of went into a little bit of detail there, but, uh, curious about if you're able to expand on the target setting methodology, um, and if you dip into any, I guess, like non-financial, um, targets and like, I guess, different level setting, um, I'm actually literally have it open right now, working on our like next fiscal year's, um, targets for the hospital system I worked for. Um, so yeah, just really curious to get like your perspective on some of the non-financial, um, metrics that you guys keep track of and if and how you do. Um, set for those.
Yeah, we do. We're actually building a platform to do general hierarchical forecasting for any metric at Stripe. And so we've expanded this. We do some sales planning right now, which is a little bit financial. Um, we do it for our active users and we're using, looking at things like, uh, like uptime as well, like how, how much, um, should we expect, you know, Stripe to be live and yeah, so the, we generally find that the approaches, oh, and we also do a lot in ops too, so capacity planning for our support volume, support tickets. Um, we've found that the methods are pretty similar across, you know, all of these types of problems is there's a lot of nuance in terms of, uh, making sure that you capture the data correctly and capture, actually solve the business problem, right? You can't just sort of throw a general forecasting method at something, but, um, but we have found in general that the, the hierarchical format works really well in a business setting and that, you know, you often care about some more detailed dimensions and then you also care about adding up to the whole.
Exploring large language models
Crystal, I see you asked a question in the chat too. Do you want to jump in?
And maybe I miss, I can ask that, that one too. And if you want to add any context, feel free, but the question was any exploration of chat GPT to see, uh, to identify any possible or meaningful use.
It's funny you ask because I think it's a really good question and I think it's to identify any possible or meaningful use. It's funny you ask. Um, yes, we are playing with it. Um, we actually were playing with it before it was cool. I have to say some went back when it was just GPT three before, um, some of the hype this fall, uh, we are, uh, experimenting with it in our support flow. So to be able to help agents answer questions, um, more accurately and much more quickly, uh, for our users. So we are still early days, but we're pretty enthusiastic. Um, we've done a lot of NLP work generally on the upside. So we classify our issues using BERT, which is sort of a previous generation natural language processing model. Um, and you know, that allows us to know what kind of, uh, questions are getting asked and what topics are support issues are about, which we then use to direct a lot of internal efforts. Like where should we be, you know, making updates or, uh, bug fixes in our product based on what the support tickets are about.
Career path: from computational immunology to fintech
Thank you. I want to make sure I get over to some of the anonymous questions too. And just remind people, if you want to use Slido, you can use that link in the chat, but somebody had asked earlier, I would love to learn Dara, how you have found the transition from computational immunology to finance and operations.
Yeah. Yeah. I'll try to try to not blather on for too long about this. Um, you know, everyone, a lot of data scientists have interesting or unusual career paths. I'm no exception here. So I was, I did my PhD in computational immunology. Um, it actually started out as a PhD in pure immunology. So I was a wet lab biologist, you know, pipetting and doing experiments. Um, I realized pretty quickly in my PhD, um, I took an R class and a Python class. And, um, then I took, you know, more CS courses and I did more statistics and machine learning. Um, you know, realized pretty quickly that that was a direction that would enable me to be a better biologist. And also just an area of a lot of interest for me.
Um, you know, I was lucky enough to be able to do a ton of coursework and really direct my research that direction. So by the time I finished my PhD, I was doing fully computational work. Um, when I finished up, I thought, what do I want to do next? Um, I, uh, really enjoyed academia, had a great time there, thought very seriously about academic postdocs. Um, and then also thought, you know, I really liked this data side of what I've been doing. I really liked the statistics and the ML and all the programming that I've been doing. Um, and so I looked at data science positions as well. So I applied to both. Um, and in the end I, uh, decided to take a data science position because I felt like it was much more different than what I had done before and that I would learn more in that position just because, um, it would really force me to, uh, continue to build the skills that I had started to develop in my PhD and put me into an, you know, entirely new problem space that I hadn't been before.
Um, so that's what I did. I went first to a small health insurance startup called collective health. I was there for two years. Um, a great thing about going to a startup is you learn a lot of data engineering. And so I, you know, spent a lot of time building pipelines and really getting to know, uh, you know, sort of bolstering how you actually write code in a, uh, environment with other people, right. When your PhD, it's very much just sort of you and your notebook. And this was now actually having to, uh, write in a more production environment. Um, so learned a lot there and I did health insurance thinking that it would be somewhat related to my PhD work, which was on HIV and the immune system. Um, it turns out not really that much, that much, uh, overlap between those two subject areas.
Um, but I really loved the data science side of things and I really loved having impact in business and being able to see the results of my work more directly. Um, so after a couple of years, I decided to move over to Stripe. Um, Stripe was very appealing because it was, um, it was still, it was big enough to have invested in data infrastructure. There was a ton of data available. Um, you know, I was going to be able to do more statistical and ML type work, but it was still small enough that there were a lot of unsolved problems, which is actually, you know, five years later, still true with Stripe today. Um, and so that's what made me decide to, that I, you know, was more interested in the data problems rather than focused on a particular domain.
Um, it has turned out that financial payments, um, are a complex system, just like the immune system is a complex system. So even though I do not use my domain knowledge for my PhD that much, I still use a lot of the skills that I learned. I still use, you know, a lot of the, um, how to present, how to do research, how to think about problems, how to break down complex problems, um, all the time. And so, you know, really have found that, um, if you think about it the right way, you know, skills are pretty transferable.
And so, you know, really have found that, um, if you think about it the right way, you know, skills are pretty transferable.
Getting stakeholder buy-in for forecasting
Absolutely. Um, I, I'm starting to see a bunch of other questions coming in too. And, uh, Alan, I saw you had asked one earlier. Do you want to jump in?
Sure. Hi, Dara. Um, I I'm really interested in the, in part of the forecasting where if you have to reconcile like SME input from people who like historically would have been setting targets like salespeople and their leaders, and I don't know what the organization is, how it sort of maps to that, but what's, what's the process sort of buy-in from them that the forecasting you're doing seems like a realistic enough picture of the world that they buy it and that they'll go sort of, you know, work along it. Um, or if their input gets incorporated earlier in the process somewhere. So they're brought along kind of from the beginning in a way, just curious how that kind of gets negotiated out, um, organization.
Super interesting question. Yeah. And it's, um, it's ongoing sort of evolution for how we do this interaction because it's, um, it's very important to get the input from stakeholders, right? There might be a big sales deal coming down the pipeline that no statistical model will know about, right? There might be some big product launch plans that, um, you know, that we're just not going to be able to learn from history. And so, um, we find that interaction very important. And when we started doing this too, when we started doing the, you know, we call it the statistical forecasting and the adjustments, you know, it was all done in spreadsheets by finance folks. And we did really have to prove our value of, you know, how does, um, a statistical model actually help.
Um, so our current system is we have, you know, the statistical model. We actually do, um, you know, that's all done by data science. Um, we do backtesting and reports to be able to, you know, build confidence in each level of the hierarchy so that people can understand where do they expect accuracy, where do they expect less accuracy, um, and then we have a whole process with the finance and strategy team where they, um, make adjustment to the forecast and, you know, are pretty clear about where they are doing adjustments and how, uh, how it's affecting the total. And then we track both the statistical versus the adjusted forecast. And what we've learned over time is that making adjustments in certain ways or certain places is very important and very effective. Um, and we sort of encouraged the finance and strategy team to focus their
efforts there, whereas, you know, in other places, the statistical model is quite accurate and it's best to spend effort elsewhere.
Um, so that's, uh, that's the baseline of how we think about it. Um, you know, again, ever evolving, um, but also we also have an embedded model, right? So we sit with finance and strategy. We work really closely with them. Um, we really understand their problems. It's very much a like partnership type model rather than a, you know, throw a model over the wall type model.
It's very much a like partnership type model rather than a, you know, throw a model over the wall type model.
Yeah, it does. It's really, um, and it sounds like for having been sort of driving it for relatively small handful of years, it sounds really mature and really, um, like, uh, uh, comprehensive in a, in a bunch of ways that I think are really impressive. So yeah, it's cool.
Thank you. It's, uh, we still have a lot more that we would like to do for sure. There's a whole literature about adjustments and the best way to do that. And we applied some of those things, but there's so much more that I think we could do to really maximize our effort.
Thank you. I know you just said you sit with the finance and strategy team. So I was curious, is there like a general data science team for the whole company or is there data science team, like pockets within each group?
Yep. So general data science team, and I'm using sit with, to be clear, I'm using sit with someone here. We used to sit with them when we were fully in office and now it's, um, you know, sit with them on zoom, but, um, centralized data science team. So we have a hybrid embedded model. Um, data scientists has a partner team that they work really closely with, right? So I'm on the business side with the finance and operations teams. Others might work with product or engineering teams. Um, and we really, you know, act like members of that team, get to know their problems, get to under, you know, attend their meetings, their offsides, understand what's going on with them and do joint planning together. And so we like to think it's, you know, best of both worlds, making sure we are solving real business problems. But the centralized data science team keeps us pushing the envelope technologically and innovatively as well.
Getting stakeholder buy-in for statistical models
Yeah, that's really cool. Uh, deep shy. So you had a question over on Slido. Do you want to ask that here?
Sure. I, um, the druid cause Alan's question was slightly similar, but I'll ask, um, slightly different way. So you do get like finance and stakeholder buy-in, but sometimes if you, when you just recently have started statistical finance models, what happens when you statistical finance models, what happens if it deflates what the previous finance predictions were? And have you had such a scenario? Cause those buy-ins are like so difficult and I've had set scenarios as well. So how do you get a buy-in there?
Yeah. Um, it's a great question. As much as you can bring in confidence intervals or backtesting, you know, statistics and show here's what we expect. Um, you know, those of course are only as good as your input data. And that can be challenging when you're first starting out, but that is, you know, finance. Folks are very quantitative, right? They're very data oriented. They want to know how, at least, you know, our, our finance folks are. Um, so that helps, uh, there's, you know, in our experience, it did take some time just to build trust. And so it was making sure that we had all the very detailed tracking in place. And so it wasn't saying, Hey, we're going to replace your model. It was saying, let's run them in parallel. Right. And let's just sort of track, you know, the statistical model, the finance model, maybe some combination of them. We want to do an adjusted model. Um, and let's, you know, look for six months at how those go and see, um, you know, let's just sort of be objective about it if we can, and that, um, that ultimately was, you know, very helpful for us in building some trust and confidence, but it is, it is, you know, a relationship that just, it does take some time to build.
Sharing forecast results
Definitely. Thanks Tara. Thank you all for all the great questions too. I see, um, one on Slido was with your forecasting, how do you share results to those who make decisions? Is it through Excel, PowerPoint? Do you ever use shiny or SharePoint, Power BI?
Ah, I would love it. If we had shiny, we don't have it set up. Suddenly it's striped. So we have our own internal rolled, um, forecasting site, not forecasting dashboarding tool. That is a SQL based and allows us to, um, sort of build, uh, pretty nice dashboards that way on our data. So we use a lot of that. Um, we will, you know, some, we, uh, we'll sometimes, you know, make sort of bespoke plots in R or Python and, um, share those. We do a lot of PowerPoint or, you know, Google slides presentations as well. Um, and sometimes we, you know, when we're working directly with FNS, there certainly are sort of queries and spreadsheets and, you know, that we try to speak their language where we can. So while we might operate in more of a SQL land, they might be operating of a spreadsheet land. And so we try to interface between those things.
IC versus management
Um, Abigail, I love your question into the, in the chat. I know we get to hear a lot of different perspectives on this, this one and love to have you ask that here.
Oh, I don't know if we can hear you yet, so I can ask it and feel free to jump in after if you want. But the question was, how do you feel about being a manager versus being in a purely SME individual contributor role? Do you still get to code?
Um, these days, not as much when I, so I started as an IC at Stripe and, um, I transitioned about a year into my tenure. And when I first transitioned, you know, managing a small team, I still did about 50% coding. Um, and now I, you know, manage usually between eight to 10 folks. So I do, you know, more pure management stuff. Um, I think, you know, I've really enjoyed the transition. I've find that I think, you know, the key question to ask yourself, if you're thinking about IC versus management is, um, how motivated you will be by enabling other folks to do work versus, um, how, how motivated are you by, you know, solving problems and shipping work, uh, yourself. And most people would be, you know, motivated by both of those things, but it's really sort of trying to figure out for yourself, um, where you want to prioritize spending your time and how helpful, um, you know, each of those roles might be to you in achieving that goal. That's my short answer. I'm happy, happy to go on more. These are great questions.
Model sophistication and forecasting approach
Thank you. Um, Eugene, I see you had a question in the chat. Do you want to jump in next?
Sure. It's a, um, my question is just about the sophistication of modeling. So you, you seem to discuss quite a few rather sophisticated models in your forecasting, uh, as you gave the, your short sort of summary of, of what you're using. And I'm just curious, has the increase in, I assume you started simple, right? So has the increase in sophistication actually significantly changed, uh, your results or your outcomes, uh, as you've, you know, as the models have evolved, yeah, this is, um, it's a good question.
So our forecast, I think the way that we do the hierarchical reconciliation is very important. And, um, you know, that has certainly increased in sophistication over time, underlying models themselves. We actually don't use anything most of the time. Um, they're not super fancy forecasting models. We're not doing, um, you know, some of the things that are winning the M5 forecasting competition these days. It's, you know, sort of bread and butter, um, ARIMA and exponential smoothing, um, for the most part, we, you know, we'll, we'll play with other models too. Um, and we have found that for our particular business problem, that works really well because we have a decent amount of history. We have, you know, very important to capture seasonality. We're doing monthly forecasts. So, uh, if we are able to capture what we need to capture for those models. So the choice of, uh, the choice of forecasting model is so dependent on the problem that you have and the, um, you know, there, there's so many things to consider about your input data. I think if we had, um, you know, a lot of external regressors that we were trying to model a lot of sort of parallel processes happening, um, you know, then probably different kinds of models, potentially more of the deep learning forecasting models that have come into play recently, uh, might be really helpful to us as well. But, you know, for the most part, um, you know, a lot of, a lot of bread and butter forecasting models work well for us.
Handling unforeseen factors and forecast accuracy
Thank you. And Trevor, I know you had a somewhat similar question, so I just wanted to see if that addressed your question too, or if you want to add on to that.
Yeah. Hi, Rachel. It is indeed a connected question, you know, because the, so most forecasting models look at historical data to predict the future, right? So that is assuming that the past is the future, right? And of course we understand that can change, right? So for example, when COVID came along, it disrupted many models, right? Nobody could have forecasted that COVID would have had the effect that it had on businesses, right? So I'm just wondering, um, to what extent, Dara, there are factors that would not have been, been in your models that would have affected maybe the accuracy of your models, that's one. And two, we use historical data to make predictions at the end of the day. Let's say, let's say you're forecasting sales, right? And you'd have forecasted it to be a certain amount, right? Based on historical data. But then when reality happens at the end of a period, you get the actual numbers. So I want to know two things, really. How close are you typically to your forecast and how do you manage factors that weren't considered in the model in the first place because of, you know, you weren't thinking about those. Um, I hope I'm clear.
Yeah. Sort of some fundamental problems of forecasting in a business setting. We do include external regressors. And, um, now of course, like we didn't, we didn't predict COVID when I look back now, like forecasting was so easy pre COVID and now post COVID it's, you know, a whole different ball game. Um, so now, you know, we do include regressors that try to sort of, um, account for the past COVID changes in our data. Um, what we don't want is to, you know, continue forecasting more COVID in the future. So we do have some sort of external things that we use that help us to predict that. We also do factor in macroeconomic variables on top of our statistical forecast that, you know, Again, you still have to make your own predictions and sort of decide as a company, what you think might happen. Uh, we do sometimes do scenario based planning as well, you know, baseline and here's sort of the upside or the downside scenario to try to give the business some framework for how to think about possibilities.
Um, Accuracy and, you know, how, what do we do when the forecast is off? So with the hierarchical forecast, you know, the general pattern is that we are not that accurate at the lowest levels of the hierarchy generally, you know, for forecasting, um, one cohort, one region, one business type that gets to be a pretty small set of data. We don't have as much history, um, or there may be erratic patterns in it. And so we don't have as much accuracy, uh, down there, but then when you add up, um, to the total, you know, you have a much more robust data set and we tend to be more accurate at the higher levels of the forecast. Um, we do a lot of tracking, we create a lot of dashboards. We try to be really clear with, you know, where we're doing well and where we're not, um, we do re-forecast throughout the year to, um, be able to sort of update our best guess on where we think we're going to land given more data has come in throughout the year. Um, so that is helpful. And then we just sit side by side with finance and strategy and do the analysis and try to understand, you know, when the forecast was off, what are the segments in which it was off? Can we identify the root cause for that? And how can we improve our forecast model for the next iteration? So no, no magic bullet certainly, but you know, a lot of continual efforts to, you know, try to improve over time.
Support ticket forecasting and metrics
Thank you. Thank you very much. And I meant to say this earlier too. I know that I get to know everybody when I see people every week, but when you ask your question, feel free to introduce yourself again. I know maybe we haven't done introductions in a long time too, so feel free to throw that in there as well, if you want to, don't have to. Um, but Nico, I saw you had asked a question in the chat a bit earlier. Do you want to ask that one next?
Yeah, sure. Thank you very much. So I'm going with the introduction first. I am, I work, um, in a small data science team, uh, JP Morgan. Um, I was interested, you mentioned Dara, thank you so much for being here. Um, you talk about that you did before some kind of, um, um, analysis on tickets on incidents. I would like to know if you can mention some kind of the, maybe the metrics that you track, and if so, if you also do some kind of statistical process control to those.
You said support tickets? That's what you're asking about? Yeah.
Yeah. Um, yeah, so we do, we do the forecasting for the support tickets and we also do some ML modeling, uh, around improving our support processes. So I talked about our issue classification model, you know, we're working with GPT now too. Um, you know, some of the things we track, uh, total number of tickets, how long does it take to solve a ticket on average? Um, you know, what is our, our overall, like how many tickets do we get to within SLA, whatever we set that at, um, we are very focused on user experience. So we track CSAT, you know, customer CSAT and what's, uh, what are we hearing from users in terms of their, um, their overall satisfaction? You know, we look a lot to it, um, which tickets, you know, by some metrics that we have, which tickets do we think we did really great on user experience? And which ones do we think we did really terribly? And, you know, try to sort of, um, expand the first bucket and shrink the second one. Um, so there's a lot more to it, but those are some of the, some of the things that we think about. Uh, we don't exactly do process controls. We do help with capacity planning. Um, so with our forecast, we feed into, um, how much vendor, uh, sort of capacity we need to order. We have, you know, a global network of vendors that, uh, answer our support tickets. Um, so we, we work with them too, but I think there's actually a lot more room for even more data science work there.
Awesome. Thank you. Um, Adrian, I, I think you had a question a little bit earlier. We didn't get to yet too. Do you want to jump in next?
Yeah. Hey, thanks. Um, I'm an analytics manager here at Wayfair. I, um, I asked a question earlier and it might be a little bit untimely now, but I'll go back. Um, you were talking about the forecasting and specifically you're talking about how you integrate it or you allow it to be influenced by the SMEs near the end of the process. And I'm curious, or actually straight up adjusted by SMEs at the end of the process.
Um, I'm curious how much front loading that SME adjustment might take place like financial growth forecast projections or, um, uh, marketing, uh, budgets, things like this, how much of that actually takes place at the beginning and how much of that initial forecasting is really just if the past repeats itself, here's what would happen.
And if there's not a perfect answer, because you said there is a data science that sort of does that piece, maybe do you have a favorite infrastructure or framework or resource to see how these get navigated?
So the question is about, you know, how much does sort of the finance team make their own projections that, um, they might add to our forecast versus how much is it sort of after the fact, once the forecast is in place?
Yeah, yeah, exactly. I've seen, I've seen both of these sort of cases happen where finance has a 33% growth and that's where we're going. So you really better track on that because it's sort of a self-fulfilling prophecy, but I've also seen it where we start with the vacuum. And I'm curious what your experience has been. I'm curious if there's a best way.
Gotcha. Yeah. So I think our approach has typically been to divide the forecast into segments. Um, so there are, you know, I alluded to this earlier. There are places where the statistical models do really well, right? We have enough history. Um, there's not a lot of sort of external factors that influence that. It's sort of a continuous time series that we can forecast pretty well.
Um, and so we, uh, sort of yeah, explain that to the finance and strategy team and work with them and, you know, show them the statistics on it. Um, and then we find that there are other places where, um, that kind of financial modeling is very fruitful and it's very helpful to, um, adjusting the forecast, right, where there are things that the statistical model will do less well, or might not know about, or, um, you know, really could use some adjustment and that's how, um, in those places, you know, we really, um, open it up and encourage adjustments actually think we could have even more process control. Uh, we were just talking about controls around this and, you know, around how we do the adjustment process. That's something I'd like us to continue to work on.
Um, but that's generally how it works. So the finance team does do some modeling and does have their, um, sort of priors, if you will, that they're, uh, adjusting the forecast by, uh, but we, you know, we work together to make sure that they are spending their effort in the right ways when they do this.
Incorporating macroeconomic factors
Thank you. You, I'm sorry. I didn't notice your hand on zoom before. Do you want to jump in? Sure. Uh, hi, there are, uh, you'll call here. So first, uh, thank you for your time. Thank you for all the insights. You'll chart a similar background here. I'm a data science professional. I work in a finance company. And before that you'll, you'll work in computational immunology. And for me, I work in biochemistry. So I'm a similar pathway here.
So I have one question. So because we both know that for forecasting models in the finance field, one of the challenges is to incorporate microeconomic factors such as GDP, unemployment rate, inflation, all these types of things. So I'm wondering for all the forecasting models you mentioned, do you do either of them using microeconomic variables? And if, uh, if it does, what do you think is the best practice to incorporate such variables?
Yep. Yep. Great question. Um, yeah, this is very important to us and, you know, with, um, 2022, for example, became even more important for us to understand the effect of the macroeconomy on our forecast. Um, so we do, we currently have this as sort of a second layer of modeling. Um, that's, uh, you know, sort of, um, an overlay that we put on our statistical model where we say, um, you know, we do our best to learn, uh, as much as we can from the past about how the, you know, things like GDP might affect our forecast and, um, apply those, uh, as a secondary model to our model.
Um, one thing that's worked really well is having scenarios here, right? Because, uh, you know, we, we can make our best predictions, but in the end, it's still predictions with these macroeconomic factors, right? It's not as if incorporating them magically makes your forecast work. You still have to, uh, decide what the future is going to look like. And so having, um, you know, different scenarios, sort of high, medium, low, or however, we've set it up different ways, different cycles to give the business some indication of, you know, we, we can't predict the future either. We don't know what's going to happen with the macroeconomy, but, um, this, at least allows them to say, okay, if, you know, if, um, something goes well, here's what we might expect to happen. If we, you know, want to plan to a sort of worst case scenario, here's how we might do that. And you can imagine in, you know, different sort of business decision-making scenarios, they might want to use different ones.
It's not as if incorporating them magically makes your forecast work. You still have to, uh, decide what the future is going to look like.
Mentorship and learning culture
Thank you. And Marlene, I see your hands raised too. Oh, sorry. Is it there? Yeah, we can hear you. Yeah. Um, so, um, hi, long time caller first time listener. No, I'm just kidding. Dara, thanks for presenting this morning. Um, sorry, the coffee's just kicked in gang. So I'm all over the place.
Um, I, you know, I, I've shared on numerous calls before that I'm just kind of an army of one in my company. We don't have a formal data science team. We're far from the machine. Area of, of tech. And, um, right now my focus is just building a culture that understands that there's a need for data science and giving some practical uses for automation and, um, better analytics.
So my question to you is, it's more around mentorship. I wanted to know if you guys had any kind of culture that promotes a mentorship. Um, I know that you guys have, you know, a lot of, you know, culture that promotes a mentorship. Um, do you say, um, have a senior data scientist that takes a junior under their wing or even like a fledgling data scientist, right? Just somebody who's got some great analytic prowess that somehow gets built, come into the fold. I mean, I don't know, take that question where you want to go, but it's around mentorship essentially. Thanks, Dara.
Totally. Yeah. Thank you for asking. Uh, we do have a mentorship program. So we have, uh, um, there is one within data science. Our team is, what are we around? 170, 180, uh, folks right now. So we've gotten big enough that, uh, we, you know, do you have sort of a spectrum of more junior to more senior data scientists, um, we do pair them up and allow them to, um, to, you know, exchange, uh, knowledge and mentorship.
Um, more formally, we have, uh, some years we have an intern program as well, where we've, um, taken in, you know, folks who are sort of finishing up some level of schooling and had them as, you know, summer interns, um, on data science, um, you know, a lot of them have converted them to full-time data scientists and that has worked out really well for us.
We also have a very strong learning culture on our data science team. And so we, um, you know, part of having a centralized data science team brings this benefit. So we do a lot of lunch and learns, um, where, you know, teach each other about our current work. We have a learning group where we, you know, we'll read a blog post or, you know, at some points we've actually gone through a whole Coursera course together, or, um, you know, depending on the interests of the group, we'll sort of get, uh, work together on those.
Um, and then we have, you know, various sort of targeted seminars at Stripe too as well. So there's the ML seminar, there's a more general tech seminar. Um, sometimes we'll bring in external speakers as well. Um, so, uh, I think that can be, you know, that's probably not unique to Stripe. I think that's something that data science brings to a company really well is most data scientists are motivated by continual learning, continuing to expand your skills, getting to try out new methods, getting to, um, you know, just, uh, continue to both add both depth and breadth to your technical skillset. Um, so, uh, yeah, I've found that to be, you know, a really positive part of working as part of a data science team.
And I see Adrian had a follow-up there on, on which Coursera course did the team do together? Uh, we, this was like two, three years ago. We did the deep learning course at the time. There was just, I think one general deep learning course probably has expanded now.
Nice. And kind of, I guess, follow up question to Marlene's too. I was poking around on LinkedIn and saw a few roles at Stripe. And I noticed that one of them mentioned like helping to build the data science community there. And so aside from like some of the learning and getting together to learn together, are there other ways that people highlight their work, like internal meetup groups or things like that, or Slack or Teams channels?
Yep. We have, um, uh, so we have a very big, like written culture at Stripe. Um, and that includes emails. And so we send what we call shipped emails company-wide. So you finish a project, you're encouraged to write it up and send it to, uh, if it's a very big project, you send it to the whole company. If it's sort of a medium sized project, we might send it to the data science team and our partner teams, um, but strong culture of, of shipping and celebrating work that way, as well as making sure that your work has as much impact as possible, sometimes, you know, sharing it with another team or another person, you know, actually helps you find another use case for something that you've built.
Um, so strong culture sharing there. Um, we have, you know, various levels of team meetings as well. So full data science team meeting where we do sort of lightning talks, you know, a few short talks, since that's a big group about recent work, and then we have sort of our medium meeting where medium meeting like our, our, you know, maybe 30 or 40 people, um, where we, you know, give, uh, maybe a 15 minute presentation on more polished work. And then in my team meeting of 10 people, we do very early stage technical feedback. And so for that one, you know, we require that you do not prepare anything for it and just, you know, put up your code, put up your doc, put up your random notes on something and talk through, uh, something very early stage that you're thinking about, um, and we found that one in particular to be really helpful to folks, this sort of low stakes, um, environment where you can bounce your ideas off of each other, especially at COVID times, that's been really helpful to, you know, building some team camaraderie and also, um, sharing good feedback.
Balancing routine work with exploration
Um, Alan, I see you just put something to the chat. You want to jump in next? Sure. Uh, I've got, I'm thinking of lots of, lots of questions today. Um, I I'm curious to how, if you have a balance that you can maintain between the sort of routine things like financial forecasting that has an annual cadence or a quarterly requirement or whatever, with room to like experiment and learn new methods, um, try out tools, shop those around into the organization. And how do you, how do you meet all those needed commitments with, how do you balance that with like room to explore room to grow for your team to make sure that they're, um, you know, uh, able to try things out that may not have a known business connection yet, but, but, you know, um, you want to give them that, that, that kind of opportunity, like, is that something you're able to do? And if so, uh, how do you do it?
Yeah. A million dollar question. Um, so we do a few, a few strategies here. Um, this also is, you know, something that we continually work on and try to promote. Um, we do, uh, planning each half with our partners. Um, and so we aim to get, and when we do this sort of biannual planning, we aim to get the most important projects to them on our roadmap so that we can jointly decide what those are. Um, and that allowed, you know, typically there within those larger projects, there's some room for, you know, new methods or innovation experimentation while still working towards something that is key to the business and that, you know, partners have decided is important to them.
Um, and so that works well for, uh, you know, trying to build in a bit more of that sort of, um, longer term project that, you know, allows the data scientists to dig in on the methodology side. But then, you know, of course we build flexibility into the process too. Things come up, you know, there may be, um, urgent things there are all the time that come up that very important as well that, you know, folks work on. Um, and so we, you know, just sort of continually evaluate those and we're, you know, I spend a ton of time just having conversations about resourcing and making trade-offs on what the team is working on, um, you know, I see that as a key role of a manager.
We also do, we have a good tradition of quarterly career chats. Um, so I do quarterly chats with everyone on my team, talk about their, you know, chance to zoom out and talk longer term about their interests. And from there I find out, um, you know, what kind of projects are they interested in that they're not currently working on or other things they're working on that they're less interested in. Um, there's of course, you know, you have to find the business need, right? We're not a research team. We don't do sort of, um, you know, pure R&D work, um, but often by just having a good grasp on where people want to go and what they're interested in. You know, when I hear about new projects, I'm able to help target those appropriately to folks.
Awesome. Thanks. A lot of that sounds familiar and stuff that I'm trying to do better as well. Yeah, it's a continual process for sure. Yeah.
I was just making a note of the quarterly career chats too. And I was just curious how, like, how do you frame that with your direct reports? Like, is that an actual formal process that they go through or just informal conversation? So the first time when I have a new direct report, either new to Stripe or new to reporting to me, I ask them, you know, it's fairly, you know, a more structured chat. We go through their career history and ask a lot of questions about the decisions that they've made along their way and their interests and, you know, what brought them to Stripe and, you know, a lot of, um, sort of just getting to understand their career motivations.
Um, for subsequent ones, um, it is more open. Um, a lot of folks, you know, have a more structured idea of what they want from their career. And we might talk through that plan each time. Um, I'll often come with, you know, questions or new frameworks that I'm thinking about from how to think about your career and bring those. Um, you know, some folks are either earlier or more open or less sure about their career. And in that case, you know, I might direct the conversation a little bit more. So it's fairly tailored to the individual.
R and Python usage at Stripe
Thank you. Okay. So I see two anonymous questions that are kind of on the same topic. So let me combine them both here. Um, but the questions are around, do you support both R and Python users on your team and what does that split look like? Um, actually let me ask that part first. So I don't, there are too many questions that one.
We do, we do. We, um, we used to all of our forecasting are actually in recently, you know, Python has stepped it up a bit and we've been doing a lot more in Python. Um, part of the reason for that is that Stripe is somewhat more Python based in our infrastructure. And so, um, doing it in Python allows us to integrate with the rest of Stripe systems, uh, much more easily. Um, we do, you know, support R here. We have our notebooks. We have, uh, the ability to schedule jobs in R. Um, but I'd say it's, you know, more like 20 to 30% of data science usage and Python is a little bit more. Um, which, you know, R is close to my heart. That's the programming language that I started out in, in grad school. And, um, you know, I could live in the tidy verse all day, but, um, you know, we use it a bit more for sort of analytical exploratory analysis work. I would say in Python, a bit more for, you know, in production jobs.
In the, the followup question there was, do you encourage and support folks who want to learn one while being more proficient in the other? Yeah, certainly. I just had someone on my team who, uh, was more proficient in R, but wanted to learn more Python. And we sort of designed a project for her that allowed her to do that.
Communicating forecast uncertainty
Oh, awesome. Um, another anonymous question from earlier was, um, based on the fact that there would be underlying uncertainties in the forecast, how do you make your partner teams aware of and consider those uncertainties?
Uh, yeah, yeah. A complex question. Um, we talk about it a lot with them. We share the backtesting results. Those are usually helpful as sort of quantitative, you know, MAPE doesn't tell you everything, but it tells you something about, um, you know, when we looked over the past year, here's the amount of error that we saw. Here's how you might think about that going forward. Um, we, there's a lot of, you know, sort of fancier joint uncertainty throughout the hierarchy, uh, methodologies that we are exploring as well. I think uncertainty is one of the most useful concepts for, you know, forecasting within the business. Uh, but it's very hard to, uh, to, to sort of propagate the right level of uncertainty throughout the hierarchy. Um, you know, because you introduce uncertainty, both at the modeling, the initial modeling, you know, various points and in the reconciliation. And so, um, so that's, you know, somewhat, somewhat more of an interesting problem, um, to get to, uh, uncertain to get to uncertainty bands that you're actually confident in. Um, so we do that, you know, we explored from the statistical side, but we also from the business side, I just sort of take the perspective of, you know, you should, um, here's what we've seen historically, you know, you should expect some uncertainty in the forecast and then, you know, really digging in when we do see forecast misses.
Managing a remote, distributed team
I saw on LinkedIn, I think it was a post maybe from a year ago where you were presenting on like managing a remote and globally. Distributed data science team. And it was curious if you have any tips that you'd like to share with us in terms of managing a team that's, that's all over.
Yeah. Yeah. And this was true even pre COVID. So I have folks in Dublin, Ireland, uh, in New York, DC, I'm in Minnesota, folks in SF and Seattle, I think that's it. So it's, you know, we're very distributed, have been for a while. Um, I talked a bit about the early stage technical feedback meetings that I actually been a surprisingly useful thing to do for, um, just for making sure that we have that sort of weekly touch point with the team. Um, you know, you can do zoom happy hours and those kinds of things, and those are nice, but it's nice to have a little bit of structure and topic to know what we're trying to do and to be able to have. Um, you know, somewhat of a technical conversation in a, you know, again, very low stakes environment. We don't expect you to have a polished presentation, but, you know, much more of a rainstorm type atmosphere. Um, so having that touch point has been really nice.
Um, what else do we do? Snacks, people love snacks. So I will, um, there are a number of services that, you know, you can send snacks, uh, to people that just, you know, I think there's a, there's a certain, you know, nice feeling that people get by sharing the same snacks on zoom. It's a good one. Trying to remember what other, what are our other strategies? I think just, oh, focusing on onboarding. That's another big one. Um, with, you know, when folks start remotely, that's a very different experience. Then it used to be where they were in the office, you know, among everybody in a starting class, seeing everyone. Um, so I focus really closely on new folks. I meet with them every day when they're starting. I meet with them every day when they're starting. I, uh, set up meetings with the relevant people that they need to meet. I make sure they have the resources that they need. I focus, you know, the first month or so just focus really closely on new folks and making sure they get integrated. Cause it doesn't happen naturally. Um, in a remote world, you really have to work at that.
Um, in a remote world, you really have to work at that.
Absolutely. I love the point of like, even just sharing the same snacks, like something little like that and Libby, I see you timed your comment perfectly to when Dara was actually talking about onboarding too. So just want to see if you had any follow-up question on that Libby.
No, that was really, really well-timed. I asked it and then Dara answered it. Um, yeah, I think onboarding is a really tough thing in data science in general and then extra tough when everybody's remote. Right. And so I was kind of wondering what, you know, emphasis you put on that and how it worked and if you buddy people up and, um, things like that. But you really hit the nail on the head that it has to be intentional. Does not going to happen spontaneously, right? People aren't going to just integrate into a group and seamlessly work with everybody. There has to be some intention there. So that was good to hear. If you have any expansion on that, I'd love to hear, but if not, you did a great job answering.
Yeah, you hit most of the points. We do have spin-up buddies. And so people get paired up with an existing data scientist or data analyst that, um, you know, they can meet with and ask any question to at any point. There's a Stripe wide sort of onboarding curriculum that, you know, teaches people about payments and, um, the Stripe product. And then we have a data science specific curriculum as well. You know, how to set up your environment. How do we use Git at Stripe? You know, some of the more technical classes that we run. And those are both pretty helpful. But yeah, from there, it is very much just like making sure to meet with the right. Oh, and we have a spin-up project. So that's the other thing that we do, you know, very intentionally. We plan a sort of, uh, well-scoped project that we give to new folks that should allow them some room for exploration. It should be important to the business, but not, um, business critical need it next week type of project so that they can, you know, ship something that is important, but not feel that sort of time pressure immediately. They need to have some time to, you know, spin up on our systems and meet folks.
That's great. Who comes up with that project? The manager does. Okay. And just do like the newly onboarded people kind of like work together on that and share results. They do. Cause there's enough engineers on the data science side. It's typically just sort of one person per project.
Best career advice
Cool. So I know we're getting to the end of our time here, but one question I, I love to ask people when we have time is what is the best career advice that you've ever received? Or maybe not best, but something that sticks in your mind.
Something that sticks in my mind is to think about choosing the wild card when you're presented with multiple choices. So, you know, for me, when I was choosing my PhD advisor, I had two rotation, you do three rotations, right? I had two rotations that I had planned ahead of time that I knew I was very interested in the research that would go really well. You know, I thought, and they, they did both go really well and I really liked those labs. And then the third one was you know, this brand new professor, I would be the first grad student. It was a little bit outside the kind of research that I had done before. So I thought, you know, I'll just do this wild card rotation. And of course, you know, that was the lab that I ended up joining. I was the first grad student and ended up having just this fantastic experience with my advisor and with the research. And that also, you know, has held true as I've made other career decisions along the way, you know, choosing to go to Stripe rather than stay in something in biology, choosing to go to data science rather than do an academic postdoc. You know, sometimes that wild card choice is, can feel a little scary, but also kind of enticing. And, you know, for me, just trying to not be afraid to choose that option among choices has worked out well.
You know, sometimes that wild card choice is, can feel a little scary, but also kind of enticing. And, you know, for me, just trying to not be afraid to choose that option among choices has worked out well.
Recommended books and frameworks
Thank you. That's great advice. And Priyanka, I like your comment in the chat too, when I asked that question to also frame it as it could be advice that you've been given or that you have given somebody else too.
I just wanted to double check that I haven't missed anybody's questions in the chat or on Slido. So feel free to raise your hand here or just unmute as well. I'll give people a few seconds here. But, but Dara, when you were talking about some of the conversations with you, you have with your direct reports on their career growth, you mentioned some like recommended reading or resources that you give them and was curious if that's something you could share with us or if you have any books that you'd recommend.
Sure. So, I mean, so we are at Stripe, there's often, you know, more than one book that we at Stripe, there's often, you know, manager workshops or trainings. And so I'm, you know, constantly in a state of learning as well about, about things. So I'll just share a couple that are sort of top of mind. There's lots of, I mean, like for books on, you know, sort of management and actually just general career books. One of my favorites is, what's it called? The Manager's Path by Camille Fournier. And she takes, she was the CTO at Rent the Runway. And so she takes you through how you can think about your career from, as a manager, from, you know, managing an intern to managing a few folks, managing a larger team, all the way sort of up, up, up to being an executive. I think that's very useful both for ICs and managers to understand, you know, how to think about your career at various levels. It's certainly one that I revisit from time to time, you know, as I've moved through my career. So I found that, I find that one to be super useful.
The High Output Management book by Andy Grove is a classic. It's one that we reference a lot at Stripe as well. These, again, are more management focused, but I think good resources for thinking about your career. One that I, this framework that I have been using in career conversations recently is kind of interesting. It's called the SCARF framework. It's a neurobiology thing, and it's about how you respond to stressful situations. So it's like, I'm not sure if I can remember what they all are now, but it's things like connectedness, autonomy, relationships, and thinking about what's important to you in your work. And so I use that and sort of ask people to talk about what resonates with them in the SCARF framework. And it's funny how, you know, different people really value different things when it comes to their work and what's important to them. And so I found that a very helpful framework for learning about folks and for, you know, folks to sort of self-identify what's important to them.
Thank you. I was trying to find the link. So it's SCARF, S-C-A-R-F. Okay. I like to share some of the links as we go. And the book that you said by Andy Grove, what was the name of that one? High Output Management. High Output Management. Okay, awesome.
Thank you so much, Dara, for joining us here. If people do want to get connected with you or keep in touch, is LinkedIn the best way? Yeah, it's a great way to find me. Okay, awesome. Thank you so much for sharing your experience with us and answering all of our questions. Thank you, everybody, for the great questions as well. Thanks, Paul. Great to meet you all. Have a great rest of the day, everybody. Thank you so much. Bye.