Resources
Dec 17, 2021 |
1h 8min
|
381 views
|

Data Science Hangout | Jarus Singh, Pandora | Human in the Loop

The Data Science Hangout is a weekly, free-to-join open conversation for current and aspiring data science leaders.

An accomplished leader in the space will join us each week and answer whatever questions the audience may have.

We were recently joined by Jarus Singh, Director, Quantitative Analytics at Pandora.

A few key snippets from our conversation:
01:28 - Start of session
6:33 - Human in the loop
14:14 - Working with stakeholders and teaching communication
25:47 - What does your tech environment look like?
28:46 - Presenting work; pretty vs. impactful
33:18 - Skills for data science leadership
43:01 - Having data science rolling up to the CFO
51:41 - Getting motivated by personal projects
49:00 - Applying our hobbies to work: cooking
1:01:25 - Going with your passion

► Subscribe to Our Channel Here: https://bit.ly/2TzgcOu
► Add the Data Science Hangout to your calendar: https://www.addevent.com/event/Qv9211919

Follow Us Here:
Website: https://www.rstudio.com
LinkedIn:https://www.linkedin.com/company/rstu...
Twitter: https://twitter.com/rstudio

image: thumbnail.jpg

Transcript#

This transcript was generated automatically and may contain errors.

Thank you, everybody, for joining, and I know we just hit the top of the hour, so welcome to the Data Science Hangout. I know it's a bunch of familiar faces here, but anybody who's joining for the first time, welcome. This is an open space for the whole data science community, current and aspiring data science leaders to really connect and chat about some of the more human centric questions around data science leadership. We really, I'm just going to mute people if there's a little bit of background noise, but we really want to create spaces where everybody can participate and hear from everyone. So you can jump in live and ask questions, you can put questions in the chat. And we also have a Slido link where you can ask anonymous questions too. So if you don't want to be a part of the recording, it's a great way to ask questions too, because we will share this up to YouTube as well for anybody who missed it. But with that, I'm so excited to be joined by my co host for today, Jarus Singh. Jarus is a Director of Quantitative Analysis at Pandora. And Jarus, I'd love to turn it over to you and have you introduce yourself and maybe start off by sharing a bit about your team and the work that you do.

Yeah, thanks, Rachel. Thanks for having me today. Really excited for this. So I lead a team of three. We're quantitative analysts. And what's a little unusual is that we're embedded in a finance team. And as a quick aside, when I first joined, I was very skeptical of being on a data team sort of embedded in the finance org, just because they're not historically known for, I think, putting as much emphasis on data or the right emphasis on data relative to sort of other orgs that may have more of a history of doing so. But it's actually worked out really, really well for us. So, you know, the way I view the team is that we have three sort of pillars that we're responsible for. One is reporting. So what happened? And this is all the metrics that Pandora cares about that informs its financial model. So it would be users. It would be how much people are listening. What platforms are they listening on? What are the demographics of listeners? All of these have impacts on what we pay for spitting the tracks or, you know, the revenue that we can make against people when they're listening on the free tier from advertisements or, you know, the propensity to join a trial and then eventually subscribe. So that's the first pillar is reporting. Second is forecasting. What does the future hold? We do that using a lot of R and we try to rely on algorithmic and automated techniques as best as possible, but realize that, like, they can't do everything for you. So there are times where, you know, you need sort of human in the loop style modeling where someone has to look at it and say, OK, I'm getting information from this team saying they're going to change their strategy and we need to make sure that we're not relying entirely on an algorithm that is not aware of sort of that change that's coming. And then the last pillar, I'd say, is data driven decision making. So we have a wealth of information about our users. If we're going to go out and do marketing or special deals with, you know, it could be a telecom company or a device manufacturer, we want to understand the most we can about the sort of users they're going to bring in or the deal terms so we can use our knowledge of Pandora's data to come up with a very refined estimate of what we think would happen under those under those deals.

Cool. I'm trying to think of myself as a Pandora user, like in the app itself. Is there something that maybe the data science team has worked on that I would see as a user directly? Yeah. So my team is mostly dealing with like business metrics, I would say. So there's not a lot of stuff we've done where I'd be like, oh, I worked on that. It's more like every time you go in and engage with the app, it's like we're trying to predict how much we think you're going to use it and sort of the potential you have towards the business. LTV as well is important. But yeah, you know, we're not selecting the next song. So good to kind of level set for this discussion before I get questions about about that.

Yeah. Thanks, Jairus, for the intro. While we're asking for a few questions from the audience and waiting for those to come in, I'd love to hear about something that you're most excited about in regards to data science as you look forward to 2022.

Yeah, I guess like generally in the space, what I'm seeing is and this is from sort of some data groups that I'm in and just reading blog posts is that data folks, I think they know they have a lot to offer to the table, but they're not always in the room where decisions are being made and they're not always able to inform those decisions in the way that I think they can and should. And what I see is the space sort of maturing in that. Like, I personally believe that data folks are going to get more ownership over things that sort of should be in their purview, but aren't necessarily at all companies. Some companies are doing this very well where they're using data where data can help. But I think not all companies are on that stage of maturity. So I'm expecting sort of more maturity from the space where people are getting more sophisticated about using data when sort of applicable. And to throw out a company example that I've heard really great things about in the past and seeing Hilary Parker talk for when she was there would be Stitch Fix. Right. So the way she kind of presented them is that anything that can be informed by like the data scientist skill set well or better than what you would get, you know, leaving that as someone else's decision who doesn't have a skill set that's being put in the data team's hands. So I thought that was a really cool way of framing it. And I think more companies are going to be moving that direction.

I personally believe that data folks are going to get more ownership over things that sort of should be in their purview, but aren't necessarily at all companies.

Human in the loop

Awesome, thank you. I see Arafat just asked a question in the chat and asked that I read it out loud, and so I just want to say if anybody uses the Zoom chat and you want me to read it out loud, just put a little star at the end so we know.

But Arafat asked, could you share some more detail about the project that you shared earlier or refer to a resource or book?

Yeah, of course. So Human in the Loop is sort of this framework that says you can rely on any automated system or algorithms to generate some of a result, but ultimately a human has to be in the loop somewhere at some stage, right, verifying the output, changing it as necessary and is basically able to improve upon the system if it were a computer alone or a human alone. So I can give some examples of that. And another one's going to come from Stitch Fix. I feel like I work there even though I don't, but they have some really cool use cases.

So the first one I'll give is from our world, right, where we're using a lot of time series techniques to forecast the future. And the time series technique is if it's univariate, we'll just look at historical data. How has this been trending? What is expected seasonality? And we'll extrapolate that forward. So that makes sense when the future state of the world is sort of structurally similar to what we've experienced in the past. But if we were to make a change and it's an anticipated change, that's where the human has to get involved. So an example I'll give you is we were growing a certain platform on the on the service, you know, pretty quickly over time, and it looks like it was a predictable series. But we tried to make some changes to the user experience there that we thought would actually dampen growth. They would have an adverse effect on growth going forward. Now, the model has no idea that that's happening, right? There was no way for us to sort of pass that in. So we create sort of what we would expect to happen under the old regime. And then we layered in assumptions we got about the new one. And sorry for being vague, but I shouldn't probably go into too much more detail. So that's where it's like you're accounting for the fact that the algorithm doesn't know this change is happening in the future. And you use an alternate set of data and assumptions to do that tweak.

A really cool stitch fix example I heard is when they're actually selecting clothes for folks. So there's a lot of feedback that people can give on prior sets of clothes they've gotten, which a computer may or may not be good at really understanding. So if the NLP side of their business isn't fantastic, it may not realize like, OK, this person's OK getting pants, but like they still want some, but they don't want it always. And a human can do a good job of saying, OK, maybe I should send pants in a kit every other kid or every every third or something. So it's an area at which you would not expect whatever algorithms you have to perform well. A human has to intervene and and kind of adjust what you would get. And you get a better sort of outcome with that.

In terms of resources, I'm not sure if it's addressed in this book, but it's one of our favorites. It's by Rob Hendeman and it's on forecasting. If you go to Otex.org, it's actually available for free. They're forecasting principles and practice. That's what my team relies on a lot for a lot of the work that we've done.

I think Tori maybe shared that book earlier as well. Oh, gotcha. One that's really helpful, too.

That's interesting about Stitch Fix, too, because I use Stitch Fix and I am one of the people giving that feedback and they make it seem like it is a human interacting with you on that. There was an anonymous question. Oh, sorry. You go ahead, Frank.

Thanks, Rachel. I'm going to pop in and just add another recommendation from a previous one of our calls who Paul from Kentucky, I forget his last name. But when Paul was talking, he recommended Prediction Machines and I recently finished that book and that was heavy on there's kind of an economics twist with how do humans work in the future with algorithms as forecasting improves. So that's another, I think, really solid recommendation, Prediction Machines.

Awesome. I went through a few of the recordings to write out some of the books that have been mentioned in the past, and we are working on getting better at sharing all this information that's being shared here. So I know someone had asked earlier, too, about how do we know like who's speaking which week as well? And we're working on creating a site for that so you can see all the resources shared and clips and stuff. So more to come on that.

But there was an anonymous question, Jairus, around how did you get into data science and what specifically attracted you to your current role?

Yeah, that's a good question. I guess I would say like I started in college that this would be the beginning of the journey, like studying economics and mathematics. And it's funny, like you see a lot of data science concepts covered then, but they weren't branded as data science in school. I guess we're behind on the branding when it comes to those things. So I feel like I was picking up the techniques there. And my first role was in economic consulting, which is a very esoteric area. But it had to do with it was in the litigation space, companies suing companies, companies suing governments, governments suing companies and trying to understand like what were the actual damages associated with wrongdoing and was there wrongdoing? And you could rely on econometric analysis to do that. But still, this is not this is not data science. And I think where I made the leap into data science was while I was doing that work, where I think I got most interested and motivated was was when I started learning how to do functions. I think that's where it really clicked for me of like you could write a piece of code that makes your ability to do this in the future much, much easier and I can make my life easier. And I started to basically talk to friends who were who were going into data science from my network and realized like that is a space in which that sort of attitude gets rewarded. So it's like, OK, I'm interested in this. I'm excited by this. Let's start to find jobs that like like reward that, which I think is broadly the data science space. Right. So I made the jump into tech and started working on picking up the skills. And I've been really fortunate to have sort of a series of folks. And this was a long time ago. So I think it was easier to say like, hey, I know this language, but like I'll learn this one if you need me to. And I had, you know, a few lucky breaks that enabled me to get where I am. And I'm obviously really appreciative of that. So that's how I ended up in this space.

Working with stakeholders and hiring

That's awesome. Thank you. I see there's a question early around hiring, too, and I know we wanted to dive deeper into this topic as well. Jairus was on an earlier webinar on hiring best practices. But the question was, what skill sets do you look for when hiring technically and also personality wise?

Yeah. Yeah, that's a it's a tough question. So I'm going to answer specifically for my team at Pandora what like what we do and hopefully this is helpful. Don't necessarily extrapolate this to like all of the data science ecosystem. It's obviously very broad and people look for different things. But what I look for at the technical level is somebody has to be sort of like maybe facile is the best word. They have to be very comfortable with, I think, the common coding concepts that are going to pop up on the job. And I want them to sort of proactively motivate. Right. I don't necessarily want to be the person that says, like, you should write a function here, right? Like the person should be comfortable enough with those concepts to sort of motivate them in their day to day work.

On the personal side, what is tricky for us is that, you know, we end up working quite a bit with stakeholders and very early on when I hire people, I may have them in the room with stakeholders alone. Like, obviously, I'm there to guide. But the ideal is to get them at the point where they're communicating well and presenting well, where they can handle that themselves. And then they can also sort of scope the sorts of questions that they get or follow up requests that they get from the stakeholders. So ideally, I want to get someone who says, OK, you know, I see you're asking for this, but if I understand your problem correctly, like I'd rather do this. I think this is actually the thing that's going to solve it for you. So people who are able to do that and what I've found is that you have folks who sometimes are very good at the technical side, but they're not as good at the other sort

of skills that we're looking for and vice versa. So when we do our interviewing, we kind of try to emphasize both. And we understand that you're not going to be, you know, necessarily like 10 out of 10 on both. But we really do need a healthy balance because otherwise it's very difficult to sort of do what we do.

Other beneficial personality traits I look for in addition to communication are and this sort of comes out during the interview is interest in the sort of problems that we work on. I think it's really good to signal that during an interview.

And what I found is that folks who do that during the interview process, you know, tend to continue to do it on the job. I haven't seen very many good actors. So when they're like, oh, you know, do you all look at this data or do you do that or have you thought about this? Those folks tend to do very well in the role because they bring that sort of brainstorming mentality to the role. And what I found is they're able to solve things in ways that, you know, I wouldn't have thought of. Other team members wouldn't have thought of. It really does help out a lot.

So that's another thing that I look for versus the sort of passive applicant who's like, yeah, I don't know how to like, what do you guys do? I'll just I'll just continue to sort of implement that.

Definitely, I think that was a tip that you had mentioned in the webinar to just making sure to actually research the like field that you're in to make sure that you're aware of what are the kinds of problems that they're facing.

Totally. Yeah, I'd say that's probably the most like common piece of advice. I give folks I work with a boot camp as well. So I do calls with folks who are looking for jobs. And then I had one yesterday just with a friend in my network. And that's one of the things I say is like understand the business just because I mean, obviously it gives you stuff to talk about during the interview, you know, when it comes to like small talk. But also like one of the most common data science questions I see is like, OK, what metrics would you look at if you were working here? And a lot of folks are very caught off guard because they're like, I didn't realize I had to kind of solve that beforehand. Like I was hoping you would tell me and then we would do a problem on that. So I think it's nice to sort of bring that to the table. And it makes any follow up questions a lot easier because you're already framing them in the context of, you know, the place you're applying.

Building communication and presentation skills

I really like that you said you put new hires into the room with the stakeholders as well to like start that back and forth communication. And I see there was a question that asked, how do you build confidence in communication and presentation skills within people?

Yeah, so. What I would recommend is have like a mentor at your company, it doesn't have to be your manager, but somebody who has those skills and reach out to them and say, I would love to go over a presentation with you or just invite them to your meeting. Sometimes that's a great way to to cheat at getting feedback. Oh, I need you. I need you in the room to see this, even though you may not necessarily. But someone who has those skills who can sort of guide you on it, because that's what I try to do for my team, just having the I guess it came from the consulting background. I don't know where it came from, but wherever I picked it up, you know, I do think that's one of the advantages I have. And of course, being in management, my technical skills have atrophied. So they're much better at that side of things.

But just the ability to make myself available and give them feedback before important meetings or sit in the meetings and guide them. And then also give them the heads up that that's happening because not everyone likes to be caught off guard while say, OK, like, you know, let's move on to this topic because I think someone's interested. Or can you explain that more? I don't think everyone got it. Stuff like that. And over time, I think that reinforcement really helps out because eventually it's kind of like training wheels like you won't need them anymore. You know, you've gotten enough practice with an advocate or a mentor that you can do it yourself.

I guess one thing to add on it is. Data scientists, I think, on average, tend to overestimate how much everyone else knows about data science. So if you're like, am I over explaining it or under explaining it? Odds are you're under explaining it. And again, depending on who's in the room, you know, again, we're thinking like stakeholder management, folks who generally don't have a background in in the data side of things, I would err on the side of over explanation and then always give them an out. Right. You say, if you already know this or I'm being redundant, I will skip the slide or skip this section and we can move on.

Data scientists, I think, on average, tend to overestimate how much everyone else knows about data science. So if you're like, am I over explaining it or under explaining it? Odds are you're under explaining it.

That's really helpful, too, and just want to say again, if anybody wants to ask a question live, Robert is teaching me to be a big proponent of the raising hand on Zoom as well. We're trying that in internal meetings or just unmuting yourself as well. And I can take that as as raising your hand.

But, Ian, I see you had asked a question in the chat that you want me to read out, and it was, has the pandemic wreaked havoc with your time series models? Some weeks back, Elaine McVeigh discussed about how difficult time series modeling became in the CPG sector. Are you experiencing this as well?

Pandemic impact on time series models

We did. Yeah, it's the effect is much more muted now. But at the beginning of the pandemic, there was, you know, a huge change in the way in which people listen to Pandora that, of course, is now, you know, when extrapolated forward, having a lot of problems. So what I was kind of proud of was our team has, you know, the knowledge of how our algorithms work. Not only do we implement them, but we understand how they work. And we were able to very quickly identify, like, look, the outputs of these are not going to make sense. And so we had to sort of pull alternative models online, which I think we did within two weeks or so, which I'm pretty impressed by. And we, you know, we had probably some silly manual model we were using like in between then in order to account for it.

But I was really proud of the way in which we sort of pivoted quickly. And I would say, like, if you're dealing with business side data, you should consider that a part of your job, because, again, there are folks who place a lot of emphasis on, you know, can I put these things into production and can I run them? But if you're not making sure that they're resilient to things like this or you don't have a plan for how you would fix it, I realize not everyone wants to plan like possible pandemic into like every month of the work that they do. But if you don't have some way of pivoting quickly, you know, in the eyes of the folks you work for, they're like, wow, that person's like not really doing their job. And, you know, we pay them so much because they have all these skills. But like in some ways, you want to be resilient to these sorts of changes. So I was really happy with how we dealt with it.

It is still causing problems on models that our partner teams work on just because, you know, we've decided we want to interpolate data and fill it in. But due to the nature of the structure, it's just like a lot to put there. And so we're trying to figure out, like, how much effort should we put into this sort of interpolation and how much should we just sort of, you know, hand wave and maybe put less sophistication behind what we what we put in the historicals. But, yeah, it did affect us a lot. So great, great question, Ian.

How do you know when your output wasn't making sense? Was it all accuracy metrics?

Yeah, that's the way we determined it. And accuracy for us, you know, we have a good idea of how accurate like we should be and then what's sort of abnormal. And we entered abnormality pretty quickly.

So what I can tell you all about Pandora usage is like a lot of it we found is associated with routine. And so when routines get disrupted, you see large changes in how people listen. And we saw that happening in March with shutdowns and with panic buying. You know, you're buying toilet paper, you may not remember to put on Pandora in your car when you're stressed out. So we did see that. And, yeah, accuracy is what we use. And it was just like this seasonal kink that, you know, you would not expect. But for, you know, major disruption happening. So that's how we saw it.

It was kind of interesting to see like it happened kind of at different places throughout the nation. And we tried to line that up with sort of how serious folks were taking it. But at the end of the day, that was kind of acute analysis. There's nothing we can do with it. You know, where a really bad ongoing joke we had is like in that first two weeks, it was like, unless you're developing the vaccine, this is not going to change. So we need to just adapt sort of what we're looking at. You know, we're not going to be able to do anything to sort of solve against this while people have much greater concerns on their minds.

I know you mentioned the serious exam and Stitcher side as well. So I was curious, like being one big happy family, if the data science teams ever like work together, share insights as well. Like I was thinking with the pandemic, did people start listening to podcasts more versus music at certain times?

Yeah, there were small changes in what people listen to. And the most obvious one is like health or news podcasts saw a spike. And it would have happened naturally. But also, obviously, like as a company, we started to promote those, right? We figured they were top of mind. So, you know, they did well, both by, I think, being relevant and by the company realizing. And this is, again, human in the loop, right? You need somebody. I believe it was a marketing led initiative to say, like, this is something that I think is culturally relevant. People want to know about. You know, the algorithm is not magically going to surface this podcast, which has not been as popular lately. And so they sort of pushed it in terms of like the type of music people listen to.

We had a very honestly, I don't think the evidence was really good, but someone was definitely pushing the like people want to like they're panicky. They want calming music. And so they were looking at like, oh, this like, you know, this calm genre that we have tagged is like increasing a little bit. But I don't know if I would have necessarily made that case. It fit with the narrative of what was going on, but I'm not sure it was actually showing up or if it was just noise.

Cool. Thank you. Fritz, I see you asked a question in the chat, and I'd love to turn it over to you to have you ask it live, if that's OK. Or I could read it as well. So that always said you said you don't have a microphone. I'm happy to read it. The question was, what is the technological environment like at Pandora? How do you deploy models that your team develops?

Tech environment at Pandora

Yeah, so we're on Google Cloud Platform for all of the sort of raw and upstream data that my team relies on and the data engineers get it there for us. So that's where we start working with Pandora data and we'll do aggregations there. You know, we have to handle ourselves. Sometimes logic needs to be applied to classifying various platforms. We don't want to deal with the laundry list that they have. So from there, we will get it into BigQuery or Postgres server we've just used for a long time and we've not migrated our jobs over to BigQuery. And then we'll connect to that with RStudio. So we're using RStudio either server for sort of the bigger projects that we have. And then we'll use or if RS Connect is needed for a shiny dashboard and then sometimes a personal laptop just when it like doesn't really matter how much firepower you need.

So we'll use sort of both instances and personal can also be good for dev sometimes if you want to use it for that. And then for visualization or end product, you know, sometimes it's just dumping the data back into something like BigQuery or Postgres. Sometimes it's, you know, a CSV that other folks in finance can ingest and use in Excel or something that can be uploaded to Anna plan, which is how we manage a lot of our financial reporting or a shiny dashboard is what we sort of default to. And then we've inherited some work that's in Tableau. So we'll do that as well.

And what I would say is the way I like to explain it to stakeholders who are used to working with folks doing a lot of manual reporting is I say like one, you know, I'm reporting is I say like one or zero click to send this report to give them an idea of, you know, you may have folks putting together stuff in Excel and pulling manually from different sources. We're able to sort of create this whole sort of pipeline or ecosystem for a lot of the problems that we do that require one click or zero clicks. The one clicks usually just to check to make sure everything makes sense before you send the report out. Yeah, so that's our ecosystem.

Thank you, Bruno. I see you had a question in the chat, too. Can I pass the mic over to you in case people are tired of just hearing me talk and release?

I see you've unmuted, but we couldn't hear you yet. There you go. Yep, I'm here. Great. Hi, thanks for doing this question. From your experience, if you had to sell a data science project in an organization, which is the most likely to succeed? An absolute baller presentation of result, but average predictive power, or an academic paper level long form presentation, you know, like no image, nothing, but now some predictive power, which has the most value to executives?

Presenting work to executives

Yeah, I'll kick it back to you, Bruno. So when you say to sell a data science project, do you mean something that's been created and you're trying to get credit for it? Or it's like, here's something I want to go out and do?

So say you build a team, you are your data scientist, because it's the buzzword of the week. And then the executive spent a lot of money to build this team. And then you come back with a result, let's say one or two years down the line, to justify the reason for your existence. Right. Does that makes a better context?

Totally, yeah. So you've been doing a lot of projects during this time, and you want to communicate your the value of the business to them. Okay, so I will answer your question. And then I'll take a sort of tangent related to that question. I would say like, the first thing is, it's dependent on who is the executive that you're presenting to. But by and large, at most companies, my experience has been the executives who hire or green light data science teams are not well versed in data science themselves. So a lot of I think the onus is on like management and leadership and data science to explain to them the value. And I think, you know, a good executive is not going to try and like pick you apart for not presenting stuff while they care about ultimately the bottom line, usually.

So, you know, better results should work, but I think it's the way in which you frame them.

So there's the idea of like making it pretty and making it impactful. Pretty, you know, the iBankers are good at, the management consultants are good at, and execs are used to working with them, and they're used to seeing things that way. But if you were to do like a summary or output of a table that shows how much money you're saving the company through your optimization efforts, and that's in the millions of dollars, my hunch is that you don't really need to, you know, put that number in the same font as everything else in the deck and make sure it's standardized. At the end of the day, I'd say that's what matters.

So where I see sort of the, yeah, dollars speak, I saw you put in chat, where I see this sort of failure in like trying to show how good the data science team is or how much they're doing is sometimes they will focus a lot on, you know, metrics that we're very comfortable with and are sort of easier to calculate. And a classic example is we had a model that had 80% accuracy. I made it 84. But without doing some work to explain like what is the value of that to the business, you know, executives can sometimes be skeptical. Also, like they may not be well-versed in the problem space. So they might say 80 to 84, like that's not very good, right? Like that doesn't seem like you did a very good job because they haven't been working at those problems as long as you have and understand sort of the hairiness of them.

So I guess more broadly than just dollars, I would say like speaking the language that they care about and are receptive to. If you're in people analytics, it might be what are the things that we've done that have resulted in employee retention, which again, ultimately makes its way down to dollars. But make sure to coach it in like what they care about and what they're expecting. Because the way I like to view data science is, you know, at the end of the day, like they're hiring us because we have a skill set. It's a tool and it's a tool being applied to problems of the business. It doesn't matter how fancy your tool is if it's not solving the problems that the business has. So try to view it through their lens. You know, I've tried to do that throughout my career and I found, you know, that tends to work. That tends to work well.

It doesn't matter how fancy your tool is if it's not solving the problems that the business has. So try to view it through their lens.

Thank you so much, Sharice. I'm marking to clip that one. I want to go back and share that. Yeah, edit appropriately. I know I was around. I was a little all over the place because I have strong feelings about it and it actually just reminded me of one additional thing and now I'm blanking on. Maybe I'll remember it while we take the next. No, I mean, I mean, save that piece to share it with everybody. Yeah, that was incredibly helpful.

Skills for data science leadership

There was a great question earlier that was, what are some small steps someone can take to become a more effective data scientist or leader in the long term? It seems like there's so much to learn and dive into. Yeah. So to be a leader, I'm going to think about that piece particularly. So like, what is the difference between a good data scientist and a good data science leader? And this relates a little bit to what Bruno was asking.

So and I sort of had this as a rude awakening during my career, which was as a data scientist, right? A lot of emphasis is placed on like, what are the tools and techniques that you have? You know, why are they better than doing it manually or doing it under, you know, older styles of, say, like an Excel based report or what have you? Like, why do we need you and why do you do what you do? And so much emphasis was placed on that. But less so, you know, on the communication, my manager was handling it. And certainly on like project scope, right? It wasn't really my job to say, like, this project should entail this or it should do that. Or if we're trying to solve this problem, we should do this. Obviously, I was allowed to give some feedback, but ultimately, it wasn't my call.

Where I see like data science leadership, like what people who are doing well in that space, you know, the skills they have to develop and they may not be developing as an IC or obviously the communication presentation, being able to understand what are the needs of their stakeholders and then how can or cannot data solve those problems. So I say can, obviously, because that's the job. But can't is also really important as well. Because if you have a stakeholder come to you and say, we want to train a model that or can you build a model that does XYZ? Like, it's going to be great. And you say, yeah, I don't actually see how that's going to help the business. And you can sit down with them and explain why or say, historically, we've tried to develop models here and we haven't had much success. There's nothing I can think of that I think would do better. So instead of just sitting down and going and do that, you're the person who has some say around, you know, I think this is more likely to be successful and actually advance the needs of the business.

And so, Rachel, that totally ties into what I was saying earlier, when you interview day one or day zero or negative one, depending how you look at it, if you understand the business, I think it makes you much, much better able to do that than if you don't really know the inner workings of it. So the idea of proposing here are the projects I think are most likely to succeed and then getting people on board with that.

Maybe this is a lazy analogy, but it almost feels somewhat like VC-like, like you're a data VC in some ways. Like I'm betting on these three projects to deliver value for the business and like I need you to buy into them so I can have someone go out and do them and help us be sort of most successful. I really like explaining it that way, as being like a data VC.

Dealing with fear of job displacement

First, I see you just asked another question in the chat that I'll read out, which was how do you deal with people who feel a threat from machine models? Those who are afraid of having their jobs displaced by data models. Yeah, that's a tough one. I haven't had a ton of experience with that, but I would say right now the state we're at for a lot of sort of data informed projects at a company is that human in the loop is a great way to go. And if a human is in the loop, then you're not really threatening someone's job. The idea is I think that you are advancing the value that they offer because you're giving them tools that they can use in order to do their job better.

That is not always the case, so it's sort of dependent on how things are structured. You can't tell someone, oh, it's human in the loop and then cut them out entirely. But for a lot of things that we see, and I mentioned that example earlier where like we knew given the cultural climate and the global pandemic, people would be more interested in health and news related to that than history. We need smart, good marketers to say, okay, I'm going to let the algorithms pick certain things to put in front of people, but I need to reserve this space for something that I personally am willing to bet folks are really interested in or getting increased engagement. So that's how they work with it. I know I gave the Stitch Fix example earlier as well.

Again, that's not always the case. It's kind of a cop out to say like, oh, everything will be human in the loop and we'll all be fine. But in general, what happens with the progress we make on the machine side is that we kind of let people add more value because it's displacing sort of the easiest tasks. Again, this is on average, not always, but that's what I've been seeing, not just in my company, but at others as well. So I would say it's not so much threatening to your job as a whole, but it does mean that if you're somebody who's been doing a job without a machine learning model, you need to understand how do I work with this thing? Because, you know, it's not going to go away.

A/B testing human interventions

I know on previous weeks, we've also talked about the importance of testing and A-B testing and how sometimes we don't always do that or give it enough time to actually consider it a test. Yeah. And I'm just curious, like thinking of human in the loop project or aspect and having people recommend certain things like, oh, during the pandemic, people will look at health podcasts or something. How do you kind of push back if that's not actually the case or how do you test that out and have that conversation?

So the idea like is having a human, like someone intervening like better than the alternative. Or if somebody feels strongly that, for example, they're going to listen to health related things when that's not actually the case, how do you communicate that? Gotcha. Yeah. So this is where we get into the fuzzy side of things very quickly, right? Because when you have, you know, a well-behaved algorithm, a recommendation system, right? You can say with some confidence, like here's how much I expect people to like whatever content I'm giving them. We have a long history of how well this is done and we expect it to perform more or less just as well in the future, right?

And so when you have somebody, you know, a human saying, well, I want to change this or I want to change that, it really has to be done case by case because there's nothing in our framework that says, oh, you know, retune the algorithm, but for a pandemic, right? We do not have the ability to do that. So they really always get dealt on a case by case basis. But Rachel, you raise a really good point that I do know there are stakeholders who probably put too much. They put too much faith in the machines and they don't do enough of that. Alternatively, you have people who do the opposite, right? Who always want to override the machines and they do so to their own detriment.

I'd say I agree with you that like A-B testing these approaches is a great idea to sort of create a historical record of like how good are our overrides. But the tricky thing about them sometimes is that you're not always able to do an A-B test. So you can't always actually figure out like what is the utility of, you know, having someone doing this intervention versus just letting the machine make its recommendations. But yeah, I would do it if possible. Sometimes it's tricky or not possible. But if it is possible, you can, what I would do is I would try to create a history of, you know, this person has tried to override recommendations. They did it in this period, in this period. And over time, they either, you know, batted above average or below average. And we can use that information to inform should we do more intervention in the future.

Advocating for the data team in decision making

That's really helpful. Thank you. See, there's a great question I had missed earlier. I'm just scrolling up. That was how do you advocate for the Pandora data team being involved in decision making? Yeah. So this is where being in finance has been hugely beneficial. And I mentioned I was skeptical when I first joined because finance teams don't, you're not generally associated with data prowess, no offense finance teams, or maybe offense taken. So you all can, you know, go out and get better and prove me wrong.

But being embedded in finance has helped with that hugely because at the end of the day, the decisions that we are making are coached in those terms, terms that everybody cares about, obviously, the CFO and the CEO as well. And so once you've got that framing down of like, we're trying to understand how, you know, if this initiative, proposed initiative will be breakeven, or we want to understand what's the utility of missing our forecasts, like how bad is 1% miss and accuracy? How bad is 2%? Like, what are the downstream effects of making these mistakes? That has been hugely beneficial.

So I also find that the finance problems are generally a little more well scoped. And this is for better or for worse. It's just very easy to understand, like, yeah, we're doing breakeven on this or what have you. It's a little trickier when it gets to someone like product where you say, well, we think we should build this. Do we have data to support it? You know, sometimes the answer is no, but you have to go out and build it anyways. So I would say the advocacy that I had to do over time was not me trying to convince people of my team's worth. It was more just like producing, hopefully a drumbeat, constant drumbeat of wins and then showing people like, look, this is what we do and how we do it. And you need to trust us and involve us in the process. And we've had times where folks have tried to go around us and try to sort of sell a decision without our involvement. And, you know, our leaders have always backed us up and said, like, no, you need to involve, you know, this group or the finance team or the group within the finance team to make sure that that's the right decision.

Awesome. Thank you. I just want to check that I haven't missed anyone's question yet so far and would love to have people unmute or come on live too.

Yeah, I guess while you all are thinking, I might add, I do expect more organizations to start embedding data people in the finance org, or if there's like a hybrid org, right? So you have the data org where we're decentralized. But if you have the hybrid org, so data org, but everyone has sort of different groups that they support like dotted lines to ultimately that reporting into the CFO, I think is really powerful. And I've been seeing it more often just talking to folks in this space. And I think that's going to become more popular.

So Brian said in the chat backing up the finance team can be very helpful because there's power there. I suspect going forward, you're going to have more and more finance teams looking to build out data folks under them.

Robert, I see you have your hand raised. Can I turn it over to you? Yeah. Sorry, my camera's off. I'm getting a bite to eat, but great stuff so far on that topic. And this is something I've been thinking about personally. What should data scientists or really just anyone who doesn't work in a financial function, what are some basic things that they should understand about finance within a business or assets that they should learn how to comfortably kind of look at?

Yeah, that's a great question, Robert. I would say the basics will get you a long way. So revenue, cost, and then profit, which is sort of revenue minus cost. And then specifically, how does your company generate those revenues? How are those costs generated against those revenues? And then what are the ways in which your company is looking to sort of change over time or improve over time to improve those things? And then the last question that you can answer, right, is how can I use data to do that? And then that's you kind of doing the job.

But it is interesting how many folks you talk to with a very strong data skill set who, you know, as I mentioned earlier, like, how do you put a value on accuracy improvement? So they've built their whole careers around, I'm going to learn models that can improve,

you know, the accuracy over whatever historical thing you were using. And that's me doing a really good job. But in the eyes of the stakeholder, if there isn't someone to do the translation of, like, here's what those accuracy benefits mean in terms of dollars, and it doesn't have to be dollars today, it could be dollars tomorrow, it's really difficult to sort of make that case of what it's worth.

So those are the big ones, I would say. The thing I would add is, like, you don't need to go too far down the rabbit hole in terms of, like, for me, I guess it's on the margins, which is most important for us, right? So like the marginal revenue of doing something, the marginal cost, I have not spent a lot of time worrying about, like, how do you look at, like, headcount and finance against them? Like, it's just not really a part of the job. And I think you can skip over that because it's not something you would be usually applying, you know, your data prowess to, but often it's on the margin. So you sell one additional product, or in our case, you get someone to listen, you know, 10 additional hours on the ad supported tier, like, what does that work?

Ideal automation and institutional knowledge

Thank you, Jairus. Bruno, I see you have your hand raised as well.

Yeah, just to come back, second question. What difficult part of your role would you rather a computer do instead of you? So, you know, you allocate your time during the day, there's stuff that's enjoyable, there's other stuff that, you know, why I'm still doing this. So what is this part that the human process you find suboptimal?

Okay, and is this something like in an ideal world? Or like, this is the next thing I'd want someone to go out and develop? Because those are very different answers. No, no, it's, yeah, ideal world, like the job of the future. The job of the future.

What I would love for, and this is almost, this is goofy, but, you know, there's always more ideas than there is actual capacity to do work at an organization is what I've always found, you know, and everybody loves to bring ideas to the table and say, hey, that, oh, I mentioned that a while ago, you know, like, I'm so great. But there's not that many people can actually sit down and implement them, right? You always have more ideas.

So in an ideal world, I would love a little, like, email scraper to go through some of the suggestions I get and say something like, hey, we did that before and it didn't work. Or like, hey, we did that before and we're doing it right now. Like, that's not a new thing, it's already happening. Or that's a great idea, let's do that. And that that's a great idea will probably be 1% of the emails I get.

But yeah, you know, it's the value of institutional knowledge. And you can argue once, you know, a computer does that, I would not have a job. But it is frustrating to say, like, yeah, that's how our things work. Or no, like, that's not possible. That sort of, you know, sorting of all the various initiatives that we get pitched, that would be nice. And I think there may be some low-hanging fruit because the same ideas do get recycled every few years. And so to just say, like, hey, you know, it's almost a Slack bot. Like, we tried that. Like, check this wiki page on the results.

So in an ideal world, I would love a little, like, email scraper to go through some of the suggestions I get and say something like, hey, we did that before and it didn't work. Or like, hey, we did that before and we're doing it right now. Like, that's not a new thing, it's already happening. Or that's a great idea, let's do that. And that that's a great idea will probably be 1% of the emails I get.

I hope that wasn't too silly. I like that idea. Automatically taking it from your email, either putting it in your to-do list or not. Yeah, yeah. But especially for stuff that we've done or already tried, because people love to think that, like, oh, no one's thought of this before, I'm revolutionizing things. Like, no, we've done that before. It either worked and we continue to do it, or it didn't, and we're not going to do it again.

Hobbies, cooking, and data science thinking

Frank, I see you have your hand raised, too. Oh, you might be muted.

So, Jairus, I was just wondering, and maybe you mentioned this in your intro in the first five minutes. If I missed it, I apologize. But do you have hobbies or passions outside of your data science work at Pandora that you use for creative ideas or inspirational thinking or just ways of looking at the world that you then apply back into your work?

Yes, to the first half. I haven't quite figured out how I want to apply it yet, but I think the applications are there. And I guess I'll share a cute example of me doing an explore-exploit algorithm manually. So, I love cooking. And I'm cooking kind of nonstop. And so, I do it every day. Even if I'm really tired, I find energy for it. So, it's a very sticky hobby that I find really rewarding.

And there are a lot of cooking analogies about the data science process. But for me, I think what has been most useful about understanding data or maybe technology more broadly is there's all these sort of things in the cooking process that are kind of tweakable. And I personally think the modern way in which people interact with recipes doesn't really account for that. And so, what I've been playing on the side, and I don't know if this is a data thing or an app or something that I'm not smart enough to build, is how do you better represent that in a way that people who are either very good chefs or very good data people can better interact with the system?

So, as an example, I'll give you is like I've tried a bunch of different ways of making the same recipe. But when you go to a website, you just see the one that someone did that they considered to be best. And you don't really see information about how many things do they have to do to get here? And is there useful information about those things? Because if you read comments in recipes, a policy I have is read the most upvoted comment and do what that person says always. Your default should always be do what they say. So, classically, you don't need that step or that's too much salt. Do what they say and then adjust the recipe because the recipe that gets posted on the blog is often not final.

So, I like that a lot. And I'd love to just see the ecosystem get somewhere where I think it's accounting for the fact that everybody's kind of experimenting. So, in two years, if I'm working on a startup in stealth, that's what I'm doing. Now you know.

But yeah, it's a good question. I haven't been able to, I think, like draw tight enough connections with data. What I do recommend folks who are getting into this space to do is I say, you know, pick a project you're motivated about. And most people are motivated about their own data. So, the two areas I like to point them to are health and personal finance. Again, you have to have some prior interest in them. But those are great ones because it's like on the one hand, you know, do I want to muck around with an iris data set, even though I don't really know what an iris looks like? Or do I want to figure out where I'm overspending and can I predict future spend? Like, it's kind of, I think, a little more fun and exciting and tangible.

I like how you flipped around, right? You're like, well, I use all these data science ideas and how I think about the data science rather than apply it to my passions to get better at my passions. Yeah. So, maybe it's just like the overarching style of thinking. Yeah. Hugh says GitHub for recipes. Yes, Hugh. That's what I'm going for, basically. So, stay tuned.

Generalists vs. specialists

I see Libby had asked a question in the chat, too. That was, do you find it more beneficial to hire generalists or specialists when a team is first being built for a new project?

Yeah, good question, Libby. My experience is generalists. But again, that's me as a leader saying I know that I am often unable to predict what future needs of the team will be. And in order to accept that, you can't hire specialists because if they're specialized in an area that gets de-emphasized or something, they're not going to be as good of a fit as someone who's a generalist. So, my team hires generalists. Obviously, it's team dependent. But I mentioned before, right, we don't expect you to be 10 out of 10 in any specific thing, but you need to be like seven or eight on a bunch of things because you are going to get pulled in different directions and sort of have to do different parts of projects.

The science in data science

Awesome. Thank you. Rick, I see you have your hand raised as well. Can I turn it over to you? Thanks, Rachel. A really great conversation so far. I had a couple of points I wanted to mention, but one thing that I wanted to jump in on was this idea of the cooking and the data algorithms. And I'm a biologist, and we use the same analogy all the time that working in the laboratory and doing experiments and mixing reagents is just like being in the kitchen at home.

But I think the difference is that at home, when you cook, when you read a cookbook, people don't know why there is salt or what's the difference between baking powder and baking soda, for example, or how the humidity in the air affects what you're cooking. But in the laboratory, you know every single reagent and you know exactly how the concentration is going to affect the outcome of your experiment. So why do I put this much magnesium in? What is it doing? What is EDTA? And what are all these chemicals? And I think it's a similar thing with data scientists that they look at, OK, just get an answer and just focus on this metric and how everything is connected and the pulleys under the hood. Like, if I increase this, how does that affect the result? I don't think that's played. I mean, my impression, at least, is that I don't think it's been played enough around with when people are learning. They just learn the ropes and they don't learn to be playful and really experiment and see how those things do, how these affect the results.

So like when I teach stuff, I try and make a simulation, like make a shiny app that allows people to just simulate some data. And what if I had this in my data? What if I had this distribution? What if I change this parameter? How does that affect the output at the end to get a more intuitive feeling? Do you think that would be something that can work? I think it would work for cookie as well as well, actually. Yeah, that. So that's a great suggestion.

I have someone on my team who is really good at that style of thinking, and I am notoriously bad at it. And the reason I'm bad at it, and maybe it's a good thing, is that, like, if you chase down all those things, like many of them don't work or are not beneficial. And in my the way my mind works is like, oh, you're going to waste your time, like, you know, 2% chance that'll make sense. But I think it's really important to have folks on the team who do have those skills. So, like, we kind of compliment each other because she'll come up with a lot of suggestions and I'll say, OK, of the 40 you just said in two minutes, let's do like 10 of those. But these are the 10 I think I have the most faith in.

But I do think that playfulness is really important on the job or just the creativity in general. Because as you mentioned, right, you do see people who are just like, just get the metric, just get the answer. And I think if you're not understanding how it fits into the greater whole, right, again, understand the business, understand your role in the business and what you're trying to do, you're not you're not going to be successful. You know, again, I like analogies like, would you rather have a precise cupcake or a delicious cupcake? Right. I think I think we all know which one we prefer.

I think what Rick's getting at is that's the scientist part of data scientist. And we call them that for two reasons. There is an element of science and experimentation. And I think what's often left out of the picture is hypothesis testing. And, you know, everybody wants to throw out all the statistics and just go to some auto ML and be done with it. And it's kind of like, well, you got to have some reasonable hypothesis before you get started. Otherwise, you will, as you say, Jared, you're going to wind up going down these rabbit holes and two percent loops and everything else. And I think that's because my background is in chemistry, too. And I was involved in making materials and stuff. So it's just like cooking. There is that creative element. But it's the same thing with data science modeling. Like, you know, how do we get it a little bit better? Or what can we learn from this last model to make the next one better?

I know we're getting to the top of the hour here, but I love this discussion that's going on. So if anybody has time to stay on, definitely stay here with us. But, Jairus, I also want to be respectful of your time. And I don't know if you have a meeting to run to right after. I don't know either. I'm going to check real quick. But, yeah, if anyone has any other cooking or data related things they want to mention, let me just check real quick.

There's a few questions still coming through from Slido, too. So I'll just ask one quick one. Someone said, totally out of context, but would be a big help. Does Pandora sponsor work visas? I don't know. I'm sorry. I wish I did. I should have that information handy. But if you are interested in a role, feel free to reach out on the side. Yeah, it's on LinkedIn. I can look into it for you.

Pursuing passions and feeding the soul

Awesome. And, Hugh, I see you had also asked a question earlier, if you want to jump in and ask that one. Yeah, sure thing. So you had mentioned a little while ago about kind of chasing some of the stuff you're interested in, like cooking or at work, making time to focus on something you're interested in rather than the day-to-day work. I was just curious, have you noticed that paying longer term dividend or paying more dividends in the long term by kind of chasing your interest or your passion rather than that work that's right in front of you?

Yeah, so I guess in a world in which, like, I'm trading off, like, do I want to spend some time in the kitchen or do I want to do more studying or more work or something? Yeah, I would say, like, for my happiness and I think for everyone's happiness, I think it's important to do things that you care about and also recognize that even if you care very deeply about something, it may not be healthy to do it all the time. So there are folks I've seen. I have one friend in the data space who loves it, and it seems like if 100 percent of his time were dedicated to it, he would be the happiest clam in the entire ocean. But most people aren't like that, and I think that's fair to recognize.

You know, when I do cook, I'm not always doing it with the lens of, like, how does this help me in the data world? But I think I pull, like, sort of my knowledge of the data world to the way in which I approach my cooking, which is helpful. Yeah, at the end of the day, I would say feed your soul. And if you know what your soul wants, go do that. And if you don't know what your soul wants, try and figure it out. Experiment. But yeah, I don't think I could do the alternative. I'd be very unpleasant to be around.

Yeah, at the end of the day, I would say feed your soul. And if you know what your soul wants, go do that. And if you don't know what your soul wants, try and figure it out. Experiment.

Thank you. I was thinking more in the context of a work environment, too.

I'm supposed to be working on forecasts for something. You know, that's my long term. You know, that's my current task. But I'm really more interested in chasing deep diving into graph databases or something else where I have an interest. And really, you know, like, long term, that ends up, you know, paying much more dividends than if I just chased kind of the work that's directly in front of me.

Got it. Sorry, I didn't do a good job of asking earlier. No, no, of course. No, thank you. That's helpful. So I guess it's the context that like you want to do the stuff you're passionate about to eventually get like a role in that area.

Let's get a role in more, you know, chase your interest at work. And rather than necessarily just doing exactly that kind of this is exactly what I'm supposed to be doing. This is the task I'm assigned to. You have to do a certain amount of that. But exploring your interest as well. Long term and kind of chasing some of those things you're really curious or passionate about and bringing those into the work that you do to kind of see the payoff there. Or do you see a payoff?

Yeah, totally. So the way I've seen, I have not been great at this myself. But the way I've seen people who do this well is one, they do the core job and they do it well and they understand sort of what's me doing a good job or is a great job or is an amazing job. And they're generally smart enough people to pick where on that spectrum they want to be. So these are usually very bright people. And they say, I could do an amazing job, but I'm just going to do a good job. And then I'm going to invest that remaining 20% of my time or whatever it is into stuff I care about.

And I'm going to do it in a way that I can sort of spin to my manager or leadership is like this has potential. We should look into it. And when you do that and you're doing a good job, I found it sort of rare for someone to say like, no, I don't want you doing that, right? Because you might leave. And now they have someone not doing a good job. So I found that that's the way to sort of make it like politically palatable is to say like, look, I'm already doing all the stuff I need to do. But here's where I want to sink some extra time into.

And then you kind of coach it in a way that like is useful to them. So research and testing gets thrown around a lot. I want to research these techniques. I think they might be helpful. If you're doing a good job, it's usually easy to sell. If you're not doing a good job at your work, it's harder to say, well, I want to try this and I want to try that. So I found the folks who are really good are folks who, again, they're kind of picking the level of effort on the core responsibilities. And then they're just coming up with a decent pitch for the stuff they're passionate about and just tying it to the business in some way.

And then they're just coming up with a decent pitch for the stuff they're passionate about and just tying it to the business in some way.

Awesome. Thank you. That's incredible. We used to use the 80-20 rule at a place that I worked at. And I was able to sell it to the other management. It was like 80% of our work is going to keep the lights on and keep the business going. But 20% of the work is going to be on what is the next thing down the road that it's going to be. And understanding that that's a long-term investment.

And it could be as simple as like when I was working with some of my junior data scientists, I'm going to take a course in deep learning for this. Granted, we're not using that in the group today and everything else, but we might need it down the road. And that's an investment. And I think as a leader, you have to encourage that. And it can't be you only do it on nights and weekends on your own time. And so I used to have an expression is Friday is my day. And so I would pitch that to my team members. Friday is your day to do something new. You just need to tell me what you're doing. And obviously, we got to keep the lights on too. And you'll know that pretty quickly if the lights aren't being kept on.

But it's a way to orient people to grow themselves. Otherwise, it just never happens. Or you force people to do it on their own, which is now you're just putting all the learning on them. And as a leader or as an organization, you expect to take everything from them and not give them anything back. And that doesn't really help people grow either.

Yeah, that's great, Brian. I think formally carving it out. And I think that was also a Google thing, right? Like 20% time is the best way to go. I have not been successful in being able to do that. But I think it's the nature of sometimes we get requests that are urgent. And you can't say like, this is my time. But as a result, I think we've done a lot of the automation to make sort of what could take X hours is now maybe even X over two. And so that frees up a lot of space for other stuff that we work on.

And at the end of the day, I think the pitch is important. So you say, someone on my team wants to learn XYZ. Here's the ways in which it could be helpful. And again, if you're doing generally well in your role, leadership's usually happy to just say, yeah, go for it. Yep. And I understand you're in finance. And I've worked in finance and everything else. And the other way I tell people to look at it is you can't do that. Spend your first half hour of the day doing it. So five days a week, if you work from 8 to 8.30, you check your emails, make sure the building's not burning down, the whole organization isn't in flames. And then you work for a half hour on this little project. That's it. The end of a week, you've done two and a half hours. Or at the end of two weeks, you've done a lot. And then it's that little bit of work goes a long way over a period of time too. And you're not really interfering with the works at all.

And I try to tell people to do that also. If you can't, if we don't have time or anything else, do that. That's the first thing you should do before you even do anything else is grow a little bit. Yeah, I love that. I love putting it first too.

Yeah, I used to have other hobbies that I would try to do, mostly failing. But I would try to do it at the beginning of the day because you always have energy at the beginning of the day.

Closing questions

Jairus, I asked you if you had a meeting to go to, and then I did a bad job of listening for the answer of that. Do you have time for one more question? I can do one more question. Okay. I saw that I missed the question from Chris earlier. Sorry about that. But it was when we were talking about modeling. It was, what are you using for your modeling today?

Sorry, can you repeat? It was, what are you using for your modeling? Is that like- Is this around packages or- Yeah. Yeah. Gotcha. Forecast is what we use for a lot of stuff. We've tried to implement profit as well with mixed success. Again, we're doing a lot of time series work. So those are the ones we generally rely on. And then for plotting, I think we're still doing ggplot for a lot of stuff. And honestly, this is probably a better question for my team because I have not been in the code as much as of late. But yeah, that, then BigQuery connectors as well. But yeah, forecast is the one we rely on. And I see Steven saying, yeah, profit is not great. It looks like he had a sort of similar experience. We brought it online for some questions and it did not deliver as promised. But forecast, and then again, human in the loop, you know, where necessary.

Profit gets very hard on data science LinkedIn. I'm just reading the comments. Yeah. So forecast is the one we've had the most success with.

Awesome. Thank you so much, Jairus. This has been so fun. I really appreciate your time and sharing your insights with all of us. Yeah. Thanks for having me and thanks for showing up, everyone. It was really great to see you and chat with you. It's been a lot of fun. Yeah. And if anyone has follow-up questions for you or what's the best way to get in contact? Is it LinkedIn? Yeah, I think LinkedIn or Twitter. Yeah, those work. Awesome. Thank you all. Have a great rest of the day. Hopefully see you next week. Bye.