image: thumbnail.jpg
From Physics PhD to MLOps builder - Julia Silge - The Data Scientist Show #087
Daliana's Twitter: https://twitter.com/DalianaLiu
Daliana’s LinkedIn: https://www.linkedin.com/in/dalianaliu/
Julia’s LinkedIn: https://www.linkedin.com/in/juliasilge/
Julia’s Website: https://juliasilge.com/
00:00:00 Introduction
00:00:51 Getting into data science
00:04:45 Transition from data centers to engineering manager
00:13:59 Common challenges in tool development
00:17:33 Challenges with survey data
00:26:42 Engineering skills for data scientists
00:28:54 Balancing roles
00:34:44 Developing skills in Exploratory Data Analysis (EDA)
00:39:14 Python vs. R for data analysis
00:44:35 Exciting aspects in career and personal life
Transcript#
This transcript was generated automatically and may contain errors.
Hello everyone, welcome to The Data Scientist Show. Today we have Julia Silge. Julia is a data scientist and engineering manager at PositPBC, formerly known as RStudio, where she leads a team of developers building fluent, cohesive, open-source software for machine learning and MLOps. Before Posit, she finished a PhD in astrophysics, worked for several years in the non-profit space and was a data scientist at Stack Overflow, where some of her most public work involved the annual developer survey.
Today we'll talk about MLOps tools, challenges in survey data, text analysis and balancing her interest in data science and engineering. If you like the show, subscribe to the channel, leave a comment and give me a five-star review. Welcome to the show, Julia. Thank you for having me. I'm really glad to be here.
Getting into data science
So Julia, how did you get into data science? Great, like you mentioned, my academic background is in physics and astronomy, and I was working in research and was realizing that the academic world was not gonna be where it was for me, not as fit for the long-term. My path at that point was a bit circuitous. I worked for an ed-tech startup. I actually was a stay-at-home mom for a few years, but eventually I started to see some people who were similar to me in background, like with this physics and astronomy, kind of making a transition into data science.
And I thought, I wondered, I talked to them and I was like, wait a minute. So there's a job out there where what I do is make plots, talk to people about analytical results, analyze data, and that's your actual job. That was my favorite part of when I was doing astronomy. So I thought this is gonna be a pretty good fit for me, and I decided to try to make this transition.
I, at the time, was a little bit underemployed. I actually had been laid off just a few months before this, and I was employed as a contractor, and I took about six months, and I took basically every MOOC that exists, every massive online course that exists to really learn some of the modern data science languages. I had a pretty strong background and programming background from the work in physics and astronomy, but I had never taken any formal stats course.
I didn't know modern data science languages like Python, SQL, R. I didn't have machine learning experience because when I came up through physics and astronomy, it was not as common for people to use those kinds of machine learning tools. I took a bit of time to do a lot of self-study, and then as part of that sort of self-study to try to make this transition, I started writing a blog.
And my vision for this blog was that it would be projects that I could talk to people about during, say, interviews or for jobs, like job interviews, because my resume looked a little weird, and I thought I really need to have evidence that I can do this job, like evidence that I can do this job. It turned out that blogging opened huge doors for me, both in terms of jobs, but also with open source collaboration. I met people that I eventually wrote books with. It really was a big part of my transition was doing this public-facing kind of work.
It turned out that blogging opened huge doors for me, both in terms of jobs, but also with open source collaboration. I met people that I eventually wrote books with.
I did get a job. So the first job that I got was in the nonprofit space, which I think is a really interesting way for people who are transitioning in, because often nonprofits have quite a lot of data, but they don't have as many resources for how to best use that data, how to best take advantage of that. Of course, salaries are not high in the nonprofit space, but it was really a great place for me to get that first data science title, that first job where that was my title, and to demonstrate that I could do this in a real org.
So that's a little bit of how I got to that first kind of data science role. After that, I worked as a data science practitioner, like with the title data scientist, for several years at different kinds of orgs, moved from the nonprofit space into tech itself, and then moved to Posit about four years ago now.
Oh, nice. I definitely want to ask more about your blogs, and that's a great story to use as evidence to get jobs, and I also used to write a blog on Medium. It's really fun, and you can turn that into so many different type of formats, and people reach out to you for collaboration, and you would never imagine those opportunities if you don't put yourself out there.
From data scientist to engineering manager
So previously you were data scientist, now you are a engineering manager, and you're building machine learning tools, so how does that transition look like, and maybe also tell us what are you working on? So when my title was data scientist, I noticed that I would spend about 80% of my time doing data analysis, and I would spend about 20% of my time building tools. This could be like an internal tool at the organization where I was, like an internal package or library, or something to make the other people in my org more effective. It also was open source tools. I contributed to open source software.
I started pretty early, actually, in my transition into data science, and I noticed that I really am motivated by thinking about the systems people use to get their work done, the tools, like people's really practical, the nitty gritty of how do people do their jobs, like how do people do their jobs, and I was always interested in kind of that systems thinking like how do we go about making people more effective when they're doing text analysis, when they're building a machine learning model.
I was happy then, because I got to do data analysis, and then kind of 80-20 data analysis tool building. When I was kind of at a transition point looking for my next thing, I was really excited to see this job at Posit, but I knew it would be a change. I knew it would be a change, because basically it's that flipped. So now I would say I spend 80% of my time focused on tools for people who are doing data science, and maybe 20% of my time now actually doing data analysis itself.
One thing I notice about myself is that I do like to be involved in both. I really do like to be involved in both. I think if I ever had a job that was all one or all the other, I would not like that as much. It's been interesting to step back from a role where my primary role is doing data analysis or data science into a role where my primary focus is how can we build effective, fluent tools for people to be able to do the kind of tasks that they do.
So I like doing a little bit of both, and then now, so I started out when I joined Posit just focusing on tools for model development, like building machine learning models, and then transitioned into a leadership role and building tools for machine learning operations, what people call MLOps. So kind of that process of you've already trained a model, now what do you do? Like, congrats, you trained a model. What are you gonna do now? Like, how do you deploy that model? How do you version that model? How do you maintain that model in the long run? And know when it's time to retrain it. So that's the kind of work I do now, which has been, it's a really interesting space.
Vetiver: the MLOps tool
And like I said, I love thinking about the really practical parts of how people do their jobs, and that I get to think about that a lot now. Yeah, and for people who are mostly familiar with RStudio, the MLOps tool you're building is available in both R and Python. Yeah, that's right. So when I joined the company I'm at now, it was called RStudio. And a couple years ago, the company rebranded. So the new name of it is Posit, PositPBC. We're a public benefit corporation. And part of that rebranding was to clarify that we're not a company that just makes tools for R. We're like a data science company.
So we have tools, the two main languages that we support are Python and R, but also some Julia, Observable. Basically, tools that people use to deal with data. What are the languages? The name change, the rebranding, in some ways is bittersweet, because many people have such positive associations with RStudio. So it's bittersweet, but I think really important, because it more clearly communicates the kind of work that we do.
So the project that I work on now, it's called Vetiver. And if you are like a perfume person or a candle person, you may have heard that word, Vetiver. So it's a stabilizing ingredient in perfumery. So the idea is like in a perfume, you might have like very, lots of volatile kind of fragrances, but Vetiver is one that's a stabilizing ingredient. And so the metaphor here is that Vetiver, the project, is for stabilizing your models. For making your models might be these more, these more volatile things, but Vetiver helps you have confidence in the reliability of the model.
So it's a project for Python and for R. And what it focuses on is you have a model trained and what do you need to do to maintain it in production, reliably and efficiently. So the three main sort of tasks and approach are versioning your model. So that's, oh, I trained it this week and I got this result, but I trained it last week and I got these results. Like how do I reliably know where, or keep those models versions really organized in like a model registry. So versioning is the first one.
The second one is deploying. So that's that process of getting the model out of the computational environment where you developed it. Maybe think of that as like your laptop, or maybe you might have developed it in like a server kind of environment, but you developed it somewhere and then you need to lift it out with all the like computational pieces that it needs and you put it somewhere else so that it can be in a production environment where it's integrated into the infrastructure of your org. So picture that as like one way that you can do that is with Docker. There are other ways as well that Vetiver supports, but that might be a good way to think of it.
So instead of it's on my laptop, you're like, ah, it's in a Docker container and is ready to make predictions because we have captured all the software requirements. So versioning is the first piece. So think of like a model registry. Deploying is the second piece. So think about getting it off your laptop, maybe like into a Docker container. And then the last piece is model monitoring. So this is, the model is in production. It's serving predictions in whatever is appropriate for your organization. And the, but you need to know, is the model performing as you expect it to? Is, do you, is it starting to degrade? Is it starting, are you having that sort of model drift so that you need to address it? So that third piece is the monitoring and maintenance piece.
So it's pretty interesting to work on it because it is, it's a space where there's a lot of uncertainty about what are the approach, what are the approaches that are important and what are the concerns that people have? And so it's been really interesting to be able to work on these tools, learn how to make, for example, like how can I make it, how can my team make it just as easy as possible for someone to put their model in a Docker container? Even someone who maybe isn't quite used to Docker yet, like how can we make it as easy as possible for them to take that next step?
Yeah, I like the perfume metaphor. By the way, you smell great, Julia. Thank you. Part of the reason why this ended up with the name is because I am actually into perfume and candles and stuff like that. So I love the metaphor, but thank you. Thank you so much.
Understanding user needs for MLOps tooling
Yeah, when you say you want to make this tool easy for people who say even they're just new to Docker, so how do you connect with a user to identify their struggles? Do you like interview them or do you read some surveys? Yeah, so before we started deciding exactly what would be the shape of the MLOps tool we would build, I did a round of user interviews. I went to both people who were customers of Posit, so these are people who use our professional products, and also people in the open source ecosystem, people who maybe are users of Scikit-learn and our tidy models, or people who are more, who are not our customers. So I did rounds of user interviews and I had an organized, like I designed an interview so that we could really understand what do people like about the tools they're trying to use now? What are the deficits that exist? For the kind of people who we know are our kind of core user persona, what do they need, what do they need?
I also did quite a bit of reading, actually. There's some really interesting, a growing body of interesting research on MLOps and what it is. There's a couple really great books that came out recently. Chip Wynn's book is one that's so good. Some really interesting papers. There's one that was like an interview study and published like what is MLOps and how is it different from maybe regular software engineering? So those were the kind of things that I used to decide what to build, would be the nature of what to build.
Can you give us some example? What are some common challenges people face or you mentioned how is it different from regular software engineer or DevOps? Yeah, one thing that I feel like was just hammered home to me, both through my own interviews and through the reading that I did, in many organizations, the people who are responsible for this are the people who are also developing the model. So in many places, if you're someone who develops a model, and by that I mean that process of hyperparameter tuning, evaluation, like how do I train the model? They are often the people who also are responsible for operationalizing that model. But many of the tools that are out there for MLOps are really built with a software engineer user in mind, and don't so much really acknowledge the iterative exploratory nature of what data science is like,
what machine learning is like. The paper, we can add it to your show notes, I think that was one of the big takeaways from it, from this interview study that was done of people who do MLOps. It's like a big difference between what you might call regular software engineering and the process of MLOps, is that it is more iterative and interactive in nature.
And so you have to build tools for that kind of person, for that kind of person, that kind of user persona. Another, just as a really concrete example, if you have an app, like just a software app that does not have machine learning in it, you, and someone talks about the performance of it, they're probably talking about things like latency, how fast does the app run, how much RAM does the app use?
But when you're talking about a machine learning app, an API that serves a model or something like that, you do have to think about all those things. You have to think about latency, you have to think about how much memory it uses, how fast it returns predictions, but you also have to think about performance in terms of the statistical characteristics of the model. And because people who do machine learning, who do MLOps, have to think about both of those things, the tools that they need to use, forefront those concerns, have to really, have to prioritize that sort of whole universe of the things that people have to think about when it comes to machine learning.
It has been interesting to notice who has responded well to our tools that we've been building, who thinks it's a good fit. And it is usually people who have the data experience and are needing to take their model the last mile. It's like they need to get it the last mile so that it is useful to their organization. And that's the user we think of when we build this.
Challenges of survey data
Yeah, and you mentioned you read some surveys to learn about user frustration, and you also worked on survey data yourself while you were at Stack Overflow. I remember reading those surveys published by Stack Overflow, it's very interesting to learn developers' habits and patterns. And when we think about survey data, it seems very simple, you just aggregate them. But we know you have data from different sources, the formats are different, you're dealing with missing data. Can you tell us what are the challenges when you work with survey data, and how do you get an interesting story that's also accurate?
Yeah, it is, so working on the Stack Overflow developer survey was honestly quite stressful at the time, partly because it was fairly public. It was fairly public. People would pay quite a bit of attention to it, and it was enormous. On the years that I worked on the survey, it would have on the order of 70,000 responses, 90,000 responses, and so it's an online opt-in survey.
So in this way, it's different from surveys that you might hear about through a government statistical organizations, where they want a sample that is reflective of the population of that country, and they work very hard so that the people who take the survey are, they know how to map from the responses they get to the population. Now, there's no census out there of developers. No one knows actually how many there are, or what their demographic characteristics are like, and when you have an online opt-in survey, one of the aspects of it that I found most challenging was how to accurately communicate about the nature of those results, and this is something that Stack Overflow got a fair amount of criticism for over the year, and some of that I think was quite fair.
So a big piece of this would be gender representation in that survey. So I don't remember the exact numbers, but you would see when I would work on that survey, the first year that I worked on that survey, it was 5% of the respondents were women and gender minorities, very low, and so when you would, if you are a woman in tech and you see that, you think, this is not representing me, like this survey is not representing me.
So one of my priorities when I worked on the survey, first of all, we invested in trying to increase that proportion. We would go to groups like PyLadies, we would go to groups that were, you would find women in tech and say, hey, can you share this with your members, like in your Slack, so that, because we want to hear, in this example, like we want to hear from women and gender minorities in tech, and we did, we bumped up the proportion, like we doubled it, we doubled the proportion of respondents but the thing is, we don't actually know what the real answer is, we don't actually know.
You can, in the US, you can get some data from the Bureau of Labor Statistics to try to, but even that, like how they define a software engineer or a developer isn't actually like a perfect mapping to probably have you and I understand what a, so it's actually kind of an open question. So the first piece is like data collection. How can we responsibly collect data? How can we try to make sure we're hearing from the important voices? And then the second piece, how do we analyze the data and how do we present the data?
So one thing we found, or one thing I really prioritized is that, I'm just gonna stick with this example, the gender example, although it applies in other ways as well, like other kind of characteristics of who people are, but I'm just, for this example, I'm gonna stick with the gender one. We would try to disaggregate results and talk about differences and say, oh, okay, we see that some questions on the Stack Overflow Developer Survey were about salary. So we're like, okay, can we talk about differences in reported salaries by gender? And is it the same for people who are backend developers versus DevOps versus data scientists? Are there gender differences across different experience levels?
Like this was actually something we did find. Women who were early in their careers, we didn't see differences in salary. So like young women are earning the same as young men who are in tech, but if you looked at, that was not true for very experienced women. So the most experienced women, like the women who are further in their careers, older, earned less than men with the same experience and the same kind of role. If you were like in IT versus data scientists, you know this. So if you controlled for those things, you would see, ah, we don't see a difference for early career people and we do see a difference for late career people.
Like this was actually something we did find. Women who were early in their careers, we didn't see differences in salary. So like young women are earning the same as young men who are in tech, but if you looked at, that was not true for very experienced women.
So I prioritized in the analysis trying to do the best we could given that we had an opt-in online survey, which is not representative of the population, partly because we didn't know. There was nowhere for us to go to get the right, what would we expect? What's the right proportion? We don't know. So we tried to prioritize understanding differences.
Yeah, I think that makes sense. For example, I know there are definitely more male developers than female, but you want to hear a female developer's voice, but you also cannot just rebalance that to 50-50, then that's not a fair representation. Although as a woman, we want to make sure our voice are heard. So yeah, I can see the challenges there. And I also like that when you break into the age group, you see the story.
Yeah. Yeah, so did you investigate, have some assumptions on the reason behind it? So I think, I certainly talk to people about it. So some people when they hear that result, they think, great, there was a problem in the past, but we don't have that problem now. That's what they say. That's some people when they hear that data, that's what they think. Okay, maybe 20 years ago, there was more unfairness in tech and now we're okay. Now we're okay, like the people who are doing this. So that's one possible answer.
Another possible reason that can contribute to this is that companies don't do a great job of supporting women in career growth, that women hit ceilings. And actually, I do see this happening, that women as they grow in their career, often get shunted to less technical roles, often get shunted to, oh, okay, I see that you're very good at talking to people. So let's put you in a management role. And if women, I think especially when women go to management roles too early in their career, it puts a ceiling on them because they are not perceived as having the technical credibility to be able to really grow a very long time in their career.
So the data that we had could not answer why, but there are different ways to think about it. Of course, it might not just be one thing. It might be multiple things. Maybe the situation is better today than 20 years ago. And maybe also companies don't do a great job of retaining women and supporting women in their career growth. It can both be true.
Technical challenges and serving results via API
Yeah, what are some other interesting challenges when you worked on the survey data? So another interesting, more technical challenge, like less sort of people problem and more technical problem is that at the time at Stack Overflow, the data science ecosystem was all like R and Python. And at the time Stack Overflow had no Python, no R in production anywhere. So I was doing the analysis with these data science tools, but we needed to get the results to production, to the website. And there was no way for me to use R, for me to use Python in that.
So Stack Overflow is a pretty interesting website. It is super high performance, like really fast website that has been carefully built with these really fast technologies. So Python's too slow, R's too slow. It's not a good fit. What I found was the best way for me to be able to deliver those results was to deliver those results in an API. So in R, you can use Plumber, in Python, you can use FastAPI to deliver like results from a situation like a computer over here that has R or Python available. And then you can publish an API that another, like you might think of it as a survey microservice maybe. And then the like high performance website can slurp up the results when it needs to, update it on whatever time period is appropriate and get those results in.
I did that actually with not only numbers, but also with plots. Like I would serve SVG plots in an API to get slurped up into the website. And so that was sometimes when people think about data science tools, like data science skills, you don't think as much about, can I serve my results in an API? But that is, I feel like someone who has like the data chops and then also is able to really back that up with some level of engineering chop. Sure, I can publish an API for you. Like, where do you want it? That is something that the survey particularly showed me how much that can scale your impact in an organization.
I feel like someone who has like the data chops and then also is able to really back that up with some level of engineering chop. Sure, I can publish an API for you. Like, where do you want it? That is something that the survey particularly showed me how much that can scale your impact in an organization.
Yeah, especially if you work for a startup, you don't have a full engineering team to support you. So if you can't get your results out by yourself, then nobody can see it, nobody can use it. So having that engineering skill is mostly, I don't think you have to have every piece of engineering skill map exactly to the software engineer skill set, but I would call it kind of like on demand.
Engineering skills for data scientists
What do you think about engineering skills for data scientists? So I think that there are some that are table stakes, table stakes, and I think those are things like in your language of choice, being very skilled and effective at managing the nitty gritty of installation that you are someone who can build, that you are someone that effectively manages the software dependencies of your project. Maybe that seems even like a funny thing to say, but I run into people who really struggle with this piece, who really struggle with understanding like dependency management of projects and how to manage that. And it is a whole skill, like it is a whole skill. So that one I think is like table stakes, being able to be competent and get, right? Like that's table stakes.
Then I think the things that will really, like if you can learn some combination of these to really push you forward, I think one of them is being someone who not only can manage dependencies, but who has packaging skill. So you can build a Python package, you can build an R package to package up code and be able to run it somewhere else so that you can then install it. So that, so not only can you manage it, but you can be the creator of it. So you are someone who understands REST APIs, like I just mentioned, like that's another really big piece. You are someone who maybe has some of the DevOps skills, like you understand Docker and when is appropriate to use Docker and how to use that. So these are skills that are not specifically about data, right? They're not specifically about statistics or machine learning or data analysis, but they're about how do you operationalize the data analysis that you're doing.
Data storyteller or tool builder?
And previously you were in data science and now you're an engineering manager and you talked about you like the type of roles when you can dabble a little bit of both. So which persona do you identify with more, the data storyteller or the tool builder?
Oh, that's a really tough question for me. I, that's a very interesting question because my day job pays me for the tool builder piece. Like that is what my primary professional, and that's what I'm paid for, I guess. That's what's in my job description is the tool builder piece. But I think that my primary, how I think about myself is still on the practitioner side, is still, oh, I'm someone who does data science. Like you introduced me as like data scientist and engineering manager, but I'm actually not a data scientist at my company. Like we have data scientists and I'm not one of them. Like I am- But isn't your soul, you identify with it. I do, no, I think that's true. And I think it affects how I decide to build tools.
I think the fact that that is my primary identity really informs the kinds of tools I wanna build, what I care about those tools, how much I care about documentation and communicating about those tools effectively. So that is a super tough question, but I think that it is most honest to say that I, my primary professional identity is still on the practitioner side.
Text analysis and the blog
Yeah, and you're still maintaining your blog. Yeah, I read your blog. You tell a great story by interesting visualizations. And I know you like literature. You're passionate about tax analysis. Can you share a few interesting stories from your blog?
Sure, I do. I always want to spend more time on my blog than I do, but it has been very valuable to maintain it, to keep it and to continue to invest in it as I'm able, partly because I do love that really applied practice side. And you're right. One of the things that I love the most is text as data. It's interesting in the era of LLMs to have so much more of the data science ecosystem thinking about text as data, but I love analyzing text, both from the very exploratory data analysis stage through to training models.
And I am interested in how these largest, biggest models are being trained and used. I will say, though, I spend most of my time a little bit lower in the hierarchy, like a little bit more towards exploratory work, visualization, and not those hard to interpret giant bot models, but models that are more transparent, that are more easy to interpret and know what they mean.
One of my recent blog posts was about Taylor Swift's lyrics. And so treating the lyrics of Taylor Swift as a text data set and it builds model with that data. And so the particular model is an unsupervised model called a topic model. And so topic models treat documents as mixtures of topics and each topic is a mixture of words. And it is, I feel like it's a really underrated tool for when you have a corpus of language, open-ended survey questions. There are many ways you can apply this, but in this like fun example, I looked at the lyrics of Taylor Swift and looked to see, so treated the different songs as different documents and then tried to understand how are these related to each other?
And if I remember correctly, like some of these results were like that the early albums of Taylor Swift have a lot in common in terms of the lyric content. If you look at the words that are there, like words about romance. And then if you look, reputation is really unique. It is lyrically unique compared to her other works, which makes sense. And then like Folklore and Evermore have a lot in common. They're like two of the albums that are the closest in terms of their lyric content, which definitely also makes sense. Those are albums that have a lot in common. So I love that as an example. I mean, those techniques can be applied in maybe more mundane or common ways, but it's really fun to look at something like Taylor Swift lyrics and do that. I actually did that the week after I went to see the movie. I went to see the concert film and it was like a little fun Taylor Swift moment for me.
Exploratory data analysis skills
Yeah, that's great. I think a lot of people don't know what kind of project they should start with. You can just pick something that interests you. Yeah, absolutely. And I think actually the most fun part lies in the exploratory analysis. There are a lot of low-hanging fruits. And if you want to get started on your public portfolio projects, you don't have to go into the very complicated machine learning model. Start with something, exploratory analysis, analyzing, I don't know, your favorite food or some, I think there's public data about Yelp stars and things like that.
And to add onto that, I think it is that actually those are incredibly valuable skills. If you're someone who's very quick and good at EDA, like those people are incredible. Like those people are very impactful in the organizations where they are, for sure. Because especially when it comes to analytics, the leaders need the insight fast. Being quick, being familiar with the tools and also sometimes you do need to put on like investigator hat to evaluate the definition of things. Those are important.
So how did you hone your skill set? How do you develop your skills in doing EDAs? Yeah, this is a very interesting question because it is exploratory in nature. It is kind of hard to say, what is the right thing to do? And when do you know you've done a good job? And when do you know you're done? When do you know you're done? It will never be done. You always have another question.
Yeah, so one of my favorite books that kind of approaches the process of this is an R-specific book, but I think that it, at least those first couple chapters would really benefit anyone, even if they use Python or another Julia or another language. So the book is called R for Data Science. It's by Hadley Wickham. And it is the first couple chapters talk about the iterative nature of say, you read data and you import data and then you do this process of visualize, summarize, model. And this process tends to loop back on itself multiple times where then, oh, I saw, oh, I see this result. Let me go and make another plot. Let me do another summarization. Let me build another maybe like simple model to learn what I need to, what I can about this data set.
And it is a matter of judgment and kind of taste. Like when you're done and when it's time to say communicate, when it's time to then say, okay, stakeholder, here is this result here. So I think what has helped me kind of develop instincts around that is knowing like being able to look at how people respond to what I give them and also to be sensitive to where people are coming from. Are we in a situation where someone wants like a ton of detail? Like they wanna see a lot or are we in a situation where someone wants some like top line TLDR kind of situation about what some results are. So I think practice, I think acknowledging that you do have to use your judgment in knowing when you're done, when something is done. And I think experience of seeing what has landed really well with people and when did people either want more or less.
Tools for presenting analysis
Yeah. For example, when you used to present your analysis to a leadership, do you have a process? Did you have your favorite toolkits? Yeah, I was a huge user of R Markdown at the time. And so what I would use today would be Quarto. So Quarto is like sort of a next generation iteration of what R Markdown is. And Quarto works for Python, R, Observable, like Julia. For that, I feel like what the superpower that gave me was the ability to quickly generate reproducible reports.
was the ability to quickly generate reproducible reports.
And so I would do things like write in like what would be like a Quarto doc now. So text code, making the plots, making the results. And then I would run the whole thing, render the whole thing to either a PDF or Google Docs. Or if I was doing a presentation, I would make slides and I could do it reproducibly. I never had to do, like the superpower it gave me is that I was never like copying and pasting stuff or showing someone outdated results. Like I had really reproducible practices.
The other thing that was like a tool that was like a superpower for me was Shiny. So Shiny is a tool for making interactive apps for Python and for R. Being someone who knew how to use Shiny, let me make, picture a interactive data app that a stakeholder could then come and they could- Drag and drop. Yeah, yeah, yeah. Shiny made me so effective. Shiny made me so effective. And it's funny, at the time I was not, like I did not work for my company now, but part of the reason I wanted to come to my company is because it was the company that made these tools. So I don't personally work on Quarto or Shiny, like I work on Bedivere, like I said, but I know coming from me now, it's sure you're gonna say that, but literally those were like my superpowers when I was a data science practitioner. Like being able to generate reproducible reports really quickly, being able to make interactive apps really quickly, like it made people view me as incredibly productive and helpful to the org.
Python vs R
Yeah. And you mentioned now Posit supports both Python and R packages and here's comes the hot topic. Okay. Python or R, which one should you use? Especially I think now with the MMOps, I don't think a lot of people use R. So what's your take on that?
Yeah, so I think that the right answer is the one that you are more productive in. If you are someone who is more productive in Python, then that is the right tool for you. And if you are someone, I think it depends on your own skills and what is a better fit for you. I do also think that sometimes the tooling is better in one than the other. Like a real strength of R is that every kind of statistical model that exists, there's an R package for that. And so if you have needs around, say like specific statistical needs around the kind of models that you build, or like often R is used a lot in situations where you have fairly sophisticated statistical needs. You can't just like brute force something. Like you need to be quite careful. That's why it's used a lot in clinical trials, why it's used a lot in like human, like research with like people, because you're looking for like subtle effects and you need to make sure your stats are rock solid.
And so if you are in that situation, you might want to consider, hey, can I make sure that this will work well? That I can use R in a way that is good. There are, of course, amazing tools in Python as well. I think one that I often will turn to Python for, just as an example, is spaCy. spaCy is a, like we talked about, I love text data. spaCy is a project and package for natural language processing in Python that is amazing. Like it's amazing, I love it. The people who built it did such a good job. I'm such a fan. And it is just for Python. And so that is something that I would turn to Python for to use the tool.
So I think the two things to think about are, which one are you better at? So for example, I am just way better with the tidyverse than I am with like pandas or pollers or any, like I'm just way faster. I'm just way faster. It matches how my brain thinks. And so if I need to do data munging, data EDA, if I need to do EDA, I'm gonna use the tidyverse because I'm so effective with it. So there's one piece that's like, what are you effective with? Use that. And then there's the other piece of the specific task you're doing, where are the best tools? Where are the best tools? And I will find myself picking up different tools depending on what it is that I want to do.
I do think people will, depending on companies and organizations, sometimes we'll get really locked in to one, like we are an all Python shop. We are an all R kind of shop. And so sometimes that also is part of the reality that like, oh, at this company, they've made a decision that all of our data science needs to be in Python. And so that's also sometimes a constraint that comes in is like where you are working, maybe you don't have complete freedom to decide what tool you are going to use.
So one of the things I'm really been happy about, I feel like lately is people acknowledging more these kinds of realities and not, I am not someone who is interested or willing to engage in one is good and one is bad. They're different. The trade-offs are different. And there are other really interesting projects out there too which we may all be wanting to use in five years. Julia hasn't, like some people who use Julia, love Julia, right? There's really cool stuff out there. It's been really interesting to see Mojo come out recently, this like new, really fast kind of language. So I think it's, I am very professionally associated with R but it is for sure not the only language I use.
And I think that it's good not to get, I think it's good to realize once you have learned one programming language is not that hard to learn another one. The things that are the hardest are the, at least this is my opinion, the things that are the hardest moving when you enter into a different language is usually not the syntax. It is usually what is that language's packaging ecosystem like? Yeah. And how is it different than the one you're used to? So I feel like that's the hardest piece usually is like wrapping your mind around a different set of constraints in terms of how packages are distributed and dependencies are managed.
It is usually what is that language's packaging ecosystem like? And how is it different than the one you're used to? So I feel like that's the hardest piece usually is like wrapping your mind around a different set of constraints in terms of how packages are distributed and dependencies are managed.
Yeah, I think that's a great point. A lot of times we're trapped in the grammar or how does Python count from zero and R from one? But it's a bigger problem, the ecosystem. And also I think another great point you brought up is you don't have to abandon the language you're most familiar with just because another language gets more popular. I think it does help you to be more productive, more collaborative with our team if you learn more popular language, in this case, maybe Python. But if R is your strength, you still can use it, do some quick analysis. And like you mentioned, I know for example, time series, causal inference, there are more packages in R.
Yeah, those are other areas where it has its strengths as well. Yeah, so before we wrap up, what are something you're excited about in your career or personal life this year? Oh yeah, that's a great question. One thing in my professional life I'm really excited about this year is that I'm keynoting at SciPy. Oh wow. Yeah, so this July I'm giving a keynote at SciPy and it is one of the themes at SciPy this year is scientific computing across different languages. And so I'm really excited to get to go
and talk about like my own experiences building projects across different languages. And what are things, they're less about a specific language and more about scientific computing generally. And then what are the things that come up that are different? Like when you're working with a Python user, when you're working with an R user, what are the things that actually do have to be different? So I'm super excited about giving that keynote. I'm super honored that they asked me. So that is one thing.
And then I guess another thing, this is also conference related. So my company is having its conference in August and I'm helping me and one of the people on my team, Isabel, is our teaching a workshop on using Vetiver. So it'll be the second time we've taught this workshop. And it's so fun to sit down with people in a room and help them get from a place where they have not done something before, like deployed a model and get them to that first deployment. It's just really exciting. So I'm looking forward to that as well.
Yeah, that's awesome. Looking forward as well. Thank you, Julia, for coming to the show today. Thank you so much for having me. It was a pleasure to chat with you.
