Resources
Apr 30, 2024 |
1h 26min
|
2.7k views
|

779: The Tidyverse of Essential R Libraries and their Python Analogues — with Dr. Hadley Wickham

#Tidyverse #RProgramming #RLibraries

Tidyverse, ggplot2, and the secret to a tech company’s longevity: Hadley Wickham talks to @JonKrohnLearns about Posit’s rebrand, Tidyverse and why it needs to be in every data scientist’s toolkit, and why getting your hands dirty with open-source projects can be so lucrative for your career.

This episode is brought to you by Intel and HPE Ezmeral Software (https://bit.ly/hpeintel). Interested in sponsoring a SuperDataScience Podcast episode? Visit https://passionfroot.me/superdatascience for sponsorship information.

In this episode you will learn:
• [00:00:00] Introduction
• [00:02:55] All about the Tidyverse
• [00:15:19] Hadley’s favorite R libraries
• [00:28:39] The goal of Posit
• [00:34:12] On bringing multiple programming languages together
• [00:50:19] The principles for a long-lasting tech company
• [00:53:34] How Hadley developed ggplot2
• [01:03:52] How to contribute to the open-source community

Additional materials: https://www.superdatascience.com/779

image: thumbnail.jpg

Transcript#

This transcript was generated automatically and may contain errors.

Do you personally encourage teams to be multilingual? Or do you often write in multiple different languages, maybe even within the same project?

I think most people tend to be like, you know, like 90% R or 90% Python. But in general, it's just better to be pragmatic. And I think that's one area where I think generative AI is like really interesting. It's really easy to translate between them. Maybe this means the barriers between them are going to erode a little bit more.

If we were going to have like a metric that me and my team are going to try and optimize, like I think the metric that I want to optimize is like the amount of time you spend in like flow states.

Dr. Hadley, welcome to the Super Data Science Podcast. It is a surreal experience for me to have you here. We have, I have seen you in person. In fact, we have actually seen each other in person, I'm sure of this. And so let me tell you these stories. We didn't, I didn't tell you this before we started recording. So you're just getting this on air.

Circa 2014, in New York at an O'Reilly Strata Hadoop World Conference, you did some kind of hands on training, it might have been a half day training, they use the the air, the flight times data set. Does that all track? Yeah. So I did that I was in the audience for that. And then a couple of years later, in 2016, at the joint statistical meetings in Chicago,

there was an announcement from RStudio, that Hadley Wickham would be at the RStudio booth at a certain window of time. And I walked by a couple times. And we made eye contact and you kind of smiled friendly. But I was too nervous to talk to you. I didn't know what to say.

You were at that time and still today, one of the most iconic people in data science to me. And I was just like, well, you know, there's no what do I do? What do I say? How do I introduce this? And so now I finally know what to say. I've got a question to ask you. Hadley, welcome to the super data science podcast. Where in the world are you calling in from? I'm calling in from Houston, Texas. Nice. It is truly such an honor to have you on the show. You were on the show in the past.

So four years ago, you were on the program. And that was specifically episode number 337. But at that time, our host was Kirill Aromenko. And the timestamp on this is pretty interesting, because that episode was published in February 2020. So it was a pre-pandemic world.

A very different, different time. Exactly. So all of that has passed and we're almost back to normal, except that so many data scientists are working from home. All right. So straight into the technical content. Hadley, if there's one word that is most associated with you, it's got to be tidy. In 2014, you wrote a highly cited paper called Tidy Data. And you're also an author of the popular tidyverse, a collection of packages that share a high level design philosophy.

Last but not least, you have been writing a book called Tidy Data Principles, which is to be completed next year in 2025. What does tidy mean in the context of data programming? And yeah, what's the guiding principle? That's definitely not a question that Chad GPT would have asked you.

What "tidy" means

Tidy to me is about having things that are kind of like well organized and like well broken down into kind of like little pieces that you can then like reassemble, like Legos. I think that's been a motivation for a lot of my work is like, how do you take some like big, maybe kind of vaguely ill-defined problem and then break it down into like concrete pieces that you can actually get stuck into?

And experiment with and play around with and iterate towards a final solution. All right. And yeah, so in your tidy data paper, you draw parallels between tidy data and the principles of relational databases, specifically Codd's relational algebra. What is Codd's relational algebra? Could you elaborate on how database design can benefit statisticians and data analysts in their work?

Yeah, so you can go and like look up on like Wikipedia or somewhere what Codd's relational like algorithm actually algebra actually is, but like I can never remember it, but I think there's a. I don't know, like I.

Yeah, it's one of those kind of very precise definitions where every like word makes sense individually, but when you string them together in a sentence, it's very hard to understand what it means. But I was kind of.

And I sort of got into like relational data because my dad had done a lot of database design for capturing data about cows in particular, and this idea about cows, yeah, for cattle breeding.

And this idea of like making sure that kind of each facts, each unique fact is recorded like once in a data set rather than having it like potentially like either split across multiple places or reported recorded in multiple different ways in different places.

And so I think like that, like the ideas of Codd's relational algebra are really important. Like you want to have your you know, you don't have inconsistencies in your data, but it's really difficult for folks who are not like trained in databases and computer science to get the idea of the algebra.

And so a lot of that, like the idea of the tidy data was like, how can I frame this in a way that like makes more sense to statisticians and data scientists and people working with data. And to me, like it's like this, like, it's just like you've got a rectangle and all tidy data really is.

We put the variables and the columns and you put the observations in the rows and you kind of wonder like that legitimately took me like eight years to figure out. It seems so simple in hindsight, but it's just one of those things that like once you figure it out and explain it to other people, it makes a lot of sense, but it takes a while to get there.

We put the variables and the columns and you put the observations in the rows and you kind of wonder like that legitimately took me like eight years to figure out. It seems so simple in hindsight, but it's just one of those things that like once you figure it out and explain it to other people, it makes a lot of sense, but it takes a while to get there.

It takes a while, even for me as my first time using the tidy data principles, which must have been many years ago now is probably a decade ago or more. But that first time kind of wrapping your head around shaping the data in this tidy way, it is so different from the way that we're typically taught in university.

And so therefore, that first time doing things in a tidy way, exactly as you described, where each piece of information is only replicated once as opposed to having a whole column where the variable name, let's say you have a binary outcome and every row of this giant table is like outcome zero or outcome one.

It's so wasteful, especially if that's like a string. It's incredibly inefficient. And so popping over to the tidy principles where that goes away is transformative. But it did it. I remember the first time trying to like melt data.

And I'm like, what? And even when I saw it the first time, it took me a while to figure out and I was kind of like, this isn't right. Like I needed to kind of, I felt like I needed to change it into the way that I'm used to, to even work with it.

Yeah, it's really, it's really interesting at the moment. I'm on the sort of run the program committee for Posit Comp. So we decide on like what talks we're going to have and how we're going to arrange them. And so the program committee is mostly like data scientists.

And so the way we think we like, we do a lot of organizing data and like Google sheets because it's so easy to collaboratively edit, but it's always in like a tidy format.

So that, and what that typically means is you can't actually get a shape. You can't really understand the shape of the whole program without like joining three different things together.

And it's interesting, like when I share it with my colleagues in like marketing who have to then turn this into the website, like the way they want to put that in a Google sheet is just like so totally different. There's like merge cells, there's like colors. It's just like, I get it, but just like the, you could not like compute on data in that form, but it's so much easier to look at as a human.

Are you able to, when you're in a data science company like Posit, are you able to kind of dictate, you know, marketing, you've got to do things our way, the data scientist way, or is that, it's an, that's an impossible.

It's impossible. I mean, I, you know, I, I don't think I could dictate it. I could certainly do more to like try and persuade people or like, you know, help them do their jobs better, but it's like a lot of, a lot of work and you're, you know, fighting.

Like the fact that none of their tools think about data in this way. And I don't really want to have to go and create like the tidyverse for marketing and the tidyverse for finance and like every other, every other field.

We like, we do, I think we have like more, certainly more penetration of like Quarto and other kind of those, those sort of document generation tools, but there's still quite a high bar for folks who are like, if you're not used to using Git and GitHub, like collaborating in this way is pretty tough.

Even though that final product can be pretty nice. And that's, that's one of the things we, I sort of hope for the future of Quarto is like these tools for, you know, scientific documents that let you mingle text and code, but also can like work with Google dogs and multiple people can be like contributing to them.

You can comment on them. You can share them with non-technical folks.

Yeah. Quarto is something that we also have as a whole topic area later for discussion. It is a great tool that I think anybody can be using within data science or amongst data analysts, particularly.

Have there been persistent challenges that you faced, like technical hurdles or adoption resistant conceptual misunderstanding? Has there been any of that over the years?

With tidy data in particular, sorry.

Tidy data. Not too much. I think there's definitely some areas where there's just such strong conventions for the field of having things that I would think want to be in a single column spread across multiple columns.

There's some types of data where it's just like arranging things in non tidy forms. It's just much, much more memory efficient.

But like by and large, I think most people have like looked at the tidy data framework and even if they like don't agree with all the tools or don't use, I'll still find that like framing really, really useful.

And I think that thought of like, let's separate that out into a separate step rather than trying to make every other tool do a little bit of tidying just makes it makes life so, so much easier.

100%. And for people out there who haven't had the tidy experience, it is absolutely worth wrapping your brain around it. Because once you do get used to it, everything becomes so much easier.

And all of the all of the tools in your tidyverse work so seamlessly together. It's like, yeah, it's like, I don't know, when I was a kid, you know, you'd have some computer video games that were just garbage and buggy.

And you'd constantly like, you know, you'd be able to walk through walls or whatever. And then you get your for me, it was like having a Super Nintendo for the first time. And like nothing ever crashes and everything just works.

And the tidyverse is kind of like that you get the data formatted in that way. And you're just it's just smooth sailing.

Yeah, that's the I mean, we've we've one of the things we've sort of wondered about, like, if we're going to have like a metric that me and my teams would, we're going to try and optimize, like, I think the metric that I want to optimize is like the amount of time you spend in like, blow state, we are just like thinking of stuff you want to do with data, like code just kind of like flows out of your fingers.

And it all just works. And, you know, we're certainly still some distance from that perfect utopia. Over time, it really feels like the amount of time you can kind of stay stay in that flow state, stay answering asking questions of the data you actually care about, not like, how do I get this, this thing into this other function, so I can actually just do what I want to do.

Ready to master some of the most powerful machine learning tools used in business and in industry.

machine learning level two. Packed with over six hours of content and hands on exercises. This course will transform you into an expert in the ultra popular gradient boosting models, XGBoost, light GBM, and CatBoost. Tackle real world challenges and gain expertise in ensemble methods, decision trees and advanced techniques for solving complex regression and classification problems. Available exclusively at superdatascience.com. This course is your key to advancing your machine learning career.

Enroll now at superdatascience.com slash level two at superdatascience.com slash level and number two.

Exactly.

Making strategic choices explicit

On that note, in a recent article about strategies and function design, you have discussed the importance of making strategic choices explicit to users.

So hopefully this is ringing a bell.

Do you think that this kind of this kind of concept of providing of providing a clear presentation of strategies in the way that you're designing a tool or function that that actually enhances the user's experience or functionality maybe allows them to be in that flow state for more of the time that they're programming because they understand the thinking behind the strategic choices in the tools?

Yeah, I think so. And kind of framing that even like a little more broadly, like one of the things that me and my teams would think about and talk about quite a lot is like how much do you want to force you to learn some like new concept that's, you know, really going to that might really like a better mental model that's going to help you in the long run.

But until you learn what that thing is, it's going to like the code is going to be a bit of a mystery for you. And that kind of balance of like we want to we want you to like learn some new ideas like this idea of tidy data.

Like they're clearly like this is big, pretty clearly a big payoff of getting that concept into your head. And then versus other times, like, are we just teaching you some kind of like technical jargon that's like, you know, really useful for us, but maybe it's just more like junk to fill your brain up with.

So that's one of the things we're going to think about a lot, like how like what do we like? How much do we want to accommodate your existing mental model versus how much we want to give you a new and better mental model like either possibly kind of against your will.

Amongst all of the libraries that you've developed in the tidyverse. So there's things like reshape and plier for shaping the data and and being able to have pipelines of data within this tidy framework. Is there any particular library that is near and dear to your heart that you feel like this was one that either I don't know, conceptually developing it like you must kind of have favorites, right? Like they're not equal to you, right?

Yeah, I think I mean, I have to say one of my favorites is like dbplier, which allows you to write like R code, dplier code, and then translate that translates that automatically to SQL. And one of the things I kind of love about it is this sort of combination of like this really kind of deep technical knowledge of R that you need to make this work in terms of like translating the R code. And there's also kind of like there's no way to do it perfectly like you cannot translate perfectly every piece of R code to equivalent SQL. There's also this kind of like how do I carve out like the biggest benefit with the smallest amount of work and I think the combination of those two things is something I really enjoy.

It's kind of related to that like there's bits of the test that package like one of the one of the test that is a package for doing unit testing in R. And one of the things I worked on a couple of years ago was this package called Waldo, which is all about like concisely describing the difference between two objects. And that's kind of like a similar problem. Like you've got this like deep technical understanding of like the language and all the objects and like writing C code iterate through them. And then how do you like present that to the user in like a way that helps them see the differences as easily as possible. So that kind of tension there between like this again programming and like human psychology. I just find that like really interesting and fun to explore.

Yeah, it must be interesting to think yourself. Well, there's clearly this need to be addressed and it's a it's an impossible problem to solve perfectly. So how can we get most of the way there in a way that will satisfy most people?

And those are two libraries that I definitely need to spend more time with. I don't think I've used to be prior or test that actually. Despite my life, it's also just one of those things where I'm like, it seems like a miracle that it works so well. Like, I think the thing that's fascinating to me is that like it really reveals like for the kind of core data, this core like stuff you do to vectors and data science, like whether you're summarizing them or you're like filtering them. It's basically the same code and like R or Python or SQL or JavaScript. There's like pretty you can express pretty much the same things in every single language in a way that is surprising and interesting.

There's like pretty you can express pretty much the same things in every single language in a way that is surprising and interesting.

Shiny and reactive programming

That is cool. Beyond the libraries that we just mentioned in the tidyverse, one favorite of mine is shiny because it allows you to so rapidly build interactive Web applications for data analysis, especially compared to any other Web development framework that I've tried. And I don't have experience, much experience developing any kind of Web tools, but I can very easily use shiny to get a Web application up and running for people to be to have a self-service dashboard. That's a clicking point. Do you want to talk a bit about shiny? And actually, I think something about it is that it's now. And we're going to talk about positive soon and the change from RStudio. But this is something that works kind of across programming languages now.

Yeah, exactly. So there's now shiny for R and shiny for Python. And they, you know, they're completely separate code bases. But the idea that really unifies them is this idea of reactive programming. And the idea of reactive programming, I think, at its heart is pretty simple. You've got like a bunch of inputs to your app, things that people can change. And you've got a bunch of outputs. And what reactive programming does is just like automatically figures out what's the minimal amount of work to do when you change one of the inputs to update the needed outputs. And that again, that's that's that's one of these ideas like tidy data that takes your takes a little while to get your head around. It's like probably it's quite possibly an idea you've never encountered before in programming. It works a little bit differently to things you might have encountered. But like once you get that idea, it just gives you this incredible tool set to create apps that like where things just work. And you don't have to worry about like things either like updating too often doing a bunch of needless work and making your app too slow or just failing to update. And so you've got these mysterious like bugs in your app where things don't change. And you expect them to, which is like one of the most frustrating things to try and debug when something doesn't happen.

That that is fun. Yeah. So, yeah, shiny. Really, really, really cool. It allows you to spin up basically like that Super Nintendo game that I was just describing. It just kind of works like you think it should. People don't walk through walls accidentally as they're using your your dashboard that you developed in literally minutes.

Yeah, it's funny. Like, I remember talking to Joe Chang, who wrote shiny like very early on. And I was like, Joey, you think I use this like website like you can use like use Ruby for that. Use PHP for that. Like why on earth would a data scientist want to make a website? And now it's like so obvious because you don't want to give decision makers in the organization just like a PDF. You want to give them like a little interactive app. And there's just been so many examples of people just like really impressing their bosses with shiny. Because you can like whip up something in a couple of hours that looks like a polished app. It does exactly what you want. I remember a very early phone call from a shiny user saying like we saved him a quarter of a million dollars. Because instead of going and like finding a like a contractor to implement a web app and a dashboard, he just did it himself over a weekend. And that not only is that a cost and time benefit, but also that like if you as a data scientist can do it yourself, you don't have to try and communicate to someone else exactly what you want.

That is tough. It is. Well, and this also allows you to make changes yourself. You know, if if you notice an issue or a user complains to you, you can just go in and fix it as opposed to needing to be a middleman, a middle person, I guess.

Yeah, I think one of the interesting things about dashboards is like if your dashboard is successful, like people are going to demand changes to it like very, very quickly. But if you have a really, really good dashboard, that means like there's going to be like two or three execs in your company who now want to like make a bunch of tweaks to it. And if that's like some weeks long process where you've got to figure it out and communicate to some like web engineering team that just like kills the whole thing. And I think with how often executives think they want a dashboard and then relate it to how often they actually use it. That is another strong point for using shiny because you're not wasting weeks or months developing a dashboard. You're you're wasting hours of days. I mean, just in general, that whole iteration, the more you can do to like increase your iteration speed, like the more effective it makes you. Because, again, like it's so hard to predict in advance, like what's the thing that's going to be valuable? There's definitely a lot to be said to just like trying out a ton of things and seeing what sticks rather than like doing a bunch of upfront planning and just hoping desperately that you've got a really good mental model of the world and your idea works.

Why use R today

So we are going to talk about, as I already mentioned, the positive name change. We'll end up talking about Python a bit for our listeners. If there are listeners out there who don't already use R, why should they be using it for me? I can actually give one example, which is for me, for data visualizations. I still find I can do things way more quickly, have much more fun making visualizations in R and get exactly what I want. There had been in the past attempts to create a ggplot2 style Python library, but the one that I had been using became deprecated and harder and harder to use. It never had all the functionality of your ggplot2 anyway. Anyway, so that's like my big example. I don't know if you have big examples of why people might want to use R still today.

On the topic of ggplot2 specifically, I think the best Python equivalent is Plot9. That's actually by a developer, Hasan Kabiraji, that we've been sponsoring at RStudio. I think that's the best possible realization of ggplot2 you can get in Python.

But I think there's things about the design of the R language that just make certain tasks much easier and more natural to express in R code than you'll ever be able to do in Python. I think that comes down to at the heart of it, R is more of a special purpose programming language. It's designed from the ground up to support statistics and data science. I think that has a lot of benefits, particularly if you've never programmed before. I think you can get up and running in R using R to do data science. You can do that without learning a ton of programming. You can get up and running pretty quickly. Then there's just things about the language that other languages look at R and they're like, oh my god, that's a terrible idea or that makes me want to throw up in my mouth. There's just so many things that are so well placed to support interactive data science where you really want that fast and fluid cycle where you're trying things out. That obviously lends to maybe a little bit of weaknesses. I'm like, now I've got this thing, I just wanted to do the same thing again and again and again and again. R tends to be a little bit magical. It tries to guess a little bit more of what you want and that's great when you're working interactively and it guesses correctly. It's not so great when you're working on a server somewhere else and it guesses the wrong thing. Everything about R I think makes it such a fluid environment for really exploring your data, digging into it, figuring out what's going on.

The hybrid cloud promises freedom, but for AI and analytics, it can feel like juggling chainsaws. Data silos hold you back, tool sets clash, and managing resources across environments becomes a nightmare. This is where HPE Ezmeral software steps in. On May 2nd, join a free webinar powered by Intel Xeon scalable processors and HP Enterprise designed for data science professionals facing these hybrid cloud AI challenges. You'll learn how to empower AI model building and training with seamless access to global data sets, leverage built-in connectors and a curated open source ecosystem to focus on innovation, and trade in Frankenstein infrastructure for consistent environments that simplify resource management and accelerate any AI journey. See the show notes for the link to this free webinar on May 2nd.

R vs Python and the Posit rebrand

Speaking of differences between R and Python, I seem to remember, and you can correct me if I'm wrong about this, but I feel like you have a famous tweet from years ago where you say, somebody says something like, and it must have been a famous poster themself that you responded to, and I can't remember, it might have been like Wes McKinney or somebody like that saying that one of the advantages of Python is that it's faster than R. And then you have this super famous reply of, what is that, and I will make it faster. Do you know what I'm talking about?

I don't, but I know I've heard things like that in the past. Yeah, it's kind of a misperception because Python isn't actually that fast itself. I mean, languages like Julia have come up to be faster than Python.

Yeah, I think one of the reasons, often the biggest, you have the worst arguments with your family and not with strangers. With people who are so similar to you, you tend to have more friction than with people who are really different. I think because R and Python are actually really close together in the spectrum of programming languages. It's so easy to see all of the little things that look weird to you as opposed to looking at some programming language that's miles away, and it just looks totally different. You can't, I just think that, I don't know, I think there's something to that. It's because we're close, you can see these little noises. Certainly, when I see things in Python that people are like, wow, that's really cool. I'm like, challenge accepted. I will make that better in R.

Yeah, exactly. Let's dig into that a bit now. For 11 years, you've been the chief scientist at Posit, makers of open-source software for data science, scientific research, technical communication communities. Many R users will know Posit as the makers of RStudio, a full-featured integrated development environment IDE for R, which I myself have been using for as long as I can remember. Basically, as long as I have been typing, I have been using RStudio.

RStudio, as you actually kind of let slip earlier in this episode when you were talking about Joe Chang, I think. Oh, no, no, no. You were talking about plot nine and how you were like, RStudio is supporting. Wait, no, Posit. That's two years ago, the company name has changed to Posit. From a distance, I mean, I don't even think it's from a distance. I think this is explicitly related to how Posit is now supporting more than just R. Is that right?

Yeah, yeah. I mean, the goal of RStudio and now Posit has always been to be this kind of like a company with a long-term vision. Like we talk internally about this sort of idea of a hundred-year company. And when you think about a company like that, like obviously no programming language is going to be around in a hundred years' time. When we started with R, that's something that's near and dear to many of our hearts. It will always be. But we also want it in the name of our company to like embrace that there are now other languages and there will be even more languages in the future.

And I kind of think about this as like the Burlington Coat Factory problem. I don't know if you know Burlington Coat Factory, but we have a lot of ads for them on television. But for a long time, they were like, no, it's like Burlington Coat Factory. It's not just coats. And for us to go into customers and say like, buy RStudio. It's not just R. It's hard to make that story. So I really, really wanted to say like, hey, for a long time now, like our products have supported not just R but Python and Julia and other tools. We don't want to lock ourselves into this. We're going to be R forever regardless of what happens with the rest of the world.

So renaming deposit, rebranding deposit was really about saying like we're in this for the long haul. And we care about data scientists regardless of what tool they're using.

Piping in R vs. method chaining in Python

One of my favorite things that you can do really well thanks to the dplyr library that you led development of is piping. And so you can extremely easily have functions pass it. I mean, just like if people are familiar with Unix programming pipes there where you have output from one function goes the input to a next function. And prior to me discovering dplyr, which was probably around 2010. Does that make sense? Prior to that, I would have so many variables in my workspace. It was just such a pain to keep them all straight. And you just end up in these weird situations where like should I be investing time thinking about the name of this intermediate variable? Am I going to use this later? Or should I just name it like intermediate variable 15 and have really ugly code?

So piping gets rid of all that where you can read the flows like a sentence. You're like, okay, this preprocessing step happens, then this next, and you can just see it so easily it makes it so elegant to read. Do you think we'll get to a point where and I have used some kinds of piping attempts in Python, but my experience of that has never been. And I guess it's been a few years since I've tried. Yeah. But it seems like it's never been as smooth or as easy as with R. And maybe that's related to what you were talking about earlier with data visualization.

So one, kind of like the native equivalent of piping in Python is like method chaining. You know, like if you're using pandas, you do something dot something. Yeah, pandas does it. Dot something. But the big difference between like method chaining and the pipe is in method chaining, like all of those methods have to come from the same class. They have to live in the same library, the same package. Whereas with piping, they can come from any package.

And I think the thing that's really interesting about that is that that is meant like Python is tended to have like these fewer, bigger packages like pandas and scikit-learn, matplotlib, like kind of everything in order to work with method chaining, like everything has to be glommed into this one giant package. Where with R, you know, because you can combine things from different packages, like the equivalent of pandas is kind of like dplyr and tidyr and readr and a bunch of other things. It's way easier to add extensions to ggplot2 than matplotlib that work exactly the same way because you can just combine them with different pieces. So I think this is one of these like sort of interesting, like subtle differences in language design that leads to, you know, fairly big impacts on the user experience and kind of almost even how the community has to work together and form.

And I think the thing that's really interesting about that is that that is meant like Python is tended to have like these fewer, bigger packages like pandas and scikit-learn, matplotlib, like kind of everything in order to work with method chaining, like everything has to be glommed into this one giant package. Where with R, you know, because you can combine things from different packages, like the equivalent of pandas is kind of like dplyr and tidyr and readr and a bunch of other things.

Yeah, it makes perfect sense. And it's actually your explanation is so simple for how that's happened. And that had kind of escaped my attention as to why it works so well in R.

Marrying Python and R — progress four years on

When you were last on this podcast four years ago, you said that you wanted to marry the Python and R languages. Four years on, how do you assess the progress made in achieving this dream, especially through projects like Apache Arrow? Yeah, I think we've come a long, a long way. And like Arrow has made a big difference in just being able to like seamlessly move data from one platform to another, you know, one programming language to another. I think the other, and then kind of coupled with that, I think the other technologies that's really, really interesting is DuckDB. You can, you know, you can use DuckDB from R, you can use it from Python and you don't have to have like a database file. You have like a directory full of parquet files. And it means that like, you know, people are using the same kinds of tools, just expressing them, you know, in the language that they feel most comfortable with.

Another, you know, sort of a similar thing is like Keras and a lot of the machine learning toolkits in Python. Like the reason that they are fast is not because like Python is fast. It's because you express those high level ideas in Python and then they get compiled down to some low level machine code. And that's why packages like the Keras package for R, which is maintained by one of my colleagues at Posit, Tomasz Komarski, like it does the same thing. Like you express these ideas in R rather than Python, but then it gets compiled down to machine code using exactly the same toolkit. So I think that like we're just going to continue to see more and more of that. Like R is not fast. Python is not particularly fast. What is fast is like people really caring about stuff and Rust and C. And then you write a more user friendly interface on top of that. So the programming language is the data scientists use every day.

And those libraries that you mentioned there are super cool. In addition to the arrow that I mentioned, speaking of Wes McKinney. And we actually have. So we have a whole episode about that. Let me quickly look up the number here. So back in episode number 523, we had Wes McKinney on and he talks about the Apache Arrow project at length. Really cool one. And the other projects you mentioned there, DuckDB as well as Keras for R. Yeah. Super cool. Invaluable packages that people should be trying out on the show for sure.

Arrow's also been top of mind for me lately because I've been working on some enhancements for the big query package, which allows you to get data from big query. And previously, the only way you could get or the way that the big R query package used is downloading the data as JSON. And JSON is like a great interchange format, but it is horrendously inefficient if you're sending like data frames of data. And so thanks to some folks who have been working on the big R query storage package, which talks to the Google API using Arrow instead of JSON. Like you can download data like an order or sometimes two orders of magnitude faster because you're using a data format like specifically designed for the type of data that, you know, data scientists care about.

And that's just like it's really it's kind of nice to see that like in practice and that the kind of dream of Arrow of like you've got data over here and you want to get it over here. Let's make it as easy as possible.

Being multilingual across programming languages

Speaking of multiple languages and interchanging between them, do you personally encourage teams to be multilingual or do you often write in multiple different languages, maybe even within the same project? How do you think about that in your own? I'm like 100. I'm 100% R and as far as I can tell, I probably always will be. Like that's my job. Like if there's something that I could do better in Python, like I will write an R package. But, you know, but that's not the reality of like most people's lives and most people's jobs. I think most people tend to be like, you know, like 90% R or 90% Python. But in general, it's better to be pragmatic if there is something that's way easier to do in another language like you can, you know, you can learn the basics of R, you can learn the basics of Python pretty quickly so you can use that tool.

And I think that's one area where I think generative AI is like really interesting, like just being able to like generate code in another language quickly. And like I use it, I've been using it quite a bit for to generate JavaScript because I do the occasional web thing. And I really like it because I like I know enough about JavaScript that I can kind of look at what it produces and say like that looks right. But for me to manually like figure out would just take like so much longer. And so I think that's really, I think it's really interesting to kind of think about how that's going to affect kind of programming languages. If it's really easy to translate between them, maybe this means the barriers between them are going to erode a little bit more.

Generative AI, language barriers, and the future of programming

Yeah, I think that's right. We haven't, it's amazing that we've gotten this far in the episode without talking about generative AI yet. That's kind of a nice refreshing and actually looking at all the topics we have lined up. I don't think any of it touched on gen AI, which is kind of crazy today. But as you were talking earlier about how in a hundred years, how Posit is aiming to be a hundred year company. And we think about what programming languages will be a hundred years from now. I was kind of just, that was batting around in my head, bouncing around in my head as you were speaking. How I wonder if a hundred years from now, anybody will be programming. Because I wonder if just natural language expression of things will be so powerful. I wonder if we'll be working at all. It's just going to be like a Mad Max hellscape.

I think the other thing that's really interesting to me, though, is like if people are really going to be using like generative AI for programming a lot. Like what does that mean for new programming languages, which are not going to have any training data available for them? Like that seems like that's going to kind of raise the barrier to new languages even further. Yeah, it's also just like what happens to stack overflow. And I always sort of like this idea of like poisoning the well. Like people are stopping using stack overflow, which is kind of like fine in the short term. But where's like all the training data going to come from in the future?

Like that seems like that's going to kind of raise the barrier to new languages even further. Yeah, it's also just like what happens to stack overflow. And I always sort of like this idea of like poisoning the well. Like people are stopping using stack overflow, which is kind of like fine in the short term. But where's like all the training data going to come from in the future?

It's exciting and scary. Yeah, it is exciting and scary. I have this perhaps completely unfounded intuition that somehow it's going. And there's people way smarter than me who have spent a lot of time thinking about this who could easily crush what I'm about to just say. You might even do it right now. But somehow I have this feeling, based on how quickly issues like hallucinations have been stamped out, the jump between GPT-3.5 and GPT-4 with how much less it's hallucinating, I have this completely unscientific, uneducated intuition that somehow we're going to be fine on this front, that like we're not going to end up having a complete like there's a specific term for this. You probably know. What is this? When is this? This concept of the degradation as we are like only have like synthetic data and then Jenny, I'm feeding into itself. Yeah, I know that. Yeah, there's there's a specific term and I've used it on this podcast before. If somebody can like send me a message on social media, I'd be like, you bonehead. Two weeks ago, you talked about this with this guest that's an expert in it.

But yeah, like that concept, it's something like like degeneration, that kind of sound, that kind of vibe. But I don't know. I somehow feel like we're going to figure it out, that like even though people will probably contribute to Stack Overflow less and less, that the quality of generation that comes out. And I guess part of maybe what gives me confidence is my recent experience with CLAWD3 from Anthropic, which I have found so amazing. I went overnight from using GPT-4 all the time every day to using CLAWD3 because everything is so sharp. And the interesting thing about that is they do tons of synthetic data generation, as well as reinforcement learning with AI feedback as opposed to human feedback. And so it seems to me like going in that direction is going to be OK. But anyway, I've gotten really off piece.

Yeah, I think the thing that makes me like less optimistic is like my Tesla and, you know, like this promise of like self-driving cars, which just doesn't seem to be getting any closer. It's like every, like, I don't know, 50% of the time that I pull into our garage, like it thinks the random collection of tools on the wall is like a semi. That's about it. So I'm just like, meh. And that's like clearly something that's been like very much hyped and a bunch of money has been. It's just gonna be interesting to see, like we're clearly in this like explosive growth. And is it just going to like flatten off or is it going to keep going? Or is it going to get steeper? Who knows?

Interesting. I don't know if you knew this or not, but that's called full self-driving. It can fully self-drive. I mean. I don't know if everyone got it, but we got the full self-driving for free for a month. So I've been trying it out and it's just like sometimes it's great. But a lot of times we're like, oh, my God, this is scary. Like I don't I don't understand how people get into these like accidents with the self-driving. Because I'm like, there's no way I would trust this without like keeping my hands are like white knuckle on the steering wheel because I'm not I don't trust it. Yeah. Interesting.

Since the start of April, I've been offering my machine learning foundations curriculum live online via a series of 14 training sessions within the O'Reilly platform. My curriculum provides all the foundational mathematical knowledge you need to understand contemporary machine learning applications, including deep learning, LLMs and AI in general. The linear algebra classes are wrapping up soon, but my calculus, probability, statistics and computer science classes are still to come. The first two calculus sessions are available for registration. Now we've got the links for you in the show notes and those will cover all the essential calculus you need for machine learning. Calculus level one will be on May 22nd. Calculus level two will be on June 5th.

And registration will open soon for calculus levels three and four, which will be on June 26th and July 10th. If you don't already have access to O'Reilly, you can get a free 30 day trial by our special code, which is also in the show notes.

Yeah, well, I guess back on topic. The last thing that we were kind of when we were on piece in the flow that I was in, we were talking about multilingual usage. Something this is we don't need to have a long discussion of this, but I suspect, Hadley, that some of our listeners out there are doing their data analysis primarily in Excel. And Python was recently integrated into Excel. Do you think that in the future, Microsoft might support R2 and Excel kind of environment?

Probably not. And I have a vague. No, I don't have a vague sense that someone told me the way that it actually works. And the way the Python works in Excel is like pretty unpleasant and pretty hacky. But I think it's true that like probably not like 90 percent of like data science is like still done in Excel. And that's just like a huge audience of people like going from like a no code to environment to a code first environment. I think like so many people will benefit from that.

And it's definitely like, you know, starting to code for the first time is scary and it can feel intimidating. I think that's another place where large language models are going to come to play and be really interesting. But just like the more we can do to help people switch from that like point and click environment to something that's more reproducible, something we've got code that other people can read and critique like that's that's really, really valuable.

S7 and object-oriented programming in R

So maybe we won't be going down the Excel route. But another big innovation for our that has actually happened recently is R7. Which I heard about for the first time doing the research for this episode or reading search our researchers research. Yeah. Do you want to tell us about R7 and the problems that it's aimed to address?

So we actually renamed it to S7 relatively recently. Oh, really? It's called S7. So it's called S7 because there are two. This is like a lot of historical minutiae. But R was like the language that came before R was called S. And S was the kind of introduced object-oriented programming. And S version 3 and S version 4. So in R there are two types of object-oriented programming. S3 and S4. Two chief types.

And the kind of idea of S7 or R7 was to try and like take the add those two things together and get the best of both worlds. So S3 is like really just a very lightweight set of conventions. Like it's not like object-oriented programming in any other language. Basically, it's very, very lightweight. S4 kind of like swings too far in the other direction. It's like very formal. There's a lot of kind of boilerplate. It's quite complicated. Things can go wrong in weird ways. So the idea of S7 was really to try and find like the sweet spot in between them. Like to take the nice features that S4 had, add them on top of S3 in a backward-compatible way so that we can hopefully switch. Hopefully, we're not just adding another object-oriented programming style to R, but we're actually supplanting S3 and S4 over time. Because everything you can do in those two, you can do in S7 and you can do it more easily. And there's better documentation and tooling and that kind of stuff.

For our listeners who don't have a computer science background, what does it mean for a language to be object-oriented? And that it can have these kinds of grades from lightweight, like you were describing with S3. Yeah, I don't know. I don't know. You have objects and you program with them. I don't know. And it's especially weird in R because when you're using R, you're not really aware that you're using object-oriented programming. Unlike in Python, where I think you're much more aware that you have objects and you call methods on those objects. Object-oriented programming is much, much less important in R as a data scientist. I think you benefit from it because packages use it.

So I think the main benefit to you as a data scientist is not that you're going to be writing S7 code. But the packages that you use are going to and they're going to be able to write code faster and more correctly from the get-go. So hopefully more of a general uplift of developer productivity in R. Probably not going to affect data scientists day-to-day that much.

Posit's philosophy and longevity

All right, cool. So speaking of switching between different levels of object-oriented programming. You've already mentioned on the show how Posit has an ambition to build a company, a suite of tools that could last 100 years. What kinds of principles or philosophies do you think are critical to creating a legacy that lasts that long in technology?

Yeah, that's a good question. I don't think we know for sure. But one of the things that make us different as a company is that we are a public benefit corp, a PBC or a B corp, rather than being an LLC. And what that means is kind of fundamentally baked into our charter is that our sole goal is not to optimize shareholder revenue, which is kind of the classic LLC model. We explicitly kind of consider other stakeholders like the community and our employees as what we're trying to do as a company.

And I think that is pretty special because we can say legitimately we don't want to make products that lock you in. We want to make products that like, you know, we want to help you and we're going to sell you products that are going to help you do your job. And you're going to hopefully pay us money for those because they save you time. They allow you to do things that you couldn't otherwise do. But we're not just about that money. Like we really care about your kind of life as a data scientist. We want to build tools for you. We want to build tools, open source tools for people that don't have a bunch of money. We want to improve academia. So I think that that's part of the mission is like we're not just we're not optimizing for short term profit. We can say like we're going to take a longer view.

And, you know, part of that is also like we're not we're not a VC driven company. We don't have to explode in either a good way or a bad way. In three years time where our investors like want to get money so that we can kind of say like, hey, we have got that. We've got the time. We've got the long. But the financial stability and the kind of vision to look out a little bit longer in the future and do the right thing rather than expedient things.

But the financial stability and the kind of vision to look out a little bit longer in the future and do the right thing rather than expedient things.

It is a super cool company. There are so many amazing people who work at Posit. So many of the biggest names in data science. Work alongside you at Posit. And on top of that, you know, I know some of the investors that have invested in Posit and they're so excited about the mission that you guys have. It's an incredible company and it would not surprise me.

Another thing that I think that's been really neat is like there's also a bunch of like people at Posit that, you know, you don't kind of see from the outside who are like fantastic. Like in finance and sales and people ops that are explicitly like we couldn't afford them like usually. But they're drawn to this mission like they've done their time in corporate America. They now don't just, you know, they don't just want to work at a company that cares about itself and is optimizing its bottom line. But they want to work at a company that like is trying to do good in the world in some way. And that's that's really, really cool.

ggplot2 and the grammar of graphics

No doubt. Switching gears a bit here from Posit in general back to a topic that I assured our listeners we would get back to later on in the episode. You created ggplot2, which is to this day the most popular data visualization library for R. It's had nearly 140 million downloads at the time of recording, which is crazy. And you have an associated book, ggplot2 elegant graphics for data analysis, which is super popular. And so my understanding from using ggplot2 now for over a decade, if I remember this correctly, you can tell me what I get wrong here or elaborate on a bit. But it's based on a grammar of graphics principle, which I guess was a book. And that was like that didn't necessarily have anything to do with programming. It was just a way of expressing visual information. And you picked that up and you converted it into a programming framework.

I mean, the grammar of graphics, the book by Lee Wilkinson really felt like it was a book like written for me because I read it. I was like, wow, this is how I want to be able to think about visualizations and describe visualizations with code. And, you know, that's that's what I attempted to do, like turn the book into something that you could actually use for free with with open source software. If you go back and read the book, there's there's plenty of like deviations and places where maybe ggplot2 didn't capture the full spirit of it. But overall, I think it's a pretty a pretty high fidelity implementation.

And just that idea of like pulling apart these pieces, like not thinking about this kind of like list of graphics, like a pie chart, a line chart, a bar chart. But thinking about these components that underlie the graphic, like the data and the scales and the geoms and the stats, like just a really, really powerful way to think about visualizations that allows you to create the thing that's like tailored specifically for your problem that maybe no one has ever created before.

And just that idea of like pulling apart these pieces, like not thinking about this kind of like list of graphics, like a pie chart, a line chart, a bar chart. But thinking about these components that underlie the graphic, like the data and the scales and the geoms and the stats, like just a really, really powerful way to think about visualizations that allows you to create the thing that's like tailored specifically for your problem that maybe no one has ever created before.

Quarto and the future of data communication

And so following on from this already very popular ggplot2 package, how do you envision the future of data communication, data publication, particularly with the development of tools like Quarto, which you mentioned earlier? But we didn't really talk about in detail. So maybe kind of fit, you know, describe the kind of key features of Quarto and how that fits into this whole data visualization perspective.

Yeah, I mean, Quarto at its heart is really a tool for communication, like particularly scientific and technical communication. Because what it allows you to do is integrate like your code and its results with your like thoughts about the results. So you can write text and then integrate it with your results. And of course, like as opposed to like having a Word document where you may be copying and pasting plots, and with Quarto you end up with this document that is 100% reproducible. As the data changes, you click a button, it reruns all the code, recombines everything together, and you have this like high fidelity document.

And there's like a bunch of cool things about Quarto. Compared to its kind of predecessor R Markdown, Quarto works really well with Jupyter Notebooks as well. So if you've got a Jupyter Notebook, which is a fantastic tool for like you as a data scientist and your colleagues who are data scientists. But if you're sharing that with other people, you might not want to share all the details. Quarto will help you turn that into like a polished, it could be a polished PowerPoint presentation, polished PDF, a polished website, a polished book. But something that's really aimed at not just showing like the mechanics, which of course you can show if you want, but like the thinking behind it, the explaining what's going on with the data.

That's super cool. And I was a big user of R Markdown. I haven't used Quarto yet myself, and it sounds like I need to be, particularly for that element that you just described there of being able to hide things. Like I have a number of times in my corporate life or in my commercial life created Jupyter Notebooks in situations where, for whatever reason, say a shiny tool wasn't the right thing. There was something about what I needed to be building where it made sense to me to create it in Python, in a Jupyter Notebook. And then I hand it off to somebody who doesn't program, and they have to see all kinds of things about the notebook that are probably confusing to see. Yeah. And so to be able to hide those things with Quarto sounds really cool.

Yeah. And because Quarto is built on top of Pandoc, you can produce pretty much any type of output you can imagine, which is really, really cool. So like all the books I write today, they were written with Quarto. I make a free and open source website that everyone can read for free, and then it also creates a PDF that the publisher gets and turns into a physical book. You can turn it into an epub as well. It's just really nice to have that single source that you can then produce multiple versions of.

Engaging the broader Python and data science community

Very cool. Throughout this episode, we have been talking about, not just in the context of Quarto, but in lots of items over this episode, particularly the name change from RStudio to Deposit. We've been talking about communities beyond the R community. With the rebranding Deposit and the expanded mission, how does the company engage with and foster beyond just the R community that RStudio was already easily the best-known brand within?

That is a very good question and something I think we're trying to figure out, too, and that we've kind of had it easy for so long. I think it's fair to say RStudio is beloved in the R community. Everyone knows who we are and most people like us. Now we're coming into the Python community, which is much, much bigger. Now we're a small fish in a very, very big pond, and we're trying to figure out what we should be doing.

And then sort of on top of that, since we promote, in the last five years or ten years, since we really had to promote stuff, new things, it's all changed. We don't use Twitter anymore. It's now like, should we have a TikTok account? Just how are you going to reach out to a developer community has changed pretty profoundly from when we were really promoting the Tidyverse, promoting Shiny, and promoting R Markdown. So I think with something we're trying to learn, doing the things that I think make sense everywhere, carefully listening to the community and trying to fix their pain points, and just trying to figure out what's new, how do we find the people where they are and reach out to them.

Nice. Yeah, sounds sensible enough. Tricky question. I realize you're not in the marketing department.

Yeah, I mean, I get involved in a bunch of discussions. It's interesting. It's like much, much harder. It feels much harder to me than it used to be. I don't know whether that's because it is actually harder or just like I have kind of got out of touch with the youth of today, so I'm not sure what we should be doing.

Yeah, it's hard to know. But a funny kind of thought that came to mind for me just now, I don't know why it took me so long for this to come in, but earlier in the episode, one of the first things that we were talking about was you made this comparison of how with R and Python, that being so similar, it's kind of like a family dynamic, or because you're so similar, you end up having these heated exchanges.

And so that took me off your thought just now about how you have a sister, Charlotte, who is also a statistician. She's a longtime university statistics professor like you are. And she even works at Posit, right?

So what are your arguments like now?

Let's have your last argument with Charlotte on air. I'll play her.

No, I would say we like mostly don't talk about trying to talk about stuff.

It is funny that I don't work with the same company as you.

But how did that how did that kind of evolve? I don't actually know who's the older one. Who's the younger one?

I'm on the oldest by three years.

But we ended up having we did a few courses at college together because I actually I originally started off in medical school. So I did a three year degree, a bachelor of human biology and decided I didn't like that. So I went back and did a bachelor's of science. So we ended up kind of overlapping.

But we both I think we both kind of got the statistics of data science or a process of elimination. Like I started in medical school. She started in like physics and both, you know, eventually ended up in a place, you know, where we both did our undergrads at the University of Australia.

And I don't know, it is kind of interesting. I have a I have a sister who's two years younger as well. And we also kind of converged.

It's interesting how that can happen. I think I like the genetics.

Cocktails, Copilot, and creative routines

That's classic. That's classic. It's just a complete jarring topic change, but on your website you have listed— and we'll be sure to put these in the show notes— you have Wiccam Family Recipes, and you also

But I have actually turned those into a book as well. I'm not surprised. There's 161 cocktail recipes curated by yourself. You can browse by ingredients or browse by tags, so you can click through and say, wow, I really like banana liqueur, and find all the cocktails with that in it. So that's a really fun resource. I'm sure there are some listeners out there that will be checking that out. I got that in the show notes for you.

So that's a fun thing to check out. Also, one of the first places where I was really kind of blown away— let's bring it back to GNI again— is that that cocktail's website is powered by a YAML file that has all the cocktails and the ingredients in. And I was really impressed. When RStudio enabled Copilot integration, it just enabled it for every file, including the YAML files. So now I can just type in title and the name of the cocktail, and a good 75% of the time it completes the ingredients using exactly the same YAML syntax that I use for every other cocktail. And I was just like, wow, that's pretty neat. That's pretty cool.

That is very cool. So my question for you is, given that you obviously have interests beyond just programming, just work, how do you optimize your routine or your flows? We talked about getting into a flow state earlier in this episode, so it sounds like something that's important to you.

Also, we pulled out from a video that you made a long time ago— actually, it was just a few months ago— on the PositPBC YouTube channel, you gave us a hint that key to your routine is prioritizing creativity and productive work first. I don't know if you want to elaborate on that.

Yeah, I think one of the things we've tried to make part of the culture at RStudio and now Posit is this idea of deep work, and that it's okay to carve out three or four hours or more from your schedule to just be able to invest in some project that you need to have that level of focus on. I still manage to do that. I try and keep Tuesdays and Thursdays free from meetings so I can focus on programming challenges. Recently, it feels like I've just had a lot more meetings and more manager stuff, which I don't love. It's important. It's not something that amps me up, but I know it's important and I do it.

Yeah, I think one of the things we've tried to make part of the culture at RStudio and now Posit is this idea of deep work, and that it's okay to carve out three or four hours or more from your schedule to just be able to invest in some project that you need to have that level of focus on.

That's something I think I'm really hoping to get back to more in the coming weeks, is just carving out that time, finding some big technical problems to get my teeth into.

One of the things I have been thinking about quite a lot and trying to learn more about is R and production, or data science and production. Just trying to understand what does that mean, really? Then once I've turned solidly understood that, really thinking systematically, how can we make that easier for folks using both of our open source products and with our pro products? I'm excited to start digging into some of those technical challenges.

Yes, certainly. That is another classic quip from Python developers, is that you don't use R in production, but of course you can. Lots to dig into there, I'm sure.

Looking ahead: data contracts and generative AI

Beyond that, what else are you excited about in this fast-evolving data science field? Are there particular challenges or paradigms that are up and coming that you're really excited about?

I don't know if I'd say I'm excited about it, but interested in it is these ideas around data contracts. Now, like most folks, most or many data scientists are working hand-in-hand with data engineers who are telling them that's where they get their data from. But often that data is not produced primarily for data scientists. It's produced for some other business purpose. Then the data engineers change that data in order to optimize that other business purpose. Then all of the data scientists' dashboards and models just break with confusing and bewildering error messages.

That just seems like an interesting intersection of technical and people problems. It definitely seems to be something a lot of companies are struggling with right now. I don't think I want to solve that, but I want someone to solve it so I can be like, okay, this is a good framework for thinking about it, and then maybe I'll copy some of the tooling in R to make that a bit easier. I think that's interesting.

Generally, I think just this whole, I'm not super invested in it, but just curious about our aims and generative AI and how is that going to, particularly how is that going to affect learning R for the first time, learning data science? How can we, and teaching it too, because it fundamentally changes what questions you can put on an assignment and expect what questions a student can answer without having to put in any intellectual work of their own. I'm very much always in the favor of lean into that. If now there's a question that a computer can answer, we need to find new questions. I think that's just really, that's interesting.

It's not like, I don't know, I'm never, I don't know, I don't think I'm going to be using my own models or training models or really understand the math behind them, but just purely what are the, how does this change the way we think about teaching data science? I think it's really interesting, and how can we build this into our tools? What are the resources we should be thinking about? All that kind of stuff.

If now there's a question that a computer can answer, we need to find new questions.

Yeah, interesting, exciting, and scary. I was a professor at the University of Illinois a couple of nights ago at the time of recording. Data Universe is a new conference that was just held for the first time in New York this week at the time of recording. I met this University of Illinois professor at a dinner. He was sitting next to me, and he was doing some really cool things with Gen AI and education. He had created a whole bunch of documents.

He was describing how there's this problem increasingly with getting corporations to share real data for students to work with. He created a fake company called Nile, which was supposed to be like Amazon, and created dozens, generated dozens of business documents and data tables, and then had all of that built into a chat interface so that students could use that as a key stakeholder that they're working with in the business and create data products for. I thought that was cool.

Yeah, so many possibilities. It's going to take a while just to figure out how to use them effectively. For sure.

Listener questions

All right, so that's the end of my questions. I think we've got actually a crazy number. I'm going to have to curate them. My apologies to listeners who asked so many questions. I'm pretty sure a record number of questions for Hadley, which is cool. It shows how much of an impact you have in the community in general, Hadley. Yeah, lots of big name guests, but I've never seen anything like this. So I'm going to pick a few.

First of all, Austin Ogilvie, thank you for connecting me with Hadley. He said that he's looking forward to this episode in one of his comments on LinkedIn. I should have mentioned it at the outset that it was thanks to Austin for connecting us, that you are here. I really appreciate that.

Me sheepishly making eye contact 10 years ago and not being willing to actually talk to you at the Joyce Statistical meetings didn't cut the mustard.

So here's one from Matias Baudino. Matias is a longtime listener. He's a BI analyst at a company called Brain Technology. And this question, it ties into a lot of the R and Python stuff we've been talking about here. He's wondering if there is a combination.

And you're probably, I don't know if this is something you would think about. So is there a combination of Python libraries out there that could give you equivalent functionality to the tidyverse, or is there nothing out there like that? I will tell you that if such a thing existed, I would call it the pideyverse. But that is as far as I've gone.

I think there are two packages that are kind of close to the major pieces of the tidyverse, and that's plot nine for ggplot2 and suba that's very similar to dplyr. How do you spell that? S-I-U-B-A. Plot nine developed by Hassan, who I mentioned earlier, and suba is developed by Michael Chow, who's also at Posit now.

So I don't think we don't have any plans to try and make the pideyverse for Python. I think Python is just a different environment. It's much more diverse than R. It's not clear to me that having this kind of re-centralized cohesive set of packages would work. But me and my team remain 100% focused on R, so I hope I'm always happy to talk to people working on stuff in this area of Python and give my two cents, but not something that I'll be working on personally.

Nice. And we had Doug McLean, who's the lead data scientist at Tesco back, also a longtime listener who's asked many questions of guests in the past. Thank you for all those, Doug.

He wanted to make sure that we talked about Shiny and deploying apps in R as well as in Python, and he wanted to make sure that we talked about how much better R Markdown or now Quarto is relative to Jupyter Notebooks. So you get that. He wrote yuck next to Jupyter Notebooks.

I mean, Jupyter Notebooks are great as a tool for analysis. They're not great as a tool to share with decision makers in your organization. Quarto now really fills that gap. You can have that Jupyter Notebook and then relatively easily turn it into a polished PDF or Word document or HTML file. It's really worth looking into if you use Jupyter.

Social media and the post-Twitter landscape

Moving along from my LinkedIn post that you were coming up to, my Twitter post, someone named Econfella wrote, just tell Hadley that we miss him on Twitter. This place sucks now. And interestingly, originally Twitter had actually hidden that. It said this post may contain offensive language. Are you sure you want to view it? And so it's funny. Somebody's written some code somewhere. Someone says Twitter sucks.

Yeah, Twitter misses you, but I totally understand why you would move on. I don't even call it what it's actually called these days. Yeah, I know. They're still angry and annoyed by that whole thing, and there's kind of no obvious successor in sight. You know, I'm on, like, FosterDawn, which, I don't know, is only going to appeal to the most technical of technical people. I found it hard enough to get up and going on it. I'm on LinkedIn, which I still cringe whenever I use LinkedIn. But it's fine.

And then, like, I don't know, Threads just feels like switching one billionaire to another billionaire. So I've heard good things about Blue Sky, and I know people have been encouraging me to check it out, so maybe that would be something to try. But it's just, like, another social media platform. Yeah, that seems to be the way to probably go as a replacement for Twitter. It seems to have some momentum. But we will have your FosterDawn account linked to in the show notes for people who get that up and running to join you in there. People have figured it out. You're very welcome to message me.

Audience questions

So back into a couple other questions that we haven't already covered in the show. This is an interesting technical one from Lelouch. So does Posit still have plans of developing an online collaborative editing platform for Quarto similar to the role that Overleaf has in the LaTeX ecosystem? Yes. I'd say we definitely have plans. Like, this is part of the vision for, like, we want data scientists to be able to use Quarto, you know, to communicate with decision makers, non-technical folks so they can comment, they can change the writing. You know, I think that's still some distance away on our plan, but, like, absolutely we really want to make that. We really want to make Quarto facilitate the communication between technical and non-technical folks as well.

Nice. And a bit of a quibble here. So somebody here says that ggplot2, that name, is at odds with other names in the tidyverse. Do you think you'll ever rename or rebrand ggplot2? No, it's just a weird name, but it's, like, it's too late to change it now. I like it. It makes perfect sense to me. Grammar of Graphics Plot 2.

Bruno Rodriguez wants to know what your top three movies of all time are. Oh, God. Oh, I hate listing favorite things. I don't know what movies. Just give us one. Just give us one. I mean, I really enjoyed the Barbie movie. The Barbie movie. Yeah. There you go. For some reason I was actually thinking about today as I was, like, falling off into a nap. For some reason I was thinking about the Barbie film, and I was thinking it is odd that I haven't seen that. It must be a really good movie. Yeah, it's pretty fun. I'd recommend it.

We've talked about Wes McKinney on this show already, Hadley. What can we expect from you two teaming up now that Wes is at Posit? Can you give us any kind of hint into what you might be working on, maybe Apache Arrow kind of functionality with R or the tidyverse? I wondered if this question was going to come up, and I can't give you any hints. But I can tell you it's pretty cool. Nice. Well, that's as exciting as it gets. That's awesome.

I wondered if this question was going to come up, and I can't give you any hints. But I can tell you it's pretty cool.

Thank you so much for answering my questions and all of the audience questions as well, Hadley. Quickly, before I let you go, I know now that you hate listing favorite things, and so you don't need to frame this as a favorite, but we always ask guests for a book recommendation. And so it doesn't need to be your favorite book. Just any book recommendation that isn't your own. Thanks for that question. I was just trying to think. I mean, the book that comes to mind the most really, which I love, is Gideon the Ninth, which I would describe as like lesbian space necromancers. It's a really good book. I like it. It's written by a New Zealand author. Nice. That sounds fun. And so that one's fiction or nonfiction? Fiction.

Awesome. And in order to follow you after this episode, you already gave us some of the ways to do that, so I'll have the Fostodon link in the show notes. I'll have your LinkedIn profile in the show notes. Any other ways that people should be following you? And just like keeping up with the tidyverse blog, that's where we post everything about new releases, upcoming, full upcoming stuff.

Awesome. All right, Hadley, thank you so much for taking the time. It has been such an honor to get so much of your time to be able to speak to a personal idol of mine in this space. What you've done for the community is unreal, and that you continue to do. You have really made an unbelievable difference to so many people, letting us get into the flow of programming, dig out some really cool insights, and make a really big impact with data. So thank you so much, Hadley. Thanks, John. I really appreciate it.

Episode wrap-up

It's so cool to be able to chat with a personal idol of mine. I hope you enjoyed my conversation with Hadley Wickham, too. In today's episode, Hadley filled us in on how the goal of the tidyverse developers is to allow data scientists to spend as much time as possible in a flow state. He also let us know how two of his favorite R packages are dbplyr for converting R to SQL, and testbat for unit testing. He also talked about how plot9 allows for ggplot2-style and sayuba allows for dplyr-like data analysis in Python, while arrow, ductdb, and keras for R allow R and Python to be more closely married than ever. He also talked about how Quarto allows natural language and code to be easily blended together into beautiful documents.

As always, you can get all the show notes, including the transcript of this episode, the video recording, and the materials mentioned on the show. The URL is for Hadley's social media profiles, as well as my own at superdatascience.com slash 779. And if you live in the New York area and would like to engage with me in person, as opposed to just on social media, on May 17th, I'll be hosting a panel live at the New York R Conference that will feature iconic open-source community members Drew Conway, J.D. Long, Sumiya Kalra, and Jared Lander. There's always pizza and beers afterward, so we can catch up over a cold one then. Hadley Wickham and other huge names like Andrew Gelman, Hillary Mason, Wes McKinney, and Sean Taylor will be there too. It's so crazy. The New York R Conference is one not to miss, regardless of what programming language you do data science in.

All right, thanks to my colleagues at Nebula for supporting me while I create content like this Super Data Science episode for you. And thanks, of course, to Yvonne, Mario, Natalie, Serge, Sylvia, Zara, and Kirill on the Super Data Science team for producing another magnificent episode for us today. For enabling that super team to create this free podcast for you, we are deeply grateful to our sponsors. You can support the show by checking out our sponsors' links, which are in the show notes. And if you yourself are interested in sponsoring an episode, you can get the details on how by making your way to johnkrone.com slash podcast.

Otherwise, share this episode if you loved it with someone who might like it. Review the episode online. That, I guess, somehow helps us build traction on the show. Subscribe if you're not a subscriber already, of course. But most importantly, just keep on tuning in. I'm so grateful to have you listening, and I hope I can continue to make episodes you love for years and years to come. Until next time, keep on rocking it out there, and I'm looking forward to enjoying another round of the Super Data Science podcast with you very soon.