Transcript#

This transcript was generated automatically and may contain errors.

I am very pleased to introduce Cameron Riddell who works at Don't Use This Code and previously was in academia and studied psychology. Cameron and I met at PyCon US a few months, weeks ago. Time is fluid and it was a real delight to get to chat and learn a lot from Cameron.

So thank you so much for joining us. I'm very happy to pass it over to talk about several things. And so if you come from the R world like I do, or if you're used to Python, you might realize that Python DataFrames are slightly different than R DataFrames. And there's a lot of interesting features about them that you have to know if you're using them in the context of, you know, working with DataFrames. And so Cameron is also a co-maintainer of Narwhals, which we'll talk about a bit today as to why it exists and how it helps all of these fun facts about Python. So Cameron, I'll pass it to you. And thank you again for joining us.

Thanks so much, Isabella. Hello, everyone. My name is Cameron Riddell. Today we're going to talk about two of my favorite things, one of them being DataFrames and the second one of them being, of course, Narwhals, which I help maintain. Now, if you will bear with me, I lost my voice a little bit this past weekend. I was coaching at a lacrosse tournament and I ended up yelling for probably five hours straight and it was quite damaging on the vocal cords. So I will do my best. But in order to get started, why don't I go ahead and share my screen?

We have here a lot to talk about. And what I don't want to do is I don't want to only talk about Narwhals. While the library itself is interesting, I don't think it's the most important topic that we could talk about and that we could disseminate. Instead, we're also going to focus on the broad topic of DataFrames and why Narwhals came into being in the first place. Because if you look at the R ecosystem, I don't think there's a need for a Narwhals style tool. That said, I only have a couple of years of R under my belt. I have a decade plus in Python. So all of my R friends, if you can fact check me, that would be fantastic. But perhaps if we can understand why Narwhals needs to exist in the Python ecosystem, we can also see why maybe R doesn't need a Narwhals style tool.

What is Narwhals?

Let's go ahead and start talking Narwhals and talking DataFrames. The first thing that I want to touch on is what is Narwhals? Aside from a very cool underwater creature, the unicorn of the sea, Narwhals is our lightweight, dependency-free data and lazy frame interface. And in fact, if you're using almost any tool that deals with tabular data in Python, there's a good chance that you're already using Narwhals. So if you check your virtual environments, you might even have it installed. But that means that Narwhals is not designed necessarily for a tail-end analytical tool. What this means is that Narwhals is going to be used primarily by library authors. That's going to be the folks over at PyShiny, Greattable Extras, Plotly, Altair, Bokeh, sklearn, Pandera, and more. In fact, I think we have at this point like 28 downstream adopters. Downstream just meaning they use Narwhals because their users come to them with DataFrames. And their users say, well, I use a Pandas DataFrame or I use a Polars DataFrame. I use a DuckDB table. I use a PyArrow table because everything else is just too much for me. And so we have all these tools that need to work with tabular data. And it becomes an incredible pain to say, OK, here's the implementation for Arrow. Here's the implementation for Pandas. Here's the implementation for Polars. Here's the implementation for Moden, plus that little patch. Because although Moden is 100% drop-in replacement for Pandas, there's just some things it doesn't do.

The other audience that likes to use Narwhals, and I will say this is probably like the 5% audience here, but some subset of people who don't like writing SQL. And so if you want a more object-oriented Polars expression-like interface for using DuckDB, then Narwhals could be a route that you would go.

But here's a quick slide that I made. And this is one Isabella I actually shared at PyCon US during the maintainers track. And Isabella, I will tell you, I need to update this because we have actually merged in. There is Pandera now. That should be on the slide. And there should be one more sklearn, scikit-learn subpackage that I need to add on here.

But you can see a number of different tools are adopting and are starting to use Narwhals. Historically, these tools only supported Pandas. Some of the newer tools then said, oh, Pandas is the old thing. Polars is going to be the new thing. And then they only supported Polars. And now they say, you know what? We can support many different data frames by using Narwhals.

Live coding: computing a column sum

Let's take a look at what this looks appears to be in code.

So I'm going to come up with a quick example. And we'll do a little bit of live coding here. Actually, a good amount of live coding today. But I have a brief example where all I want to do is compute the sum of a column. And we'll see the different ways that we do that across these different tools. The first thing that we're going to need is some data. So I'll create a Pandas data frame here. And we'll just have some values. Nothing too crazy, just the numbers 1 through 5.

We'll create a Polars data frame with the same data.

And then we'll also, we can instantiate. So this is kind of a sleight of hand here, because this is not necessarily how DuckDB works. But we will go ahead and create a DuckDB pi relation. So here we have all three of these tools. And we'll go ahead, and I can print each of these out. I have a little utility helper here.

And we'll just print all of these.

And there we have our three data frames, just with one column each. If I wanted to take the sum of each of these columns within each of these data frames, well, I would take this sum for Pandas. I would do something like this. If we wanted to retain the fact that it is a data frame, I would use the double square brackets here to make sure I get a data frame back instead of a series. And I can compute the data frame for each of these columns.

And then I can print out the sum of each of these columns within each of these data frames. And I can compute the dot sum just like this. Polars, while we could use the same syntax to grab the series and compute the sum, Polars often opts to use expressions. So I would use the select method on the data frame object. And then I would grab the Polars column. That's going to be our value column. And I would take the dot sum of that column. And then for the Duck data frame, this is why I said there's a sleight of hand here, because really, this is just going to be another query.

All right, update. I did a poll in the Discord, by the way, just asking, did you learn Python or R first? We have about 22 votes in. Scroll up in the Discord and take that poll if you want to, if you still haven't. About 70, oh, more people are voting now that I mentioned it. Over 70% said R first. About 30% Python first. Excellent. And we can actually speak more about that, how the series object in Pandas or Polars is very similar to an R-style vector here. We'll wrap this up first. But we see that we have computed the sum of the numbers 1 through 5 in Pandas and Polars and DuckDB. And you look at the code that was written, and none of these look like the other. They are all completely distinct. That means if you maintain a plotting library that says, give me your data frame, I just need to compute the sum. I need to do a very simple, straightforward operation. You have to manage three separate code paths. And it becomes kind of a headache to start maintaining each of these.

Instead, what we can reach for is a little bit of narwhals here. So the way that narwhals works is we're going to take what we call the native data. This is going to be your Pandas data frame or your Polars data frame. And we convert it into a narwhals data frame here. Now, you'll notice that the narwhals data frame, even in the repper, the printouts that we have here, it's the Pandas data frame just tucked inside of a broader repper. And if we were to instead use the Polars data frame, we'd actually see the representation of the data change.

So that it matches the actual Polars data frame. This is the whole core idea behind narwhals, is we take your data frame, we then put it into an object that has a very uniform interface, meaning it doesn't matter if you put a Polars data frame in here or Pandas data frame in here, the rest of the code will look the same. And then when you're all done with your data transformations, you can spit it back out into the same type that you put it in as.

This is the whole core idea behind narwhals, is we take your data frame, we then put it into an object that has a very uniform interface, meaning it doesn't matter if you put a Polars data frame in here or Pandas data frame in here, the rest of the code will look the same.

So I can use the syntax here. I can select a column. We'll take the value column here and compute the dot sum. The great part about this is I can perform this with any data frame that I have. In fact, I can write a function here. I can write what we call an agnostic function. This takes a data frame and I'll take the column. You'll take this code here. We'll just return this.

And then we can do our agnostic sum from our Pandas data frame with our value column. And we'll do the same for the Polars data frame and the duck df here.

Here are each of our sets of results that you see have come out. Let me get rid of the column. And there's our sets of results. We can cast this back to our native types here. And we'll get back out each our Pandas data frame, our Polars data frame, and so on and so forth. Let me just fix a small typo in the code. And now we have the sum on our Pandas, the sum on our Polars, the sum of our duck db. So instead of having to maintain all of these different code paths, you just can work with one code path. And that's the beauty of narwhals.

Questions and answers

Amazing. Are you open to stopping for some questions?

Absolutely.

Okay. First of all, we have a question that I knew we would get that actually Isabella and I asked Cameron the moment we saw it before we started our session. What are you using today? We had a question from Christoph that said, what IDE is Cameron using? Are we looking at his terminal? Is this Emacs? And I said, no, I think it's Vim plus extras is what you said. Would you explain a little bit?

Absolutely. This is, yes, this is my terminal. We have a combination of tools here. And I cannot take credit for having created this. This was shared with me. And then I added a couple of additions on top of it. We have three tools that you'll see. The first tool is tmux. This is a terminal multiplexer. It lets me split my terminals into an arbitrary number of panes. So I can have a top pane, a bottom pane, a bottom, right? So on and so forth. I'm only using two panes here. As you can see in the top pane, I have Vim running as my code editor. And this is just a markdown file that we're reading through here. The last tool here is called I notify trigger. It's an I notify weight tool, meaning that I sit here and I monitor this file for any right events or any specific type of events. And when the file changes, I execute the code that's in the active cell. And that's how you see the output at the bottom.

Amazing.

As far as I know, there is no specific Vim plugin for this. There's a lot, there's a decent bit of custom scripting around this.

Okay. Isabella put some links in the chat for everybody. And as a callback, if you remember the one where Joey Marshall joined us and we worked on the Frog Australia app, what he was using was Cmux, which I believe is built on Tmux, which, you know, don't quote me on that because I don't use either. But that's that extra flexibility added to the terminal. And then also Gregory in the chat mentioned that if anybody wants something that works like this on Windows, there's Zellige. Maybe he can add a link to that and that's helpful too. And then we had Nor who asked, so does narwhals essentially act as coercion for data frames, like coercing data frames into something and into something else? We think of coercion a lot in R. So that might be a thing that we're trying to make a parallel to in our brains.

Yeah. Can you give me an example of what coercion would be like in R?

Sure. So if you have, like, let's say you have something like a zip code that is a number, right? And you don't want it to be a number because you don't want to do math on it. And you want to preserve the maybe leading zero on it. You can coerce that numeric into a string in R, and we call that coercion. Okay. So under the hood, you can coerce things. But what we actually end up doing in narwhals is we have the native data frame that you passed in. So if you have a Polar's data frame, that's going to have all of the Polar's data types. What we do for all of the narwhals operations, we map a bunch of different narwhals data types on top of those Polar's data types, meaning that you can write your narwhals expressions and provided that the assumptions we made about the data types are correct, then your code will work. At the very end, we just translate it back to the Polar's data types. So every operation that we end up doing just gets translated effectively into the native backend or into Polar's or Pandas or anything in this case. So we don't do coercion. We're not actually changing your underlying data types. We make generalized inferences about those data types so that when we say, take the sum, we can then just tell Polar's, hey, go take the sum,

and then Polars does whatever Polars is going to do. So we try to be as lightweight as possible. We're not intercepting your data types and transforming them and then doing something with them and then spitting them back out in a different format. We're using those data types to push our computation down into the backend and then let the backend do whatever the backend is going to do.

That's one of our big philosophies. We didn't want to do too much with the expression tree or with the syntax tree because that could induce some overhead. And we want to keep overhead as minimal as possible. Yeah, that goes right into Hubert's question, which was, does Narwhals impose any performance penalty? Yes. When you're wrapping things, the performance penalty is going to be unavoidable. However, the performance penalty is at a non-important portion of the code, meaning that typically for these data operations, if you have a million rows, a sum across a million rows is going to take half a second, a second, depending on the speed of your machine.

But what we do is since we just take your data frame and we wrap it into our Narwhals object, that is the expensive part. You just did the most expensive portion. It's literally going to be taking an object and just adding an attribute. The rest of the computation, we just translate it and put it back into the backend. So there's going to be a small overhead there. But the actual computation, the most expensive part, that's going to be completely running on whichever data frame engine you passed in. So the expensive part, which is the computation, we don't touch and we try to just let the backend do what it's going to do to minimize that performance penalty.

But the actual computation, the most expensive part, that's going to be completely running on whichever data frame engine you passed in. So the expensive part, which is the computation, we don't touch and we try to just let the backend do what it's going to do to minimize that performance penalty.

Syntax choices and supported backends

Isabella had asked that she was curious which syntax Narwhals decided on. Yes. So here you can see that we very clearly made a choice. We decided, actually we, as say Marco, who's the lead maintainer of Narwhals, Marco decided to use the Polars style syntax as inspiration for everything that you'll see here in Narwhals. We are not a one-to-one mapping of the Polars API, but you can see that we use .select. We have this idea of expressions. Give me a column and then give me the sum of that column. We use .select to drive that computation.

There's going to be a couple of deviations, which we can absolutely talk about, but for the large parts, we do suggest or we do follow Polars for the most part.

Great. Awesome. And I shared a link to the Hangout that we had with Marco Gorelli. Marco joined us on the Hangout. So if you would like to meet him and learn all about philosophies behind stuff and decisions, I highly recommend going and checking out that YouTube video. If you're watching this later on YouTube, I'll put the links in the description for anything that we're saying here.

Okay. All right. We had one more and then we are going to go back to letting Cameron live code some more perilously, although you're doing fantastically. And that is from Ken saying, so is it like a universal translator? Well, it can't be universal because you have to give Narwhals the different types of data frames that you want it to be compatible with, right? Are you like constantly expanding that? We have been expanding it, um, probably up to the beginning of this year.

And then we said, I think we have 11 different backends. I actually have a slide here that I can pull up. These are all of the backends that we supported, at least as of PyCon US, which as Isabella said, was that a couple of weeks ago or a couple of months ago? A couple of months ago. Some of these I haven't even heard of. Like I've heard of from an, from an, being our person, primarily I do R and Python, but I will admit primarily R. I recognize Polars and Ibis and DuckDB and Arrow and Pandas, but then I am not much of a SQL person. I avoid SQL whenever I can, unfortunately is my least favorite data manipulation language. Um, so I don't recognize all of these, but hopefully everybody else recognizes some of them.

Um, we can go through them very quickly if you'd like. Yeah. Do you want to mention a couple that only have a symbol like D? Absolutely. Yeah, absolutely. So while I'm not, and I cannot say that I'm an expert in all 11 of these, I do know the general principles and their use cases. Rapids or also known as QDF, which is Nvidia's GPU data frame implementation, which actually Polars now has an API sitting on top of, and that's been a big part of the Nvidia and Polars collaboration. Um, but QDF is its own tool for running data frame computations on your GPU.

Apache Spark. Uh, I believe Spark had a data frame interface called Koalas, but now I think it's just Spark data frames at this point. Um, but this is for running distributed data frame computations using Spark SQL frame. This is the one that I'm least familiar with probably. Uh, but SQL frame allows you to easily translate one style of SQL syntax to a variety of different SQL implementations. Moden here is a data frame tool came out probably eight years ago. That's the gas off the top of my head. So don't hold me to that. But Moden was a tool that said, Hey, pandas is kind of slow. We rewrote pandas and we made it fast. And we have the exact same API as the pandas API issue is the issue for that tool.

However, is pandas is 15 years old. Pandas has more functions than I know, probably words in the English language. It's an exaggeration, of course, but it's, it's going to be impossible to mimic the pandas API a hundred percent. It is, it is a, and this is from also the developers of Moden. It's just an impossible task that they found out, unfortunately. Now they do have most of it captured. The hard part is, do they have the bits that you care about for your upcoming project? Because as soon as you hit a wall, it drops back to pandas and then you get the slow path.

Hmm. Okay. Yeah. I think this is sometimes hard for the people who answered. I learned our first in the chat to wrap their heads around because in our, it's not a problem. Uh, data frames are native data types in our, and so that sort of like worry about is this data frame going to be covered on this backend is not something that we've ever had to think about. So I'm hoping that this is building empathy for folks who are, you know, learning about working outside of their language, that it is something that people have to worry about.

Um, and I lied. I want to throw in one more question because I think it's great. And it's from Hubert and he said, were any of these backends more challenging than others to accommodate?

I will say the hardest backend to accommodate so far has been ironically Polars. Most things we can just pass straight through because our interface is so similar to the Polars interface, but we did start supporting Polars versions that as of today are quite old, but at the time they were brand new. Polars is a very young library, so things change very quickly. And the reason I say it's hard to support is that if you look through the Narwhals source code, um, I had to do this, uh, for the histogram method or for the hist method. There's like, if you have Polars version one point, something, you know, do this. Oh, they changed it. The Polars changed in 1.2 now. So now if you have version Polars version 1.2, do something slightly different because they changed how it was implemented. And since Polars keeps changing things, we have this decision to make. Do we adopt how Polars has changed? And then do we unify the rest of the backends across that?

Or do we make the new Polars version align with what we have already implemented? And so this is a, this is a very common issue. Uh, when you try to say I have an API, it sits on top of somebody else's API. Yeah. This is one of those things that's like the best and worst thing about open source, especially a new and evolving thing is you're like, yeah, the new thing. But then also the new thing is constantly changing. Oh, great. Fantastic. I'm going to let you move ahead with some of your discussions here. I'm ready to learn. I'll pause you again later. We have more questions.

Narwhals overhead and the XKCD problem

Yes. So there's a, two things that I want to probably, we'll probably skip around a little bit if that's all right, but two things I want to address. The first is we can talk about overhead. And we can come up with probably some little example, examples to see how much latency or how much overhead does Narwhals add to any given operation. The other thing that we can, I want to also address, and this is probably the first thing that I'll get to is this. I see a very popular and famous XKCD has entered the chat. And in fact, this XKCD comes up all the time when you talk about Narwhals and you say, well, here's the issue. We have 15 things that we're trying to map and we want to unify them. We want to make one best implementation.

And if you're familiar with the Python data frame community a decade ago, 15 years ago, it was just pandas. That was it. No other data frames in Python. Everybody built their tooling around pandas. And then probably in 2018, especially maybe 2017, 2020, you had group after group after group saying we can do pandas, but I think we can do it better. And so the ecosystem changed from just pandas and pandas by itself to Polars, DuckDB, Ibis, Dask, Mode, and all those tools and more because not all of the data frames have survived until today.

I know for a fact Veix is still around, but I have not seen or heard anybody talk about it since probably at least a few years ago. And so it's not one of the backends that we support, but it's a data frame library that existed. And so this brings up that very popular XKCD of, well, there's 14 competing data frames. That's crazy. Let's just make one universal one. And then now all of a sudden you have 15 competing data frames. And this is not what we want for Narwhals.

What we want for Narwhals is to have one interface that you can opt into. It's going to let you do a well-defined subset of tasks. However, we encourage you that if you have something that falls outside that task, you can always submit a feature request, or we can provide you the guidance that maybe Narwhals isn't the tool for this use case. So we don't want Narwhals to compete with pandas or Polars. We want Narwhals to help unify some of the things that pandas and Polars can do, some of the common operations. Do you think about our primary audience, like a plotting library? They don't need to do that many complex tasks. Maybe they need to pivot some data, or maybe they need to do a group by aggregation of some sort. And we fully and completely support that. But we're not here to say, oh, we're competing with IBIS, or we're competing with Dask. Because I'll even go ahead and say, if you're coming up with an analytical project, you have a big data analysis that you're planning to do, don't use Narwhals. It's not the tool to do your data analysis. It is a tool that if you have a bunch of other users, and they all get to use their very own data frame, they keep giving them to you, and you want to write one function that can do something with any of those data frames, that's when you reach for Narwhals.

if you're coming up with an analytical project, you have a big data analysis that you're planning to do, don't use Narwhals. It's not the tool to do your data analysis. It is a tool that if you have a bunch of other users, and they all get to use their very own data frame, they keep giving them to you, and you want to write one function that can do something with any of those data frames, that's when you reach for Narwhals.

Amazing. Okay, that is the description that I think just finally landed on me for Narwhals. Thank you. But also, that makes me ask another question that I'm hoping you didn't already answer and I didn't understand, which is like, I see Marco sometimes on LinkedIn being like, hey, yay, this package just added us as a dependency. Does that mean that all of those package builders and maintainers need to add Narwhals as a dependency in their package in order for you to be their sort of universal translator?

Narwhals as a zero-dependency package

Yes, they do. But the nice thing about having Narwhals as a dependency, if we go to the Narwhals GitHub, I'm like crossing my fingers that this fact hasn't changed since the last time I looked. Because I've been a bit busy, so I haven't had a ton of time to contribute. But taking a look here at our pyproject.toml, you can see that we have zero hard dependencies. If you depend on Narwhals, you're not bringing in a bunch of data frames, libraries with you. Whereas if you tried to create your own tool that can support multiple different data frames, you might need to have a dependency on each of those data frames in order to support them, or at the very least an optional set of dependencies. So all of the dependencies that you will find are going to be optional dependencies here. And so these are the versions of each of these tools that we've seen that we support. You can see it goes back quite far. Pandas 1.34, that's quite old. It's quite old. We even have versions of Polars here before their 1.0 release, and Polars is coming up on their 2.0 release. I don't think it's released just yet, but it should be coming up soon. So if you depend on Narwhals, we will appear in your pyproject.toml or whichever packaging configuration you're using. But you're not bringing in a bunch of other hard dependencies that everybody has to install every single time.

pandas or import pandas. We try not to import a bunch of stuff and just guess. We do our best to infer what is it that you've given us so that we don't have to assume, make assumptions about what dependencies exist in any environment to keep the tool as lightweight as possible.

Okay, great. Fantastic. Well, we had a sort of follow-up from earlier that I wanted to add from Ken when we were talking about histograms. Ken says, so you mentioned histogram, which I think is the hist function. And he was asking, does Narwhals only support what that passed in DataFrame architecture supports or does Narwhals as a wrapper extend capabilities that are not present in the underlying library? What a great question, Ken. That is an absolutely fantastic question. It's one of the trickiest things that we have to balance here when designing and working on the Narwhals interface is if DuckDB doesn't

support this thing, do we implement it on our own? Like say DuckDB didn't support taking the correlation of two columns. Well, you can do the math to get there, right? Means and standard deviations, a little bit of division, a little bit of algebra, and you can make your way there. Typically, what we will do is if the math exists and it's well-defined, we will implement it. The thing that we will never, ever implement, though, is if it takes you outside of that tool. And so we can actually look at what I mean by this. If it takes you outside the tool, meaning like a Pandas series or DataFrame.apply, any of those incredibly, incredibly slow operations, we do not want to support. Because what we don't want from a user experience is, oh, I translate all my code into Narwhals, but now it runs 20 times slower than it used to. And that's just a disaster for everyone involved. It's going to result in an issue being filed and saying, why is this slow? And then we have to then give the guidance of, oh, well, if you write the code in this very slightly different way, it becomes fast again. But you hit one of the things that we only partially support or don't support completely, and that's why your code got slow. And in my eyes, it's a leaky abstraction. It means that we didn't do a good enough job of making the fast option, the fast thing, the only option or the most visible option. And so we never want the slow thing to be the default. And I think this is like part of the downfalls of Pandas.

Demonstrating Pandas slow paths

Like if we create a DataFrame here, let's do something really slow. We'll take the random number generator here. We'll make some groups. And we'll do an rng.choice. We have our ASCII lowercase. So we have all of our lowercase letters, A through Z here. I'll dump them into a list. And we'll just piecemeal this together. We'll see if this works here. That's fine. Forgot a couple of letters. So there's our letters. Now if I take this and I set the size, let's do 10,000. 10,000 by four. So now we have a two-dimensional array here. I can view into this because the arrays are fixed with strings in NumPy. But if I do a view less than the four, now it's less than four. All right. Now I have a bunch of groups or group identifiers here. We'll take the unique ones because there could be some repetitions.

Okay. So now I have a bunch of groups. And if we check the length, this should be close to 10,000. So 9,874. I'm going to create a DataFrame here. We'll have these groups. And that will be our unique groups. And then we'll do like 200 observations per group. So I'll just repeat this 200 times. And then I'll assign in some values here. Say we want uniform values from zero to 100, whose size is the length of the current DataFrame. And there should be our DataFrame with a bunch of groups here and a bunch of values. And we'll even go the extra mile here because sometimes Pandas is decent at figuring out if your columns are sorted. And it can dispatch a couple of fast paths on here. So we'll even sample this to make it real slow. We'll just shuffle all of the rows here so that they are completely out of order.

Now I'll go ahead and let's make a quick timer. Grab PerfCounter. And we'll need ContextLib. And we'll say def timed here. This takes a message. We'll do start. So once we have this timer up and running, we'll be able to time arbitrary units of code here. We'll try finally. We'll get our stop here just like that. And then we can print out start, stop, and start. We'll do our message. 30 characters, I think, should be enough. OK. This looks good.

Anybody who is new to Python, what he was wrapping around was he was just a little bit of formatting on that. So we'll just time how long does it take to assign x equals 1. We'll comment out this code here so it doesn't run. And that should be timed with a D. And I need the ContextManager. As Libby was saying, if you're unfamiliar with Python, all this is doing is it's going to execute up to the yield and then pause. So we've captured our starting point as soon as we enter this with block here. So as soon as we reach the first piece of indented line that you see, we capture the starting time. Then when we leave the indented block, when we come out here, we're going to get our start. And we're going to get our stop. And we're going to get our stop. We capture the starting time. Then when we leave the indented block, when we come out here, we advance that function one more time. So we pick up where we left off. We pick up at the yield. And we execute the rest. We capture the stop. And then we print out. So how long did it take us to assign x equals 1? Well, barely a microsecond there.

So what we can do with this is we can see the slow path that Pandas has. If we group by our groups column and we take the value, let's do the dot standard deviation, give it a slightly interesting computation. And that should be values with an S. So we'll do this. We'll say Pandas fast path. This computation took 0.13 seconds. And then Pandas also has a slow path via dot apply. And we pass in a lambda. And take the same result. We take the standard deviation of each of the series objects within the group that's being generated. But you can see it's almost nine times slower. And this is part of the downfall of Pandas. This is what we want to avoid with Narwhals of there's two different ways you can write your code. One is super fast. And the other is incredibly slow. Nine times slower means that if we up the data to I think we have about 10 million rows here. If I did my math right. Or 1 million, I should say. If we up the data to 10 million or 100 million, the difference is going to be insane. It's going to be 1 second versus 10 seconds. Or it's going to be 10 seconds versus 90 seconds. That's the difference between I can sit here and wait for this computation to finish. And well, I need to go get my fourth cup of coffee today. And I have the time because this is going to take about five minutes to complete. So that's what we want to avoid with Narwhals. We just wanted one way to write the syntax. And then you always get the fast path. So in our Narwhals back end for Pandas, we absolutely, absolutely always avoid the dot apply because it is expensive and it is slow.

Nine times slower means that if we up the data to I think we have about 10 million rows here. If we up the data to 10 million or 100 million, the difference is going to be insane. It's going to be 1 second versus 10 seconds. Or it's going to be 10 seconds versus 90 seconds.

Yeah. Oh my goodness. I am. I was just asking in the chat if any R people are seeing Cameron write Python code and feeling mental friction where you're like, Oh, this is unfamiliar to me. And I love that Gregory chimed in and said, Hey, I feel the same way when I watch people type R code because I'm used to watching Python. Okay. So I think that we have handled a bunch of questions, but I believe Isabella has one.

Narwhals and IBIS comparison

Yeah. It wasn't explicitly asked, but it was mentioned the IBIS framework. And so I was wondering if you wouldn't mind explaining a bit about what IBIS is and how it compares with Narwhals or interacts with Narwhals. Yeah. So this, this was some of the, the earliest feedback that Marco got, uh, after releasing Narwhals as well. IBIS is also this framework that says we can work with any data frame. They used to support pandas. I actually don't, I don't think they dropped support for pandas a couple of years ago. Um, and now that at the very least their default implement backend is now DuckDB, but it's another one of these tools where you can put in, you know, insert data frame type X here, and then you get one uniform style syntax. The thing that IBIS does differently is that they will actually work with the generated, uh, expression tree in order to optimize the queries that you are writing. So they do, uh, they go maybe an extra mile further, but not all of the optimizations that IBIS makes ends up being better for every single backend. Now, even in a world where we'll say that IBIS is a great, uh, is a perfect optimizer, it always performs better than the native code that you would have written. Well, now it's another API that you need to learn. And one of the things that we wanted to target with Narwhals is you don't, there's not too much to learn. It's polar syntax. So if you're already familiar with polars, then you're already familiar with Narwhals. And I will tell you, in fact, I have probably written 200 lines of Narwhals code. So you're watching me write some of like my first Narwhals code, like as an end user, because I only ever writes the backend code as a maintainer. And I'm leaning on all of the things that I know about polars in order to carry myself through working through these Narwhals examples that you're seeing here. That said, Narwhals also sits on top of IBIS. So if you do have an IBIS table, you can pass that into Narwhals, use your Narwhals syntax, and then you can recreate an IBIS table. And then you can let IBIS do all of the optimizations that it was going to do. And you don't need to remember, what was the name of that function? Again, I forgotten it. As long as you know polars, then you know Narwhals. So that's the that's the big conceptual difference. They do occupy kind of a similar space. But we were going for as minimal and lightweight as possible with Narwhals. Whereas IBIS kind of went the other direction where they wanted to make everything really fast and really shiny and nice. And part of doing that was investigating the syntax that you have written in a tree and seeing if they can reorganize things or optimize things based on that.

It's polar syntax. So if you're already familiar with polars, then you're already familiar with Narwhals.

Narwhals overhead in practice

Okay, cool. Well, I'm wondering if you have anything more, we have 15 minutes left. If there's anything more that you have on your plate that you would like to introduce us to before I ask the final big question that I have. Absolutely. Well, we can talk a little bit more about the overhead here that Narwhals introduces. In fact, we are pretty set up to do just that. So we can wrap up this example. And then we can talk a little bit about how Narwhals works under the hood and see if we can get something going in less than five minutes. It'll be a little bit of a race, so we'll find out. But in terms of overhead, the expensive parts, the most expensive part, as I shared before, is going to be the initial wrapping of the data frame. So if we have our Narwhals import here, and we time, we'll take the fast pandas standard deviation here, and we're going to compare this to a Narwhals. We'll call this the only path because the only way that you can express the same computation in Narwhals. So I'll use a Narwhals from native, capture pandas data frame that we have here, going to use, now we're entering in the polar syntax world, so it's group underscore by, not group by right next to each other, our Narwhals column of values dot standard deviation. And when we do this, oops, and that should be, that's good. Oh, nobody told me that we're grouping by on the groups and then aggregating the values. I won't hold your feet to the fire for that one in the audience here for not catching my initial mistake. But you can see that the pandas fast path is about 0.11 seconds. The Narwhals only path is about 0.15 seconds. And then the pandas, the slowest you can get pandas, in fact, if we tried, I'm sure we can make it slower, is about just one second or 0.95 seconds. So here you can start to see that overhead that we have for Narwhals. In this case, it's coming out to four tenths of a second or four one hundredths, I should say, of a second, which in the grand scheme of things, the computation of actually computing the standard deviation per group is going to be the heavy part. And that's the part that we just let the back end do what it does. We're not intercepting anything. The only thing we're doing is we're just wrapping some syntax around the outside to make it a little bit more polar's like.

How Narwhals works under the hood

And in fact, that actually wraps up nicely because this talks about, this touches on how Narwhals actually works. And it's just layers upon layers of composing different entities or different objects here. And so one day we woke up and we said, the pandas API, it's it needs some love. And they're too deep. They're not going to reorganize and restructure everything. But people really seem to like what polar's has going on. Is there any way that I can use the polar's interface, but put it, map it to a panda's data frame? And you can do that through a couple of ways. But the hardest part is to understand that you have all of these different pieces that are moving. And so what we'll do here is I'll create a very basic data frame. And we'll see how far we can get in just a minute or two. I suspect we won't get all the way to the end, but if we have a data frame here has two columns, a and B, we'll say four, five, six, just to differentiate. We can take a look here at the data frame. Now, if I want to create an object that wraps this data frame and gives it a slightly different interface, there is no dot select method on a panda's data frame. And so we need to give it something that has a dot select method. There's two ways you can do this. Typical program, you talk inheritance, or you can talk composition. We went for composition because in our case, when you're trying to unify things, going for an

inheritance pattern, it kind of inverts the chain that you want here. So we create inside of narwhals. We literally have a class. It's called pandas like data frame because we also use the same interface for moden and other things that claim to have 100 pandas compatibility. We have a class that looks like this. It's just a panda's data frame. And I'll grab our data classes import as well.

This thing takes a data frame. We'll say it takes a panda's data frame specifically.

And so now what I can do is I can take this data frame and say that we have our agnostic, I'll call it my ag data frame. I also did grad school at UC Davis, so go Aggies there. But this is our pandas data frame. And it just consumes a regular data frame.

Now the wrapper is going to be quite, quite bad here.

But this is effectively all that we do in narwhals. We take your data frame, and we put it inside of a different type. We then define the methods that we want, like a select method. A select method has access to the current data frame, and it has access to all of the expressions that you have put in. But then you have to say, well, wait a minute. What is an expression? So somewhere else in the narwhals source code, we have our pandas expression, which looks like this. And this will be another data class here.

How expressions work in narwhals

The expression, this is where the tricky part is, where a lot of the magic comes in. The expression stores a function that you can call, and the function has a very well-defined contract, meaning this function, if we grab our type hints here, this function takes in a pandas data frame, and this will return a list of pandas series objects. The idea is that an expression is just going to be a function whose only input is going to be a data frame, and it's going to return to you a bunch of series that you can horizontally stitch together and make a brand new data frame. And that's exactly how select is going to work under the hood.

The idea is that an expression is just going to be a function whose only input is going to be a data frame, and it's going to return to you a bunch of series that you can horizontally stitch together and make a brand new data frame.

Now, these functions can also have some arguments, but maybe for time, out of the interest of time, we'll avoid dealing with any dependent arguments for expression chaining, or I should say nested expressions in this case. So if I want to do write code that looks like this, we're going to select, and I'm going to, let's create a single column selection. So this is going to be our column expression. This takes in a single name here. This is going to return a function. So we'll go ahead and return that pandas column function. This is the part of that API contract I was telling you about. This thing takes in just a pandas data frame.

And we're going to go ahead. We have both pieces of information. We have the name that's already bound in the scope, and we have the data frame that's going to be passed in. We can return the data frame name just like this. In fact, this should just be a series. So this code will look like this. We could also use .lock to get to the same place. So now I have something that looks like this. I have the column expression, which as I shared earlier, just takes in a string, and then it gives you back some function that you can then evaluate on a pandas data frame.

And so once we do that, that looks good. There's our name. This should be values for A. So we have our expression, which is the results of column A. And then we have the result, which is going to be calling the expression, giving the expression, the data frame back to that return type, which then selects out just a singular column. And that means in this select method, we do a lot of, we iterate over the expressions, we pass in the current data frame to that expression, and then we horizontally stitch together all of the results. And that's how it works.

So there's moving parts here for sure. And I glossed over some of the details. We don't just have a pandas data frame. There's also the DuckDB frame and so on and so forth that we've implemented the same interface for across the board. But that's pretty much how it works. You have a couple of classes, you stitch them all together, and then you keep passing one data frame to expressions, which are really just functions that we evaluate. And the beautiful part is if you're working with like, with polars, you have a, like a lazy, a lazy frame here, the expression that gets generated is just a polar's lazy expression, or just an actual polar's expression. And so you get to stay within the confines of your lazy system, you don't have to actually concretize or collect the data, and then use narwhals operations, it just passes straight through.

This is a great example, I think of the work that developers do to abstract away all of the complicated stuff that like, end users will find really tedious and be like, this is a roadblock for me, I don't I don't want to do all of these things. And the developers are in the background just taking it, fixing it, allowing you to just write your code and have it work. So thank you so much for being the back end that allows people to just do data science. This is super amazing.

Notes for R users

And I wanted to point out a couple of things that because we have that 70% our users here in our group, there are some things that might not have made any sense to you while Cameron was typing them a second ago. So if Cameron's willing to scroll up a little bit, I wanted to mention just a couple of them. Because again, data frames are native in R, and we don't think about some of these things. We also don't have some of these things in our everyday work.

So the first one, Isabella mentioned in the chat, there was a lambda function up there. That is sort of equivalent to an anonymous function in R. It's got a little bit of a different syntax, but it's really similar. So if you know what an anonymous function is in R, yeah, there we go. You'll understand that a little bit better. Don't let that throw you off.

Another one is the way that Cameron was importing things. So he was importing multiple things from the same library, right? So he was going through and importing things bit by bit. Whereas in R, we might just call library and the whole thing, and we pull the whole thing in. So that might be another thing where you're like, what is he doing? He's also assigning aliases for his packages so that he can call them like pandas as pd. He doesn't have to type pandas each time. He doesn't have to type polars or numpy each time. He can just type pl or np by aliasing them.

Another one is decorators. So you see he has a little at symbol and then context manager. Decorators can add functionality to functions basically. And that's not something that we have in R either. So if any of these things looked a little bit weird to you or even defining classes, don't worry. They are doing a lot of the same things that are happening in R. In R, they are very abstracted away. They're very, very behind the scenes, but they are happening still.

Okay. I hope that that was helpful for anybody who needed a little bit of a translation. I really appreciate everybody hanging out with us. Cameron, thank you so much for joining, talking about DataFrames in Python, talking about narwhals. Thanks to you and Marco for doing the good work behind the scenes. We really appreciate you, and I hope that you had fun too. Absolutely. This has been a blast. Thank you all so much for the great questions, and I'm glad that I got to do a little bit of live coding for you and explore, like I said, one of my favorite topics, and that is DataFrames.