image: thumbnail.jpg
Polars, pandas, and Narwhals, oh my! | Marco Gorelli | Data Science Hangout
Marco discusses his journey into open source, highlighting the importance of finding your niche and contributing to areas that others might find "boring" but are essential for a project's success. He also emphasizes the significance of fostering welcoming and inclusive open source communities.
This episode explores several key topics:
Polars: We learn about Polars' key features like lazy execution, its performance benefits, and the power of its expressions API. Marco explains how lazy execution can lead to significant performance gains and answers questions about its relevance even when working with smaller datasets.
Narwhals: We discover how Narwhals enables developers to write tools that work seamlessly with various data frame libraries, promoting interoperability and simplifying development workflows.
Open Source Challenges: Marco addresses the challenges of maintaining work-life balance while being deeply involved in open-source projects. He offers practical advice on prioritization and managing the constant influx of tasks and requests.
Resources mentioned in the chat:
Pandas: https://pandas.pydata.org/
Polars: https://pola.rs/
Narwhals: https://narwhals-dev.github.io/narwhals/
Awesome Polars: https://github.com/ddotta/awesome-polars
Ibis: https://ibis-project.org/
Polars Plugins Tutorial: https://marcogorelli.github.io/polars-plugins-tutorial/
Understanding Polars Expressions: https://www.youtube.com/watch?v=E7cHgN9rd9c
Great Tables: https://posit-dev.github.io/great-tables/articles/intro.html
Great Tables Blog - Polars Styling: https://posit-dev.github.io/great-tables/blog/polars-styling/
Great Tables Blog - BYODF: https://posit-dev.github.io/great-tables/blog/bring-your-own-df/
Ruff: https://astral.sh/ruff
UV: https://docs.astral.sh/uv/
R/Pharma 2024: https://rinpharma.com/
PyladiesCon: https://conference.pyladies.com/
Beeminder: https://www.beeminder.com/
Todoist: https://todoist.com/
Nyctography: https://en.wikipedia.org/wiki/Nyctography
This episode offers valuable insights for anyone working with data in Python, particularly those interested in exploring the benefits of Polars and the power of Narwhals for building data-frame-agnostic tools.
āŗ Subscribe to Our Channel Here: https://bit.ly/2TzgcOu
Follow Us Here:
Website: https://www.posit.co
LinkedIn: https://www.linkedin.com/company/posit-software
To join future data science hangouts, add to your calendar here: https://pos.it/dsh (All are welcome! We'd love to see you!)
Thanks for hanging out with us! š
Transcript#
This transcript was generated automatically and may contain errors.
Welcome back to the Data Science Hangout, everybody. If we haven't had the chance to meet before, and if this is your first Data Science Hangout, I'm Rachel. I lead Customer Marketing at Posit. Posit builds enterprise solutions and open source tools for people who do data science with R and Python. We're also the company formerly called RStudio. I like to add that in there. I'm joined by my co-host here, Libby.
Hi, everybody. I'm Libby. I help Rachel out with the Hangout community, and I'm also a Posit Academy mentor. So I mentor cohort-based groups who are learning R and Python to apply those to their everyday data jobs. So we're so happy to have you joining us here today.
The Hangout is our open space to hear what's going on in the world of data across all different industries, to get to chat about data science leadership, and connect with others who are facing similar things as you. And we get together here every Thursday at the same time, same place. So if you're watching this as a recording and want to join us in the future, there'll be details to add it to your own calendar in the YouTube description below.
But thank you so much to those who have helped make this the friendly and welcoming space that it is today. We're all dedicated to keeping it that way. If you ever have feedback about your experience that you'd like to share with me anonymously, good or bad, or maybe suggestions for topics to dive deeper on, I'm going to share a Google form in the chat with you right now, but you can always reach out to me directly on LinkedIn as well.
Awesome. We love hearing from you. Doesn't matter your years of experience, your title, your industry, what programming language you use or don't use, we really encourage you to connect with other attendees in the chat and also chime in to ask questions. This is a conversation that's really driven by the community. So drop in your role, where you are, something that's fun about you, definitely your LinkedIn profile so people can find you, or your website. And then there's three different ways that you can jump in to actually ask questions or give your perspective today. You can raise your hand on Zoom. We will call you to jump in and ask questions. You can put your question right into the Zoom chat, and if you happen to be somewhere loud or your mic doesn't work today, and you want us to read it for you, just put an asterisk before or after it, we'll read it. You can also ask in Slido. There's a Slido link where you can ask anonymous questions. Isabella just put that in the chat, and we will be sure to ask those questions anonymously for you as we get to them.
With all that, thanks again for joining us. I'm so excited to be joined by our co-host today, Marco Garelli, Senior Software Engineer at QuantSight Labs. Marco is a core dev of Pandas and Polars and the author of Narwhals. And Marco, I'd love to have you just kick us off by introducing yourself and sharing a little bit about the work that you do, but also something you like to do outside of work too. Sure thing. Hello, everybody. Thanks for inviting me.
Also, I'm glad that we just surpassed 128 participants. Yes, that's right. We celebrated powers of two, not meaningless numbers like 100. Who cares about those numbers? So, well, yeah, I think you gave a good introduction of what I do at work. I work as a software engineer at a company called QuantSight Labs, which does a bit of a mixture of open source and consulting and training, a mixture of things that deliver value to the open source community without necessarily giving a profit and things that are meant to deliver a profit so that we can fund cool open source work. Outside of work, I'm lately really passionate about playing Irish music, not because I'm Irish, but because where I live, there's an Irish pub, which every week hosts a jam session where anyone who plays an instrument can go along and play some Irish music with the hosts of the jam session. So if you have an instrument, you can just go along. They're very friendly. They show you the chords, show you the songs, and that's a major source of entertainment for me outside of work these days.
Introduction to pandas, Polars, and Narwhals
Got some really good guitar music coming out. Yeah, absolutely. So, Marco, I thought it might be helpful to maybe open up with this. So for people who might not know pandas or aren't familiar with Polars, can you talk about pandas, Polars, and Narwhals and kind of let us know why it was created, what's so special about it, and maybe some of its use cases? All right.
So pandas and Polars both expose objects known as data frames. A data frame is a two-dimensional object with which you can store data, typically in columns. So you've got a collection of columns. Each column has to have the same number of elements, and within each column, all the elements need to have the same data type. People in the chat are saying that they are two cute bears, and this is true. It does seem to be a bit of a law that in data science you need to name your tools after animals. I tried to follow that tradition with Narwhals. But pandas actually derives from a panel data analysis. A panel was some structure that was in the library, but it's no longer around.
And Polars was, so it's written in a programming language called Rust, where the file extension is usually .rs. So that's why Polars is named like this. It ends with .rs and also makes a reference to the pandas library. Narwhals, on the other hand, tries to bring them together. So traditionally, most of data science, let's say that pandas has carried a lot of the weight of the world of data science on its back for the best part of a decade. And most of the time when people write tools intended for data scientists, they are writing them with pandas in mind. So maybe people can pass in arrays, maybe people can pass in pandas data frames, and quite often that's about it. But things are starting to change. A lot of users are demanding that their tools natively support Polars data frames because they're, it's newer, it's trendier, it's got a bunch of improvements over pandas. Like it's a lot stricter, it has some parallelization by default, it's got a lazy API, it's generally speaking noticeably faster and stricter in a lot of respects. It helps you avoid making a lot of bugs that you would otherwise make quite easily using pandas. So there's a lot of demand now on people making data tools to not just support pandas, but also to support Polars data frames. So how do people do that?
You can either, there's a few different strategies. So let's talk about the different strategies. Let's just double check in the chat if there's any question that's come in that I should address before going into the different strategies. Polars was a lifesaver in my dissertation work. Wow, nice. Window functions, easy to use. That's very pleasant to hear. We can talk a bit more about that. I can see some people in the chat recognizing the Mr. Weeble reference from the novel's name. Glad you're getting it. Don't worry about the chat in Shrown. Okay, let's get back to the explanation then.
So if you want to make a tool that supports both pandas and Polars, what are your strategies? So one strategy is to just write your logic using pandas. And then if your input isn't pandas, you convert to pandas and you continue like that. This kind of works, but it's a missed opportunity. It could be a lot better to just keep things native to the library that people are starting with. Another strategy could be to duplicate your logic. So you've got your logic for pandas dataframes, your logic for Polars dataframes, but then what happens? What happens next year when somebody comes across with belugas dataframes? You're going to have to have yet another set of logic for those dataframes.
So what we're trying to address with Narwhals is a third way of doing this, which is just express your dataframe logic once using a nice little unified API that's extensive enough to be useful, but not too extensive that it becomes unmaintainable. And like this, it can just dispatch to whatever input your user provides and you enable your users to achieve the concept of BYODF, which is not a system of a down song. It stands for bring your own dataframe.
it can just dispatch to whatever input your user provides and you enable your users to achieve the concept of BYODF, which is not a system of a down song. It stands for bring your own dataframe.
Lazy versus eager execution
Awesome. I wonder if you might expand upon the lazy execution part of what you said. So you talked about lazy execution, and I know from talking with you before that lazy versus eager is a thing. And it's not something that I, as a data person, have thought about very much. I don't use huge, huge data either. So maybe you can help explain that for people who don't get that intuitively, like I don't.
Absolutely. I would love to talk about that. There's also a very good question in the chat. I know you told me to not check it too much, but there was a good question. I wasn't able to not see it. And that is, what about just using PyArrow? So I'm going to make a mental note to come back to the PyArrow question. So first of all, lazy. Lazy execution. Right. This is one of the nice features that Polars has on top of Pandas. So with Pandas, everything happens eagerly. You give the library some instructions and it evaluates your instructions the moment you give them. Whereas Polars, it's got an option whereby you can give it some instructions, it can wait a bit, and then it can do them all together in the most optimal way that it can detect. If I was to tell you to cook me a recipe and I gave you the steps one at a time, and you have to do the steps one at a time, and for each step go out and buy the ingredients that you need for that step, it's going to be a pretty inefficient way of making a cake. But if I can give you the recipe beforehand and you can buy all the ingredients together, then maybe do two steps at the same time, it's going to be a much nicer cooking experience.
On the question about, I don't deal with big datasets, why should I care about lazy execution? I would say that performance doesn't matter until it does. And that when you've got something that's categorically faster, it changes the kinds of questions you can ask. So in a previous company I was working at, I remember that there was some workflow where we were making weekly forecasts or something. There was like a very complicated process, like the forecasting process took like hours. But the moment when we modernized the tool chain, used some better algorithms and we got the time from minutes, sorry, from hours to minutes, then all of a sudden the questions that product management started asking were, well, maybe we can run daily or bi-daily forecasts. So yeah, categorically better performance really changes the kinds of questions you ask. And then at some point, even if your data isn't big now, maybe it will be later. So it's a good thing to start with scalability in mind.
performance doesn't matter until it does. And that when you've got something that's categorically faster, it changes the kinds of questions you can ask.
PyArrow and data frame interoperability
Absolutely. So I know you already saw the question around PyArrow. So, and I see Abigail and Glenn both had that question. Glenn, do you want to jump in at all and add any context to that?
Yeah, I was actually specifically asking about using the PyArrow backend that you can use with Pandas and kind of that as part of the infrastructure when you're working it together with Polars. I was just kind of looking at an overall perspective of that.
Yeah, sure. Let's talk a bit about that. Before going into it, I'm just noticing there's some familiar faces, some familiar names here. Maybe I'll just say hi to the people that recognize. Hi, Barrett. We've interacted a fair bit on GitHub. Yeah, if there's anyone else that recognize, I might say hi. But in the meantime, let's go to PyArrow.
So as of Pandas 1.5, you can set PyArrow back to data types. And for some data types, that makes a big difference in Pandas. So for strings, it makes your operations a lot more efficient. And for integers, it means that you can store missing values properly. Whereas for the classical integer data types in Pandas, the moment you've got a missing value, then your column becomes a float column. And then the missing values get represented by none, but none isn't really a missing number. It's more like not a number. So yeah, lots of people excited about PyArrow.
And some people suggest this as a solution to data frame interoperability. Like why can't we just write our tools using PyArrow? And then why do we even need Narwhals? Everyone can just convert to PyArrow. So yeah, let's address this. And it goes back to Libby's question about lazy execution. So PyArrow, it's a very... So Arrow is a nice memory format. PyArrow specifically is a library in Python, which implements this memory format along with some compute functions. It's nice and but it's all eager execution. So if you have the possibility of keeping your workflow lazy, then being forced to convert to PyArrow breaks the lazy execution. It's a missed opportunity.
Furthermore, PyArrow is a fairly heavy dependency. Let's have a show of hands. Has anyone here run into dependency hell when working with Python projects? Yeah, I remember one company I worked at, I was not able to upgrade the dependencies of a project because two different libraries had pinned TQDM to different versions. Like TQDM, it's just a progress bar. And we weren't able to update our dependencies because of a progress bar, which I didn't even want to see. So since then, I've become rather anti non-required dependencies. And I think it's a pity if a dependency as heavy and as difficult to install as PyArrow becomes required in too many places. By all means, use it if it brings an advantage. But I really hope that the solution to data frame interoperability does not become PyArrow. I think we can do better.
So, Marco, I learned from you that great-tables, which our team works on, plays really nicely with
polars because they make polars expressions part of the API. And I see that you know Barrett here as well, who works on Shiny. And I just kind of wanted to open up the conversation a little bit more about that and just hear a little bit about how you found out about great-tables or how that collaboration came to be.
Sure. I'm trying to remember how I first came across it. It was probably on some social post somewhere. It might actually have been on the Polars Discord, where one of the great-tables devs, maybe Richard, maybe Michael, one of them had posted about it saying, oh, we realized that the polars expressions are really nice. Like we started with supporting pandas and then when supporting polars, we decided that we can do better than just trying to mimic what we're doing for pandas. We can go further. We can allow expressions to, we can allow users to pass expressions and like this have a really nice readable polars idiomatic, polars idiomatic experience. And yeah, I really liked it and figured, I then got a bit angry when I saw a bunch of people online saying, oh, but I like polars, but it doesn't have any way of displaying tables. I was like, okay, well, there's some great-tables, which does this really well. So maybe we should hook that into polars.
So as of polars version something one, can't remember if it's version one or some late 0.20 version, there's a dot style namespace with which you can access some great-tables functions and get your data frames beautifully displayed to you. Like I was really shocked by how beautiful these displays could be with just a few lines of code.
That's awesome. I love that like social media kind of gets a bad rap, but it's amazing what you can bump into, right? Because you just came across this.
Yeah, yeah. I mean, American Management with social media is to just put some guardrails on it. So I might be a bit more extreme than most people on this. Like, I'll tell you what I do. So I've got an app which records how long I'm spending on social media sites with my browser. And then this, this is recorded by another app called Beminder. And if I go above a certain threshold, then it actually charges me money.
I think I might need that. I was just going to say, I was going to say that's not extreme at all. I'm pretty sure we can all benefit from that. Yeah, because there are positives. And I think if you can just avoid the scrolling part of it, then you can get the positives without getting, without it being too much of a time sink. Absolutely. I was thinking that you were going to say that you like curate your LinkedIn feed and block people or add people or mute people, which is something I recommend or try to do. But that is a fantastic point. I'm gonna have to like listen to this back and take notes.
Making open source more accessible
We have a question from Slido that I would love to ask, and it is, how can we make open source more accessible, especially for underrepresented groups?
All right. I think I've got the answer. Yeah, I think if you consider the topic of women in open source, there's typically relatively few in most packages. But in Narwhals, I think we've got a far higher than average number of female contributors. And I think part of the reason is that when we approve pull requests, we give people cute animal GIFs. Now, I'm not saying that this is the solution to diversity problems in tech, but it is the differentiator and it helps make the project more fun.
And we've got a zero tolerance policy to unpleasant people. So my recommendation would be to, like, you know, if you're relying on volunteer labor, then try to make it fun. Try to encourage people. There's, I think there's a fair bit of evidence that being nice and encouraging works a lot better than being critical and responding to people with minus one the moment they make a suggestion. Yeah, if you're relying on volunteers, they're not very likely to make other suggestions if their first experience with your library is minus one, this is a terrible idea.
Yeah, if you're relying on volunteers, they're not very likely to make other suggestions if their first experience with your library is minus one, this is a terrible idea.
So yeah, be extra nice to everyone and give people cat GIFs. I mean, if another big problem that open source has is that of funding. And if we did have more funding in open source, I think it would be good to prioritize mentoring people from underrepresented groups.
I know this isn't necessarily, like, on contributing to open source packages, but just to add to this point, something that I've found really, really helpful for this in community building is to actively reach out to people as well, instead of just waiting to see, like, who is reaching out to you or who is showing up to events to actually reach out and invite people or let people know when you saw about something great that they did or a blog post they posted on LinkedIn, reach out to them and let them know. And it's a way to kind of open the door to a conversation to them one day, contributing as well. Yeah, totally. Really, really back that you can sometimes give it for granted that people are self-confident that, you know, of course, this person knows they're great, but people are often insecure and just encouraging them can go a long way.
I see that Barrett in the chat says that it was a very, with very shouted positive experience contributing to the Narwhals repo. So, yeah, thank you, Barrett. Really appreciate it. Loved your contributions. Too many fun GIFs. Yeah, you see what I'm saying.
Conference trends and boring tools done well
Well, Marco, I know when we were chatting before, you had mentioned you were at PyData, I think in Paris, and then you were going somewhere else and been to a number of conferences lately. And I was wondering, what do you see people getting excited about right now?
I can see a lot of talks about LLMs and yeah, I guess a lot of people are excited about that. It's not something I'm personally too excited about though, so not something I can really comment on, but something else on the total flip side of that, what a lot of people seem to be getting excited about is really boring tools as in boring tools, but that work exceptionally well.
So who would have thought that in 2024, it would be a good idea to re-implement PIP? But there you go, the astral folks have done it and UV is taking over Python packaging. And same with, yeah, same with RUF. So yeah, boring tools done well. I think there's a lot to be said for that. I mean, when I was chatting to Rich and Michael as well from Posit about great-tables, I think they were saying something along these lines that some people would consider displaying data frames as tables to be a quote-unquote boring topic, but there's actually just so much to it. And if you do it well, there's just so much scope to do something that gets people excited.
Getting into open source
You know, Marco, speaking of boring tools done well, I feel like that's something that you had as an entry point into open source. And I was wondering if you could talk a little bit about that, like how did you get involved in open source? Because I know the story, but I think it's really great. Right, sorry, I started laughing earlier than I should have, because I remember the conversation we had about it. Yeah. All right, we've had the IBIS question in the chat. Great. I'll need to address that at some point. That's the number one question we get about Narwhals. How does it differ from IBIS? So before we do that, yeah, entry point into open source and in particular polars. So I think when we chatted about Libby, I'd quoted to you the phrase, if there's something which you find interesting, but which other people find boring, then that's your competitive advantage. And when I started using polars, I noticed that a lot of the time zone code just hadn't really been done properly. Like there was in theory, some support for time zones, but it was typically just convert to UTC, apply an operation and then convert it back, which isn't typically correct. Like it works in some cases, but you know, if you need to report to your boss, how many sales you've made per day, then the definition of day better respect the time zone that you're selling in. If it respects the day boundaries of UTC, then it's not a very useful analysis. So yeah, I've got a bit of a niche interest in time zones. Not really sure why.
Maybe, well, I live in the UK, parents live in Italy. So I don't know, just always thinking about difficulties with scheduling and always having to remember in which direction to the sum or subtract hours. Maybe that's part of it anyway. Had a bit of a niche interest in time zones, but I didn't know Rust. So how was I to contribute to polars? Well, just by doing the time zone things that nobody else wanted to do, and in the process learning Rust.
So they were willing to overlook the fact that you didn't work in Rust because you were willing to do the time zones, which absolutely nobody wants to touch with a 20-foot pole. It was something like that. Yeah, yeah. And I think it ended up being a win-win situation.
If there's something which you find interesting, but which other people find boring, then that's your competitive advantage.
I think that's a fantastic piece of career advice. Thank you so much. They were extremely welcoming and friendly. So yeah, I'd also like to put in a good word for polars people if people want to find a nice project to contribute to.
Oh, that's good. Yes, absolutely. And I have a question in the chat here. Marlene, would you like to ask?
Oh, man, you always get me when I have food in my mouth or- It's okay. Just cracking up. Hey, Marco, thanks for joining us. My question's actually pretty simple, I think. Just wondering what's next? What's working on your side of town? Maybe a vision that you have for a new endeavor? What's coming up next?
What's next for Narwhals
Sure. So top priority for now is Plotly. So one of the main contributors to Narwhals, Francesco, he's opened a pull request to Plotly to Narwhalify their Plotly Express part of the code base. So Plotly folks have been very encouraging about that. They've marked it as a priority one in their repo as well. So it's top priority for a lot of people, but that's really at the top of my mind at the moment. And then I'd like to see where else we can go with Narwhals. There's some other projects that have showed interest. So there's the Profit Library, there's Scikit-Learn. It would be unreal if we were actually able to make it into Scikit-Learn, so I'm not holding my breath on that one. But I think at some point I'll just open a pull request, get ready for it to be rejected. If I start with low expectations, then I won't be too disappointed.
Marco, I... Oh, sorry. Go ahead, Marlene. No, just saying thanks. I was just saying, Marco, you're doing an awesome job as also being a co-host here and seeing the questions in the chat. And I know you mentioned one about Ibis, and I think it was Sunday. Sunday, do you want to jump in and ask that?
Sure. Marco, great, great, great talk so far. And it's inspiring hearing how you got into doing Rust. I've been thinking, how do I get into that space? Maybe I just need a small project, something I find fascinating sufficiently. Now, my question is about the Narwhals project here. And I mean, I like to look at different tools and see what fits best for what. But I'm always looking out for that one thing that will do most things well, so I have less of a need to jump. And recently, I've been looking at Ibis. And I like Ibis. I like Ibis direction, big data, syntax, tidy-verse-like, and all that. So how does Narwhals compare to this? And what are the benefits? What should I look out for? I'm willing to explore and see what I might find there.
Narwhals vs Ibis
Sure. Let's talk about that. So perhaps for people who don't know, Ibis is a project which describes itself as being a portable data frame library. So the idea is that they've got this Python API, which then gets translated to their intermediate representation, and they can then translate it either to SQL engines or to Polars. And like this, with just one API, you can target different engines.
And it sounds a bit like Narwhals. So how do they differ? When would I suggest using one over the other? And if you're an end user, if you need to do some complicated analysis, if you're an ML engineer or whatever, then I'm not sure that Narwhals is necessarily the best tool for that task. So with Narwhals, we're trying really to target library maintainers. And to do that, we're making decisions based on what library maintainers need. And I don't think that our API is extensive enough or that it will be extensive enough for the kinds of really customized, really complicated analyses that a lot of analysts and ML engineers are doing. So a SQL front end like Ibis might be a better choice for that. Conversely, for the kinds of tools that people are building using Narwhals, I don't think that Ibis is a very good choice. And a few reasons for that are that a lot of tools that use Narwhals
really do a lot with eager execution, whereas with Ibis, everything is lazy.
And with Narwhals, we've got both the lazy and the eager execution, so you can choose. But at some point, you may well need to do things eagerly, and we allow that.
The other is that with Ibis, every back end requires Pandas and PyArrow as required dependencies, whereas with Narwhals, we've been very strict from the beginning by saying no back end should have any extra dependencies.
Another is that with Ibis, there's no support for categorical data.
Sorry, we really need some water. I'll be right back.
Bear, I know you're listening in right now, and I know there's some work that's been done recently with Narwhals and Shiny, and I was wondering if you could tell us a little bit about it, and I'm sorry to put you on the spot. No worries, it's not a good time.
Oh no, now it works. Yeah, so Shiny, I'm sad to say we didn't get the release out for this Hangout, but Shiny will have a release in the very near future, and this means that Render DataFrame will have support for Narwhals. So if it's an eager DataFrame, so Pandas, Polars, PyArrow, and if you can get CUDF installed, then congrats. It'll also work, but yeah, it'll work right out the box, and then if you ask for any of those methods on the render object, like .data or .dataview, that'll also be in your native data that you supplied back, and that can also include Narwhals. So if you return Narwhals data, you'll get Narwhals back that's been subsetted or filtered and ordered appropriately. Thanks, Bear. You were ready to go with that fancy microphone there too.
We're on these all day. I gotta have it set up. Love it.
Do you know when, what is it, version 1.2? 1.2. It's tonight or tomorrow, but it's been slowly creeping, so we'll say next week it'll be out, and then yeah, it'll all be integrated and everything, just waiting on some other features. The Narwhals stuff is already there in Maine. Oh, thanks. Very exciting.
Should we video? No? Yes? Don't want to steal here. Oh, no. If you want to share the link, can you share the link to it in the chat? I didn't know. I don't have a public link. I just have... Okay. If we have it tomorrow or whenever, I could put it with the recording too. Perfect. We'll do that.
That works. Okay. Awesome. Marco, I didn't want to cut off your train of thought from before. Is there anything else you wanted to add from the previous one?
Yeah, sorry. I think that was the ghost of Ibis punishing me for saying that. No, but yeah. Got myself watered down. Okay. It's in good state again. Good. If you need a second, let us know.
Yeah. Just to finish, it was just that. If you've got a workflow where you want something like DataFrame X in, DataFrame X out, then that's not really what Ibis was designed for, and it doesn't preserve everything from your original DataFrame. So they're really just different tools with different use cases in mind. Thanks. I think I'll mess around with Narwhals and see how that goes.
Managing large projects and prioritization
So there was a question on Slido a little bit earlier. Someone asked, when working on large projects, I get feelings of drowning in work and not making progress. Can you give me any tips from your experience of large projects?
Ruthless prioritization. I think you need to just say no to things.
This, yeah, we've talked about career advice. So here's another career advice. I think this one might come from Warren Buffett, maybe, which was about, I think the story is that he asked someone like, what are your top 30 goals? And yeah, how many of these do you plan to address in the next year? And this person says, oh yeah, I might do the top five. Maybe I can do the top 10, top 15, if I can do it. And Warren's reply is no, no, you've got it all wrong. You need to circle the top five and everything else you consciously do not work on because it's going to distract you from the top five. So I think you just need to be ruthless in saying no to things. Like not everything can be fixed, not everything should be addressed. And keep the end goal in mind. Don't chase rabbits when you're out hunting elephants.
Keep the end goal in mind. Don't chase rabbits when you're out hunting elephants.
Life at Quansight and open-source as a drug
I like it. Thank you. We had another great Slido question and it was, can you tell us more about QuantSite? How does your time split between your open-source contributions, business consulting, et cetera, and what type of consulting engagements do you do?
Currently it's about 50-50. So yeah, I've got a contract with a company and yeah, which is just around software engineering related things. And yeah, apart from that, there's developer relations work and yeah, some unfunded open-source work.
So yeah, really grateful to my company for allowing me to spend some time on my open-source addiction. I think it's really an addiction at this point. It's a legal drug. So yeah, there's some openings. Apply to QuantSite if you're interested and reach out to me if you want a referral link. Thank you, Marco. I have heard you mention that open-source is a really legalized drug before. Can you talk about why you feel that way?
Sure. Although first I need to address the comment in the chat, which is addicted to code. No, I hate code. I want as little code as possible. I'm addicted to open-source, like just tools that solve problems and things, but code, I'm addicted to open-source. So yeah, I'm addicted to open-source if there was a way to do it without code, that would probably be better.
So sorry, I just got distracted by that and I forgot some of the questions. Do you mind repeating, please? Sure. So you have mentioned before that you find open-source to be a legalized drug and that people should maybe be careful around entering this space. And I was wondering why you said that, like what your experience has been around it with like work-life balance and stuff. Yeah, I think when you start making contributions, it can be very rewarding to fix things and to add features which are going to be used by millions of people.
On the flip side, you start feeling a bit of responsibility for things and you become really attached to the things you've done.
And it can be a bit dangerous, like if you address all of the issues, all of the pull requests that are coming in, there's just not enough time in the day to do that.
Furthermore, it's possible to keep yourself endlessly busy with relatively unimportant tasks which maybe just affect 0.001% of users. So in terms of maintaining a healthy work-life balance, I really suggest more prioritization. Like if you label an issue as low priority, it's not because you're trying to insult the person who's opened the issue, it's just that and it's not that you don't think it's important or that you don't acknowledge that it's an issue, it's just there's not time for everything.
Thank you. I mean, on that I try to distinguish between when is it, when is something raising an error when it shouldn't be and when is something giving incorrect results. Like if you're giving incorrect results, I think that's all, it's usually either high priority or critical priority, whereas something that raises an error when it shouldn't, even if it's in a fairly popular part of the code base, like if there's a workaround, I don't think it's critical priority. Like you can give yourself permission to take a break from work if there's something like that and get to it the next day.
Extending Polars for ML and plugins
Yeah, I'm just curious if something like Polars, you know, in the future, if it could potentially train ML models as well without having to leave the sort of Polars paradigm. Like, I mean, Spark works really well for distributed jobs, but it's one of those things where Spark works beautifully on my system, but then when I'm interacting with, you know, external systems, oftentimes you'll run into some kind of Spark or Spark dependency issue. And I'm primarily an R user, so I'm, you know, using Sparkly R or Spark R or something that's kind of running Python and Reticulate in the back end.
Yeah, so I'm just going to share a little resource in the chat, which is a tutorial for how to extend Polars with your own plugins, because something like training ML models is almost certainly always going to be out of scope for Polars itself. But good news is you can extend Polars, and this is what people have been doing. There's a really nice plugin called Polars DS, and there they've got lots of ML models. There's also Polars OLS. If you look at it, there's a GitHub repository called Awesome Polars. They've got a section there specifically about plugins, and people have made all kinds of extensions for ML-related things.
Cool. Have a good one, Lisa. And yeah, then like these, they just interact regularly with your Polars workflows. They're just regular expressions. They typically- This is great. Yeah, thank you. I appreciate the share.
It is, yeah. Always nice when you're able to answer a question, and the person says, this is great. Nice. Thanks. Yes, that's exactly the repo.
Organizing knowledge and managing tasks
Marco, another Slido question that was asked a little bit earlier from Ricky was, do you do anything particular to organize your knowledge in a way that you can share it with other people? And the person says, do you do anything particular to organize your knowledge or information so it doesn't grow in an unsustainable way? Any tips?
For tasks, I use Todoist, and there you can set labels and you can set priority levels to things. So yeah, pretty ruthless about checking that both on my phone and on my laptop. So there's a book called Getting Things Done, one of these self-help books about productivity, and I think most of it is not worth reading. But there is one lesson in there, which is about having a- I couldn't even remember the exact name. It was something like a capture-everything device.
So for me, having the Todoist app both on my phone and on my laptop, chances are, whenever I'm around and I think, oh, I need to do this, I just add it to my app. And one insight that I got from that book was that it's a major source of stress in your life when there's open loops. So that is to say, when there's something that needs doing and it's not captured somewhere. Like if you're just about to go to sleep and you think, oh, I really need to remember to do this tomorrow. I really need to remember to do that. It's going to stress you out. It's going to be very difficult to get to sleep. But if you know that it's in your app and the next day you're going to look at your app because that's part of your routine, that's part of your routine. And there's an app, there's another app. In my case, there's another app that charges me money. If I don't regularly look at my Todoist app and tick off boxes, then yeah, you're going to look at it. And yeah, in the long run, I think this really reduces your stress. So the other app that, if you feel like you don't have enough self-control, that you really need an app to charge you money, if you don't stick to your goals, then Beeminder is the tool for that. I can see a few people looking like they're writing this.
One insight that I got from that book was that it's a major source of stress in your life when there's open loops.
That's amazing, Marco. I feel like you hold yourself hostage a lot to get things done.
It actually reminds me of Nictography, which is what Lewis Carroll, the writer of Alice in Wonderland, invented a way to write in the dark so that he could wake up in the middle of the night and write things legibly in the dark. Go Google it if you're interested. Nictography is fascinating.
Narwhals and DuckDB
But we had a question from Thomas in the chat. Thomas, are you still here? Would you like to ask?
Yes, I am here. And I'll go ahead and roll in the other DuckDB-related question. I was curious about interaction between Narwhals and DuckDB, and I was able to find something on the roadmap about it. There was another question from Scott about competing interfaces, such as DuckDB, Pandas, Polars, PyArrow. If you could just speak to how Narwhals might work with or be competing with some of the work being done with DuckDB and others. Awesome, yeah. I wouldn't consider any of those to be our competitors, and I'm not sure that, like for some of them, I wouldn't say that the competition is on the interface but on the engine. So let's say Polars versus DuckDB, I think
there is some open competition for who's the fastest, who's the most scalable, and all of that. But on the interface they are completely different. So the way you interact with DuckDB is by writing SQL and Polars has its own API. For Narwhals itself, I'm hoping that we can get to defining a lazy-only layer of support that we should support DuckDB in. But going back to priorities, first we need to get the Plotly integration in. That's the top priority. And next year in 2025, when I've got a bit more headspace, I'd really like to focus on doing, yeah, on defining the rules behind SQL engines for Narwhals. The main difference that we have there that we need to address is that with SQL engines you don't have a defined row order, but in Polars and Pandas, the row order is always defined, even if it's a lazy data frame. So I'm going to have to figure out what to do about that. Like we've got some methods like a cumulative sum, a cumulative sum makes sense if your row order is defined, but if you don't have a row order, then you need to define what your cumulative sum is ordered by.
So what we might do is just have like a cumulative sum, which is available in Narwhals, and then cumulative sum by, which is also available. But if you want to use the lazy only layer, then you have to use cumulative sum by rather than cumulative sum.
Thank you. Scott, I see you had a question that also touched on DuckDB, and I just wanted to make sure it covered that question, or if you want to jump in.
Sure. I guess my question was more about ongoing projects, not like an individual contributor. There's always, you know, projects tend to evolve over time. People's use cases change over time. And if you're committed to a particular way of interfacing with data, such as Pandas, you don't want to change midstream to other types of interfaces because it causes changes in the code base. People who work on the project after you have to now learn a different way of interacting with things. And, you know, as you know, the exposed interfaces potentially have different models of how one works with the data. And so over time, it gets confusing. That's why I put that XKCD link in there, because now you've added another abstraction layer with another level of interfaces that supposedly are going to unify, but in fact just become an additional set of things that people have to maintain and manage. And so it wasn't so much of the implementation details, but how people actually work with the tooling and the APIs and method calls and interfaces that they have to get familiar with in order to be productive in doing the type of data analysis that they are interested in doing.
Yeah. Yeah, it's true that things, yeah, I'm familiar with the XKCD comic, the one of the, I've not clicked on the link, but I think I know which one you're talking about. The one of that is graphical standards in Python. And now there's 15 standards. Yeah, yeah. That's right. Exactly. Right. So, yeah, so my vision with Narwhals isn't that this brings everything together. It's, my vision isn't that users will use this. I think users should use, learn the best task for their, use the best tool for their task, which nowadays for most data related tasks, I think, is probably going to be either Polars or DuckDB. I really recommend learning both, to be honest.
And yeah, we're not trying to unify everything for end users, but we're just trying to address the much smaller problem of how can people write data frame agnostic tools. Thank you.
Oh yeah. So yeah, Barrett gives a nice comment about that in the chat saying that they were able to reduce the number of interfaces by just integrating Narwhals. And before they had to write adapters manually, and now there's just a single interface. What you see with a lot of data frame consuming tools is that the kind of things they're doing with data frames, it's it's a lot more limited compared to what end users are doing with data tools. So there, I think it is feasible to bring things together.
Choosing between Pandas and Polars for new projects
I wanted to just add real quick, we're trying our best to keep up with questions in the chat, but if there was anything that you had asked earlier that we didn't get a chance to cover yet, or we missed, feel free to repost it in the chat here, so we can, you know, bring it to the top as well. But I thought it might be good to go back to a little bit of what we were discussing in the beginning about Pandas and Polars, and think about, for someone starting a new data project in Python, what factors would you recommend considering when choosing between Pandas and Polars? But maybe also touching upon, you know, what are some of the things that you but maybe also touching upon, we all said dependencies being a pain, how we can maybe smooth that process as well.
Yeah, if you're starting a new project, I'd really recommend this as the best time to try out Polars. You know, if you've got an existing project, my recommendation is if it's not broken, don't fix it. But if you're starting a new project, I think it's, there's not, like, unless you need something which only Pandas does, like maybe complex number support, or some geospatial functions. But even then, DuckDB has got good support for geospatial, so you can always just get around it with that.
Yeah, so it, I think in the previous person who asked a question, there was a bit of a comment that we become attached to the code we write. And it's been a bit painful for me to maintain Pandas for years and then come to realize that actually it's probably just not the best thing going forwards for new projects. But yeah, it's just something we've got to contend with. Different tools are going to come along, and yeah, we've just got to be ready for that.
And there's a little, you can find that if you, if you install narwhals and then do import narwhals.this, there's a little Easter egg there. And one of the phrases is, our code is not irreplaceable. I just want to try to remind people, like, don't get too attached to the code you've written.
our code is not irreplaceable. I just want to try to remind people, like, don't get too attached to the code you've written.
Thank you. I was wondering what some of the key differences are that people might notice. Like, if you are used to working in Pandas and you're like, okay, I'm just going to start a new project. This is it. I'm going to try Polars. And they jump in and start learning Polars and using it instead of Pandas. What are some of the differences that they might notice? Like, I think one thing when I was looking through was, you don't need to reset the index. And that made my heart pound a little bit. I was like, wait, really? Like, that would be amazing. So I was wondering if you could talk a little bit about that.
Yeah, I love that, that the index can be powerful if you use it correctly, but nobody does. So for most people, it's just an annoyance.
So yeah, I would say the main difference when coming to Polars from Pandas is that the way you interact with data frames isn't with square brackets and series, but with expressions. And we don't have that long left. So if you want a more detailed answer, then I'm going to plug my own talk from PyData Amsterdam called Understanding Polar Expressions When You Use Pandas.
Awesome. Thank you. I'm sure we'll get that in the chat somewhere, a link to that.
Map elements and open source fun
Great. Well, I know we only have five minutes left here. Is there something that you wanted to chat about with us today, Marco, or something that you're really excited about that we haven't got a chance to cover? Okay, yeah. Well, there was just a comment about map elements in Polars. So map elements is the way that you can apply user-defined functions row by row. And it's really a last resort function. It's something you should only use if you absolutely have to. Usually it's just more efficient to use the built-in expressions API or to write a plugin if you really need to. But regarding map elements, one of the most fun experiences I've had in open source was when I just got so angry seeing some blog posts where people were saying, ah, Polars, it's not really faster than Pandas because look at this function where I used map elements and it's not very fast. I was like, okay, there needs to be a way to tell people not to use map elements. So my first suggestion was to rename it to map elements underscore slowly, but this got rejected.
I'm not really sure why. I think it was a brilliant name, but no, it didn't happen. So what we then did was we would take the lambda function which people pass in, we decompose the bytecode, we reverse engineer the bytecode to figure out what the user had passed in, and then we can emit a warning telling the user what to do instead. And yeah, that was wild. Most fun, one of the most fun open source experiences ever. We can't do it for everything. If we could, then we could just do it for them instead of emitting a warning. And I really don't want to do it for the user in cases when we can detect it because then people don't learn. So yeah, we emit the warning instead of doing it for you to teach you a lesson.
Upcoming events and closing remarks
Thank you, Marco. Are there a few upcoming workshops or conferences or talks that you might be giving that we should keep an eye out for?
So, well, in-person talks, I'm out-talked, out-conferenced for this year. I've done too much. But there will be PyLadiesCon towards the end of the year, and one of our regular contributors, Magdalena, will give a talk there, and might even lead a sprint. So if anyone's interested in contributing to Narwhals, I'd really recommend attending that.
That's great. Let's see if anybody could share that link in the chat with us. That would be awesome. And Libby did just put out a link in the chat about RnPharma happening next week, and you can go and sign up for the free workshops there. And there is a link. I think that link will go directly to the workshops, right, Libby? I think that's the main page, but if you sign up for the conference, then you can register for the workshops. And yeah, I highly recommend doing them. I am not in pharma. I attend RnPharma all the time. It's amazing, wonderful. Go watch the talks. There's so much to get from it, even if you're not in pharma.
Well, thank you so much, Marco, for taking the time to join us today and sharing your experience with us all. This has been great getting to learn from you.
Thank you so much for inviting me. This was a lot of fun.
It's always amazing how quickly an hour goes by, but Libby and I will write up some episode notes and put that with the recording. We're getting better at sharing the links that were shared in the chat as well up with the recordings, so hope to have that in the next few weeks. But I do just want to say, if this was your first Hangout today and you want to join us again, they are every Thursday from 12 to 1 Eastern Time. And I'll just put this little short link in here where you can add them to your calendar if it's not already there on the recurring calendar invite. So next week, Steph Locke at Microsoft is going to be joining us. Steph was a recent keynote speaker at Earl and came highly recommended from Hadley Wickham. We'll be talking about business as usual and how it can slow down the development cycle and what we can all do about that. Thank you all so much. Nice hanging out with you all, and I hope to see you next week.