Resources
Apr 25, 2024 |
52 min
|
2k views

Why he created pandas, the future of data systems - Wes McKinney - The Data Scientist Show #086

Wes McKinney is the co-creator of pandas library and he is the cofounder of Voltron data. Currently he is a principal Architect at Posit and an investor in data systems. Subscribe to Daliana's newsletter on www.dalianaliu.com for more on data science and career.

Daliana's Twitter: ⁠⁠https://twitter.com/DalianaLiu⁠⁠
Daliana’s LinkedIn: ⁠⁠https://www.linkedin.com/in/dalianaliu/⁠⁠
Wes' LinkedIn: https://www.linkedin.com/in/wesmckinn/

00:00:00 Introduction
00:00:39 How Pandas Started
00:06:35 Voltron Data
00:09:58 Benefits of Easy-to-Use Data Tools
00:13:15 The Rise of New Data Tools
00:18:02 Choosing Tools: Vertical or Flexible?
00:22:56 Big Models and Data Tools
00:29:24 Challenges in Building a Product
00:31:23 Becoming a Top Architect
00:34:50 Missed Aspects of Previous Roles
00:38:59 A Busy Week: Advising, Designing, Investing
00:43:37 Improving Open Source
00:45:19 How to Decide What to Work On
00:46:23 What he’s learning now
00:47:51 Excitement in Career and Life
00:48:24 Using ChatGPT for Learning
00:50:22 Future Impact Goals

image: thumbnail.jpg

Transcript#

This transcript was generated automatically and may contain errors.

Hello everyone, welcome to the Data Scientist Show. Today we have Wes McKinney. He's an open source software developer and entrepreneur focusing on data processing tools and systems. He is the co-creator of Pandas Library, the author of Python for Data Analysis, which I have a copy of. And he's the co-founder of Voltron Data. Currently he's the principal architect at Posit. He also just launched his MicroVC fund, investing in data infrastructure, AI, and ML companies. If you like the show, subscribe to the channel, leave a comment, and give us a five-star review. Welcome to the show, Wes.

Thanks for having me.

How pandas started

So I have to ask, what's the backstory of Pandas? What was the motivation?

Yeah, well, my first job out of college was in Quant Finance. I had a math degree, and it was 2007, and it was the great financial crisis that just started. And I was growing frustrated because I was under a lot of pressure to do analysis work, and I felt that I wasn't able to do it quickly enough. And I had been introduced to Python programming, and I said, hey, this programming language is really great, but it's missing some data analysis tools, like the kinds of things that I had seen some of my colleagues programmed in R. And so I wanted to have some of the same kinds of tools in Python that I saw in R. And it was also like a way for me to learn Python and have exposure to building a software project.

But it started out as tools for myself. And at a certain point, I started socializing it with my colleagues. And they also really liked using it. And then I convinced the company where I worked, AQR, to let me open source it. And so we open sourced it at the end of 2009. And so I gave my first talk to the Python community at PyCon 2010 about the project. I think that video is still online somewhere. You can go see it.

And yeah, I started grad school in 2010. And then at some point, I realized that there was a huge opportunity to make Python an important language in statistical computing and data science. And so I dropped out of grad school to work full-time on Pandas starting from like May 2011 and spent a little over a year working full-time on the project. I wrote my book, Python for Data Analysis. And yeah, it really helped fill out the features in the library and build the initial open source community.

And sometime in 2013, Changshu and I, he's one of the early developers of Pandas. He was also at AQR with me. We decided to start a company. And there were other people that we had gotten involved in Pandas development. So when we got busy with our startup, Datapad, we turned over the Pandas project to the other core developers. So we haven't been so actively involved in Pandas development since like 2013, 2014. And I've been working on other projects in the meantime, but obviously Pandas has become extremely successful. And we're all very happy that it exists. And it's been a useful tool that's helped the Python data science ecosystem get as large and widespread as it is.

And why Pandas? I got a lot of questions from my followers.

So at the time, I spent a lot of time trying to come up with a name because I was like, this is a Python data analysis toolkit. And I was working with all these econometricians who spent a lot of time talking about panel data. And so I was like, Python data analysis, panel data. I sort of was mashing around syllables and letters. And I was like, oh, there's like a panda found in here. And so initially it was going to be Panda, but then somebody suggested that Pandas was like funnier. So that was kind of the backstory. But it was, yeah, the main origin of the word was panel data.

Initially, I would say the main challenge was the SEO wasn't good. You would search for Pandas and you would not get the thing that you were looking for. And eventually, if you wrote Python Pandas, it would come up. But now when you type Pandas, it comes up in Google.

So basically, all the Python library is just like a zoo.

Yeah, exactly.

The analytics work behind pandas

So when you feel frustrated about data processing before you create a Pandas, what were some data science or analytics project you were working on?

Yeah, I was working on a small statistical modeling project that involved some financial data sets and the data was, was over, it was cross-sectional data over time. So panel data and there, the data was patchy. So there were some data issues. And so I was really the, one of the problems I was really focused on solving was making a tool that made it really easy to work with cross-sectional data over time. So time series data or cross-sectional data with a lot of patchiness. So really good support for the indexing capabilities in Pandas and the, the data realignment logic that's built into the series and data frame data structure.

Like I wanted the, all of that to work automatically so that it would like automatically deal with the data quality and data alignment issues that I was, that I was experiencing in my work. And so it was like trying to incorporate ideas of R, like the data frame data structure, the tabular data structure, but also add in the data realignment logic, the data indexing logic. And so that's why when you use Pandas now, like you have these row index data structures and column index data structures that assist you in, in, in data alignment. So you can do arithmetic between different time series that have, where maybe a time series has different missing dates in one time series or missing, there might be like missing stock tickers in one, in one series. And you can do math across these data sets that have data quality issues and handles automatically realigning things for you, which is really nice. But at the time that was, that was like the thing that I really wanted to work well and be intuitive.

So after that, you dived into building data processing tools and become a software engineer. Do you miss the analytics part of things?

Yeah, I like doing data analysis and statistics and data science work. And now, and then I still do, I do a little bit from time to time, but I haven't, I haven't been a practicing statistician or data scientist in a long time. So really just focusing on infrastructure, software and tools. So when I have an opportunity to do some analysis or do some data visualization, I always enjoy it a lot. Cause I get an opportunity to use the tools that I've built or helped to build, but yeah, I enjoy building software. And yeah, I think it's definitely been, for me, it's been an opportunity to build things that have, that have a lot of users and a lot of impact.

Starting Voltron Data

And what made you want to start Voltron Data?

So we had spent several years working on Apache Arrow. So Arrow for people that don't know is, it's a bunch of things, but it started out, we wanted to create basically a universal table format or a universal data frame format that could be transported really efficiently between data processing systems and between, from one programming language to another. So for example, being able to share large datasets efficiently between R and Python.

And we always had the aspiration of building computing engines that were Arrow based. So first we had to get Arrow adopted and successful as a data format and something that's adopted and used by many systems. And then we moved on to building compute engines that are Arrow based. And so now there's several compute engines that support Arrow natively, DuckDB, Data Fusion. There's Acero, which is part of the Arrow C++ project. So we've got these reusable, like mod, we call them modular computing engines.

And so our idea was that we wanted to create a company that could, on one hand, provide a lot of development and commercial support and drive forward the Arrow development roadmap and provide enterprise support for companies that are building on Arrow to be a partner to them in, in helping drive forward the open source project. But we also saw that there was a lot of opportunity in enabling next generation data systems to be more modular, to be Arrow based, and to be able to take advantage of hardware acceleration. So GPUs, but also there's custom Silicon like ASICs being built to accelerate data processing.

And so we recognize that we need to re-architect or help companies re-architect their data systems to be able to take advantage of these advances in computing technologies. So we created Voltron Data kind of on one hand to be a driving force in Apache Arrow, but also to build some technology to facilitate this transition to these kind of modular computing engines and taking advantage of hardware acceleration more seamlessly.

So there'd been a lot of work, initial work, exploratory work, building GPU accelerated analytics kernels through the Rapids project at NVIDIA and some work on creating accelerated processing engines for Arrow based on the, based on Rapids, so the Blazing SQL project. And so the company has spent the last, has spent the last almost four years working on making that work at scale and in production so that it can be adopted amongst large companies running in data centers.

So, and we have a large team there that over 20 people that work full time on open source and have done great things for the Arrow ecosystem and like the periphery of projects around Arrow that are helping accelerate people's data processing work. But Arrow is interesting because it's like the kind of project that most data scientists won't come into it, into contact with it directly. Like it's something that just starts getting used. Like it's getting used in Pandas now, it's being used in a lot of other projects, but it's used internally. So it's something that makes things faster. It makes things more efficient, more interoperable. Like you can now share data very easily between, between R and Python. You can use Arrow to interact with large parquet datasets that are stored in the cloud. So there's all kinds of like new use cases that have been unlocked through adoption of Arrow. But most data scientists don't need to know about Arrow. They just, they get it indirectly through the, the tools that they're already using.

Composable data stacks

So you mentioned modular data processing tools and also your, your investor, you're interested in emerging composable data stacks. So what's the benefit of the tools being modular or composable?

So the benefit of this, like this modularity or what we call composability concept is that it facilitates, it facilitates reuse where firstly, you make it easier for people to collaborate on shared, like reusable software components that, that many people can use to build many different kinds of data processing systems. And so rather than everybody building a vertically integrated system where they're responsible for building all of the constituent pieces of the system, instead there's off the shelf components that are high quality, that are state of the art that you can use to create a novel data processing system using off the shelf components.

And already like DuckDB is a classic example of you have a cutting edge analytic database system that's available as a single C++ file that you can drop into any project, or you can load it into your web application and have super fast SQL processing, basically anywhere on your phone and your web browser, really anywhere. And that reusability and kind of composability is what we call it. The composability comes from the use of open standards. And so in order to achieve composability, you need to have a standardized interface between that piece of software and your application.

And so part of what we've done in the Arrow project and in Voltron data is driving these open standards that enable and facilitate that composability and reuse. And so now there's a growing collection of open source projects that are participating in what we call the composable data stack. I helped write a paper last year with Meta called the composable data management system manifesto. Try saying that five times fast, but basically we tried to communicate a vision for, I guess we can maybe put a link to that paper in the show notes, but we tried to communicate a vision of what is, what does the future look like where data warehouses, databases, and data processing engines are built with reusable modular components.

And the idea is that you want to build a system in such a way that when something new and better comes up, comes in the future on the horizon, that you can change out the old part and put in the new part without disrupting the whole user experience so that you get things get better, but you don't have to like completely throw out the system and use something completely new. But you have, yeah, you have something that's you can hot swap components without breaking the whole, having to, yeah, throw out the whole system, throw out the baby with the bath water.

And the idea is that you want to build a system in such a way that when something new and better comes up, comes in the future on the horizon, that you can change out the old part and put in the new part without disrupting the whole user experience so that you get things get better, but you don't have to like completely throw out the system and use something completely new.

And yeah, so recently I just launched a venture fund, like a micro venture fund called composed ventures specifically to invest in companies that are helping make this happen. And as an entrepreneur, like I've, I've started a company to work, to make large contributions to toward this effort. But now there's a whole ecosystem of companies that are building technologies that are helping make it easier to build composable data systems. So it gives me another way. And aside from building new projects by investing in companies and helping other founders and open source developers, for me, that's another way that I can help accelerate this transformation of the, the, the data processing and data science ecosystem.

Why now is the right time

So what do you think is the reason that this composable data stack didn't happen, say 10 years ago? Why now is the right time?

Yeah, I see it as a natural evolution of the way that systems across many domains develop. So the best analogy I can give is the, if you look at semiconductor manufacturing, so the original model of semiconductor manufacturing is the Intel model, which is vertically integrated. So Intel built their own tools. All of their designs are proprietary. All of their chip fabrication is proprietary and in-house. And so like they control everything, everything top to bottom.

And so compare that to the new kind of the, like the new mod way of designing and building new computer processors. We have open processor architectures and specifications are ones that you can license from ARM. There's now RISC-V, which is a totally open source, freely available processor architecture, which can be used for chip fabrication. There's different number of companies that build the software that assists with chip design. So Cadence Design Systems is one company that many people have never heard of, but it's like a very valuable company that builds software for chip design. We have fabless, fabless semiconductor manufacturers like NVIDIA, one of the most valuable companies in the world doesn't own any fabs, like they have their chips manufactured by TSMC in Taiwan. And so in Taiwan and TSMC by turn has specialized in kind of taking all of these pieces and being really good at producing with high yield, producing chips, but they're dependent on ASML, which provides like the most, the world's most advanced photolithography. And ASML in Netherlands is also dependent on advanced optics from, I think from Zeiss in Germany.

And so basically what you've had is if you kind of decompose like all of these problems that like Intel was responsible from top to bottom, there's now like a specialist building these tools with around open specifications and kind of reusable systems for each layer of the stacks. You've got a specialist in photolithography, a specialist in optics, specialist in semiconductor manufacturing, software for chip design.

And so that's very much like what's going on in data systems right now is in the past, it was more expedient for somebody building a database or somebody build a data processing system to take ownership of all of the pieces in order to ship something more quickly. But now that we've gone through that first wave of progress in open source data management systems. And so now we can start to take a step back and say, okay, well, we want to make things 10 times better, but in all of these different places.

And so there was a collective recognition in the middle 2010s, I would say that it wasn't sustainable for us to continue building these vertically integrated systems. And so that's what led to, well, firstly, there were open source file formats that became widely adopted like Parquet, but then Arrow provided this in-memory computing layer, data interchange and computing layer. So that everyone realized that was something that we needed.

Then we need reusable execution engines. And so that's led to things like DuckDB, Data Fusion, Velox, so reusable execution engines. We're starting to think more about the user interface and the query optimization layer. So another project that I created at the same time as Arrow is Ibis. It's a Python project, which provides a portable data frame query layer. So you can use it to write your analytical queries and then it, depending on what backend you're using, whether you're running in memory with DuckDB or with pandas, or you're running against your data warehouse, like BigQuery or Snowflake or another cloud data warehouse, Ibis knows how to generate the SQL code or the pandas operations that you need to run that query.

And so that really helps with achieving this decoupling of concerns and systems. And so, yeah, that was like after building pandas, pandas is an example of a vertically integrated project where we were responsible for building everything top to bottom. And so having gone through that experience already, that was what really motivated me, that plus wanting to build Python interfaces to big data systems. So when I was at Cloudera, like that was like one of the things that I wanted to figure out was like, how do we build Python interfaces that can use all of these large scale systems, build Python interfaces that can work with Apache Hive or Hadoop or Spark, but not have to rewrite your code to use from one system to another.

So that's what motivated me to create Ibis, which is still going and it's been developing rapidly in the last few years, as well as Arrow, which is more of an infrastructure level project. But for Python users, Ibis is a project that you can pick up and use to author complex SQL queries and with the comfort of a DataFrame API, which is really nice.

But for Python users, Ibis is a project that you can pick up and use to author complex SQL queries and with the comfort of a DataFrame API, which is really nice.

Vertical vs. composable data tools

So like you mentioned, pandas is more like vertical and it also has a huge user base. So there is a benefit of having a vertical data processing tool. So when do you think is more appropriate for this vertical type of tools and when do you think it's better for composable data tools?

So, so one of the challenges with the vertically integrated strategy is that especially when you have a piece of software like pandas that people love as much as they do, is that they want everything to work just like, work just like pandas. And so the trouble is that the pandas API is pretty large. And so, and many of the aspects of its API, the code that you write is, is coupled to, or it contains details of how pandas is designed and implemented.

And so if people want to create, and there've been a number of projects like Modin, Koalas, that have created pandas emulation layers that translate their other operations onto some other execution environment. So there's a company Ponder Data, which used Modin to create a pandas interface to Snowflake. And now they were acquired by Snowflake and are working actively on that in Snowflake.

And so the challenge is that you can get 90 or 95% of the way there, but there are always going to be some, you're never going to be able to totally take existing pandas code and run it against another processing engine because there's details of pandas that surface in the API. And that's, what's very challenging is because whenever you want to swap out the execution, the storage, if you want to take pandas code and run it at a thousand times larger scale on like a large data set, I think Modin and Ponder are designed to help with that problem, to be able to run pandas code at scale, but there's always going to be some pandas code that it's not going to, that isn't going to be able to be translated and run because it's relying on internals, internal details of pandas data frame actually contains these NumPy arrays internally. And so you can get at them through the pandas API and that if you're working on an data warehouse or a large cluster, like that's large, like a spark cluster, like that's not going to work like it does on your laptop.

I think for a specific persona, for example, data scientists or machine learning engineers, where we have a variety of tools, we're not responsible for a large system. I think those vertical tools are useful for us to solve a specific problem. But if you are building a system to support a variety of use cases. I think having portability, particularly at the API level is especially useful for like people building systems that need to work in many different environments.

So for example, with IBIS, like IBIS was designed to give you the power and flexibility, like the full power of everything you can do with SQL and SQL is a very powerful programming language, which is why it's 50 years old. And then like people are still building databases based on SQL. So SQL is really powerful, but also SQL dialects are very different from database to database. Like even though there's the quote unquote SQL standard in practice, like SQL dialects are different from database to database.

And so the idea of IBIS is that you have a single, like a standardized Python API. That's data frame. That's a real programming language. It's Python. You get tab completion, you get type checking, like all the nice things that you get through Python, the ability to write unit tests, like the ability to write functions and reuse code in a modular way, but then at the end of the day, you can write code once and then you can run it in any of the 20 different backends, or if you're, if you're using multiple types of SQL engines in your work, maybe you're using duck DB, but also Databricks SQL or Spark SQL or Snowflake or big query from the same Python code, you can get SQL dot, like you can emit SQL strings for all of those backends and not have to rewrite your code at all, and that is really powerful.

It's especially powerful for framework developers. And so one of the types of projects we, one of the types of projects we've been talking to are people that are building like business intelligence frameworks or other data analysis frameworks that, that run on top of SQL engines for that are written in Python. And by delegating the generation of SQL or like creating these complex, complex SQL strings to IBIS, instead of authoring them by hand is saves people a lot of work. And so we can centralize that logic in one place.

There's another project, which is, which IBIS is using now, which is really cool called SQL Glot, which is like a SQL dialect transpiler. So I don't know if everyone, anyone's had the painful experience of, Oh gosh, like I've got all these SQL server queries and I need to convert them to Postgres or I need to convert them to, to Redshift and like SQL Glot as a tool that just built to do that. And it's written in Python.

Multi-modal data and the future of data tools

I think earlier on, a lot of data processing tools are focusing on structured data. And now with a larger language models, we have more unstructured data and multi-model data. So how do you think that will change the data processing tools?

So multi-modal data is something that we didn't really tackle initially in Arrow. And so Changshu, like my co-founder from Datapad and longtime collaborator on Pandas and other projects, he recently founded a company called Lance DB. And so they are creating the Lance file format, which is an Arrow compatible file format, but it's designed for multi-modal AI data. And so it has support for vector embeddings, images and kind of the data that you, you find in, in LLM applications, as well as the indexing kind of building the secondary indexes that you need to support LLM, like vector database type workloads.

And we've also seen like a whole ecosystem of new vector database products emerge, as well as vector database plugins for existing databases. There's, I think a couple of different projects for Postgres, for example. It sort of remains to be seen like whether the support for multi-modal data where it will happen mainly through specialized tools and systems, as opposed to extensions to general purpose databases.

But that's, I think an interesting area to explore, but I think projects like Lance that are starting with Arrow and layering on these kind of multi-modal data management capabilities are super interesting because it's definitely building for the composable data stack by being based on Arrow and using Arrow as much as possible. And also using things like Data Fusion, which is kind of an Arrow based Rust computing engine. It's like a reusable computing engine, which is being used in lots of different places as a modular query processing engine.

So we see a lot of data tools coming and going. So based on your experience, what are some tools you think are going to be obsolete? What are some emerging tools you think that are going to be more important?

I wonder that I might become obsolete. Because a lot of us who are writing code, I think the more optimistic way to look at it is that we'll get to spend more time doing the fun stuff as co-pilot and generative AI, like automates a lot of the boring stuff that we have to do while we're doing data analysis or doing CRUD tasks, writing code that's repetitive or transforming, forming data from one format to another, things like that.

I am very optimistic about, yeah, about having better logical separation between the APIs and the user interfaces that were used with projects, IBIS, the backend. To say that like, not to say that like the Pandas API isn't great. Like I think it is great, but it does create, pose a challenge for being able to run workloads at scale or to be able to auto essentially transpile your work workloads, like based on kind of where you need to run the code and just the size of the dataset and many other factors.

I think this is especially important as we see the hardware, like hardware heterogeneity increase. So right now, like already we have multiple GPU architectures, like Intel, AMD, and Nvidia have separate GPU computing architectures. Apple Silicon has its own like metal GPU architecture. And some of the machine learning frameworks have been optimized for all of these, for all of these frameworks. I think there's a lot of development, like development energy that's involved with building support for these different hardware acceleration frameworks.

I think that as time goes on, it's going to be easier and easier or become something that developers have to think less and less about, where basically we're able to automatically take advantage of hardware acceleration when it's there without having to explicitly opt into it. I think that's already happened to, to a great degree in deep learning where like with TensorFlow was like a pioneer in enabling hardware heterogeneous computing. Try to say that 10 times fast, where you can write your workload in TensorFlow or in PyTorch, and if you have TPUs available, it will use them. If you have GPUs available, it will use them. And so that, that enables portability across, across hardware.

And so, so that's, I think a really positive, a really positive trend. And so I think any increasingly, I think that systems that are not built with this modularity in mind and being able to run on different types of hardware seamlessly or be reusable or take advantage of these kinds of composable ideas. I think those would be increasingly be seen as like the last generation of technology that's best replaced by systems that are built for this more like kind of modern, the more modern way will be the modular composable way.

I think those would be increasingly be seen as like the last generation of technology that's best replaced by systems that are built for this more like kind of modern, the more modern way will be the modular composable way.

So, and I mentioned, I've been telling everyone I've been following pretty actively, this new company modular. It's Chris Lattner's, Chris Lattner's new company. They, he created the LLVM compiler project and there's a something called MLIR, which is like a layer on top of that's intended to make it easier to write kernels for deep learning or other machine learning workloads, but to be able to compile for different hardware targets. And so they build a whole new programming language called Mojo for that is producing really amazing results on LLM workloads.

And so I do think that the folks working on compiler technologies that are making it a lot easier, the company is called modular. So it's like similar theme, right? To kind of all the things I'm talking about. But I think that is like definitely going to be the way, like the way of the future in terms of like how we build these systems to create these abstractions where developers don't have to really think about the details of the hardware. Like nobody will need to be an expert in how NVIDIA GPUs work or how Apple Silicon GPUs work. That's something that the compiler will take care of and that we'll have expressive enough frameworks for developing the systems so that we can generate optimal code that runs on each on the type of hardware that's available and that we can upgrade the hardware independent of the software.

And because ultimately what we want, and this was like, this was why we founded Voltron data was like, we wanted to create a company that would enable that transition to hardware modularity, where you, if the hardware gets better, you can change out the hardware without disrupting the software experience. Yeah. But it turns out that getting there is like really hard. It takes a lot of money and a lot of engineering work.

Challenges at Voltron Data

So what was the biggest challenge when you were at Voltron, when you were building the product? Well, it's a many layer challenge. There's a lot of, cause there's two, so we have a really ambitious vision. There's a lot of pieces that need to, they need to fall into place. There's open source standards that have to be developed to help. I like to help with the contact points between different pieces of the stack that we're working on. So we recognized that we needed a technology to help connect compute engines with the query optimization and front end layers. And so we collaborated with a group of people to create the substrate

project, which has provided, it's even more abstract than even more abstract than Arrow, but it's, you can think about it as being like a technology that helps connect the language front ends and the query optimization layer with the compute engines so that the compute engines don't have to be, it's like, you think it wouldn't be optimal for every compute engine have to know how Python works or how R works so that it gives you like a standard way of telling the compute engine what you want to do.

But also it creates an abstraction to sort of go from a representation of the workload that you want to run to translate that into what physical operations do you want to do on the particular hardware that you're using?

So we not only needed to create like implementations and production ready enterprise systems to run real world workloads, like build it in this way, but also building the open source standards that make it possible and making sure that we aren't over specializing for the particular needs of the company.

And yeah, I mean, we built 130 person team also during COVID. And so raising over a hundred million dollars in venture capital, hiring a hundred people all during the pandemic, definitely not for the faint of heart, but we had really amazing supportive investors who were big believers in our vision. And so we were fortunate that we had the support and the capital and the people, right, people involved to make it, to make it happen.

Joining Posit as principal architect

And later you left Voltron Data and become a principal architect at Posit. What made you make this move?

So I'm still an advisor at Voltron Data. So helping out the engineering team and obviously continuing to drive the vision around composable data systems and kind of the open source technologies that are the foundation of the foundation of the company. But after spending between Ursa Labs, Ursa Computing, Voltron Data, I'd spent more than five years in entrepreneur mode working on Apache Aero. And I'd felt that I had sort of achieved like a lot of what I personally needed to contribute to that, contribute to that project and opening myself up to explore the ecosystem and look for kind of other projects to make investments in, personal investments of time and effort.

I was also interested in doing more venture investing and having had a long working relationship with Posit. It was kind of the perfect, like the stars aligning, like I can stay involved and stay involved in Voltron Data, help Voltron Data succeed, but without being in a full-time operator role running the engineering team there. So it was good timing. And in the meantime, while working on Voltron Data, RStudio rebranded to Posit, incorporated Python into its enterprise products, repositioned itself to be a polyglot data science company.

And for me, like I really have been a huge fan of JJ Allaire and Hadley Wickham and the leadership at Posit. And so to be, take a larger role in the mission of, in the mission of Posit, I think it was just the opportunity of a lifetime for me. And, and I'm in a position there where I'm able to make important contributions to Posit's product offerings to help enhance the experience, not only for Python data scientists, but for data scientists more generally, regardless of programming language.

But I also have the freedom there to continue working on critical open source technologies, work on, if I want to write another book, it's a place where there are tons of Posit employees have written books and there's a whole library of books written by folks at Posit. And so I think for me, it gives me a platform to really, I think, have a lot of impact in the data science world. So very happy with the transition. And as I mentioned, I'm investing in part-time kind of with my new firm, venture firm, and yeah, I definitely keep very busy with lots of projects, but I've been very fortunate with the connections that I've made and just really looking to do what I can to help drive forward the, the open source data science ecosystem.

So now you are, is your role more like an individual contributor at Posit?

I don't manage anyone directly. So I, yeah, I would say technically I think as a principal architect, I report to JJ and so I have, I'm not managing anyone directly. So technically that makes me an individual contributor, but I'm doing a mixture of, I am writing code. It's kind of a mixture of Python and some TypeScript and so working in multiple programming languages.

And I'm also helping with the product roadmaps around like Posit's product offerings. So Posit Connect, Posit Workbench, Posit Package Manager, sort of Quarto. I'm very interested in Quarto as a technology. Like I read my third edition of my book, like I migrated to use Quarto and so you can read it for free on my website. It's all powered by Quarto. I think that's a really important piece to help with content creation, building dashboards, building interactive documents, building interactive applications and publishing and sharing results of data analysis. So it's, yeah, I'm involved in a bunch of stuff there and yeah, it's an amazing team, so I'm very happy.

Missing coding and the return to individual contribution

So when you were a full-time co-founder of Voltron Data, did you miss writing code and kind of being a creator?

I would do some coding now and then while it was full-time at Voltron, but you know, often, yeah, I guess when you're running a company, often software development is not the best use of your time. And so sometimes if I would do write some C++ on Arrow, it would usually be like a nights and weekends kind of thing, or maybe I had a long plane flight, I'd be like, oh, okay, what am I going to do for the next five hours? Okay. I guess I'll find an issue to hack on and write some code.

I did miss coding, but I think when you're in that, when you're in an operational role like that, I found it's better to, to not get in the way. And so I would try not to pick up work that's, that was in the quote critical path, because the last thing I'd want is to be like, oh, I have a data hack. And then I put up a pull request and the pull request isn't done. And then I get really busy with something else. And then there's people who are depending on me to finish this PR and that's no good. So yeah. So I'm happy to be spending more of my time coding again. And my plan is to spend the majority of my time doing software development, doing direct hands-on coding work or writing, or, you know, beyond podcasts, beyond podcasts.

I know a lot of my engineer friends, they later become managers or founders because they want to have more impact and then they realize, oh, I miss writing code. And now I have to manage people, write performance review. And then they really kind of struggle with a change of work. And a friend of mine, he was a director of engineering. Then later he went back to become an individual contributor. He really enjoy, he has more time to write code, but he misses the direct influence you can have on your team, on other people's team. Do you miss that element?

I mean, I enjoy doing development work, I think by nature, a change agent. And so I think I like building successful software projects, building successful, like productive developer groups. And so I think that working in an architect role, I think even if you aren't directly managing, like directly managing people as an architect, like your job is to shape the culture and the roadmap of how the software project works, like the tools that it uses, like how it operates on a day-to-day basis, makes decisions. Like what is the code review culture? What is the issue, like kind of the issue management, like project planning culture, as well as identifying like high leverage projects to help the project move forward.

And so I think my most productive periods of being in that principal developer role where you're writing a lot of, you're writing a lot of code and you're kind of like the tip of the spear, like helping kind of carry the torch and lead the charge in an ambitious software project, I really enjoy doing that. And I really like working on new things. Cause people have asked me like, Wes, like you aren't really working on pandas very much anymore. And I think partly it's because a lot of the work in pandas is very important. And there's a whole large core team. There's pandas is just an enormous project. It's had thousands of different people contribute to it, but it's a very different project now than it was 15 years ago when it was brand new.

And so I really enjoy that, like exploratory trailblazing, kind of that building something new where you're like figuring, figuring out something new, like building something from scratch. And usually once a project gets more mature after four or five years, this also happened with Arrow. Like I'm not doing as much Arrow development these days, but also like Arrow has become a more mature software project. And so it made sense, like to not be such a looming presence in the project and to make space for other maintainers to grow a leadership role in the project and to not be dependent on me for kind of driving the project forward, which is great because it frees me up to think about what to build next.

And so that's part of kind of what I'm doing right now is thinking about what's next, like where's the, I think Arrow has had a huge impact in the data world. Pandas has had a big impact. I think Ibis is going to have an even bigger impact in the coming years. And so I am looking for other things that we could be doing to help transform, kind of transform the ecosystem and continue to make progress.

I think Ibis is going to have an even bigger impact in the coming years.

Managing a varied schedule

And you're in a very interesting position as a advisor to a startup you founded and as a architect posit and you're also investing in a variety of startups. So how does your week look like? Do you feel you have to constantly change context?

Well, fortunately, yeah. I try to structure my days so that like I have a block of meetings and then a block of deep work. And so I try to protect as many blocks of, of deep work in my schedule as possible. Maybe some of you have read a Paul Graham's essay, the maker's schedule, the manager's schedule kind of discusses like the Frick, the push pull between having meetings and networking and influencing other people and making time for, for doing deep technical work.

So it's a lot of calendar management and I mentioned I'm doing the venture investing, but I'm also trying to not, my goal is not to be a full-time, a full-time VC is really a part-time thing. And so I'm not out there hustling for hustling for deals. If somebody introduces a founder to me or a group of founders to me that are working on something where it's something I know about, or I could make a judgment about their tech stack or something that seemed that appeals to me, then I'll take a look, but yeah, my goal is for venture investing to take as little time as possible on the advising front. It's an ebb and flow.

I think doing podcasts and helping and also networking with other software developers helps me identify opportunities and ways to create like interesting connections, kind of new synapses between these different commercial and open source projects. So yeah, it's definitely a lot different than it was 10, 12 years ago when I was just had just pandas and didn't have a lot going on outside of that.

But I've realized that influencing other people and helping, helping align disparate efforts in the open source community, like solving the social challenges in open source are, are as important or more important than the contributions of any one person. And so even when sometimes I'd be like, Oh, it's annoying. I have so many meetings. I really would rather be coding, but I also have to recognize that like those meetings and those conversations, like they are recording podcasts, like they matter because, um, you know, having people helping, um, get people working on thinking on the same wavelength or thinking about things in, in, in compatible ways is really important.

But I've realized that influencing other people and helping, helping align disparate efforts in the open source community, like solving the social challenges in open source are, are as important or more important than the contributions of any one person.

And so like arrow is a good example of a project where, uh, it took a lot of collaboration, like a lot of people wrangling to make that project happen. But that work of, of getting people like marching to the beat of the same drum, it was essential and a project wouldn't have happened otherwise. And I think that social labor of understanding what are other people's motivations or is there an opportunity to collaborate or could there be something we could build that could make collaboration possible?

That was like, I think for a long time, one of my questions was how do we enable the Python and our communities to collaborate and before arrow, like we didn't really have a technology that facilitated that collaboration. And so now that we have a computing engine and like a whole compute and data access framework that ships both in Python and R. So now we can make an improvement to arrow and it's instantly available in both Python and R and developers in both eco and both ecosystems can benefit. So that's great.

The Python vs R language wars

Yeah, that's awesome. Because on social media, people always put Python and R against each other. Python and R is like, you have to pick a side and it's great that now in positive supports both Python and R.

Yeah, I think Hadley, Hadley Wickham and I got together back when arrow was starting in 2016 and, and talked several times and had said that we thought that the quote unquote language wars were kind of dumb and we would just want them to go away. And so one of the best things that we could do is to start working more actively together and building things that could help end the language wars. And I think that arrow is one technology that's really helped. It's really helped a lot with that.

And obviously a posit like with its products for data science teams is about making it easy for teams that are multi-language to use common technologies, for example, for report deployment or application deployment. So being able to deploy an R based shiny application and a Python based stream load application in the same platform is super useful. And like being able to deploy an R markdown or a Quarto document or a Jupyter notebook and set it up to, I want to send this, rerun this Jupyter notebook and send the report to my boss once a week to be able to do that in the same place. It's super useful. So yeah, I think the polyglot model feels to me like the model of the future. And so anybody who's not building for a multilingual environment will find themselves kind of disfavored, I think amongst users.

Improving open source funding

And if there's one thing you can change or improve for the open source community, it could be cultural or on the technical side, what would that be?

I mean, I think overall we still need more structured avenues for open source developers to get financial support and like direct funding for their work. It's definitely, things are a lot better now than they used to be. Like there's a number of GitHub sponsors. There's a new platform for open source sponsorship called Polar that I'm pretty excited about. It's different from Polar's, the Python library. It's polar.sh, the open source funding platform. There's also Tidelift and Patreon.

So there's new things that we didn't have years ago and there's organizations like NumFocus that you can donate to that helps get money into the hands of funding small projects and infrastructure improvements and open source projects. But yeah, I remember a decade ago we had the aspiration of ideally we need a hundred million dollars a year of government grants toward open source development and we're a long way from that. And a hundred million dollars a year wouldn't be enough, I think, to support the kind of maintenance and feature development and innovation that's needed in this ecosystem.

And so I think any project that has become dependent on corporate support has like an inherent vulnerability because you're in a sense, you're subject potentially to the capriciousness of a single, of a company and their ability to sustain their investment in that project going forward. And so I've heard no project say we have more funding than we know what to do,

which means I guess the answer is there's not enough funding. So I just hope that more corporations can allocate part of their R&D budget toward making targeted donations or investments in open source projects that they depend on.

And you said you like to work on new things and when you face a lot of new options, what's your philosophy or do you have a set of criteria when you want to decide what to work on next?

I try to learn from the users. I listen, or if I, if I'm building something myself, if I find that something doesn't work quite the way that I think it should, then I'll try to like talk to other people and see, is this a problem that you also think that this doesn't seem great. Do you also think this is a problem? And so, I mean, that's basically what happened with, with Arrow is that I had a suspicion that there was a problem to be solved, but it took the better part of a year till I find other people that had thought about, had the same ideas had occurred to them.

And then we had to convince each other, can I work with you? Can I work with you? And so we, we got a coalition of people together to start working on a project, but yeah, I think I trust my intuition like a lot in terms of what to spend my time on, but I definitely do collect feedback and try to get market research, talk to real users and confirm, you know, confirm my hunches about what problems are worth working on.

Learning new technologies

And what are something you are learning right now? I'm learning Rust and like a lot of people, maybe later than a lot of people, but I think there's a lot of like new Rust based tools that are coming out in the Python ecosystem. And I'm an investor in Astral, which is the company behind the rough code formatter and linter. And also they just released a new PIP replacement called UV, which is like a ultra fast kind of package management tool. And that's all implemented in Rust.

And so I'm pretty excited about the use of making it easier to develop Python extensions and to do more systems programming and address some of the shortcomings of Python. I also think it's great that Guido van Rosum is at Microsoft and is working on making CPython faster. I kind of gave some feedback also to Microsoft about their Python and Excel integration. And so I think that's all super cool. So yeah, I'm still continuing to expand my repertoire of skills. And, and yeah, as I mentioned, I'm doing some little bit of TypeScript development cause I'm interested in using DuckDB compiled to WebAssembly to do things. And that's all, it's some really cool stuff there.

And yeah, cause in the past, our company Datapad, we built a visual analytics product, but we had to build a whole data processing engine to power it. And nowadays you could just use DuckDB to do that. And so we have all these like nice tools that didn't exist a decade ago. And so I'm exploring and thinking about things that ways that they could be incorporated to, to level up the experience of, for data scientists.

Personal interests and language learning

Well, aside from all the things that we've been talking about, which I'm definitely excited about outside of programming, I like foreign languages. And so it's been a lot of time on language learning. So it's like I'm learning French and Italian right now, which I enjoy a lot. And like the tools and things for learning language learning have become so much better. And that not only is there like ton of amazing content on the internet, on YouTube and podcasts, but also using ChatGPT for language learning is tremendously useful.

As you can imagine, it's really good at translating idiomatic ideas, or if you ask it for like examples of how to say different things, but also how to say different things, like in different contexts of like, how formal is the situation you're in? What would you say to like your family or your friends versus what would you say to somebody you're doing business with somebody you don't know? And so, yeah, I found it tremendously useful. Also for, you could tell it to like, take an article that's written in one language, say, I want you to translate this, but generate a summary of it in a different language.

And so if you're interested in like, how would you express like complex ideas or ideas that you don't necessarily have the linguistic capability to express, but it can assist you in putting something into words and then you see, okay, well, how would ChatGPT say this in my target language? And so it's super useful for generating novel content.

Using AI tools for software development

I've used GPT, mainly GPT-4 for writing tedious scripts. Things, most recently I used it to do some things with the GitHub API. And I was like, yeah, okay. I could read the GitHub API and figure out how to do this on my own. And I know how to use requests and I know how to do, reuse Rust APIs and stuff. But ChatGPT knows all of that too, and has an encyclopedic knowledge of the GitHub documentation and everything. So you can ask it to write Python code for you to do pretty much any task that you want and you'll have the results in seconds rather than having to pour through the API documentation and figure it out yourself.

I think that to me is like to eliminate a lot of the drudgery associated with being a software developer and not having to spend time on boring, but straightforward tasks, definitely as a good use case for, for LLMs and things like Copilot. I still haven't set up Copilot. I've got to figure out how to hook up Copilot with Emacs since I'm an Emacs user, but I'll work on that.

I think that to me is like to eliminate a lot of the drudgery associated with being a software developer and not having to spend time on boring, but straightforward tasks, definitely as a good use case for, for LLMs and things like Copilot.

Long-term impact and goals

So say after 20 years, what kind of impact do you want to leave for the community? I think, I mean, in general, like I'm really interested in continued growth of like open source data science. I think that one of the reasons that one of the main reasons I'm at Posit is because I think they've built one of the most sustainable businesses that is largely open source data science focused. Huge fraction, I don't know the exact numbers, but like a significant fraction of the engineering team works on works primarily on open source projects.

And so I think the, and JJ, the CEO and founder, he has the aspiration of building a hundred year company that can survive him and survive all of us. And so I think given how passionate I am about solving the open source funding and sustainability dilemma, that if we, I, if I can help do that by building, my building Posit's business and helping the company grow its impact and achieve a, if be operating in terms of impact at 10 times the scale that it is now in terms of the contributions to open source and its footprint in how people do data science and in the business context, that would be very satisfying to me because it serves my goals of building impactful open source projects, but also doing it in a sustainable way and increasing.

I've had the opportunity to direct a lot of commercial funding into open source development. And I like doing that. And I, to have a large team and being able to contribute in a major way, what we've been able to do with Arrow development has been, has been really satisfying to me. And so to continue doing that and to be able to do that at an even larger scale is a, is definitely a major goal for me for the rest of my career.

Well, thanks Wes for coming to the show and I'm excited to see the growth of Posit and your work. Thank you. Thanks for having me.