Resources
Jul 28, 2023 |
1h 23min
|
5k views
|

3 Reasons to Use Tidymodels with Julia Silge

This is a recording of a virtual seminar on '3 Reasons to use Tidymodels' by Julia Silge! The event took place July 13, 2023 and was hosted by R-Ladies Philly and R-Ladies DC.

Thank you, Julia, for the wonderful talk!

Talk description:
Modeling and machine learning in R involves a bewildering array of heterogeneous packages, and establishing good statistical practice is challenging in any language. The tidymodels collection of packages offers a consistent, flexible framework for your modeling and machine learning work to address these problems. In this talk, we’ll focus on three specific reasons to consider using tidymodels. We will start with model characteristics themselves, move to the wise management of your data budget, and finish with feature engineering.

image: thumbnail.jpg

Transcript#

This transcript was generated automatically and may contain errors.

We are very excited to be hosting this event for you and want to thank you for being here. I also want to thank Our Ladies DC for collaborating with us to host this event.

We'll be starting with a short introduction. We'll be starting with a short introduction, announcements, and then I'll go ahead and introduce our speaker for today, who will then take over and talk to us about tidymodels. We can have a brief Q&A session. Go ahead and add your Q&A questions to the chat as the talk goes on. And then for the last few minutes of the event, I'll be going over the raffle winner announcement.

And so if you didn't read in the meetup page, we'll be doing a raffle. Three books for three books titled Tidymodels in R by Julia Silge will be up for raffle at the end of this talk. And you can choose if you'd like a hardcover or an e-book.

So again, welcome. We are Our Ladies Philly, a chapter of Our Ladies Global, a worldwide organization that promotes gender diversity in our community and in data science. We warmly welcome all individuals who share our values and follow our code of conduct, as well as those who have an interest in data science, regardless of skill level. We encourage everyone to join the Slack community where you have access to our help and more. Please subscribe to our YouTube channel to view recordings of past events and consider signing up as a volunteer speaker or mentor. And all skill levels are welcome.

We want to thank all of our sponsors, our consortium, Independence and O'Reilly Media. And also check out our Twitter page, Our Ladies Philly.

We have two event announcements. First, we'll be having a intro to GitHub workshop, which will be an in-person event July 20th, which is really exciting. I believe we'll also be doing a raffle for the same book at that event. You can find more details about this on our Meetup page linked below. Our Ladies DC will be hosting an online book club on September 26th. And again, the details are on the Meetup page.

Bioconductor will be having a conference in the Boston area in August and is hosting a conference in Chicago in September. So if anyone is going to any of these conferences or any other conference, let us know in the chat.

Here are the links to our Slack community, our YouTube channel, and where to volunteer as a speaker or mentor. And again, thank you to our sponsors.

And again, I'd like to announce that we'll be doing a raffle at the end of this event for this book here.

So now I'd like to briefly introduce our speaker today. Julia Szilagyi is a data scientist and software engineer at RStudio, where she works on open source modeling and MLOps tools. She is an author, an international keynote speaker, and a real-world practitioner focusing on data analysis and machine learning. Julia loves text analysis, making beautiful charts, and communicating about technical topics with diverse audiences. So thank you so much, Julia, for being here again. I'll go ahead and stop sharing my slides, and you can go ahead and take over.

Awesome. Thank you.

All right. Let me get it where I can still see the chat. Great. Great. Thank you so much for that introduction. I'm really happy to be here and to be talking about tidymodels and why I have been really glad to work on this project.

Three things that you might keep in mind that if you run into your work or you start to get concerned about in your particular domain, that you might then think, ah, I should think about using tidymodels to solve this kind of problem.

Julia's background and path to tidymodels

So I'm going to start by giving just a little bit of background of, like, who, like, where I came from and how I came to kind of work on many of these kinds of tools. So my academic background is physics and astronomy. I have a Ph.D. in astrophysics, and I was in research, eventually left academia, and then went on a little bit of a winding path of, like, what am I going to do now? Now that I'm, like, exited academia.

And around the time when this was happening in my own life and career, we really were seeing this growth in, like, this newish at the time field, data science. And people who are coming into these kinds of roles were coming in from kind of backgrounds that were weird in the way that my background was weird.

So I come to the practice of data work, the practice of machine learning, not so much from a statistical background. Like, I'm not one of these people that has, like, really solid academic statistical knowledge or theoretical statistical knowledge. And I'm not someone also who comes to my work as a data science practitioner from a software engineering background. Instead, I come as, like, oh, let me come and, like, pick up these skills as I'm preparing for jobs or, like, on the job somewhere. Can I learn about this? And can I think about it as someone who is really concerned with process, with systems, and, like, how things work together?

I love thinking about people's really practical problems and how they solve them, like, really applied kind of work. So I worked as a data scientist in a couple of different organizations. And then starting about three and a half years ago, I was hired at then our studio, would now Posit. And I was hired to work on open source software. So I contributed to open source software kind of on the side, right, like as many of us do. And we have regular jobs.

And so going to work at our studio, formerly our studio, now Posit, this was a shift from being someone who was largely a practitioner and built some tooling on the side to think about tooling as a focus and then kind of keep my hand in on, like, data science practice as I think about how people engage in these tasks. I was hired to our studio to work on tidymodels.

So tidymodels started, you know, maybe a couple years before I started being engaged in it. But by the time I joined, things were really coming together. Things were really get, like, pretty much ready for people to engage and use tidymodels in their real work. So, like, we started saying things like, well, if you're using something else, like Carrot in your project, you know, no reason to rewrite it from scratch. But if you're starting a new project, we recommend, you know, you check out tidymodels and see if it will be good.

It was very interesting because I had engaged in machine learning before in some of my jobs. But, like, you know, I did not come from an academic statistics background. And so it was really interesting to get thrown into this tooling, the tooling for machine learning and thinking about what people, sets people up for success.

What is machine learning?

So I'm, if you, you know, we love thinking about, like, what is machine learning? Like, thinking, like, okay, what, when I hear this, what is going on? And, you know, this, of course, a very wonderful and delightful SKCD about, that gives us, like, an insight into, like, what is it that we mean when we say machine learning? You know, we, so we're using data, we're using, like, big piles of data to give us some kind of answers, to give us some kind of prediction.

So instead of maybe coding decisions, like, with SL statements or something, instead we use data and use statistical methods to learn patterns from the data to be able to give us some kind of an answer. So, you know, these kind of, it's interesting to be talking about this right now, because over the past, you know, year, six months, two months, you know, we see, we see talk about what is, like, what is AI and this word being used so much more broadly.

So traditionally, AI would be the biggest outside category and includes both making decisions, you learn from data, and setting up, like, rules-based AI. So being, say, picture yourself, a picture of yourself, the sort of, if we see this big, we would, if we say that's artificial intelligence for some definition of artificial intelligence, we can have one approach that is, I'm going to code a huge set of rules. If this happens, do this. If this happens, do this. And, like, code the rules directly.

Another approach is to take a huge data set of chess games that exist, and we then put it all into some kind of statistical learning algorithm, and then we let a model learn how to play chess from the data that we have. So we have the big category, traditionally, of AI, which traditionally includes rules-based and machine learning, meaning we learn from data. And then inside of that, there's different kinds of machine learning.

So the kind that, you know, if you're hearing about in the news that are, you know, like driving these changes in large language models, they're all the way inside of that deep learning circle. The work that I'm going to talk about is in a little bit different place. It is in that place where it says dozens of different ML methods. Depending on the kind of data you have and the kind of problems you're trying to look at, that's actually where a huge amount of business value comes from, is in that place where it says dozens of different machine learning methods.

Of course, we're seeing lots of people be really excited about generating value from things over there in, like, the deep learning category, but you do have to have specific kinds of data and some kind of very large volumes of data to be able to make use of that. So we're specifically going to focus down into that place where it says dozens of different ML methods.

Another way to think about this kind of, which you might call classical machine learning, so that's kind of in contrast to deep learning methods, which are, I guess we can say, newer. We generally categorize things into two places. So unsupervised is when your data doesn't have a label. So you might do things like dimensionality reduction there, where you say – look at examples. It says, like, okay, you've got clothes. Maybe I want to cluster data. I have different kinds of clothes, and I want to look at the – none of the clothes have labels. I don't know which kind of clothes are which, but I want to use the characteristics of the clothes to clump them into a cluster. So, say, I'm going to put the pants together with the shorts with the leggings, and I'm going to put the dresses with the jumpsuits. So, like, learn how things go together. But I don't have – in that case, I don't have any labels. I don't know – I don't have something I'm trying to predict.

On the other side, supervised machine learning, then, for sure, the data – the idea there is the data has a label. That label typically is either a category or a number. So when we have – let's say we've got socks, and we want to divide them up by color. So we see here on that little visual, we're going to, like, divide the apples from the pears. The other kind of supervised – the most common kind of supervised problem is regression, where we're predicting a number. So we can predict a category or predict a number. The kind of work that I am going to be talking about here – this is all kind of like, what are we even talking about – is over on the left-hand side.

So if you can formulate a scientific or business question as a supervised machine learning problem, then there is a huge swath of tools that are out there to help you solve that. So there's – you know, there's lots of different things that fall under the category of AI, lots of different things that fall under the category of machine learning. Supervised machine learning, either classification or regression, are areas that have – that have huge infrastructure of methods and tools so that you can use that to solve some kind of a problem. Specifically, we are going to talk about using tidymodels to approach this kind of problem. These – like, if you – hey, you've got a question or a problem, can you formulate it as supervised machine learning? Then you can use a tool for it, such as tidymodels.

The tidymodels framework

So the tidymodels framework is a – you can think of it as analogous to the Tidyverse. So it's a collection of R packages. So if you think about the Tidyverse being used for data manipulation, data visualization, reshaping of data, munging of data, dealing with CSV files, dealing with dates, you know, the tidymodels framework is an analogy where all these different packages are focused on a different part of the machine learning process with a specific focus on supervised machine learning. So if you can install the tidymodels metapackage, and then just like, you know,

if you have done library tidyverse and then you've gotten ggplot2, dplyr, tidy R, if you do library tidymodels, you get a bunch of packages that get, you know, attached there so that you can use the functions from them.

So I would first say if it feels overwhelming to get started to know there's a bunch of packages and to not be sure which functions come from which individual package, I would say try to step back from that feeling of discomfort and just kind of go with it. Like, just – like, don't stress out too much if you don't know which functions come from which packages. That's definitely something that comes as you get more experience using something.

So maybe when you started, like, using tidyverse, maybe you felt, you know, maybe you're like, oh, gosh, is – where is the function coalesce from? Or where – like, oh, I think I saw people, like, talking about pivot wider and pivot longer. Is that in dplyr? Is that in tidy R? Like, no – try not to be super stressed about that, but to know it's a meta – it's a meta package with individual packages in it, and those individual packages are focused.

The reason why – so it actually is better for us, the maintainers of tidymodels, to have things modular like this. Also, it turns out, it's better for you to have the things more modular. It's better for maintenance because if each of these packages is separate, if you need to change something about, say, tuning, which you probably guess is for the tune package, so hyperparameter tuning for models, you only have to change it there. You can have smaller changes. It's easier to do releases when things are more modular. You probably experience in your own work. And when you make pieces of code more modular, things are better versus, like, one giant kind of thing that can get hard to deal with.

So it's better for maintenance. It's also better for you because it turns out if you want to do – you know, if you want to do, like, deploy a model, you don't need the example data that, you know, we use in all our packages. You don't need – you know, you don't need the tuning infrastructure, probably, if it's time for you to deploy a model. So it helps you be a – to adopt better practices in your machine learning when you have these modular pieces.

Key packages in tidymodels

So let's look at a couple of these names. So yardstick is a package for measuring how models are doing. It's a package for model metrics. So yardstick is – you know, you probably don't have to worry too much about knowing these individual things, but yardstick is the package that has all the metrics that we use to see how something is going. Tune, like I said, is a package for hyperparameter tuning.

And so what I'm going to do today is to focus on a couple of these packages and to frame it in you thinking about – you thinking about, if I run into a problem like X, what can I reach for and what kind of problems does tidymodels solve for me? So we're going to talk about three topics, three things that we're going to go through for today. So the first one, we're going to talk about what makes a model, like what actually is a model, what are the components of it, like what can we use to define a model. We're going to talk about spending your data budget wisely, how to be a good steward of the amount of data that you have. And then we're going to wrap up by talking about how we can build better features for machine learning. So these are the three topics that we're going to walk through.

What makes a model

All right, so let's start with the first one, what makes a model. So in R, modeling is really heterogeneous. It's like really, really mixed up, really, really – there are dramatic differences around model interfaces, how you go about fitting a model, and like what the execution kind of strategy is. So this is a lot less true in Python. So in Python, there is scikit-learn, which you have access to many kinds of models within scikit-learn.

In R, R has more, you know, statisticians in it, like working with it. And so there's a real strength in R is that there's a huge diversity of different kinds of models. They can live quite separately from each other. On the downside, it means if you were to pick up some package for modeling, the way that you go about like fitting and maybe executing a model can be extremely different from the next modeling package that you fit up.

So in R, the model, the norm is kind of that things are quite broken up. There's lots of packages and like the – say if you wanted to change from random forest to XGBoost, like you go to a whole different package because the implementations for those two things don't live in the same piece of software.

There have been several kind of attempts or try to kind of make a more unified kind of thing. If you ever have heard of CARET, the CARET package, C-A-R-E-T, that was by Max, my co-author on the book. And it was kind of like a first go to make a more scikit-learn-like interface to models. There's also MLR and currently like MLR3. They're kind of on their second iteration of like – or third, I don't know – of like how to go about making like a unified kind of interface to many kinds of models.

So part of tidymodels was approaching – like the part of the motivating why bother like doing tidymodels is to approach this very question and give a new answer, kind of like a modern answer, a Tidyverse-inspired answer to like how – what are we going to do when – like I have to rewrite all my code if I want to change from one kind of model to another.

The Parsnip package

So the package that approaches this is called Parsnip. So it's a – you know, it is a little bit of a joke, right, compared to CARET, like the idea of like – although that CARET is not spelled like the vegetable, but so that's where this kind of like pun or joke comes through. It's like another vegetable. So the Parsnip package is the package for setting up model specifications. And in tidymodels, there are three components to completely specifying what a model is like. The first one we might use the word model for, but what this means – we'll get into it a little bit, but this is like what kind of statistical algorithm is it. The second stage is to specify an engine, and the third is to set the mode, the mode of what kind of model that you need to have.

So let's walk through these. So first, model here means what kind of mapping are we going to make to get from our inputs to our outputs, like our predictors to our prediction. How do we get from our predictors to our prediction? How do we get from our inputs to our outputs? So one kind of model is linear regression, regular old OLS. So here I can set up a model specification here by using the function linear reg, linear reg here.

So there's lots of different kinds of things. A model – different model types. You might say model type, model algorithm. You know, we can – it's random forest is another one that I already mentioned. Like it's a different way of getting from the inputs to the outputs. Next we need to say an engine. So back here you can see it says computational engine LM because it turns out there are some defaults. But you don't have to just use the defaults. In fact, you can set your own engines, the engine that is right for you. So here instead of before it said – notice here it says computational engine LM. Here I'm saying computational engine Glymnet. So this is an R package.

So when I say LM, it's going to do ordinary least squares. It's going to use the built-in LM function that comes with R in the stats package. If I say set engine Glymnet, then it's going to do regularized regression using the R package Glymnet. So engine, think about it as – it's not about the algorithm that's being used. Instead, it is about what computational engine am I going to use to implement that computational algorithm or type, model type.

So here the first two that I told you about are both R. But not all engines are R. So we can also use SAN here. And here what we'll make is we'll use a Bayesian linear model implemented in the Bayesian engine SAN. So the engines can be things like R packages. They can be things where we're calling out not – what's actually doing the computation is not R at all, but rather I believe SAN is in C++. We can do things like Spark where it's doing off-host execution. So we separate out you deciding what kind of model type or model algorithm you're going to use from how that engine is implemented. And, of course, the idea here is that we're going to give a unified interface to all of them so that you can not have to change everything about your code if you want to switch from using, say, LM on your laptop to Spark in a Spark cluster.

And, of course, the idea here is that we're going to give a unified interface to all of them so that you can not have to change everything about your code if you want to switch from using, say, LM on your laptop to Spark in a Spark cluster.

So next we need to set a mode. So notice here that it says regression in all of these here because it turns out linear regression, the model always is predicting a number. It's always predicting a continuous property. But some model types, some model algorithms don't have to only do one kind of output. So an example of that is a decision tree. So if I call just the function decision tree here, you can see it has a default computational engine. R part is an R package that is used that has an implementation of decision trees. But notice it says unknown mode because a decision tree can be used for regression or classification.

When I first learned about this, I was confused because the idea of a decision tree being used to predict a number was confusing to me. But, you know, it depends on how complicated your tree is, like how smooth of a set of outputs that you can get. So decision trees can predict either regression or classification. And the way that we do that is we set the mode. So we can set the mode as regression here if we have a numeric continuous output, or we can set the mode as classification if we have a categorical output where we're going to predict yes versus no or red versus blue or some category versus a continuous one.

So all of the models are available. Whoops. Let me see. Can I? No, I can't. Oh, my gosh. I can't copy it. Okay. Sorry. I'll drop the links later. So if you go to tidymodels.org slash find slash parsnip, there's a list of all available models. And what you can do is you can, you know, search and sort and see what is there and what is appropriate to your use case.

So these are the three pieces that the three components or aspects of a model that make up a model specification in tidymodels. The reason why these three are the ones that are required is because that's what we need to kind of fully specify what kind of model you need to do. And a reason why you might decide to use tidymodels is because you don't necessarily know ahead of time what kind of model algorithm is going to be the best fit for your particular use case. And, in fact, you want to try a bunch. You want to try more than one, maybe a lot. And if you use tidymodels, that allows you to pretty fluently change from one kind of model algorithm to another.

By contrast, if you are someone who's like, I only ever use LM, LM solves all my problems and is good enough, then you're probably in a situation where tidymodels is not going to help you that much, where working with using tidymodels is not, you know, maybe not necessary, maybe not that important.

Unified interfaces across model implementations

Another piece of this, what makes a model specification, is about the interface to these different models. So what we have here is three different implementations for boosted trees. So the XGBoost implementation, the C5O implementation for boosted trees, and the Spark implementation for boosted trees. So they all are implementing the same or close to the same algorithm, where you make a tree, a decision tree. You train it on some of the data. You use that result there to make the next one. You do it again. You, like, boost the tree. So you, like, take a bunch of weak learners, and you put them together to make a strong learner.

The thing is, they all have different names for the arguments that are actually the same thing. Like, look at that trees one. That's when you're making a boosted tree model like XGBoost does

or like the Spark boosted tree thing does. There is a number of trees. Like, how many trees are you going to have? In XGBoost, that number is called n rounds, meaning how many rounds of boosting are you going to do. In C5O, it's called trials. In Spark, it's called max iter, like what's the maximum number of iterations will go through to boost things.

I find it kind of overwhelming if I wanted to switch from, like, XGBoost to Spark to try and figure out how these map to each other, like how these things map to each other, so that you can do an apples-to-apples comparison or, you know. So another thing that Parsnip offers you is a unified interface. We will admit it's another set of names, right, for all of these things. But it is one set of names to interface to all of them. So you only have to know trees instead of knowing all the individual different kinds of names.

So this is another reason. So if you think about what kind of problem am I working on, do I need to try different implementations to be able to know what the best option might be? If you are in that situation, then tidymodels can be a great fit for you.

Spending your data budget

All right, let's move on to the second reason to use tidymodels, the second sort of thing you might be faced with that you might think, okay, I want to do – I need to do this. How am I going to go about doing it? It's this idea of spending your data budget. When you work with – when it's time to approach some modeling problem, some machine learning problem, you have a certain amount of data. And, you know, in most of our situations, it's not infinite, right? You have a certain amount, and you want to do a good job of using it to get the best results that you can from the limited data that you have.

So the package in tidymodels that handles this data budget, these data budget kind of tasks, is called RSample. So it's like a shortening of resample. So it's tools for data splitting and data resampling.

So let's start with data splitting. So I bet many of you are familiar with this idea of splitting your data into training and testing. So you have some original quantity of data, and you need to split it randomly, typically, and you're going to keep some of your data. You're going to use to estimate the model parameters. You're going to use for learning. Another piece of the data you are going to use to estimate how well the model is performing. So it's not used for learning, but instead for prediction only, for estimating performance only. And we want those totally separate. We do not want to reuse – we do not want to use that testing data for any important things.

So tidymodels, of course, provides fluent tools for these kinds of splitting. So here I've just got a quick little example data set here that's housing prices in Sacramento, California. And here if I take that original data set that I have and I use initial split, what I have is a split object that keeps track of which of my observations went into training and which of them went into testing.

Once I have that split object, I can operate on it to get out what I need. So if I called training on the split object, I get out those 699 houses that are in my training set. If I call testing, I get the other ones that are in my testing set. So the idea here, you use the training data only during the process of model development and only at the very end to use the testing data. In fact, we would think – we would say, like, the testing data was extremely precious. We can't go about just, like, spending it willy-nilly.

The purpose of the testing set is to estimate your performance on new data, to do a final check on did my model – is my model performing the way that I thought it did.

The purpose of the testing set is to estimate your performance on new data, to do a final check on did my model – is my model performing the way that I thought it did.

So that all sounds well and good, you know, great, training, testing. But I just said a little bit ago that we might want to try a whole bunch of different models. We might want to – we might want to try XGBlues, Juice, plus Random Forest, plus Glimnet. I might want to try a lot of different things and see how they compare to each other. So what do I do? What do I do in this situation?

If I use the testing set to compare the performance for four or five models, I am up a creek. That is a bad decision, and I am in no good situation. I can't compare four or five different models. XGBlues needs to be tuned. There's a bunch of hyperparameters, and we have to try a bunch of different – there's no way to know what those hyperparameters should be from training the model one time. Instead, I have to train it a bunch of times on different model configurations and see which one is best. How do I decide? What data do I use to see which one is best? If I use the training set, I get an overly optimistic view. And, in fact, if I – depending on how my training split, I can get a wrong – I can get a wrong answer depending on exactly which observations are in the training set.

Resampling strategies

So the answer to this is resampling. So what we will do is we have our all data to start with. We divide into training and testing. And that testing data just stay – is set aside. Think of it as precious. We do not touch it until the very end of our model development process. And, instead, we resample our training set to create little simulated versions of that training set. So for each resample, I divide into an analysis set, which is an analog of training data, and my assessment set, which is an analog of the testing data. And I can use all those resamples for hyperparameter tuning or to do something like comparing. Say I want to compare random forest with glimnet with a decision tree. I want to compare them all and see which one is best. And we do that by using our resamples, using all of them and then aggregating the results there.

There's a huge number of ways to make resamples of how to divide up your data. One of them is called – one of the most common and great default kind of one is cross-validation. So if I – the way cross-validation works is that I have my pile of data, my pile of observations. In this example, it is the training set there, that yellow-orange pentagon-y thing, that training set. So these 30 things are the training set. And I can randomly split them up into a number of folds. So here we're doing three for this example. And so I randomly assign as I go along which one goes into which fold, yeah, into which fold, and then I divide them out so that in fold one, the first time I go through, I hold out one-third of the data, and the other two-thirds go into the analysis set.

So notice I'm going to fit the model using two-thirds of the data in this case, and then I will estimate performance using held-out data that was not used in the performance. So here where it says estimate performance using as we go along there, that's the equivalent of the assessment set. It's like a little test set. It is not used to train. It is instead used to measure performance, the test. So in the second fold, we hold out a different set, one-third, and train on the other two-thirds. On the third fold, we hold out the last third and do this. So this is how cross-validation works. In tidymodels, the function for it, it's called vfoldcv, and the default is actually to make 10 splits, because it turns out that works great for many situations. If you have the amount of data that you should be doing cross-validation, 10 is a good default.

You notice here this might be a little too small for cross-validation because I don't have that many observations in my little test sets there and my assessment sets. So I might, for this data, I want to think about a different approach. But let's talk about what it is. So we have the same amount of testing data, sorry, training data, and then we make 10 different resamples from it. We divide it into 10 pieces, and then we hold out the first, use the other nine-tenths, hold out the second, use the other nine-tenths, hold out the third, and so forth. That's how we make these cross-validation resamples. What this lets us do is to use the training set for tuning, for comparing models, for making decisions, before having that very precious test set at the end.

For something like this, it might be a better idea, for a smallish data set like this, it might be a better idea to use a different approach for resampling, like bootstrapping. So it is a different way of making these kinds of resamples. This approach is, again, showing an example of three. So the idea is you take your original training set that had 30 in this example, 30 examples in it, and you draw with replacements until you get to 30 again. So notice that in the first bootstrap iteration, the first observation is in there twice. So you draw with replacement so that it goes back into the bucket before you take another one. So back in the bucket, you take another one. So in the second one, notice three is in there three times. That can definitely happen. And then whatever doesn't get picked, whatever doesn't get picked to go into the analysis set, that goes into the assessment set. So here we are always going to train on 30, and then we will estimate performance on, you know, whatever kind of number, you know, is leftover, whatever the leftovers are there.

So bootstrapping, the default, you know, here's the function for that. Bootstrapping, the default is 25, bigger. And you can see that for my smaller data set, this seems like a better idea. So cross-validation and bootstrap resampling, they have different kind of questions, or different kind of bias-variance tradeoffs is what it comes down to. So it can be a good choice in different situations.

Q&A: reproducibility and setting a seed

Let's see, do you have a question about what I'm talking about right now? Yeah, we have a few questions in the chat. I'm going to save some of them for the end, hoping we have some time. But there's one relevant to what you're presenting right now. How does the setC function ensure that we can create reproducibility?

Yeah, great. Yeah, so these all involve randomness, right? Like we're pulling from these distributions randomly. And so if you ran this, and then, well, I have a setC here. And so what that means is that you will get the same result every time that you run this, if you run these together with setC. Here, oh, I have setC there too. But if you were to not run with setC, if you were just to call bootstraps, bootstraps, bootstraps, like you would get different versions every time.

Now, we would hope we're using robust enough statistical methods that we're not seeing huge differences. Like we pick a totally different model if we set the seed, if we set the seed different. That means something's wrong, right? Like if you choose a totally different model, if you set a different seed, that means something's not going great. However, it always is a good idea to set your seed when you have randomness, just so that you get the same answer that you get the next time, like down to the same like exact numeric answer versus being like, oh, my gosh, my metrics are a little bit different this time than what I got last time. So great question about reproducibility and setting a seed.

So that number there, 1, 2, 3, is not a special number. Like I could put, you know, the year I was born, this year. You know, like I can put any kind of number there. But what it does is it sets the random number generator. So computers have a hard time making random numbers, but we have these algorithms that give us close to, as close as possible to truly random numbers. And it's a stream, like R has a random number generator stream. So if you've ever seen RNG stream, that's what that means. It's like the stream of random numbers. Now, if you don't set the seed, the next time you go into the stream and say, give me a random number, you'll get the next one. Whereas if you do set the seed, you always start at the same place in the random number generator seed, which is not like you would be bad if you got a totally different result at the end, like you chose a different model. But it's still helpful in the process of model development so that you get the same answers every time.

All right. So there's lots of different resampling methods. And what these allow you to do is to spend your data wisely. You have a certain amount of data, and then you need to make the best decisions possible about how to spend it. You spend some data on model estimation, on learning those parameters of that model, whether it's XGBoost or linear regression. And then you spend other data on how is that model performing? How is that model performing? I have to know. I can't use the same data that I use for training because you can memorize that data. These different kinds of – so, like, really good options include cross-validation.

So that's that V full CV. Bootstraps are a really great option. You can do Monte Carlo. It's okay, right? Leave one out. Cross-validation is maybe not the best option. But it is there as an option.

If you have a ton of data, just a ton, then a validation split can be a good option. So that's basically picture this image, but you just have one of those. You just – instead of having B, see how it says resample one, resample two, resample B, instead of doing it, you just do one split. And then you can use that validation set to tune, to choose models.

And so here, pictures are just being one row instead of ten rows or 25 rows, just one row. And that can work if you have so much data that it is a waste of time to do things ten times because you get a good estimate of how well a model is performing with one assessment set.

Recap of reasons to use tidymodels

Okay, so we talked about building models. Like, you may choose to use tidy models because you need to try a bunch of different models. You may choose to use tidy models because your data budget is limited and you want to use good statistical practices to spend your data in a wise way. Tidy models provide you with these tools and guardrails, like when it comes time to fit, to tune, to – we're about to talk about feature engineering. When it comes time to do all these things, tidy models provide you with tools and guardrails, like keeping you from doing the wrong thing in many situations so that you can spend your data budget wisely.

Feature engineering with Recipes

The last of these three topics that I want to talk about, like why might you decide to use tidy models, is to build better features. So the package in tidy models that handles feature engineering is called Recipes. It's the Recipes package. And it uses a certain, I guess, analogy or metaphor for what feature engineering is like.

So, you know, we use the phrase pre-processing kind of to be synonymous with feature engineering. Your data, you have data, and you need it to go into a model, but it's not quite ready to go into the model yet. You need to do some stuff first to it before it can go in.

This is not so much data cleaning. So data pre-processing or feature engineering is different from data cleaning or more basic data preparation. Like if it's something that you could use dplyr to do, that's not really, like two lines of dplyr to do. That's not really what we mean by data pre-processing. Instead, we're mostly talking about statistical transformations of your data.

So the analogy here is that you have a pantry, and in your pantry are various variables. They're like the ingredients. So the ingredients for your feature engineering are like those are the things you're starting with. You've got response, an outcome, an output. You've got predictors or inputs here.

So we write a recipe that uses our ingredients, our components, our variables, and says what to do. And it's like a recipe like step one, step two, step three. So here, this little example, it's like first, I'm going to impute missing data using K nearest neighbors. Notice this is like a statistical transformation. We learn it. We learn it from training data. It's an unsupervised algorithm, actually, if you remember all the way back from the beginning. So we learn which, if there's missing data in some variable, which one is that most like with nearest neighbors? Let's put that value in there.

Then it's step scale. So that's where you're dividing each of the variables by a standard deviation. So they all have standard deviation of one. Again, this is a learned statistical transformation. You use the training data to learn what the standard deviation is, then to divide.

So you write out a recipe. Then you say, okay, here's some training data, and I'm going to learn what those transformations are. So that's what there is, number three. So number one is what are my variables. Number two is what's my recipe. Let's write it down. Number three is let me use my training data to learn those statistical transformations. And then number four is I can now apply this to other data. Say I could apply it to new data at prediction time. You can think about applying it to the testing data.

Because these steps, especially these examples that are here, these steps learn from data. So imagine a situation where you used your whole training set to learn these, and then you try to estimate. You, again, were like, wait, how well did that work? So, again, you have to do this inside of a resampling loop. I don't mean literally you have to write a loop, but you have to use your resampled data for feature engineering just like you have to use your resampled data for model estimation.

These count as parameters of your model broadly defined. They're not a parameter of the decision tree, but they're a parameter of the feature engineering, and they have to be learned. One reason to use tidy models is that it provides you with the guardrails and the tools to not have data leakage, to not have overly optimistic results from your feature engineering.

One reason to use tidy models is that it provides you with the guardrails and the tools to not have data leakage, to not have overly optimistic results from your feature engineering.

So, I mean, I think in the machine learning practice world, not very many people are falling into traps anymore around, you know, training on the whole data set and not having an estimate of how it does at the end. But it's tricky to keep data leakage out of your feature engineering as well. So, there's good support for that here.

Q&A: handling missing data and class imbalance

So, let's – yes, Jessie. You touched on this a little bit, but can you give a little bit more information about how tidy models compared to traditional ML handles missing data, class imbalances, and the preprocessing in general?

Yeah, that's exactly what this is about. So, tidy models provides functions, or they're functions, but they're technically what we call like recipe steps for all of these kinds of things. So, in the basic recipes package, there are feature engineering preprocessing steps for things like imputing missing data. I think someone said class imbalance. So, there's a recipes add-on package. It's called CMIS. This is what it's called, T-H-E-M-I-S. So, that is within the tidy models, you know, framework, and it provides steps all for handling class imbalance, like upsample, downsample, all the different kinds of algorithms, or many different kinds of algorithms that are out there for dealing with class imbalance.

So, yes, those are handled with recipe steps, which are these modular pieces that you can put together in a fairly sensible way. You like write out, do this, do this, do this, do this, and the input of one can go to the output of another, and you can just like go through and, you know, you're not having to do much of this manually. Instead, you can specify it as a recipe.

So, let's look at what that would look like. So, here is where we're taking our ingredients. So, if I've got that same Sacramento housing data I had before, I say, okay, my output, my outcome, my response, that's the price. That's the thing that I'm predicting. The input here used a dot, which means everything else. So, I start with my ingredients. So, price is going to explain everything else. And then I build my preprocessor step by step.

So, here, what this first one does is if you have a factor and you see a new level in it at prediction time, this just kind of puts the mean in there. So, it's a handler for new levels and factors that you didn't see during training time. SEPDUMMY creates dummy or indicator variables. So, let's say that you have a bunch of factors and you need to convert them to indicator variables like a bunch of ones and zeros. This does that. SEPVV removes variables that have zero variance. So, models like, especially the linear models, behave very, like they can't converge if you have, they will error. If you have variables that have zero variance, meaning they're all the same value, like picture yourself doing resampling and you end up with a, like there's some factor in it that's, you know, red and blue. And you end up with all reds in one fold, one resampling fold. Then what this does is it removes that so that you don't, so that your model can fit anyway, even though that turned out that way.

So, we basically, it's basically like all kinds of different, we just specify here, here, here. You do have to think about the order a little bit. Like, for example, you can't do SEPNOVEL after SEPDUMMY. Because SEPDUMMY means, what SEPVDUMMY does is it converts all your factor variables into numeric variables. So, if you did SEPNOVEL after, you know, it doesn't do anything because all your factor variables are gone. But that's the way recipes work, you know, like in real life, you have to do things in the right order. So, the analogy holds up fairly well there. You have to be thoughtful about the order in which you set up a recipe.

Types of feature engineering steps

So, I just showed you a couple. But there are a lot of different ways of doing this. So, you can, we have those categorical predictors, things like a predictor that say red, green, blue, or some kind of factor. You can encode those in various ways to numeric values. So, I talked about dummy or indicator variables. That's like everything to 1s and 0s. Like, if you have three levels, you go to three columns, and it's all like 1, 0, 0, 0, 1, 0, 0, 0, 1, like that. There's other ways of encoding categorical predictors, like using an effect encoding. That's like a supervised feature engineering approach, which, again, means you want to be really careful about how you use resampling to see how that goes.

You know, real basic things, like centering and scaling variables. Some model estimation approaches need everything on the same scale. Anything, basically, that uses like a distance metric inside of it, nearest neighbor models are like this. Support vector machines are like this. They use a distance metric, so you need to have everything on the same scale.

You got to handle class imbalance, like someone asked a question about. So, you know, if you have a very imbalanced kind of training set, where you have a lot of one class and very few of another class, you can handle that somewhere, either by upsampling the minority class or downsampling the majority class. And there are different algorithms to decide how exactly to go about doing that.

Imputing missing data. If you have missing data, you don't have to throw it out, depending on if you have some missing data in your training set, in your data that you want to use for machine learning. You don't necessarily have to throw those examples out. Instead, you can use some kind of algorithm like nearest neighbors or just a plain old median or, you know, there's lots of different ways to impute missing data that you have.

You can perform dimensionality reduction. So you can say you had a lot of features, just a lot, too many, in fact, one might say. One thing to do is to use an unsupervised approach, like maybe there's structure in there. Some of them are correlated with each other. You can use dimensionality reduction to project that high dimensional space, lots and lots of predictors, down to a lower dimensional space. And then you can use a model estimation, you know, algorithm like linear regression or, you know, anything like that, to then learn your outcome from your inputs.

So there's just a lot more. I think, you know, when I talked about models at the beginning, you're like, oh, my gosh, there's so many models out there. How, like, I have to try a bunch of them to see what might work best. I feel like there's even more options that things do when it comes to feature engineering, but on the positive side for that, usually you know more about your data. And feature engineering is really a place where often we use our domain knowledge by how to create these better features.

And what tidy models lets us do is to, you know, to write something like this where we write with a series of steps, how do I need to get from my beginning data to my data that is ready to go into a model, some kind of machine learning algorithm. So here, notice, we normalize, and then we're doing principal component analysis, which is a kind of dimensionality reduction here. So the output of that, well, okay, let's say, so here's this recipe, I can put that recipe together with a model. And the recommended way we do that, we have for doing this is in what's called a workflow. A workflow, a modeling workflow, puts together the components that you need to, that you have in a connected thing that you can then walk around with and use, for example, within resampling.

So here, what I'm doing is I'm just, I'm fitting it to my training data. So notice here that the coefficients of my model are those principal components, those principal components that I learned.

So, you know, I started with like more features, like what I showed you in the, at the beginning when I was talking about spending your data budget. And at the end, what I actually predict my, what I fit my model with is with these. And then I can also predict with those, even with my input data as it is.

So we put those things together in a way that lets us use, lets us have good statistical process through that whole thing.

Wrapping up: three reasons to use tidymodels

So there is lots more to learn. These, in terms of actually using tidymodels, I really like thinking about focusing on these three items as things that might be motivating for you to think about whether tidymodels is a good fit for your particular use case. I mean, they are very common things that we run into like, oh, I don't ahead of time know which kind of model I should use. Oh, I have a limited amount of data. I need to make sure I'm using it responsibly so that I'm not fooled into thinking my model is going to perform better than it does in the end. Or I need, I really got to do some feature engineering, some pre-processing of my data to get it ready for my modeling.

These are very common and great things to keep in mind as you might want to, as you might want to think about what modeling approach you'll take. So with that, I will say thank you so much.

So here are some resources that are up here for learning more about, for learning more about tidymodels. So I have a YouTube channel that is pretty much all tidymodels screencasts there. I blog a fair amount about tidymodels. And tidymodels.org is a great place for getting started. And then if you really want to dig deeper, really like get into the meat, you can see Tidy Modeling with R at tmwr.org.

And that also is the book that is being given away, in the raffle tonight. So that's a great option if you're like, it's time for me to really learn beginning to end the details, like really dig deep into it.

So with that, I will say thank you so much. I will stop sharing so that I can maybe take a look at the questions. You can tell the sun is setting here and I'm starting to be in like, my sun sets.

Q&A: cross-validation for time series and spatial data

Yes, go ahead. You want to help me help with these questions? Awesome. So one of the questions from earlier, we were talking a little bit about cross-validation is how does it handle, how does it work for time series modeling?

So there are some, one great thing about no longer sharing is I can find links easier. There are methods for a time-based data. So there, I think a lot of the, there's a couple of them, a couple of different kinds, probably the one that you would want to use the most are these sliding windows. So if you picture yourself having, you know, picture yourself having a hundred time series data. So the way that you do cross-validation with time series data is that you make these sliding windows and you train on, on say the first part of the window and then about a test on the second bit, and then you move it down like this. And then that helped that like gives you say, if you did 10 folds, 10 sliding windows, you get then get 10 estimates of performance and you can aggregate those. If you're out how, how it goes. So those are the sliding windows.

The other ones are the, a rolling origin, which is a little bit, it works. It works slightly different than the sliding window, but there are these different, these different methods for when you have time series data. The other thing to keep in mind is you need, you need different approaches when you have spatial data. With both spatial data and with time, time series data, the thing is that objects that are close together, either in space or time are more similar. Then then in random, like they have, you know, it's called auto correlation. Like things are similar things that are, that are near each other are similar to each other. So you have to do a slightly like more careful job of data splitting so that you do not get again, fooled into thinking you have better performance than you do.

So I dropped a little much for that. I dropped very comprehensive. Yeah. I dropped the link for the spatial sample one as well, which is really worked really good. Thank you.

Q&A: tidymodels vs. Keras and computational performance

We have a couple of more high level questions. So I'll start with one from the beginning. How would you say tidymodels simplifies the process compared to something like Keras, which relies on TensorFlow and can be computationally expensive and time consuming. So how does it help with that particular part computation and,

Yes. So, so tidymodels actually uses Keras in some situations. So if you do a multi, what is it? Multi-layer perceptron? No, no, no. Oh my gosh. I am totally blanking on the name. Here. No, I'm finding it. Okay. So a multi layer perceptron, which is a single layer neural network, you can actually do that with Keras and tidymodels. So I would say,

I would say it does. If you're doing something that's computationally expensive, tidymodels doesn't solve that problem. It doesn't solve the problem that the, the underlying thing you're doing, like there's no performance gains. There's no changes to how models are implemented. So you, you will get, you will have the same kind of performance. If you use Keras directly, versus if you use this tidymodels interface to Keras. So tidymodels goals are not about increasing performance. You'll get the same performance as what you might use elsewhere.

So tidymodels goals are not about increasing performance. You'll get the same performance as what you might use elsewhere.

It tidymodels provides interfaces to many models that are much faster than the ones that you have access to in Keras. So it's, it's a little bit like different, different priorities or different kinds of approaches. So in tidymodels, you can try, you know, linear regression or extra boosts, which are much faster than, you know, learning a multi, you know, like maybe a, like a deep neural network. So we have different, it's like different goals. You might be more likely to use tidymodels. If you have rectangular data and more likely to use something like Keras or Pytorch, if you have say image data or, you know, video data, some language data, although you can use tidymodels for language data as well.

So interesting question. I would say there are, there are no performance benefits. Hope there's not really performance, you know, on benefits. There's like, it's not worse performance, but it is. The goals are to give you a uniface to try lots of things, not necessarily to make the performance better.

Q&A: future challenges for tidymodels

Thank you. And then we have another one about the field of data sciences, constantly involving. What new challenges do you foresee for tidymodels? And how do you think the ecosystem will address them?

That's a great question. So tidymodels focuses on rectangular data, because that is, that is like just really the bread and butter. I think even today with the rise of large language models, like I still think in most people's organizations, what's really providing value to really helping people make decisions is models based on rectangular data. Not like, let me go to this large language model and ask it what I should do. You know, like that's not really solving people's problems, at least as of today.

So I think the, when I say, when I think about like, what are the, like, what are next steps? What are ways tidymodels is moving forward and helping people with their real problems. I think a lot like in these areas where people are using their own rectangular data to make decisions, to serve, make decisions, serve predictions, that type of thing. I think a lot of the questions are around a explainability and B. Issues around measuring, like how models affect different groups differently. So maybe things under the, the, you know, the category of like fairness, like measuring fairness. Can you, can we do anything to ameliorate unfair problems with models?

So tidymodels integrates. Well, with many of the explainer explainability methods that are out there. One of the chapters of that very book that's up there is all about using explainability methods. There's also a chapter on model trust. When, like, can we use what we know about the model? Can we use what we know about new data to understand when we can have high trust in a model predictions? And when we can have low trust, an area that is newer for tidymodels is fairness metrics.

I met, I mentioned before the yardstick package. I didn't talk about it a ton, except for this introducing it. The yardstick package is what provides metrics. And most of the fairness metrics, like what, what they involve is taking on reg quote, regular metric and seeing how is it different across different groups, maybe different protected classes. So there, so actually there's work going on right now in tune and yardstick are the two places that is happening to provide, to provide support for these sort of group wise metrics, which are often known as fairness metrics so that you can measure differences in model performance across groups.

So an example of this, you know, is like say you have, say you have people of different education levels in your data set, say one, three, right. And can you measure in your model performing better for the people with higher education, right. Compared to the people with lower education. So you can do measure that, understand that. And then of course, then you lead into this as a, what are you going to do about that?

I love that one. That's really cool. So none of that is on, to be clear, none of that's on cram yet, but it's all actively being worked on like NPRs and stuff like that. I'm looking forward to hearing more about that. I think a lot of organizations are starting to look in that into that. It's a big priority right now. So I'm excited. Yeah, I agree. I agree.

Q&A: real-world modeling experience and tidymodels in practice

The book provides a lot of practical examples and case studies. Can you share a particularly interesting or challenging real world problem that you've tackled using tidymodels and how the framework helped you overcome it? Yeah, that's a great question. So I was a working practitioner, like real life kind of working practitioner before tidymodels was really mature. Like I would say tidymodels has become very mature. Like as, as during what I've been working on, it.

So when I did real world, like machine learning, like, for example, my last job, I was at stack overflow and some of the models that I worked on and was responsible for would be at the time. Stack overflow had a, had a jobs board slash hiring platform. And so like we showed people jobs in the sidebar and, and, and those were, there were two ways to decide what job to show. You can either use the content on the page. So you're looking at an R question, show people a data science job, or you can show people content based on the, the, the, the person, the user.

So in that case, you say, okay, this person is looking at a, you know, it's looking at a CSS question. Right now, but I know overall they look at R and shiny questions like this person. So I'm not going to show them a front end web developer job. I'm going to show them a data science job because overall they look at R or R and shiny things. So I know they're just, you know, they're just looking at CSS to solve their shiny CSS problem. Right. And so that is an example of a kind of modeling, you know, kind of model I was responsible for and worked on.

So that I didn't use tidymodels because I didn't want to show people So that I didn't use tidymodels because it was not really mature enough. I think. So my, I like try to keep my hand in with modeling often comes from teaching from demos. So I think, you know, I really like doing some of the, the tidy Tuesday data sets. It's not the same as a real problem at work. Right. It's not the same, but it does kind of give some nice ways to show examples. And it kind of had to keep, you know, keep, keep the muscles working the like I train. I'm not just, I don't want to be a tool builder. Who's too disconnected from the people who are using the tool. So let me try to stay up on it. So that's how I answer that question.

Q&A: ethics and privacy in healthcare data

Thank you. And then what ethical or privacy considerations are important when working with healthcare data in particular? That's a great question. That's a great question. So I have not worked in healthcare as an industry. So I certainly don't have the kind of real world experience. I've worked like in tech. I've worked in the nonprofit sphere. So, so there were some privacy issues around some of the data and the nonprofit area, but certainly, certainly. I mean, probably not at the level of, you know, I've never worked anywhere where like the data I had to be like, There were super concerns about, I even had it on my laptop, you know, like I like, like I could only access it through certain through a certain server with like very strict lockdown kind of rules.

I have not had that, that kind of experience, even, even though I'm thinking back to like my last job before I was that positive when I was at SAC overflow, even that, I mean, like you can think about privacy implications of user data at stack overflow, right? Like those are, we have, we have enough data on people's activities that even that can be sensitive.

So I, I know that. So I don't have, I don't have practical real life experience of how to answer the question about healthcare data, but in general, like I feel like the privacy, like privacy and ethical considerations around data, you know, come, come into play maybe more than we would like to think. Maybe it's more important than we would like to think. Even, even when we're not in those examples, like, like healthcare data, like people's private health, health data.

And I think one, you know, like there's questions like, do we need to collect this or keep it at all? You know, like that's a good kind of first question for many of these things. And then like, how responsible are we in how we, how we store and expose some of this data?

I worked, so I worked on tidymodels for like just tidymodels for a year, year and a half. And then I have been working on tools for, which I didn't really talk about much today, tools for deploying models. So ML ops kinds of tasks.

And it's very interesting to think about the privacy implications when it comes time to deploy a model, because often you want a deployment service to be able to serve predictions without too much risk of sensitive training data being exposed. And it turns out that you can expose sensitive training data, even through your model being deployed, depending on what you save, what like, what binary objects do you say where, what, what information do you allow even to be returned as an error? You know? So it's something that's really interesting to think about and to try to build safe defaults, safe defaults and to build in tools for people to make kind of use those guardrails.

And it turns out that you can expose sensitive training data, even through your model being deployed, depending on what you save, what like, what binary objects do you say where, what, what information do you allow even to be returned as an error?

Ensemble learning and model stacking

Thank you so much. We can crowdsource a bit more on like healthcare in particular. I think we can get some users to chime in as well, but that's great. We have a more specific one. So what are, or how does tidymodels support ensemble learning and model stacking methods?

Yes. So maybe, I don't know if this person, if this was like a softball kind of question, but there's a, there's a, there's a tidymodels package called stacks. And so what the stacks package does is it provides an interface to model on assembling or model stacking. So it, I showed you a workflow kind of towards the end of that talk. Like we put together feature engineering and model together, model estimate, like model estimation together. And what stacks does is provide an interface to ensembling workflows. So you can, you can ensemble different combinations of preprocessors and models here. So it is pretty nice. It works really well.

I'm stacking or ensembling. What, what it gets you is like you squeeze out the very last bit of every last possible bit of model performance out of the data that you have. So I, you, you briefly heard me say we ensemble weak learners into strong learners. Like that's how XG boosts works is that you take or boosted trees in general, you take a bunch of decision trees and each one is a weak learner, but you stack them all. You boost them. And then you combine them and you end up with a strong learner. Well, you can do that, not just with decision trees. You can do that with any arbitrary model. So you can stack up linear regression plus XG boosts, plus, you know, anything you can, you can stack everything up all together. And then you get, you know, you, you, you increase your model performance this way. And by that, I mean, something like accuracy or, you know, RMSE or something where you're like, how good of a job is my model doing? You can, you can bump that up often by assembling at the cost of having training, have to train like 10 times more models, but sometimes that is worth it.

Best practices and tips for tidymodels

That's really cool. I'll have to look into that a bit more. So we have one last question. And this is a pretty open ended one, but do you have any other best practices, tips or tricks for using tidymodels?

Yeah. Okay. So what, so I think that the ones that are the, that I think are the most common, the problems, like the most common problems, like that I myself have done are messing up around feature engineering. I kind of already mentioned this, but thinking of feature engineering as something that you do one time, instead of something that you do over and over within the resampling loop that you do it, you do it one time with the training data versus you do it 10 times with the resamples. I think that is the most common, like, like error that people make and that you can avoid that in tidymodels by using workflows, you know, using the infrastructure that exists there.

So any other like best practice kind of, kind of situation or kind of advice? There's a, if you're ever like, oh gosh, I don't even know how to get started. There's a, there's a, our markdown template for getting started with tidymodels. Let me, I'll drop you to the blog post about that. You can look at it. It's kind of a nice, it's a nice little, a nice little, where did it go? Let me find the blog posts on it.

It is yeah, here. Yeah, here we go. So here, this is, I think a nice little, a nice little template for, you know, like just getting started using your, using your something to like use good practices. I think that is a good, a good approach. And I think if I remember right, someone had a blog post where they walked, they walked through it.

So it is that that's something that comes to mind that maybe not as many people know about that. It's, it's, it's like gavs you take that first step to be able to like, I, I'm not sure what to start. Am I going to forget one of the steps that like, let you walk through that. So I'll say that. I think that's a good one.

Closing remarks and raffle

That's all the questions that we have in the chat. I think it's time for the moment. Everybody's been waiting for Jessica. Do you want to get the raffle results going?

Yes. I'm so excited. So I'll go ahead and click the screen. And I'd also like to thank you again, Julia for today's incredible talk. We had really great engagement and attendees. So thank you so much again. Thank you so much for having me.

I really hope we can do more of these O'Reilly collaborations with authors. It's been really great. O'Reilly and you as well. Thank you. And also check out our meetup page for our future events. Coming up. And I just want to thank the rest of the organizing team as well as our ladies. For helping us put this all together. So with that, I believe. That is. That's it for us. So thanks everyone. Bye-bye.