Resources
Dec 9, 2021 |
1h 2min
|
4k views
|

RStudio Connect | Cut down on the grunt work. Deliver insights more effectively with RStudio

You built a great data science product. Something your stakeholders want and have been asking for.

But operationalizing that data product is hard.

How do you deliver it to interested stakeholders in the format that they want with the specific insights that they need? And what if you need to deliver your data products to multiple stakeholders, all with slightly different requirements?

How do you make sure that the data product is always up to date? And when the data is updated, how will your stakeholders know?

This is why we built RStudio Connect. RStudio Connect is the best way for data science teams to curate and share data products with their organizations. It comes with all of the bells and whistles that professional data science teams typically need to turn their analyses into value.

In this live event, we’ll do a deep dive into one of RStudio Connect’s most powerful features: scheduling and distributing data science products.

This feature enables data science teams to take all of their data science products and insights and deliver them to everyone that needs them to make decisions. Always on time, in the format that they want, and as efficiently as possible.

Specifically: we will show how a data scientist can deploy an R Markdown or Jupyter Notebook to RStudio Connect, then schedule the data product to update and get delivered to interested stakeholders. All without the traditional work associated with updating data pipelines and updating reports.

Put simply: we think RStudio gives data scientists, data engineers, and business users superpowers, and we can’t wait to show it off to you.

image: thumbnail.jpg

Transcript#

This transcript was generated automatically and may contain errors.

Hi, everybody. Thank you so much for joining us for today's RStudio Data Science Live event. My name is Tom Mock. I'm going to be your host for today. I'm our customer enablement lead at RStudio. I'm going to be going through a lot of different content specifically related to RStudio Connect.

We're also running kind of a fun experiment in streaming to many different locations. So a lot of the time when we've been talking to our customers or people interested in learning about open source data science, it might be coming from YouTube or LinkedIn or Twitter. So we're trying to meet everyone where they want to be in streaming to all those different platforms. So we're hoping everything goes really well today. But if you do have any hiccups, please bear with us. We're really excited about trying out these new different streams and seeing if we can have a good time with it.

For today, this will be recorded so you can always watch the recording afterwards on YouTube. And the content will be available in terms of the slides as well as the code I show here on the screen. So the slides are going to be at colorado.rstudio.com rsc.automate. And we can actually send that out through the chat. And then we'll also have the code examples I'm using as available as well. And those will be shared in the chat.

So all these different possibilities in terms of like what you can build with open source are in Python and then going further.

So again, my name is Tom Mock. We are trying out this new data science live is kind of a webinar style event, but going to many different locations. So hopefully this kind of again is an exciting moment for you. And you get to kind of join along with us as we're learning and sharing some knowledge for today.

Before I get started, I also want to give a quick shout out to my colleague Alex Gold, who's one of those solutions engineering managers here at RStudio. He put together this great bike share example that we're going to be walking through in depth along with a few other pieces of code. And then also my friend Katie Masiello, who's a customer success manager at RStudio. And she put together a great point blank demo content that we're going to be showing as well today as part of kind of the data validation, ETL scripts and other content. And kind of automation of data analysis and data quality and validation.

Overview of RStudio Connect

So as far as the core idea of what we're going to be talking about today, we're going to be looking at RStudio Connect, which is kind of our premier platform for building and sharing, or I guess in this case, sharing data science products and building kind of insights among your organization. The core idea is that RStudio Connect makes it easy to share your data products that you build in both R and in Python. For your data scientists, they get to stay and use their preferred open source data science tooling. So again, R and Python, but they're also able to get these things off of their laptop and share these insights to impact decision making across the organization.

So they might be creating things like APIs with Plumber or sharing datasets as a pin or versioning models with pins or deploying shiny applications for interactive web applications. Or lastly, what we'll be talking a lot about today, sharing R Markdown reports or automating and scheduling scripts or R scripts through R Markdown documents.

Now for our Python colleagues, in terms of the Python data scientists, they also have a rich workflow that they can kind of put things on to connect and use. So specifically, we'll be talking a little bit about Jupyter today, which is a great notebook that actually supports Python and R inside it, just like R Markdown can support R and Python. And there's other data products you can publish to connect like Streamlet, Dash, Bokeh for interactive web apps, or Flask and FastAPI for publishing APIs or RESTful APIs in Python.

Now, as far as getting these things out to your stakeholders, there's what we call kind of the easy button. You can just kind of go into a data product, click publish, and then send it out to your organization. Your decision makers, whether they're business stakeholders, executives, or other colleagues can then access the things you're creating via a web portal with authentication and with kind of automatic scaling within your server component. So lots of ability to create and share data products with our tooling, and that's what we'll be talking a little bit about today.

I'm seeing a lot of fun comments from people saying where they're joining us from. It's always exciting to see people from around the world. I think that's also one of the exciting things about doing these live webinars is we're able to kind of reach people across the world. So thank you so much for sharing all that. I'm joining you from Texas here in the U.S. and excited to see some folks from around the world.

Notebooks and automation on Connect

Now, as far as kind of the data products we're talking about today, we're basically talking almost exclusively about notebooks on RStudio Connect. So the idea here is you can automate the execution of arbitrary, basically any R and Python code you want with notebooks. So you can have R Markdown for native R code or Jupyter for native Python code in the RStudio Connect ecosystem.

Now, when I talk about automation in a little bit or scheduling, what I'm really saying here is that Connect gives you this ability to schedule the re-execution of that code. So not only can you create the initial document, publish that, or share it wherever you want, but you can also have it re-execute on a schedule. So I have lots of scripts that I run either daily or weekly that clean up some data, pull it together, and then save the data out, basically ETL jobs. What Connect gives me the ability to do is to specify things like a specific time zone. I'm in, you know, Texas, so I'm in GMT minus six, and I want things to execute at a specific time. So let's say 1037 a.m. on December 9th, which is about 30 minutes from now, give or take a few minutes. And I want this to run every single day, and I want it to republish the output, basically override itself when it's published.

I also have the ability to do things like send an email from Connect, and this would actually send out an email to myself or to colleagues. Now, when I say arbitrary R code or arbitrary Python code up here, that really means essentially any R and Python code you can write, you can embed into these documents. So I could have things like HTML widgets or leaflet plots or interactive kind of basic JavaScript inside my R Markdown document. And in Python, you have similar libraries, so things like Plotly for interactive graphics, as well as kind of the execution of the code itself. So maybe retraining a model or saving a data set out. All these things that we're going to be talking about today, and I'm very excited to go through it all.

Now, you might ask, why notebooks as opposed to maybe a .R or a .py file? Well, notebooks are very powerful in that you can achieve the primary goal, which might be actually executing your Python or R code and very useful side effects. So you could also have things like writing up to sync or to separate output. So you might be generating a RDS file or CSV file or a pickle file in Python or graphics or tables or whatever else like that is your primary goal. But your useful side effects are that you're also creating a nice document or a nice report that comes along with it. And that's what the power of a notebook gives you.

And that's what the power of a notebook gives you. So you get this rich self-contained output and report. Basically, the notebook is the data product in addition to whatever it's actually doing.

So you get this rich self-contained output and report. Basically, the notebook is the data product in addition to whatever it's actually doing. It's actually self-documenting. It's showing you when it occurred. It's telling you all these things about like the code you executed and the context it was run in.

There will be a lot of demo components. I do have some slides to get through, and then we'll go into some of the demos. And importantly, you know, these assets you create are shareable and hostable, meaning that not only can you create a notebook, but with Kinect, you can actually put it on there, share it out, host it there, and people can access it through the web. They're easy to read in terms of it's a typical document. They're self-documenting and easy to use as the data scientist and the end user.

So importantly, R Markdown and Jupyter often and almost always show the code that's used to execute along with all the embedded outputs in line. So for reproducibility standpoint, you kind of know what's going on. And what was the code that was run when this was scheduled? Now, you can hide some of that code execution if you're reporting to business users. We'll also talk a little bit about that.

And then lastly, R Markdown and things scheduled on RStudio Connect can also generate additional side effects. Things like sending an email. So not only can it execute the notebook, execute the R code or the Python code, it's also sending out this email or kind of message. It's also sending out this email or kind of many emails to all your colleagues, updating them only when something happens. So kind of meeting where they are in their inbox rather than asking them to go visit, say, a dashboard or something else.

Why RStudio Connect?

Now, one more slide, then I'll jump into a little bit of a demo in terms of what the actual platform looks like. But why RStudio Connect? Obviously, like part of the beauty of this is that with open source data science, you can build your own tooling. And that's fantastic. RStudio Connect is a great solution in that you can use multiple languages there. So R and Python native, as long with embedding things like SQL or Spark through Sparkly R or PySpark. You have environment isolation in terms of you're limiting the packages and libraries to specific versions. So capturing that environment. And when you publish it, that environment is kind of trapped and isolated with it.

You have your enterprise-grade authentication, so you can make IT happy and have all the things you're sharing be secure. From a data science perspective, you also have things like push-button deployment for easily deploying it out, as well as things that are more robust like Git or continuous integration or continuous deployment from, say, Azure DevOps or something.

So we'll be going into all of that. And I'll be showing a little bit more of the content that we'll be talking about today. The last part before we get into a little bit of the demo is the re-execution of R and Python code on demand or on the schedule. So you can actually show a little bit of the manual re-execution or scheduling automation of tasks.

Demo: Connect platform walkthrough

So let's actually flip over to Connect, and I'll show you a little bit what that looks like.

So I've hopped into Connect. You might have actually seen for that brief moment that I'm showing my slides here, and that I have this ability to schedule and distribute data science products. This is actually a data science product running on RStudio Connect, so my slides are on Connect. But more typically, rather than slides, I might actually create something like a basic report that's being scheduled.

So here I've got an older script that I've been running for a while, just looking at the weather forecast in Boston. So RStudio's headquarters are in Boston, so let's take a look at that. So this is a very basic report in terms of it's just got the title, it's got a plot confirmation here, and it's saying a little bit about a data set and a very basic plot. Not the most exciting thing.

The actual benefit is that this is taking that data and writing it out to what we call a PIN, or an external data set that's being updated. So this could be running on RStudio Connect, and you could actually be saving data to Connect, or you could be saving data out to Amazon S3, or to Azure Data Lake, or to SharePoint, or wherever you really want to save it. There's a lot of different options.

So this report that I'm looking at actually has the ability to show me this output, so I have this confirmation when I go and visit it that, yes, it ran, and I have some little basic information about it, but I still have access to the raw data, which is the purpose of this report, is scraping and collecting this data and storing it so I can look at it in the future.

You can see here on the right inside Connect that I already have the document scheduled, so it's going to be running in my time zone in Chicago. I started it just before Thanksgiving in preparation for today's kind of presentation, and I have it run every morning at 6 a.m. Just in case I ever had to go to the Boston office, I want to know before I fly out, what's the weather like? And it's going to run every single day and publish an output after it's generated.

Now, this is overriding itself, so you can see that for today, it was run as of December 9th, but with Connect, I can also go up here and look at the history, so I can see every run that occurred and look at older timeframes. So I have these snapshots of what the report looked like, even though the data is being updated to a separate environment. So these reports themselves, kind of as a side effect, are very beneficial in terms of I can go look at the different pieces and kind of make sure that things are going as expected, in addition to them actually executing properly.

Another benefit of Connect in terms of more from the admin perspective is if I go in here to the admin panel and look at scheduled content, I can see all the different things that have been scheduled for my entire team. So you can see there's a lot of stuff that we've scheduled here for our demo server, and some of these are very common. So some of them are occurring down to the minute, and some of them are occurring maybe once a day or a few times a day. So I get this nice little snapshot and be like, hey, maybe I don't want to have this be running every single minute. I can go check on that, view the content, and make sure I actually want it to be executing, as well as the timeframes it's executing, making sure things are overlaying and that we have everything going nicely.

What to automate

So let's hop back into the slides real quick, and I'm going to talk a little bit on automation, and then we'll go back into a demo portion and actually show a little bit of code. So I showed you a little bit about how RStudio Connect works, how you can schedule things, some of the benefits of the admin pane and kind of seeing all the different automation that's possible.

But what are the things you actually want to automate or things that you want to schedule? So there's a spectrum of needs here. In terms of things you might publish or automate. So you might have data updates like I just showed, like you have this ETL script where it's taking data from an API, saving it, and then appending it to a database or to some flat file where it's storing it for later. You might have a report that's running. So you just want to report on that data that's being updated. And actually, the part of automation here is creating a new report without you having to manually run it. And then delivering that to your colleagues.

Another common use case we see is automated model training, updates, or batch scoring. So here you're either training an update, training a model, updating an existing model, or sending that model out to actually do some scoring. So like running through, forming some predictions and then saving those scores out into a database. And then lastly, something I'll talk about in a little bit is automated email delivery with warnings or updates.

or just nice to knows.

So not only do you have the actual report running and connect handling the environments and the creation of those reports, but sometimes you want optional reports. Basically, you want to know only when something fails or only when something goes above or below a certain threshold. So by combining emails along with your reporting or your automation of data science, you can basically get your warning only when it's needed. So you have a high signal to noise ratio and you're not just getting updates all the time, but only when something occurs.

So you have a high signal to noise ratio and you're not just getting updates all the time, but only when something occurs.

Demo: pinning data to RStudio Connect

So as part of these automated data updates, we're going to share my desktop here and I'm going to show you just a quick example of a pin real quick. For pins, this is going to be a very straightforward kind of code, but I'm just taking a dataset and saving it out to RStudio Connect.

So this is actually, you might see a .R file. So here I'm loading the pins library, which allows me to take datasets and push them out to a remote location. And I'm loading dplyr just so I have the pipe, but maybe I want to do some data manipulation as well. So number one, I'm actually going to form a connection to what we call a board, where a board is a collection of pins or a collection of datasets that you're storing.

There's native connectors to things like AWS S3 or Azure AD, Azure Data Lake Service, as well as Google Cloud, Kaggle, and a few different other locations. So I can just call a board register RStudio Connect. It's going to automatically read in my R environment for the URL and my credentials. And then it's going to write out a specific dataset to this board. So I say, this board that I just connect to, take a dataset. In this case, we'll just take MT cars, a little toy dataset, and write it out to pins, write it out on this remote location.

So if I execute all this code, it's going to connect to Colorado, which is our demo server. It's going to take that dataset and writing it out to MT cars. Now, you might say, like, nothing exciting happened. I just saved the dataset out somewhere. But the benefit being, is that it's now hosted on RStudio Connect.

So if I go to content for our RStudio Connect demo server, let it load for a second, I can actually show you that dataset we just created. So now I have this dataset. And of course, I can download the dataset if I want to by clicking on it. It gives me a nice little preview. But more importantly, I have this code, meaning I can read it from anywhere. So I can read this from a scheduled document. I can read it from a different computer or from a different server. I can give this to someone else and they can use it. So I have this dataset that can be shared.

So this basic idea of just taking a data from my laptop or from an API or somewhere else, putting it up into somewhere where we can all use it, makes it more consistent and more, basically, readier for this automation task.

Packages for data access

So the packages we're talking about, I talked a little bit about pins. We're going to talk a little bit more about Arvest and the HTTR package for accessing web data. Arvest specifically allows us to access like websites or embedded data in websites and kind of download that or scrape it. And HTTR actually allows us to query APIs. So you might have like a RESTful API that you're sending some credentials to and accessing data from.

We also have dbplyr and sparklyr for accessing, writing to, or reading from SQL databases or Spark clusters or connections. And then pins I talked about briefly for uploading files, data, or arbitrary R objects. I wanted to talk about pins first because you can see that it was easy to get started in terms of I just loaded my library, made my connection, and then uploaded the dataset. But these other options are very valid as well and might be more useful in, say, a data engineering context where you're pulling data from other sources and then saving them out somewhere else.

So the pins package makes it easy to publish things like data, models, and other objects and make it easy to share across projects and with your colleagues. So what I showed you was using Connect as a board and writing a data frame out to it. So we first formed this connection. I'm using board RStudio Connect, but it could very easily, you know, be a different cloud provider or somewhere else besides RStudio Connect. And then I take that board and I write it out somewhere else. So here I write out our tidy sales data, save it as sales summary, and it's going to be a type of RDS.

So I can basically specify what is the file type. Maybe I want a CSV so it's operational between Python and R, or I want to upload a Parquet file so it's more efficiently stored, or what other file I want to write, I have the ability to specify that file type and upload it or write it out. And then the real benefit here is then in a downstream automated report, I can actually connect back to that board and read it in.

So you might have heard of people having friction when they're scheduling something or trying to automate a task, and they're not able to access the data. So having PINS available to access very specific files, update them in place, or overwrite them is a very powerful workflow. The other benefit is that while PINS is really used a lot as a way to share data across a team, you can upload any, basically any arbitrary R object. So that means you could save model objects. You could train a model, save it as an RDS, and save that as a PIN. You can then version the PIN, and you can access it downstream and look at the very different uses of that model, whether it's loading it into an API or loading it into a Shiny application and then using it there.

I mentioned this just a bit ago, but PINS are not limited to just RStudio Connect. While you can access data there, you are very often going to be pulling in data from, say, a cloud provider, local folders, or in some organizations, things like OneDrive or SharePoint, and you can kind of move between those different data sets and move data in and out according to the locations you want to go. One benefit specifically for RStudio Connect is that you are kind of keeping your data close to your data products, across those different things. So you would be using, say, a separate login for AWS versus Azure versus GCP. For RStudio Connect, you're using the same credentials to log in and access your data throughout.

And then it's not just limited to RDS files or R files. You can also upload things like JSON, CSVs, Arrow, or QS files, and anything else that can't be read and write natively, you could just upload and download those. So I really like using, say, Parquet files because they're used a lot in downstream kind of data engineering or data lake environments, and they're very efficient to store and very fast to read. So the efficiency there is beneficial in terms of a Parquet file might actually be like one-tenth the size of a CSV, so it's cheaper to upload, download, and still really, really fast to read in with the Arrow package.

Working with SQL databases and dbplyr

Rather than working with a bunch of flat files, you might be working with a SQL database, and you can write either native SQL code in R notebooks or in Python notebooks, or you can use something like dbplyr, which is a wrapper around dplyr, to actually translate your R code into SQL on the backend. So this is very powerful for using the exact same syntax for in-memory operations that your dplyr you know and love, but also writing out in SQL databases with the same dplyr code.

Of course, you can always escape out to, you know, writing native SQL chunks or SQL code if you wanted to do that, but dbplyr is very powerful for not having to do that mental model switching of moving from R to SQL back to R and vice versa, just sticking to dplyr.

As far as what dbplyr can look like, you can use dplyr again against SQL databases and using R or PySpark to work against Spark clusters in R and Python. So first, just like we showed kind of with pins, is you need to form a connection. Basically, you need to give it your credentials, who am I, and who am I going to connect to? So we'll have in this example, we're connecting to a MySQL database, and then I'm making a connection to a very specific table within that database, and with this API package and with this table command, I can then do my dplyr queries and it'll actually execute them in the database.

So I've saved this flights database as a table connection. I can look at the flights database, group by specific thing, summarize it, and get the results back. So this is essentially the same dplyr code you know and love, but it's being translated into Python code.

And then lastly, you might have like a data automation process where you're accessing web data. So maybe you're, you know, scraping web pages or you're sending curl queries out to a RESTful API with HTTR. Or there's also the GraphQL API package you could use, ghql from our OpenSci for those GraphQL API endpoints that you might run into.

The table or tbl function is from the dpi function or dpi package. So when you load this initially dplyr, you have dbi to form the connection and then you can form the table command around it and then just execute your dplyr code.

ETL and ELT automation

So we talked a lot about these different data sources. What this really is is like you're doing some type of ETL automation. So you're going to extract data from a directory database, from your customer data, marketing data, business data, machine data, API, whatever. You're just grabbing different data from these heterogeneous data sources. Then while the data is kind of streaming in, you're transforming it into a standardized useful format so it's ready for your downstream analysis or your downstream applications and systems. Then you're going to load the data or the L in ETL, load the data into the long-term store, most typically a database, warehouse, or data lake as a flat file like Parquet, for example.

So this can be done with R Markdown or Jupyter in terms of like you could have a SQL R or Python code executing as I showed before with those packages that's bringing in data from all over the place, transforming the data in memory, and then loading the data into your long-term store.

So actually writing out to say an alternative, you might actually do something like extract load transform or ELT. Here we're doing the extracting the raw data itself and then just loading it into a storage area, basically saving it as it is. We're not transforming it ahead of time. We're just saving the raw files or loading them in as they are. So downstream applications in this case or other processes are then going to transform or otherwise process as raw or semi-structured as opposed to being kind of ready for analysis by your data analyst.

Now, the benefit here is I mentioned Apache Arrow Project a little bit before. Having the ability to write out to files like Parquet is really useful here or to JSON, for example, as you're going to be getting potentially data from APIs that are stored as JSON or you want to have the ability to read files. R and Python both have rich support for doing this within the cloud environments. You might be extracting the data, loading it into a cloud environment and then transforming it later when you're ready.

So with R, you can go to AWS, Azure, or GCP with these specific packages. And then pins, again, has wrappers around all the different cloud providers so you can wrap around cloud APIs or command line interfaces or software developer kits. For example, things like the Boto3 package for Amazon Web Services S3. So both R and Python have this rich experience of writing, doing ETL and ELT work that you can schedule and automate.

The last part, and just kind of alluded to it, but Arrow also gives you ability to efficiently read and write files and transmit them around. So you can do compressed formats along with more basic formats like CSV and even more complex objects like Feather or IPC. Again, this is very useful working with data warehouses and data lakes in terms of giving R as well as Python the ability to manipulate and work with Parquet files, which are very commonly used in some of the enterprise-grade data lakes or data warehousing software.

Lastly, with the Arrow and writing out of data sets, you can also partition your data and transmit the data into different separations and individual structuring of the data files in, say, like a high format where it's separated by month or year or something else like that.

Data validation with pointblank

Now, the last part in terms of automating data, and we're going to be jumping into a live demo of all of this together here in just one second, is point blank for data validation. So as you're pulling data in, obviously you can have SQL code that the data is kind of as you expected, and this is what point blank and R is for, is not only bringing the data in and doing your extract and transformation, but validating the data as you're loading it. So when you're creating these reports, yes, the goal is to extract the data, load it into a database. But in the middle, you also have a report being generated saying,

so brief overview here, and then we'll jump into the demo in terms of what this all looks like. So if you're scheduling data set updates

or automated data pipelines, you're bringing in data from all over the place. SQL databases, potentially flat files or individual files like CSVs or web APIs, these get read into R or Python notebooks that are being executed on Connect and you're feeding those in in isolation of them.

Once the report is executed, then you have a filled out report. Basically, you can fill it with all this information about what was the data you brought in? What was the date? Was it successful? What did the data look like? And then you have that beneficial side effect or that intentional side effect of writing the data out to the outputs or the sink wherever you're putting it.

Data validation with Point Blank

So we talked about this. We've talked a lot about code and kind of the different packages. I want to show you a quick example of, say, like point blank and what this might actually look like. So point blank, again, lets me do data validation on either in memory or in database data. So as part of your ETL process, you might pull in some data.

As we're seeing here, this is like bird strike data from a different API that Katie Masiello put together. And it's got a couple different documents here. So it's got some reporting that's going on. It's doing some scanning and analysis of the data. And by using our markdown to do this and scheduling it on Connect, you have the ability to build out this robust analysis and this robust reporting around it in addition to actually saving the data out.

So we'll look at the daily report real quick here. So here for the API, so it's actually going and finding an API, querying it, saying here's the timeframe it's being queried at, throwing in our API key, which is being added here, and then getting some type of response. So looking at the response, exploring how long it took to run, and then doing a data validation here.

So because it's done in our markdown, you have this ability to have basically green and red indicators for was it successful or did it fail? So you've used Point Blank to create this agent response. It's going to look over all the different parts of the table, interrogate and say, did it pass all the different validations? And are validations passed? If they failed, it would give us a warning about where they failed.

Now as far as does the data make sense, we might have more robust things saying like was the data between a range or was it greater than or equal to? So Point Blank also gives you the ability to query specific columns. So here I've got, we're querying to say, should these values be between 0 and 500? We should never have negative values here, and 500 is the theoretical limit that we've ever seen.

So these should all pass, and we have no evaluation issues because it's saying yes, it's between these ranges. Or maybe it should be greater than or equal to in terms of, it should be above a value rather than between a value.

For any rows that failed or any observations that were concerning, we can actually download a CSE file and look at those different rows that were wrong. So I can actually open up the CSV file and look at it, and it will show me the data points where those were failed. There's only 45 failure rows, so not too worrisome out of the many rows that we have here, but you have that escape hatch if you wanted to use it.

So all to say that like when you're doing this reporting or you're doing this automation, not only can you bring the data in, you can write it out, but you can also have these robust reports in the middle, basically doing the validation and doing the reporting around it that are scheduled out. So you could have, you know, not just the effect, but all of the in-between reporting, and lastly, having like a nice graphic at the end in addition to the tables and the warnings and the data validation steps that you're doing.

End-to-end data science pipelines

Now, this example may not, you know, obviously like you may not be doing as much of this. Maybe you're doing more machine learning as opposed to data extraction and ETL work. So a more complex kind of data science focused task, as opposed to maybe data engineering or data preparation, is this end-to-end example from my colleague, Alex Gold.

So here, we're actually doing something similar. We're bringing data in from an API, scheduling it to be cleaned, updated, and saved out to both a database and to a PIN, and then loading that into an API downstream that's running in production. And all of this is being executed inside of RStudio Connect.

So this is where we're going to spend most of our time remaining, is kind of diving into a little bit of this example and how you can kind of schedule these out and what they look like running on RStudio Connect. So let's close out a few of these examples. Because we're done with those.

Now, for this bike share, you know, there's a lot going on. So this is the overview of what it looks like. So again, bringing data in, data import, cleaning, and training some models, saving it out to a database, and then saving it out as an API and a Shiny application that are being fed the downstream data.

Because it's on Connect, I have this kind of landing page I've created or that we've created with Connect widgets, basically give me the ability to write about this project. You can imagine that with a data science project of this scale, you have a bunch of different scheduled jobs.

Not only do you have the ETL tasks, things we've covered a little bit before, but you also have modeling. So maybe you're scoring the data and doing batch scoring and writing out the values to a database or training the model itself, or you want to send out a specific email if a criteria is met or not met. And then lastly, you have the actual data sets that you're creating.

So the pins I've saved out, so these are the static files that are being uploaded with the bike model as a XGBoost saved out, the scoring and the error, and then information just about the metadata about the bike station and environment. What this is basically doing is grabbing a bunch of metadata around these bikes and bike shares in Seattle and saying, okay, what bikes are actually going to be available in the future? That's why we call it the bike share example.

Going back down here to our scheduled jobs, let's just show real quick the ETL step and then we're going to move on a little bit. So here, again, you have a bit more bare bones report. It's basically just saying, hey, you know, here's the analysis. It ran about a week ago and was executed on a schedule.

It's using the ODBC package to form a connection to a database. We form a connection with the database and then clean up the data and write it back out. So we're grabbing data from both the database and a pin and an API, and then we're writing out the data with dbplyr. So our query that we're actually running is a group by, summarize, interjoin, and then writing out the raw SQL code and showing that query.

So I don't know about you, but writing out this SQL code by hand as opposed to writing in dplyr, I'd much rather write it in dplyr and get the rendered SQL code through dbplyr here. It's a bit more easy to stay in the mental model of R as opposed to switching between R and SQL.

And then lastly, we have kind of the data saved out, and we're actually writing it to the database. You can see there's about 10 million rows that have been changed here and updated, so a lot of data that we're working with.

Scheduling documents on RStudio Connect

Now, as far as scheduling those, I did briefly show scheduling those documents before, but let's just show you what scheduling it would look like just so we know how to do that again.

So I'm going back into Connect, and we're going to show the bike share and show the bike scoring this time. So here, I'm actually looking at the document running on RStudio Connect, and it has been scheduled to execute on a scheduling. So here, we're actually scoring the data and moving on to that step as opposed to doing ETL.

Now, again, this is very easy. I just publish the code and the actual R Markdown document, and I can go in here and schedule it to occur at a very specific time, 3 a.m. every morning so that it's not conflicting with any other interactive work that I'm doing today. So this would be a specific start time running at the daily schedule, although I could change it to be weekly down to the minute or even more frequently, and it's going to publish the output here again. So I can look at the history and look at previous versions of this report that ran, say, back in March as opposed to more recently.

Automated reporting for data scientists and business users

So this idea in terms of like we've shown the bike share, we've shown the ETL work that's going on there, and we hinted at that there's these, you know, more robust kind of data science pipelines or model training. What we're showing here is really automated reporting in terms of we're moving beyond just the ETL work and saying, okay, now I'm actually doing model training or reporting on the model or sending out reports to my data science colleagues or my business users.

So for this automated reporting, you can, again, schedule our Markdown in Jupyter, which can be used as a report. And these reports have many different purposes. They might be intended just for the data scientist or their team, meaning that they're going to show all the code. The metadata there is so that the data scientists and the data analysts working with the data know what's going on.

Alternatively, there might be reports for business users or executives, people who just want to consume the final output or they want to get some type of like insight or they want to get a report delivered to them as opposed to understanding what the code is doing. They're more interested in what the actual output is. And obviously, there's going to be a spectrum between these where you might have reports that are useful for both the data scientists and the business users or vice versa.

All are valid in terms of like you're thinking about here what is shown in the report, what is the ultimate purpose of the report, and the persona it's being created for. So our reports for data scientists would be fairly bare bones, you know, in terms of like I'm showing this report, it's not the most beautiful thing, but it's telling me, hey, this report runs after the data has been cleaned, it's running an XGBoost model, and then it's writing out and loading the model and scoring it. Great. It's doing all the different steps that are expected and the data scientist is happy with that.

The report itself, in this case, is more of a side effect. The primary purpose is like automating the model training and model scoring. But again, that report is a useful side effect. You have the report that's beautiful or that you can build different components into to visually check or to report on the data science pipeline that you've created, as well as, you know, writing out the output or writing out the model or the data somewhere else.

So again, this could be ETL or ELT work, data cleaning, data preparation, or the example we just showed is more of like model training, batch scoring, maybe data imputation or validation that we showed with point blank. All those things are really intended for data scientists or data analysts. While the business users might be interested in knowing that the data is valid and has been validated, they're probably more likely in receiving a clean, pretty report at the end of the day.

So as far as the examples from the bike share, I showed the ETL cleaning process and the ETL into a database. Again, using those packages we talked about with the tidyverse and hater and dbplyr and pins to basically pull all these different data sources together, execute some models together, and then save it out somewhere.

As far as the model training and batch scoring, we're doing that with like an XGBoost model, scheduling the training to execute, batch scoring it out, and then writing the scores out to a database as well as saving out the trained model. So all of that, again, is occurring inside Connect and can be done with R or Python. I could replace R Markdown with say, Jupyter and do a very similar process within Connect as well.

Styling reports for business users

Now, closing out, we'll talk a little bit more about business users and show an example of how we can make those reports a little bit prettier and kind of how to meet them where they want to be. So for business users, these reports are going out to potentially like non-technical colleagues who are decision makers or business analysts or executives. And here, you're more worried about packaging up the data or your story, your visualization, your tables, but you're not as worried about showing the code, although some people may want to see the code.

So how do you make this report that fits your company style or one that's attractive and useful for them? We showed those reports earlier. They were pretty bare bones and not that many people would be excited about receiving just a black and white report with a lot of code being printed out. Like, they might be confused about some of that.

So we can do, and the examples I'll show here in one second, are using different themes or using different customization of the appearance. So our Markdown theme, say, from the Bootstrap lib or BS lib package. This allows you to customize the entire appearance of our Markdown as well as Shiny applications or other kind of downstream effects. So custom themes, you know, customizing small components as well as bringing in some of the boot swatch themes, which are entire themes that have been created by the community.

So let's show a quick example of this. I'll hop back into our studio. And let's first run a very basic report. Again, all of this code is available on the GitHub repository that's linked in the slides. But let's take a no-theme report and just knit that out.

Very simple report. We're not going to blow anyone's minds with this report, but just showing what it looks like. It's a basic report. It shows the code, it's got a graphic, and it's got an interactive graphic and a table. Pretty bare bones in terms of it's black and white and not really themed at all. And that's fine. This might be useful for the data science team, but the business users might expect more. So what we can do with them is using Bootstrap Lib, we can actually create a custom theme.

So I can make one more aligned with, say, like the RStudio brand and use these background colors of like a light gray or doing a RStudio blue color, and I can render that out.

So here I'm taking our Markdown document and just changing the HTML document output and applying Bootstrap Lib theme to it. So if I knit this document, this RMarkdown doc, it's going to look a little bit different. Still fairly bare boned, but you can see that it's got this kind of gray background, the fonts have been changed, but the graphics haven't been changed in terms of we still have this white background on the graphic.

So we're missing a component here of translating the RMarkdown theming into the graphics themselves. The table looks okay. It's, you know, gray and blue, but we do have this nice kind of like RStudio blue at the bottom. So there's one more step we can do here.

We've themed essentially the RMarkdown document, but we need to translate that theming into, you know, the plot, into Plotly or the table or ggplot. So there's this chunk here with thematic or the thematic package, which allows me to pass all of this theming that I've done to the RMarkdown automatically into ggplot and Plotly graphics. So I've made this chunk eval to true. I can knit this, and we'll take a look at it again now.

So on this report, we can see that the graphic completely blends into the document in terms of there's not a white background that's making it contrast. We've actually applied the theming of the overall document into the graphic. It's applied this alternative color scale like a blue-green-yellow that's a bit more visually appealing than the default graphics or default colors that ggplot loses. And this works for both interactive graphics and Plotly and ggplots that we see here.

The table itself, you can see the fonts have changed. So we have this Spira sans code that I really like, and then it's changed the colors of the overall kind of navigation of this table.

So while these might seem like small changes that you're doing, they're actually very powerful for connecting with your business users and helping them actually want to see the reports or kind of working with them and the theming that you're working on.

So I showed what this looked like interactively, but as far as Bootstrap Lib and the things you can change, you can look at the Bootstrap theme function from BS Lib, and this shows you you can change the background and secondary colors as well as fonts for the heading, base, code, and scaling of those fonts. The YAML header I used in that R Markdown document looked like this. So HTML document with a specific theme, and I changed the background, foreground, and a few other different components, and it can even bring in Google fonts for you. So you have access to hundreds of different fonts that you can embed inside your document.

That second component that we talked about was thematic, which was actually passing this theme into your plots, and that's what thematic is providing with thematic R Markdown.

Using RMarkdown Distill for beautiful defaults

Now, you may not even want to worry about doing a lot of customization. You could use something like R Markdown Distill. So this is like scientific and technical writing native to the web. It's got amazing defaults, and it looks really, really nice. So this gives you the ability to very quickly take the exact same code you're running, but make it look better.

So let's go back into the R Markdown document, in this case, a Distill doc that we're working on, and we'll knit that out. Exact same code as before. You know, we're not doing anything crazy. We're just, you know, showing that other part. And now we have a nice-looking report with very good defaults. So this instantaneously, even though it's black and white, to me, it's much more visually appealing.

So let's go back into our R Markdown document, and we'll take that R Markdown that we're working on, and it's much more visually appealing. I've got kind of this floating table of contents where I can navigate back and forth between the different components. I've got nice fonts that it defaults to, and it's kind of making this smaller center portion and assigning things to these columns where it's a bit nicer to work with.

So with essentially no work on my part, just by changing one chunk of the YAML header, I can make a nicer-looking report, or if I wanted to go very deep, I could go full customization with Bootstrap Lib and make these reports look even better.

Sending emails with Blastula and RStudio Connect

Now the last portion we have in the last few minutes that we have for today is talking about emails and sending out emails from RStudio Connect with the Blastula package. So Blastula makes it easy to produce and send HTML emails from R. So this is very powerful in being able to combine things like ggplots or tables or data and embed them into emails that you're sending to your colleagues.

So emails can obviously be sent through SMTP, so you can do something like send them via Gmail, but more commonly we see a lot of customers use RStudio Connect to send these emails through the built-in mail server. So this uses your actually existing company email and uses that type of authentication as opposed to being limited to Gmail or something external to your company.

It's really easy to kind of render these emails. Basically, you take Render Connect email with a specific body of the document and then attach that email to the body of the document in terms of render it and attach it. So it's not only going to show it inline in terms of your actual email will be shown when they open their inbox, but you can attach different things to it, whether that's a copy of the HTML or a PDF report or an Excel file, whatever you want to attach to that.

Now, we'll jump back over and talk about the Connect Live examples and what that looks like. When I say built-in mail server, what I really mean here is that you can install a mail server and send it through Connect so you don't have to rely on an external thing like Gmail. You can have Connect talk to a mail server protocol through your company.

So let's hop back over to here. What these emails actually look like is what I'll show briefly.

And let's do the main document. Now, these emails could go out to one person or they could go out to an entire host of people. So part of what Connect is doing is letting you bring in things like your authentication groups so you can send to your entire executive team or the entire sales team or the entire data science team and send it out to all of them.

So here we have an R Markdown report. This R Markdown report is not special in any sense. We can knit it and take a look at the repository. The main thing we're looking at here is this is what the actual R Markdown document looks like. So you could schedule this to be executed and it would generate this, which is intended for the data scientist. It's actually like, here's what we were doing, here's talking about the report and showing some of the data.

Here at the bottom, and I'll zoom in a little bit, it has some information saying we can actually attach the email. That was the render Connect email and attach the email that we were doing. It kind of jumped the gun there, but it actually sent it out and this is what the email would look like inside your actual email client.

So this would actually be the body of the email. You not only have text and headers, but you can embed ggplot graphics, you can print data, or even embed specific tables. So you could use like the GT package to theme a table and attach it there.

And then it gives you this nice footer saying, hey, this content was generated, is everything in line? And then you can actually go to the live examples and click back into the report that's actually out there.

So that was the preview. Now in terms of if I were inside Connect, like let's actually go to an email one.

So I'm going to go back into this and talk a little bit about the batch sending of emails because I know that was a question that came up. So let's do content I own.

And let's say I had this weather collector and I wanted to send an email to somebody. Because Connect has a connection to an email server that's been installed alongside Connect, I could email the report and say, just send me a copy of this report. I can basically say do it now. But more often what I would do is like schedule the document and have it send an email out to people. So every time it's run, it sends the email out.

As far as who it's being sent to, that is up to you. I'm the only person looking at this document right now, but maybe I want to add my colleague Kelly or I want to add the entire solutions team at RStudio to receive this. So now all these people being brought in from our authentication protocol will be receiving the email if I want them to.

So I can go back to schedule. I can say send email after update. And I can send it to all the viewers and all the collaborators. Basically send it to everybody who is interested and have all this batch email being sent out.

Conditional email sending

Now that might be useful in terms of sending these emails, but what if that email gets too noisy running every single day? Maybe I don't want to see it every day. And that's also where Connect can help you or Blastula can help you.

So this idea of conditional execution is basically, yes, every time a doc is rendered, it can be useful to send out an email. But sometimes you want that higher signal to noise ratio basically saying only send an email if a criteria is met or not. So maybe data quality is below expectation, or your model is predicting a value outside a specific range, or your data set is too small. It should be thousands of rows, and it's 20 rows. Basically whatever logical criteria you want to define, you can build that logic into your R Markdown doc, schedule it, and then send out emails conditionally based on that.

But sometimes you want that higher signal to noise ratio basically saying only send an email if a criteria is met or not.

So just a few examples of code. You know, here's an example basically saying if predictions are outside of a range or greater than a predicted value, then send me this email basically saying, oh no, the model is drifting or the model values are too high. Generate this other report and email it to me. Otherwise, if values are below that, it just doesn't send the email. It still executes the code. It still renders your report, but it doesn't warn anyone because nothing is there to be warned about. The report will just be, you know, kicking along and scheduled as it rolls.

Another option is rather than writing that kind of if else with like the suppress of the scheduled email, you could have something more along this idea where you have the option of sending one report or a basic report. So here we have a value just in like pseudocode saying if the value is above some type of norm, then give me a true or a false. And then we have an if else statement saying if that value is high, then send the high alert email. Basically, render this specific email, send it out to me, and give me all this extra metadata about the report. Otherwise, just send the basic email saying, yeah, everything went along fine. Here's the basic report for the day. And then it will actually send out that email and attach the email and do everything else. Basically, send along the email file that you want to bring along.

So not only can you look at those different files, but just showing you what a R Markdown email looks like. So let's look at this. The actual code itself just looks like any other R Markdown document. And really, that's the beauty of Blastula is the only thing you're really changing up here is Blastula, Blastula email as the output format. So it's going to say, you know, load the tidyverse, bring in some data from the parent document that's rendering it, and then let's create a table at the end.

There's really not much else to learn. You're just using all the other skills you've learned with R and with R Markdown to write the document. And Blastula is handling the conversion of this R Markdown into the actual, you know, email that you've just created. So now I've opened up the actual rendered R Markdown as an email.

All right. So that was a deep, rapid dive through a lot of different things. There's a lot of different powerful things you can do with Connect and scheduling. The one part I want to touch on before we leave is I've shown a lot of R Markdown, but you can do very similar things with, if I get over to it, with Jupyter.

So I do, don't want to leave those people out in the cold in terms of you can also do similar things natively in Python. So here we have some stock examples where we're using Matplotlib and Pandas to go over to these documents, pull them in, and generate, you know, some graphics and save some data out. So this native Python code in Jupyter Notebook can also be scheduled, and it will run every weekday and then publish the output. We have the same ability to look at older versions and do your access controls and scheduling of those documents.

A second one might be, again, doing model training. So rather than just relying on Pandas, you could also bring in XGBoost or sklearn, scikit-learn, to actually train a model in Python natively and do training and testing split, training the model and saving it out. So this can also be done and run on our Studio Connect as a Jupyter Notebook with the same scheduling that you're worried about.

Wrapping up and resources

All right. So we're right here at the end. I've linked to all the different content here. Again, the slides are available, and I'll put those into chat. So that is the link to colorado.rstudio.com and the slides, which have all the sublinks and link out to the code.

There's information about Blastul emails and in custom emails, pins, making beautiful reports with Distil, Thematic, and BS Lib. You can go a bit deeper into the bike share example and that kind of piecemeal, deep example of an end-to-end data science solution. Some other different examples about data formats.

And then here on the right, we have Connect scheduling and email and updating, and then two other webinars that might be interesting to you. These already occurred, but there's a Beyond Dashboard Fatigue webinar with additional content. Again, open source code where you can adapt it and kind of use it in your own organization. And then this Rethinking Reporting with Automation is another great webinar from an outsider

to our RStudio where they actually talked about really focusing on that kind of business stakeholder. I talked a little bit today about business stakeholders and data science colleagues, but this Rethinking Reporting does a really good job of diving deep into making beautiful reports that are very useful.

If you just want to learn more about Connect, you can click on this link which takes you to the Connect landing page and talks a little bit about what RStudio Connect can do for your organization.

With that, thank you so much for hanging around. That was a full hour, so always good to hang out for that time. Thank you for kind of being part of this experiment of doing some more live content webinars. And hopefully you're able to join us from very different locations. Maybe join from YouTube or LinkedIn or Twitter or somewhere else. Very excited to kind of be meeting our different community members where they want to be meeting. So thanks for your time today.

I'll look and see if there's any other questions real quick. And it looks like my colleague Kelly, I just want to say thank you so much for answering a lot of these questions. Looks like we're in good shape there. And thank you so much for your time and for hanging out with me for an hour.

The slides, again, I will share those. They will be, I can actually share the GitHub repository that has links to the slide, the YouTube video, as well as the slides themselves. So this has all the source code. It'll link to the YouTube video, which is the primary long-term storage, the slides that I demoed today. And then you can see all this demo code that you can adapt, modify, or play around with and look at it a while. So you can download from here. You could clone it and edit it released under a CC BY 2.0 license.

Oh, thanks so much, y'all. It looks like Ahmad gave me some help. So again, because I'm hosting this on Connect, I'm going to flip this over to anyone can access it, no login required. So go ahead and take a look at it. Thank you, Ahmad, for the shout out there. So it should be public and accessible for everyone now.

Have a wonderful day. Thanks for joining us on another data science livestream. And we will see you next time. So I'll say goodbye from here. Thanks again. And if you have other suggestions, feel free to reach out to me on Twitter at Thomas underscore mock or on LinkedIn. I'd love to hear from y'all in the community. What is the next thing you want to see in terms of like what type of content would be useful for you or would be helpful for your learning and your education? So thanks again. Have a great day. Stay safe and happy holidays.