Resources
Oct 30, 2021 |
1h 10min
|
4.6k views
|

Leveraging R & Python in Tableau with RStudio Connect | James Blair | RStudio

Leveraging R & Python in Tableau with RStudio Connect
Overview Demo / Q&A with James Blair

Tableau combines the ease of drag-and-drop visual analytics with an open, extensible platform. RStudio develops free and open tools for data science, including the world’s most popular IDE for R. RStudio also develops an enterprise-ready, modular data science platform to help data science teams using R and Python scale and share their work.

Now, with new functionality in RStudio Connect, users can have the best of both worlds. Tableau users can call R and Python APIs from Tableau calculated fields, getting access to all the power and analytic depth of these open-source data science ecosystems in real-time.

For Tableau users, this makes it easy to add dynamic, advanced analytic features from R and Python to a Tableau dashboard, such as scoring predictive models on Tableau data. They can leverage all the great work done by their organization’s data science team and even call both R and Python APIs from a single dashboard.

Data science teams can continue to use the code-first development and deployment tools from RStudio that they know and love. Using these tools, they can build and share R APIs (using the plumber package) and Python APIs (using the FastAPI framework).

Speaker Bio:
James is a Solutions Engineer at RStudio, where he focuses on helping RStudio commercial customers successfully manage RStudio products. He is passionate about connecting R to other toolchains through tools like ODBC and APIs. He has a background in statistics and data science and finds any excuse he can to write R code.

A few other helpful links:
Tableau Integration Documentation: https://docs.rstudio.com/rsc/integration/tableau/

Tableau / RStudio Connect Blog Post: https://blog.rstudio.com/2021/10/12/rstudio-connect-2021-09-0-tableau-analytics-extensions/

Embedding Shiny Apps in Tableau using shinytableau blog: https://blog.rstudio.com/2021/10/21/embedding-shiny-apps-in-tableau-dashboards-using-shinytableau/

James' slides: https://github.com/blairj09-talks/rstudio-tableau-webinar

image: thumbnail.jpg

Transcript#

This transcript was generated automatically and may contain errors.

Thanks, I'm really happy to be here and excited to share the work that we've been, we spent a lot of time on what we're going to be sharing today. So we're excited to have a chance to talk to you all about it and demonstrate some of what we've been doing. But like Rachel indicated, the idea here is going to be leveraging R and Python in Tableau using RStudio Connect.

So this will be kind of, you know, the contents of today will heavily feature RStudio Connect, which is a paid product. But there's also some open source offerings that exist based on the work that we've done. And we'll highlight those as well. And to kind of start off with, I want to, I want to say that I think, you know, when we look at RStudio, and when we look at the R language itself, and Python as well, I think a lot of, or at least one common theme that I see is this idea of being able to use the right tool for the job, right?

There's an ever increasing number of R packages and ever increasing number of Python packages that exist to make data analysis work more efficient, more straightforward, and things of that nature. And at RStudio, we've really embraced that. And we've extended that idea of using the right tool for the job to our professional tooling. So tools like RStudio Workbench offer additional environments beyond just RStudio for doing your work in that includes tools like VS Code and Jupyter Notebooks and JupyterLab. The reticulate package was created in an effort to make it again, allow flexibility in determining what what tools the best tool for the job that you do you happen to be working on.

Just to kind of highlight this, here's an example of some of the different, you know, packages and frameworks that we see in use today. On the R and Python side, and all of these that we have listed here on the screen are packages that have support inside of RStudio Connect as a publishing platform. So data scientists and analysts and developers today can build tooling in Shiny or Plumber or Flask or FastAPI or Dash or Streamlit or TensorFlow or any of these different numbers of content types, share these types of content on RStudio Connect and allow business users to browse, investigate, explore, make use of whatever it is that's been developed and shared in that environment.

Now, we're going to continue this kind of theme of extending our tooling and our capabilities by bringing Tableau into this mixture as well, right? Again, following this theme of what what tool is the right tool for the job. And there are many use cases where Tableau makes a lot of sense when we're looking at creating dynamic data visualizations that can be easily shared throughout our organization and easily accessed from business users all the way up to high-level executives. This is something that we've heard a lot from our customers, this request for better Tableau integration or just questions that we receive are, you know, how do we use Tableau alongside RStudio? How does RStudio position itself alongside Tableau?

Questions like that. And this is something that we've, like I said, spent a lot of time thinking about and are excited to share the work that we've done here today. Now, here's kind of where we're going, right? Here's an example. We'll look at an example of this same dashboard again, kind of towards the end of our conversation today. But in this particular example here that we're looking at, we have this customer analysis that's looking at customer sales performance and comparing profit and sales and determining which customers are performing well and which customers maybe need a little bit of help.

But the really interesting thing about this particular setup here is the fact that there are these two different views here, these two scatter plots that are based on extensions that are built in R and Python and then hosted inside of RStudio Connect. So, on one side, we have an R extension that's doing some outlier detection. So, you'll notice that these points here are colored a little bit differently. The blue points are non-outliers, the orange points are outliers. And that outlier detection is happening in real time based on an extension built in R that's being hosted on RStudio Connect. And we're going to showcase what that process looks like as we go through our conversation today.

The other component here is this other extension that's built in Python. This extension is built on top of a machine learning model that takes some set of inputs about a given customer and their transactional history and then predicts their expected profit as an output. And so, on this left-hand scatter plot, what we're looking at is actual profit for a given transaction versus predicted profit. And that can allow us to do things like identify customers that are over or underperforming. And again, all of this is being driven in real time by extensions executing R and Python code, all hosted on RStudio Connect. And we're going to look at how we get there as we go through the conversation today.

And again, all of this is being driven in real time by extensions executing R and Python code, all hosted on RStudio Connect.

Background: existing Tableau–R/Python integrations

So, before we go there, let's step back to the beginning a little bit and kind of look at what has led us to the point where we are today and the work that we've done to make these types of extensions possible. And so, to really kind of start at the beginning, there are two existing mechanisms for integrating Python and R code into a Tableau workbook. Tabpy is an open source Tableau package, or excuse me, open source Python package that is built and maintained and developed by Tableau themselves. And this basically exists to allow you to execute arbitrary Python code from a Tableau workbook, view the results response back into your Tableau environment. RServe is a very similar type of functionality where you open up an RServe server in some environment, and then you submit arbitrary code for execution into that environment.

And here's an example pull from Tableau's documentation on how these types of extensions might be invoked. So, we have this script function here, and then we, in this case, this is an extension using Tabpy. So, we define our entire Python process or entire Python script here in this string. We submit that to our interpreter that ingests the string, interprets and executes the code, returns the result, and then we pass in individual values or arrays of values from our Tableau workbook.

Now, there's a few challenges with this approach. I'm not going to enumerate everything, but one of them that immediately becomes apparent is the fact that if I'm a Tableau user, like, I have to know at least enough R or Python to be able to write my code into this particular extension or into this script function. So, that makes it a little bit challenging because I can't, like, I have to have some knowledge of these scripting languages to really use this, or I have to know somebody that has enough knowledge to give me some sort of copy and pasteable piece of code that I can then execute here. And then beyond that, the development process of this code can be a little bit difficult, right? There's not really a good way for me to validate this code. There's not really a good way for me to debug this code if there are errors. I'm not really working inside of a development environment. Rather, I'm working inside of Tableau and passing in some sort of plain text string that is then going to be executed as code on the back end. A couple of other challenges that we've heard from customers about security with this type of model can be a little bit difficult to maintain the appropriate levels of enterprise security, particularly when it comes to RSERVE. And so, again, we've heard from customers that these types of integrations, being able to execute R code, being able to execute Python code in a secure, manageable way from Tableau Workbooks is something that we want.

The Tableau Analytics Extensions API

Now, moving forward, right? Again, we've started back at the beginning. Tableau released this Tableau Analytics Extensions API, I think it's now been a couple of years ago. And this is really kind of the building block for all the work that we're going to highlight today. This Analytics Extensions API is actually the kind of the foundational piece of the TabPy integration that they've done. And so, this same framework is what we've used to provide access to the extensions that we're going to highlight today.

I'm not going to go into too much detail here. The goal here today is not to be a comprehensive deep dive into all the nitty gritty components of the Tableau Analytics Extensions API, but rather to give enough information to understand kind of what's going on and why we've decided to go in the direction that we've gone with the extensions that we're providing access to. The main thing I think to understand here is that this Extensions API is really just a contract that exists between Tableau and some external service, right? It doesn't matter what that service is. It doesn't matter what it's implemented in. It doesn't matter what language is it. It doesn't matter what it's doing, right? As long as it obeys the contract, it works.

And the contract is that Tableau will make two different requests to two different endpoints. It's going to make a request to an info endpoint to understand information about the service that's running. And it's going to make a request to the evaluate endpoint that is expected to return some modified version of the input data, right? So, this evaluate request is the request that then returns back or that submits data from Tableau to the service. The service is expected to respond in a way that Tableau can understand and reinsert that data back into the workbook of the Tableau environment. There's a lot more detail and documentation on the Tableau Analytics Extensions GitHub site that walks through all the different components here. But I think it's useful to understand just for the context of our conversation here today, that this external service, again, can be whatever it needs to be, as long as it follows the guidelines set forth in this Tableau Analytics Extension.

Introducing the plumber.tableau package

So, what do we do when we have, you know, some sort of repeatable behavior that we want to provide access to end users? You know, again, if we want to create an easy way for users to build extensions that obey this contract, what's the best way for us to do that? And the natural response as an R user and R developer is to put it in a package, right? Like, let's see if we can make some sort of a package or create some sort of reproducible behavior that will make it easy for people to build tools that meet this contract that Tableau is expecting us to meet. And so, that's exactly what we've done. We have a couple of different packages that we're going to highlight and introduce today. The first is a package called Plumber Tableau. This is an R package that is built on top of the Tableau, or excuse me, built on top of the Plumber package. So, if you've ever used Plumber before, Plumber is an R package for building interactive web APIs using R. And this Plumber Tableau package simply extends Plumber a little bit by adding a few additional tools that will allow us to easily make extensions for Tableau.

I'm going to go ahead and switch over into a development environment now and kind of showcase what this looks like and how we might go about building this up and highlight some of the features of what this package offers and how it works.

Okay. So, here's, again, this is my development environment. We're inside of RStudio Workbench here. If I wanted to create a new, we're going to just use a really simple use case here. Let's say that I want to create some sort of a function that takes some input, some string input, and capitalizes it. Okay. So, if I have, we'll call this CAPS. This is a function, and we want some string.

And then, in R, this is a fairly straightforward task to do. We could just say we're going to call this toUpperStringInput, right? And really, we're just wrapping this toUpper function here. And if we run this, we end up with this CAPS function. If we say CAPS World, we end up with the capitalized version of that string, right? Again, very basic, very straightforward. We're not trying to do anything terribly complex here because the focus is less on what we're doing and more on how we turn this into an extension. So, let's say now that we want to take this function that we just created and turn this into a Tableau extension that we can use from inside of a Tableau workbook.

We can come in here, and we can say, let's first bring in a couple of packages. So, we need the Plumber package. We also need the Plumber Tableau package in here, okay? And then, Plumber is run. If you haven't seen Plumber before, some of this might feel a little bit unfamiliar, but it hopefully should seem fairly straightforward. Plumber works off of special comments that identify different attributes of the API. So, here we can say the API title is example Tableau extension.

All right, okay. And we could say that the API description is simple virtualization extension. Okay, there we have that. And then, for this particular endpoint, we need to define a couple of things. One is, let's give it a brief description. So, capitalize input. And then, we need to define how this particular function will be invoked. So, we're going to respond to post requests at capitalize. So, this here, again, this is all, we haven't done anything unique to Tableau yet. This is all just standard Plumber definitions. But here, we're identifying that this is going to respond to post requests at this capitalize path.

And then, there are a few new kind of special comments that we've introduced with the Plumber Tableau package. The first is Tableau r. And this allows us to define arguments that we expect Tableau to submit. And in this case, that Tableau argument is this string input value that we have here. So, here we can say, we're going to call this string input. We expect this to be a character value. And then, these are the strings. And then, we also have Tableau return. We're going to return a character value. And this character value is a capitalize. Let's call this capitalize. Okay. So, here's the, so far, all we've done is created this endpoint here that allows us

to submit data and return and receive back the capitalized input. And if we run this right now, let's save it. So, let's save this as Plumber r. Okay. So, if we run this API inside of RStudio, we end up with this nice little, you know, user interface over here on the right-hand side. We can see our little capitalized endpoint here. We can come in and we can try it out. We can say the string input is hello world. We can go ahead and execute this. And we should see this output that is the capitalized version of that input. Right? Everything's good.

Now, there's one critical piece that we're missing in order for this to be a valid extension for Tableau. And that is, we need to do one more thing here. And this is highlighted in all the documentation for the package. But we can use this little plumber annotation. And then we add Tableau extension.

Okay. So, this Tableau extension function modifies our existing API so that it meets the contract that Tableau has established. Right? We need to be able to respond to requests in a certain way. And we need to be able to accept certain types of requests. This function here is what allows this package to handle all that for you. The goal of this package has been to make it as easy as possible for our developers to build these Tableau extensions without needing to worry about how Tableau is actually formatting its requests and how it's actually expecting response. All we need to do is write our code and then let the package take care of the rest. And if we run this API again, we'll see that things have changed a little bit in our user interface. We have some links that have appeared up here that give us access to additional pieces of documentation. And if we open up our endpoint here, it's actually changed quite a bit because now we are using an example request that matches the type of request that Tableau would send.

So, Tableau sends requests that are JSON objects that contain two different fields or two different values, script and data. And here we formatted an example request that will mimic what Tableau actually said. So, in here, if we wanted to test this out again, we could come in and add hello world into our data values. We could try this out. Let me find my execute button. There we go. Try this out. And again, we see back this hello world type response.

Building a Tableau extension in R

Okay. Now, we have done everything we need to to build a Tableau extension in R. And I want to highlight a couple of things. One, it was really a matter of writing an R function, adding a few comments, and loading the right libraries. The only really critical piece here that might be a little bit unfamiliar is the fact that we have this requirement that we use this Tableau extension modifier here. That just is, again, this is an instruction to plumber to say, look, take the existing API that we've already defined and modify it using this function. And this function comes from this plumber Tableau package. But once we've done that, we have a running, capable, fully functioning Tableau extension. The only challenge now is that this extension isn't in a place that Tableau can reach out to, right? This extension is still living in my development environment. I need somewhere where this extension can be hosted. And that's where RStudio Connect comes in.

The only challenge now is that this extension isn't in a place that Tableau can reach out to, right? This extension is still living in my development environment. I need somewhere where this extension can be hosted. And that's where RStudio Connect comes in.

So, if I take this extension that I've just built, we can come up here and we can say, let's publish this to RStudio Connect. We're going to publish our plumber API. I'm going to call this example. Let's call it example. All right. We'll go ahead and publish.

This is going to deploy this API to RStudio Connect. Because this is really just a plumber API at its core, there's not anything different or any sort of unique adjustment that I need to make in order for this to publish. We can see that it published relatively quickly. And if I come open up RStudio Connect and refresh here, we should see this example Tableau extension listed here at the top. Now, if we open this here, we'll notice immediately that things are a little bit different. One, we're not seeing if you've ever published a plumber API to RStudio Connect before, you're likely familiar with the fact that in most cases, what you see here is the same sort of interface that we saw inside of RStudio, this Swagger interface that we see over here. When we publish Tableau extensions to RStudio Connect, instead of that Swagger interface, what we get is we get this Tableau user guide. And the intention here is that this is now documentation for the Tableau developer so that they know how to use the extension that we just published. All of this information here is information that, again, is specific to the Tableau developer to say, here's how you might use this particular extension. In fact, this extension here can be used by copying and pasting this line into a calculated field inside of a Tableau workbook.

Using the extension in Tableau

In fact, let's try this. Let's come into Tableau. We're going to create a new workbook in here. This will take just a minute. So this is Tableau Online. I'm going to create this new workbook. This extension we created, remember, just takes some input text and capitalizes it. So we're not doing anything really exciting or crazy. It doesn't really matter the data source that we select to connect to. So I'm just going to connect to this superstore data set that Tableau comes with automatically. Let's save our workbook so that we can... I'll just call this example extension Tableau. Go ahead and hit save. Then the other thing that we need to do is we need to set up what type of analytics extension we want this workbook to use. I'm not going to go through the entire process of how you set this up for Tableau Online. We have detailed instructions on what needs to happen from the administrative level to make sure that RStudio Connect is set up appropriately to interact with Tableau. But from my workbook, I can come in here and say manage analytics extensions connections, choose the connection that I want, and then I'm ready to go.

So now I just need to create my extension. The first thing I'm going to do just to make things a little bit more interesting for us is I'm going to create this parameter. We'll call this text input. We'll expect this to be a string. Example value will be hello world. And then we'll go ahead and show this here. So over here on the right-hand side, now we have this little text input. Let me zoom in a little bit, maybe. So we have this text input that shows over here on the right-hand side. And now let's create some part of our workbook that references this extension. So we'll come in. We'll create a calculated field. We'll call this to upper. This can be named whatever. And then let's come back into RStudio Connect, and we can just grab this little script right here, drop it into our workbook. Let me expand this a little bit.

And then string input is the name of the parameter in our R function, if we remember correctly. But now we need to make an adjustment because now that we're in Tableau, this has some sort of a, oops, this has a different name now that we're in Tableau. So we're going to call this text, I think is what we called it. There we go. Go ahead and hit OK. Now we have this function to upper. Let's drag it into our worksheet. This is now going to submit a request to our extension being hosted on RStudio Connect. That request will include this hello world string and expect that request to come back as an uppercase version. We see in fact that it did. Now it's formatted a little bit funny because Tableau has by default selected that this value should be mapped to color. We're going to change this to label so that we'll get some text output here in our workbook.

And now we see this to upper field contains the value hello world. And if we adjust this and say something like hi there instead, then we'll see that this field adjusts to say hi there. Not anything crazy. We're just taking lowercase letters and making them capitalized. But the interesting thing here is the fact that we have real-time interaction happening between our Tableau environment and our RStudio Connect workbook.

Debugging and logging

Now, one thing that I will highlight here that I think is kind of interesting is if we come back to RStudio Connect, we might want to, because we have two different components here. We have a Tableau environment that as the R developer we may or may not have control over or be involved in. We have the Tableau environment and then we also have this extension being hosted on Connect. And sometimes it can be difficult to diagnose if something goes wrong what is actually happening. How do we debug these types of extensions? And one of the things that I think is really kind of interesting and unique to the plumber Tableau package is that it has some built-in debug logging that we can actually choose to activate on the RStudio Connect side. So if I come in here, I can come into this variables panel here. I can set the debugme environment variable to contain the value plumber Tableau.

Go ahead and add this. Save this change. What this does is this inserts this new environment variable called debugme with the value of plumber Tableau into RStudio Connect for this particular extension. And when I do that and I look at my logs here, I end up with, let's go back to our to our little workbook here. Let's send another request. Let's say good morning. It's morning where I am. I know we have folks from all over the world here. So it might be afternoon, it might be evening, it might be the middle of the night, but for me it's morning. So we'll say good morning here. And if I come back into RStudio Connect now, let's go ahead and refresh the page.

Okay. So if I look over here on the right-hand side, I can see this output that wasn't here before. I should have highlighted that this wasn't here before, but I see this plumber Tableau, all these plumber Tableau messages in the log output. And this is messaging that was enabled by us setting that environment variable inside of the environment variables panel here in RStudio Connect. So now what we've done is we've increased the amount of logging that we're getting out of this extension so that we can diagnose anything that might be going wrong. And all this information, so we get back information about what types of requests are being made. Here we have a post request being made to evaluate. Again, we talked earlier about how Tableau makes every request to evaluate. We've done some work in both the R package and RStudio Connect to intercept those requests and reroute them to the right location. I'm not going to go into all the detail about what's actually happening behind the scenes, but there's further information about what is going on if you're curious in the documentation. So we have this post request coming in.

We can see that the body of the request is right here. It's this JSON object that includes good morning as this data value. And then we can see that we're rerouting this evaluate request to capitalize because that's what we actually requested in our extension. We can see that if we look at our... Let's come back in here. If we look at the definition of our to upper function inside of Tableau, we can see that we request the capitalize endpoint on this particular extension. So here's the extension name. We're requesting capitalize. That's how RStudio Connect and the R package know where to send this request when it's received. And then the argument that we're passing in again is this text input here.

One other thing that I'll highlight here, and then we're going to go back and look at some of the other tooling that exists here, is if we come back into our access settings here, we can actually adjust the URL for this. This feels really clunky, right? Content, a bunch of random numbers and letters, and then capitalize. So if instead we wanted something that was a little bit more straightforward, we could come in here and we could say, I'm going to call this James Tableau webinar. Go ahead and save that. Once we make this change, the documentation will reload and we see that our URL has adjusted, right? So now if we wanted to use this extension from Tableau, we could use this much more easy-to-understand script. We could bring that back in. Let's just go ahead and make the change here. Grab the whole thing. I thought I grabbed just the path. Let's fix that. All right. So now we have a much easier-to-read path to our extension here. Everything else remains the same. We can go ahead and apply this, and we'll see that the behavior remains consistent with what we saw before. We're submitting requests here. Let's just verify again, and we see hello world returned back over here.

FastAPI Tableau for Python

Okay, excellent. So let's come back to our slides now. That's plumber Tableau, right? Now, I mentioned there were two open source components to this. One is what we just looked at, which is plumber Tableau. The other is this new Python package that we've created called FastAPI Tableau. Now, as the name would suggest, this is a Python package built as an extension to the FastAPI Python package that allows us to do the same exact thing that we've just done in R, but using Python, right? We can build these Tableau extensions using Python, using a Python framework that we're comfortable and familiar with, and we can submit those, and we can publish those extensions to RStudio Connect. I'm not going to walk through that whole process again on the Python side, just because it very closely mirrors and mimics what we did on the R side, but I will pull open an example for us to look at. So let me just showcase what this would look like here.

So here is, and all of these examples that we walk through are available in a GitHub repository, I'll share the link to that at the end of the talk here, so you'll have access to all the example code that we've explored. Let me find what I'm looking for.

There we go. Okay, so if we look at this, all right, so here is a Python script that does the exact same thing that we did on the R side, takes some input text, capitalizes it, and returns the result. Again, not anything exciting in terms of functionality, but this does highlight how we would use this FastAPI Tableau package to build these kinds of extensions. And what's really kind of cool about this is the fact that the only real difference between a Tableau extension built using FastAPI Tableau and a regular FastAPI router is that we import FastAPI Tableau instead of FastAPI,

and we instantiate our app with FastAPI Tableau instead of FastAPI. These two changes are the only things that change between a traditional FastAPI router and one that can function as a Tableau analytics extension. And again, if you're curious about the technical details and implementation of the package and what the package is actually doing behind the scenes, the GitHub documentation walks through some of that and can highlight the components there. But this extension here, this Python script, again, does the same thing. If we look at this on RStudio Connect, we've already published a version of this. Let me pull that open here. Let's come back to just content I own.

Okay. So we have, here's our capitalized endpoint that we've created in Python. You'll notice when this pulls open that this documentation and interface looks different from what we saw on the R side. The goal is that those two differences, those two packages, Plumber Tableau and FastAPI Tableau, are going to converge as we continue to kind of iterate on them. It so happens that the way that we went through our development cycle, the R package was developed first, the Python package was developed second. We introduced some new things into the Python package that we're now working on porting back into the R package. And those changes will be made in the near future.

So one of those changes is this difference in the UI that we're seeing here. But we have the same functionality, right? We can click on this link here, using Tableau workbooks, we can see the same kind of documentation that we saw previously on the R side. And then we can also do things like view the traditional Swagger documentation if we wanted to do things like troubleshoot requests on RStudio Connect and investigate if those requests can be served.

How Tableau analytics extensions work with RStudio Connect

Okay. So we've already kind of, you know, we've talked through this, but let's just highlight it once again, right? Tableau, the Tableau Analytics Extensions API is basically just a way that Tableau sends requests to some service and expects a response back. And what we've done with Plumber Tableau and FastAPI Tableau and some product changes that we made in RStudio Connect is we've allowed this other service to be RStudio Connect. I can publish FastAPI Tableau extensions, I can publish Plumber Tableau extensions onto RStudio Connect, and I can use those extensions from within my Tableau workbooks.

And we've looked at what an example of this looks like on the RStudio Connect site, right? I have documentation available to me, I can interact with my API, I can see exactly how a Tableau user should leverage that API, leverage that extension, I can view documentation, I can turn on debugging, there's lots of things that I can do once this has been published to RStudio Connect that allows me to continue to both serve my end users, my Tableau developers, as well as do any additional debugging or any additional work that I need to do on the development side.

So, here's a kind of a more comprehensive example. We'll take a look at this dashboard here in just a moment. But here's an example where, and this goes back to the Tableau workbook that we looked at kind of at the very beginning, where we are submitting information about a given customer and their transactions to a machine learning model that was built in Python, or generating predictions based on that input data, submitting those predictions back to Tableau, and then visualizing them in that Tableau environment. And here we see, again, the same pattern. We use this script function in our calculated fields, and then we define the path to the endpoint that we want to, or the path to the extension that we want to use, and then the values from our workbook that we want to pass into that extension.

Comparing old and new approaches

So, let's contrast for a moment the two different approaches, right? We talked about how, in the beginning, we have tabpy, we have rserv as existing integration points for executing R and Python code within Tableau. The pattern for those extensions is what we're seeing here, right? I have some script function, and then the first argument to that script function is the full script that I want executed. In this case, it's the entire Python script that allows me to do this clustering of data points.

Just to reiterate some of the challenges of this particular approach, it requires that, as the Tableau user, I have the ability to, or I have the understanding and the knowledge, to write the script, or I have the ability to rely on somebody else that can write the script, and then I can copy and paste it in. I don't really have a great way of debugging this, right? I can submit the script. If it errors out, Tableau will tell me that an error occurred, but it doesn't give me a lot of additional information about where that error happened or what the cause of the error was, and it becomes a very kind of repetitive trial and error process to get something like this working. I also don't have any sort of syntax highlighting or any other developer help when it comes to writing and creating the script inside of Tableau.

The last thing, this is just, this is my own kind of, you know, again, personal perspective on this, is it's also a little bit nerve-wracking to me that this is just arbitrary code execution in some sort of remote environment. I don't know what I'm, I could put anything in this string, and my remote server is going to try to execute that. Hopefully, I'm not putting anything malicious here, but if I am, it's still going to try to execute it on the back end.

This new approach that we're suggesting and that we've highlighted today is that instead of submitting arbitrary code for execution, we instead provide a path to a predefined extension that we would like to execute. In this case, that extension, if we were going to take the same example where we're doing this cluster analysis of spatial data points, we could create that extension using something like FastAPI Tableau, we could host that extension on in an environment like RStudio Connect, and then we could simply refer to that by name. We're reaching out to this location in our extension environment, which is RStudio Connect, and we're passing these values in. And hopefully, we're able to kind of see the difference in these two approaches.

Right here, even if I don't have any understanding of Python or R or FastAPI or Plumber or any of these tools, I can still use this extension. If I'm really great at Tableau, but I never spend time in R, I can still make use of what my R developer has done if I want or Python developer has done, whatever the case is, because all I need to know is what the name of the extension is and then what values I want to I need to pass into it.

Right here, even if I don't have any understanding of Python or R or FastAPI or Plumber or any of these tools, I can still use this extension. If I'm really great at Tableau, but I never spend time in R, I can still make use of what my R developer has done if I want or Python developer has done, whatever the case is, because all I need to know is what the name of the extension is and then what values I want to I need to pass into it.

Advantages of the RStudio Connect approach

So, what are some of the advantages of this particular approach? Well, we've talked about some of them, but just to kind of revisit a few. One is, this is the only solution that exists today where you could use both R and Python extensions from the same exact Tableau workbook. Right? The other solutions, the existing solutions today, like RServe and TabPy work. We've seen customers utilize those, but you have to choose one or the other. You can't support RServe and TabPy execution from the same workbook or from the same Tableau worksheet.

It has to be one or the other. With the extensions that are hosted on RStudio Connect, because you can host R and Python extensions in the same location, you can use R and Python extensions alongside one another inside the same Tableau workbook. You also have the advantage of the enterprise-level security that RStudio Connect provides. Access control, you have a very, all processes on RStudio Connect are sandboxed. You're not allowing arbitrary code execution, but rather you have a predefined functionality within an extension. That extension gets invoked when it's called from Tableau, but Tableau users don't have the ability to pass arbitrary code into RStudio Connect. And then all of these extensions are behind the authentication layer that RStudio Connect implements. So, when we set up RStudio Connect to refer to these extensions, we do so by supplying an API key that is then used whenever these extensions are invoked.

This third point is one that I think is most significant, and that's the fact that R and Python developers can be separate from your Tableau developers, right? We could have a team of statisticians and data scientists that are building these kinds of extensions and providing them on RStudio Connect, and those developers can be entirely separate from the business analysts that are using Tableau or the Tableau developers that are building the workbooks. I don't have to be an expert in both sides, right? Personally, I find myself doing a lot of R and Python development. I don't spend a lot of time in Tableau, right? I don't consider myself a great Tableau developer, even now after spending a lot of time in the product as we've worked through this process.

But what that means is I can use what works for me, right? I can use the tool that's right for me, and I can build machine learning models, I can build business logic, I can build whatever I want to using R and Python, and then I can easily make that functionality available to my Tableau developers, who might not spend a lot of time in R and Python, but they're really, really good at Tableau, right? This is the idea that we started with at the beginning. Let's use the right tool for the job, and this is all about enabling that choice even further.

The last point that I'll mention here is environment management provided by RStudio Connect. So, one of the challenges that exists today with tools like RServe and TabPy is how do we make sure that the right packages, the right modules, the right environments are in place so that when we submit code to be executed, it executes in the manner we expect, right? If I try to load the tidyverse, but my RServe environment doesn't have the tidyverse, then I've got to go through some process to contact the admin of that environment, have them make the tidyverse available, then try publishing or then try executing my code again. RStudio Connect is different in the sense that any time we publish any extension, whether it's built in Python or R, the environment is automatically recreated and managed for us on the RStudio Connect side. All that happens automatically. I can have different versions of different packages used by different extensions, and it's all being managed by RStudio Connect.

The other thing that I didn't highlight here, but I think it's worth calling out, is that we also have the scalability introduced by RStudio Connect. So, if I have an extension that's used in lots of workbooks, and there's lots of users on those workbooks all the time, and so that extension gets lots of traffic, RStudio Connect can spin up new processes to support the load against that traffic as needed.

Resources and shinytableau

Some resources, and again, we'll distribute slides and everything like that so that you can have the links here, but there's a blog post that we put out a couple weeks ago that highlights the RStudio Connect release and the Tableau integration that it includes, and then there are links here to the Plumber Tableau GitHub repository, the FastAPI Tableau GitHub repository. This collection of examples here at the bottom is a repository of Tableau examples that we're building out. It includes the Capitalize example that we talked about today, as well as the other workbook that we briefly looked at. And then one thing that I didn't get a chance to really work in here today, but I will leave here as a reference, and you can kind of go check it out, is the shinytableau package. This is kind of a package still in somewhat of a beta release, but it allows you to create a different type of extension for Tableau where you can embed Shiny visualizations within your Tableau workbook. So the extensions we focused on today support real-time execution from R and Python back into Tableau. Shiny Tableau is focused on a different problem, and that is embedding additional visualizations into a Tableau workbook supported by Shiny, and the link here will take you to the documentation for that package.

Finally, if you want to take a look at the resources that we've walked through today, or revisit the slides, or anything like that, this link will take you to a GitHub repository that includes all the information that we've gone through today. So it includes the slides, and then the slides have links out to the additional resources that we've discussed. So feel free to jot that down and go visit that and take a look. I will, in fact, I'm going to drop it in the chat here, just so that you have...

There's a lot of great questions on Slido, too, James.

Yes, I will pull that up and take a look. I can see things scrolling.

I can read them out.

Okay, yeah, that would be great.

Yeah, I'll read them out to you. And thank you, Kelly O'Brien from the RStudio team was helping to answer some of those questions as they came in as well. But one that had actually the most upvotes was, are there any plans to allow integration between RStudio Connect and Power BI?

Yeah, no, that's a great question. So that's a question I've asked myself a lot of times, too. I can't speak to anything concrete right now. It is something that I plan on looking at, because I want to know the answer to that. But I haven't investigated it enough to have any sort of definitive answer, other than it is something that I plan on looking at. That's the most I can say about it right now. But I would love to see that happen, right? I would love to see a similar style kind of integration happen on that side of things. But it's not anything that we've made any real progress on at this point. But it is, like I said, it's on my radar for the very near future to at least start exploring that idea.

Awesome. Thank you, James. And also, just a quick note, because I know this was also a popular question, the recording will be shared to everyone. We share the Meetup recordings up to the RStudio YouTube as well. And so I'll include that link that James shared on the YouTube recording, too. The next most upvoted question, James, is what are the limits on what kind of data I can pass from Tableau to R? For example, can I pass something more complicated than a single string?

Yeah. Yeah. Great question. Let me see. So this was behaving a little bit slowly earlier. But let me see if this is working now. Yeah. So here's an example. We didn't get to this because I wasn't sure if it was going to behave. But it looks like it's decided that it's going to behave now. So this is kind of what we were looking at a little bit earlier with a screenshot, where we have this kind of superstore data set here that contains a bunch of transactional data for customers. And we're trying to identify customers that are outliers. And so what we have is we have this extension posted on RStudio Connect in R that does some outlier detection, given some input data. And in this case, what we're sending, I don't think I have logging enabled on that extension, actually. But we're actually sending, I think it's like 19,000 records of data from Tableau over to the extension. And then the extension's computing across that data and then coming back into Tableau. So you can do much more than just a simple string.

Apologies if we kind of got a little bit sidetracked with the simplicity of the original

example. But you can pass any number of things. So let me actually pull open this other one, too, here, if it'll open up for me. So the other example that we have in this particular workbook, this workbook and all of its extensions are included in the example repository that I link out to from the slides. So you can go kind of poke around and see how these extensions are implemented and everything like that. But this other example that I'm pulling up right now is a Python extension that has a really, really simple machine learning model behind the scenes that takes a bunch of input data. Let's look at the definition here.

Here we go. So we're passing over all of this information from Tableau, the days to ship actual, days to ship schedule, quantity, sales, discount. And then all of this gets passed as individual arrays over into our extension. The way that both Python and the respective Python and R packages have been implemented, there's a lot of kind of cool stuff that we do behind the scenes to make it really easy to ingest these types of requests so that they're automatically parsed and ready for use inside the body of your functions. But the idea is on the Python side, in fact, I'll just pull this open because I've got it right here. On the Python side of things, if we look at this, I'm not going to go through this whole script because I don't think it's, it doesn't make sense in the scope of the webinar right now. But if we look at the actual definition of the endpoint, we're just defining the values that we're expecting from Tableau, building those into this list, and then running them through this model pipeline that does a little bit of feature engineering, runs the data through a model and generates a prediction.

And on the Tableau side, when we look at this, we're getting this, oops, where did it go? Here we go. On the Tableau side, when we take a look, what we're getting is we have profit, which comes from our original data set on the x-axis. And then we have predicted profit, which is the output of our extension on the y-axis. And this allows us to identify, okay, who's overperforming based on what we would expect them to do, who's underperforming, things like that. But to go back to the original question, this example is much more involved than just a single string. We've got multiple numeric values, arrays of values that we're submitting to the extension, and then generating a result and bringing it back into Tableau. The one thing that I will note is that you can only return a single array of values back to Tableau. You can't return a large data set containing multiple columns and things. Tableau will only accept back a single array of length equal to the length of the data submitted.

Q&A: extensions, Shiny Tableau, and security

Thanks, James. Vanessa asked a question, can a visualization hosted on Tableau public leverage extensions hosted privately on RStudio Connect?

Great question. It cannot. Tableau public doesn't support analytics extensions for a variety of reasons. Tableau online does, but obviously, that's different from Tableau public. So, you can't do this on Tableau public, unfortunately.

Thank you. And then also, is there a way to host Shiny apps through Tableau server?

There's not. So, we didn't spend time, just I wanted to kind of keep things a little simple, but we didn't spend time with shinytableau, but I did want to highlight it. Let me bring it up here. So, shinytableau, to be clear, does not allow you to host Shiny applications on Tableau server. But what it does allow you to do, let me zoom in a little bit, as this kind of screenshot suggests, is you can embed Shiny applications that are hosted on RStudio Connect into your Tableau environment. So, RStudio Connect is still hosting the actual application and visualization, but it's being embedded into your Tableau environment. And it's more than just like a static embedding. It's actually a really interesting package. It might be worth another meetup or another discussion. That's something we'll discuss internally. But what shinytableau allows you to do is build these extensions, these visualization extensions that are hosted on Connect, then embedded inside Tableau, and they're responsive to changes in your workbook. So, if you have filters that get applied to your workbook, the Shiny embedded visualization will change to reflect the new values based on those filters and things.

So, shinytableau, to be clear, does not allow you to host Shiny applications on Tableau server. But what it does allow you to do, let me zoom in a little bit, as this kind of screenshot suggests, is you can embed Shiny applications that are hosted on RStudio Connect into your Tableau environment.

Thanks, James. There's quite a few questions, if it's okay if we go over a few minutes. Yeah, well, I would say yes, but my daughter is performing her Halloween program at 11. So, I'm going to jump off and go and go catch her for that. Yeah, go ahead. I'll try and get as many questions. And I know Kelly is on as well, who may be able to help too. But John asks, can we use this as a way to clean data as part of the data refreshing?

Yeah. So, I know Tableau has their Tableau prep process that is kind of like that initial data load, data refresh process. I haven't spent a lot of time, again, it's on my short list of things to look at, about what role R can play there. These extensions, I don't see playing a role there, at least as things currently stand. These are meant to be, once we have our data inside of our workbook and we're using that data, how do we get real-time interactivity with R and Python? That's the focus of these extensions. I do think that there's a space for R, and I do think there's even existing functionality with tools like R and Python in Tableau prep. But I'm not, like I said, I haven't spent enough time there to know kind of what the existing landscape looks like, as well as any things that we might try to bring to that landscape as well. But it is on my short list and something that I plan to look at.

Thanks, James. So, let's see, we have time for two more questions, because I want you to be able to run right before the top of the hour. One was from Jackson. Would you need to create separate security permissions for RStudio Connect extension versus dashboards built in Tableau?

Yeah. So, I'm not sure if this question is about the security that we need to provide on the Tableau side or on the RStudio Connect side. So, I'll speak, I'm more familiar with Connect, so I'll just assume that we're going in that direction. But on the Connect side... And Jackson, feel free to jump in, too. I can allow it so you can unmute yourself, too.

Yeah, yeah, it's really around the security to call the API, and then if it's, I guess, around embedding that permission on the Tableau dashboard, so that, I guess, like, do we need... The number of users need to also have permissions in RStudio Connect and also a Tableau, you know, license.

Yeah, yeah, good question. So, Tableau users are going to need to have a license to, like, view the dashboards, but you're not going to need a license, like, like a Tableau license isn't necessarily invoked when you're calling these extensions, if that makes sense. So, on the Tableau side, when you set up these extensions, the way that you do that is from the admin side of either Tableau Server or Tableau Online. You could also do this on Tableau Desktop as well, but I'm focusing on Server and Online here. You can come in and create a new extension or, yeah, a new connection, and then here you'd put in, you know, this is the RStudio Connect or whatever you want to call it. You give it the hostname and port of your RStudio Connect server, and then this username, and this is all in the documentation, but this username is RStudio Connect, and then this password is a Connect user API key, and the important thing to understand is that this API key is the key that will be used for every extension that you're accessing on this host. So, what our recommended approach is, is to create some sort of, like, quote-unquote service account user on your Connect environment that's called Tableau or Tableau extensions or extension user or something to that effect, and so you would come in here after you create that user, you'd generate a key for that user and use that key here, and then on the Connect side, for any extension you wanted to make available, you would just add, and we haven't set this up on our environment yet, but we're in the process of it, you'd come in here and add, like, that Tableau user or Tableau extension user or something like that.

Thanks, James. I don't want you to miss the show, so we'll let you run, and thank you so much for your awesome presentation, but we have Kelly O'Brien on still from the RStudio team, and Kelly was helping to answer the questions on slide 02, so I can ask some of the questions to Kelly. Well, thank you, everyone. Appreciate the time today.

Closing remarks and further questions with Kelly

Thanks, James. And also want to let people know, as we approach the top of the hour, if other people have to drop, though we do have another event next week with Tom Mock from RStudio team, who's actually going to be walking through an RStudio-hosted evaluation, and so that includes RStudio Connect, which you saw today, which I know many people on the call probably already have, but if you don't, I'll just put that in the chat as well, and if anyone else has other questions, too, or something that wasn't covered today, you can always feel free to reach out to me directly. I'll just put my email in the chat, too, and I can always help connect you with someone from the RStudio team, but thank you so much, Kelly, for jumping in here.

One of the let me just go through a few of the other questions. One that someone had asked is if we had paid for RStudio Connect, what is the incremental cost of adding Tableau? And I saw you had answered that there.

Okay, cool. Yeah, so I answered that one in the chat. We are not currently resellers of Tableau today in any capacity. That might be something we explore as we do a partnership with them potentially, but today we have no official relationship with Tableau, so you have to go through them to figure that out, unfortunately, so I don't have any information there.

Thanks, Kelly. Brody had asked, could something like Plotly or HideShutter be put into Tableau via these APIs?

Yeah, so for that one, I would explore probably iframing that application that's been hosted somewhere else or potentially that shinytableau package that James previewed a little bit. Shiny Tableau is a little bit different than iframing because it involves actually building out a dashboard extension, and that's a little bit more complex, but the documentation site for shinytableau has a little bit of a video that Joe Chang put together of a demo of what that looked like, so if you're interested in learning more about Shiny Tableau, I would start with the documentation site and watching that preview video.

Thanks, Kelly. I see that Prava asked, are there any enterprises known to the RStudio team that have connected RStudio Connect to an enterprise-wide Tableau deployment? I know it might be a bit early for using this particularly, but do you know, Kelly?

Yeah, I don't think we have any examples of customers who have set all of that up and are willing to talk about their implementation at this point, but we can definitely start thinking about that with our product marketing team, and I'm sure that Rachel would love to have somebody, if anybody on this call is interested in setting something up, once you are in a good place to share what you've done, coming back to this group and talking about the experience would be excellent.

I was just going to say that, Kelly. If anybody is going to start playing around with this right away, I would love to welcome you back to the meetup. One other question is, is there a way to somehow reverse engineer the Tableau dashboards to R code? They said these tools seem geared towards R to Tableau. We already have a Tableau process in place, though.

They are for sure geared towards R to Tableau and not the other way around, and we are opinionated in that respect, trying to help out the R users specifically and Python users as well. We have a companion package called FastAPI Tableau also for Python, but I would be quite interested if there was anyone in the community who was playing around with methods of taking Tableau dashboards and making, turning them into shiny or generalized R visualizations or code. That would be very cool.

I'll also say, if anybody would like to chat more about this, too, or schedule a virtual coffee or something, I'll invite you as well, Kelly. Feel free to just email me if you're interested, and I'll just put my email again in there.

A few more questions. Do you have to host extensions on RStudio Connect, or are there other options for hosting?

You don't. The packages are, like, generalized and open source and meant to be, you know, like, work really well with RStudio Connect, but that's not a requirement. The things that are nice that you get with RStudio Connect is management of those API keys, so your authorization and authentication needs, and the automatic handling of requests to that evaluate endpoint that get rerouted to the actual extensions that you want to be sending your request to. So, that gets handled on the Connect side for you. If you were to host these outside of Connect, you would have to write that logic out yourself.

Thank you, Kelly. Another question that had a few upvotes, too, was from Vanessa, and Vanessa asked, could you speak some to why, if an organization has R tooling and staff capabilities, they wouldn't just build visualizations in Shiny?

I replied to this one, too. I said, we would love it for more people to just build visualizations in Shiny, but we know that lots of organizations have already made big investments in Tableau, and so we want to be, you know, good citizens of the analytic infrastructure marketplace and space and recognize that there have been those big investments and show how the RStudio tooling and the R and open source R and Python packages can play well in that space.

So, long answer to just say, yeah, we would love for more people to just build Shiny.

Thanks, Kelly. And if other people have ideas, too, or things you'd like to share, feel free to use the chat, too. I see a few people were commenting in there as well.

Sparklyr R and Tableau integration

And another question, Kelly, is still new to all this, but if we're working with Sparklyr R, can we push to Tableau through that as well? That's a really good question. I think that, like, there's nothing preventing that from being a pattern that you explore. The things that you want to be kind of cognizant of when you're interacting with Sparklyr R, as you probably are aware, is that you want to be sure that your code is not pulling down more data than you need at any given time. So, that's always a concern when you're working with big data and you're trying to pull things down into visualizations. And I think that James and Edgar are really great people to start building out some examples of what that might look like on the RStudio side. So, I would definitely reach out on the community page about that, specifically if you have any tactical questions around how to get started.

Analytics extensions and Connect limits

Awesome. Is there a limit to how many analytics extensions can be used on Tableau's server site? Would this work for only one extension?

Yeah. So, Tableau's server sites have recently lifted the restriction around a single extension. I think when I answered this, I interpreted it from, like, the Connect perspective. On the Connect side, you can publish as many of these Plumber, Tableau, and FastAPI Tableau extensions as you want. And they get routed from requests coming through the evaluate endpoint once you've set them up to work with Tableau. So, there's sort of an unlimited set on the Connect side. And I believe the restrictions from the Tableau side for the number of extensions you can register has also been lifted per site. But definitely check on the Tableau, or I defer to the Tableau documentation on that.

Great. Thank you. I saw you also shared documentation for this question in the chat, which was, does this work with Tableau's server also? Yes. And we have integration docs on our docs.rstudio.com site. So, works with Tableau server, Tableau online, and Tableau desktop. Essentially should work anywhere Tableau allows Tableau analytic extensions to function. Those are the three places we know of.

Authentication between Tableau and Connect

Thank you. I'm not sure if this question was answered in another way earlier by James, but Jay asks, can you elaborate on how authentication works between Tableau and Connect? Yeah, hopefully what James demoed right before he hopped off was an answer for that. But yeah, like at a high level, you have to set up an API key, hopefully that represents a service account user on the Connect side. And then you register that as an admin on the Tableau site. And the instructions for doing all that are at that same docs page that I just said that we just linked.

Great. Thank you. And Kelly, let me know if you have to run to a meeting. I'm just firing these questions. I probably have to hop off in a couple of minutes. Yeah. Okay. We'll take two more then. There was an anonymous question that said with RServe, I've been unable to publish Tableau dashboards that were using R in a server that hosted Tableau dashboards. Does the Plumber package address that?

Yeah, if you can use Tableau analytic extensions and serve Plumber Tableau or FastAPI Tableau, either on RStudio Connect or somewhere else, then this should be a replacement for RServe.

Data limitations and API restrictions

Thank you, Kelly. And we'll take one more. I'll say two more questions here. Derek asks, what are the limitations of the API from a data perspective? How much data can be passed in return? And what are the data type restrictions? So, James would be much better suited to answer this one. There are a couple of different script definitions from the Tableau side that you have. So, you have like some restrictions there. Most of the restrictions are on the Tableau side. And from the API perspective, we've added a bunch of tools for you to turn on debug logging at various different levels so that you can see the latency and what actual data is being passed through the request. And that should be hopefully really helpful to you when you're trying to figure out whether you've overdone it. You're sending too much. And the actual, the ability of Tableau to process the data you're sending over from Connect is going to fall over. We're only able to expose what's happening from Connect's perspective, though. Tableau in many respects is kind of a black box when it comes to understanding what its limitations are. So, I would encourage you, if you have, if you run into those restrictions, to reach out to the folks at Tableau and make it known that you would love to see more transparency around what its capacity is.

Tableau in many respects is kind of a black box when it comes to understanding what its limitations are. So, I would encourage you, if you have, if you run into those restrictions, to reach out to the folks at Tableau and make it known that you would love to see more transparency around what its capacity is.

Retrieving pins from RStudio Connect via Tableau

Awesome. One last question I saw just came in was, we mentioned a limit on the data Tableau can receive with this API, but can Tableau retrieve a pin from RStudio Connect? So, Tableau, I believe the, I'm sure that what makes this problem difficult to solve from Tableau's perspective is that Tableau doesn't allow you to just reach out to, like, a regular Plumber API or, like, Python API. It doesn't make requests in that way. It sends two types of requests, whenever it's reaching out from Tableau Analytics Extensions, it's expecting to hit an info endpoint that gives some data about, like, what it's speaking to, and then the evaluate endpoint. So, requesting a pin could potentially be set up through, like, that interface Tableau has for registering data sources, but going through the same route to get that pins, I don't think would work, unless you have those pins available and exposed through the Analytics Extensions, and I'm not sure that the script parameter on Tableau would allow you to set that up and make that a functional data pull, but, again, James would be a better person to answer that question in more detail.

Well, thank you so much, Kelly, for jumping in here and answering some of the remaining questions. I'll sync up with Lou from our team after this, as well, and we can work on putting together a blog post with the questions, as well, to share. But I just want to say thank you all for joining today. It was great to see so many of you here. And another great event will be next week with Tom Mock, who will be going through our RStudio-hosted eval walkthrough, if anyone's interested. I did just share a Flix dashboard I kind of made to share upcoming meetups and events, in case you don't have access to meetup at work. So, if you're interested in that, you can go to Flix.org and search for meetup at work. But thank you all so much for joining. If you do ever have any other feedback or things that you'd like to see, there's a Google form in the chat, too. We'd love to hear from you all. But thank you so much for joining, and have a great rest of the day.