image: thumbnail.jpg
Workflow Demo Live Q&A - September 25th!
Links mentioned in the Q&A:
✨ Release Blurb: https://docs.posit.co/connect/news/#posit-connect-2024.08.0
✨ Security: https://docs.posit.co/connect/admin/integrations/oauth-integrations/security.html
✨ Publishing Quarto: https://docs.posit.co/connect/how-to/basic/publish-databricks-quarto-notebook/
✨ sparklyr: https://github.com/sparklyr/sparklyr?tab=readme-ov-file#connecting-through-databricks-connect-v2
✨ odbc: https://github.com/r-dbi/odbc?tab=readme-ov-file#odbc-
Helpful resources for this workflow:
✨ Full examples to get you started: https://github.com/posit-dev/posit-sdk-py/tree/main/examples/connect
✨ Admins will likely be most interested in starting here: https://docs.posit.co/connect/admin/integrations/oauth-integrations/databricks/
✨ End users will be most interested here: https://docs.posit.co/connect/user/oauth-integrations/
✨ Databricks Integrations with Python Cookbook: https://docs.posit.co/connect/cookbook/content/integrations/databricks/python/
✨ Databricks Integrations with R Cookbook: https://docs.posit.co/connect/cookbook/content/integrations/databricks/r/
✨ Snowflake Integrations with Python Cookbook: https://docs.posit.co/connect/cookbook/content/integrations/snowflake/python/
Transcript#
This transcript was generated automatically and may contain errors.
Hey everybody, thanks for jumping over here to the Q&A. We're going to give people probably another minute or so to join us over in this room.
Okay, I can see 15 people made it over here so far. Let's wait a little bit longer.
Alright, and let me start to bring my Posit friends here on stage with me as well. There we go. Well, we'll get started here and people can jump in and join us as they do. I'll go make sure everyone knows where the link is. Should have redirected most people over here. Okay, I can see a big chunk of people coming in now.
Well, thank you all so much for joining us today and thank you to Zach for such a great session. As a reminder, we host these workflow demos the last Wednesday of every month and they are all recorded. And so the recording is made available immediately after the demo. And so we now have, I believe, 19 different workflows shared there. I can't believe we've been doing it this long already. So anything from building a model annotation tool to pins workflows to building beautiful business reports. And I will share that link to the full playlist in the chat in just a second here.
I want to add while I know many people joining me already be Posit customers or using our professional products. If you're new to Posit team and would like to try it out or if you want to chat more with our team, always feel free to message me on LinkedIn or email me. But I'm also going to put a link in the chat where you can schedule time to chat with our team. And sometimes that's really helpful if there's questions that might be better suited for a more one on one conversation where we can actually dive deeper into your own workflows. So I'll share that there in the chat. Awesome.
Introductions
Well, to get us started here today, I would love to just quickly go around and do some brief introductions. And I didn't introduce myself when I kicked it off, but I'm Rachel Dempsey. I lead customer marketing here at Posit. And so I host a few different community events like this one. If you've never been to our data science hangout before, I host that every Thursday. We have a different leader from the community joining to answer questions from you all as well. So I can also share more info to that about that. But David, I'd love to have you introduce yourself next.
Hello, everyone. My name is David Kegley. I'm a software engineer on the Connect team. I've mostly focused on the integration stuff that we've been talking about today. But a lot of the Kubernetes work and office execution back end as well. Awesome. Pablo, want to go next?
Sure thing. Hi, everyone. Hi, everyone. My name is Pablo Bianco Lascaray. I work with David and Zach on the Connect team. I'm a quality assurance engineer on the team. Let me just try to make sure things work before we release them. Thank you. And Zach?
Yep. My name is Zach Verham. I just talked for 20 minutes about this feature that we're really, really excited about and really happy to be on this workflow demo to share. And I work with David and Pablo. I'm a software engineer. Before working on this feature, I was doing a lot of work on sort of what the experience is for Connect admins as they interact with the product.
How to ask questions
Thank you. So just to let people know how you can ask questions, if you haven't been in one of these before, you can ask questions right in the YouTube chat here. So where I just put, hey, into the chat, you can ask questions there and I'll pull them over. And then you can also ask questions anonymously. And so that's the short link that I have shared on the screen right now, and I'm just typing that into the chat as well. And so I see we had a few questions already. So let me go grab that over from Slido there.
Q&A
OK. One of the first questions that came in and I'm going to copy it over so it shows on the screen here. Is. Does this feature work when Connect is using the SAML authentication provider?
Yeah, I can take that one. So the well, the feature itself uses OAuth under the hood and does require that you're using OAuth with the external data sources. It does not require that Connect itself is using the any particular authentication provider. So you might be using SAML or LDAP or PAM or username and password off. Any of those are fine and any of those can be used with this new feature.
Thank you. So another question that came in on Slido is, can you deploy Quarto or Markdown reports from the RStudio IDE to Posit Connect using Snowflake or Databricks credentials?
Yeah, I can take that one as well. So the there currently this feature is limited to interactive content. So it does require that a Connect user actually visits the content to to inject the credential. So unfortunately, we can't support rendered content yet. So our Markdown and static Quarto are not supported yet, although we do have plans on the roadmap to be able to support those with a little bit of a different OAuth flow. So there should be news coming out about that soon. I don't want to talk about deadlines specifically, but I think that's one of the next things we're going to work on. So unfortunately, static and rendered content still do require environment variables, but we'll be working on that soon.
Yeah, I can add a little bit of flavor here. I was just looking at that at this particular problem this morning and figuring out what the solution will be. So this is something that we are like very much aware of and something that we want to support. And the question specifically says without setting variables. Right. But we do have a guide specifically deploying Quarto to connect using environment variables. So if anybody is curious about that, we can also provide the links. Yeah, Pablo, would you be able to share that in the chat with me and I can share it with everyone? And we won't hold you to a date here on the call.
So our Markdown and static Quarto are not supported yet, although we do have plans on the roadmap to be able to support those with a little bit of a different OAuth flow. So there should be news coming out about that soon.
OK, let's see. Another question was, is it apparent to viewers of the data that masks have been applied? Or is logic exposed anywhere for viewers to understand why they may not see all the data?
Yeah, that's a great question. So it is it is going to feel different for viewers who are used to interacting with Kinect, because up until today, any any content you run on Kinect, every viewer of that content is going to see the same view. There is no such thing as like a dynamic application where one viewer sees one thing and another viewer sees a different thing. So it is not obvious right off the bat that that what you're seeing is is correct, even though it is like the ACLs are applied at the data source. That's one of the pieces of feedback we've gotten a few times now. And I think it's something we'll be addressing probably through the dashboard. We can make it a little bit more apparent that that these integrations are in use through the dashboard. Another option is for the application author to to display some banner or something at the top, just to let folks know that what they see might differ from what someone else what someone else sees. It is a it's an explicit choice by the content author. So it's not something that's going to happen without any effort. And so it should be pretty easy for the content author to just display some banner and let folks know what to expect.
Yeah, I think. Oh, sorry. To what David was saying. Right. The important thing is that the access to the data is not managing Connect. It's managed somewhere else. So it's not necessarily being masked. It's just access control and third party.
I had a question. If if customers have feedback on this, what is the best way to share feedback or questions with the team outside of. This format, I guess. It may seem counterintuitive, but I would go ahead and create a support request. Those oftentimes get funneled to the engineering team and we'll be able to give you more specific feedback. And we can take those considerations when we start looking at our roadmap and planning new features and stuff like that. It does help us catalog the feedback a little bit better. So, you know, obviously do that if you have a problem, but it can also be useful just to do that if you have a question.
OK, thanks. Something as you were answering that I just thought to add is I know sometimes, especially at huge companies, it can be hard to know if your team is actually already using these tools as well. And so sometimes I talk with one department who might not know that the other exists. So I shared a link to talk specifically about this feature. But if you just generally want to chat with our team and just understand if your organization may already be using our tools, feel free to use that that same link. And I just put that up on the screen here, too.
Let me jump back over to I don't see any coming in in YouTube yet, so I'm going to keep going with the Slido questions. And another one was thanks for the demo. Zach mentioned that the cookbook for the OAuth sample code may be released this week. Is it released yet? And if so, can you post the link?
It's not released yet, but it definitely will be. We're working on that. And when that's available, we'll make sure that the links are available to everybody. Yeah, and we'll have links to the SDK, the Python SDK, and that has a subset of the examples that will be in the full cookbook. So if you want to look at things today, there are examples in there. There will just be a more robust set of examples that will go out later this week.
So the integrations that Connect is managing, unfortunately, are not going to be available on the open source version. There is no open source version of Connect or the closest thing we have is Connect Cloud, which is free to use and then public alpha. But we we don't have support for these OAuth integrations there yet. If you're looking to use Databricks and Snowflake from R and Python, Posit helps maintain the R odbc package, which has some really helpful Snowflake and Databricks connectors. We can share a link to that. And then Posit also helps maintain the sparklyr R package for interacting with Spark on Databricks.
Thank you. Yeah. If you don't mind sharing those links in the chat, I can share them with everybody here, too. Let's see. OK. A few authentication questions and one was similar to the SAML question. If we still use LDAP for login to Connect, can we also use this OAuth feature out to Databricks?
Thank you. And I was just multitasking and trying to copy over these odbc links into the YouTube chat as well, so everyone has them. OK, the other question, I believe, was are there any OAuth API providers that are on the shortlist to add?
I see Zach nodding. Yeah, it's hard to know sort of exactly what the timelines are for certain things. Again, right, like scoping out when things will land is hard, but we have in the immediate short term, GitHub is one that we're prioritizing. And then we are going to be looking at like BigQuery and AWS and figuring out how to set up first party integrations for those as well.
I see this person edited their question to say, for example, Azure API Management or Logic Apps or Azure Functions. So maybe if other people have those in mind as well, it'd be helpful to see that in the chat, too.
Yeah, I can add to that. Azure is supported today. That was one of the first ones that we implemented. So we have support currently for Azure Databricks and Snowflake. And then we're working on GitHub now. That should be happening pretty soon. And then after that, I think, as Zach said, it's going to be like Google and AWS probably next. But if there are other things that I didn't list, then we would love to hear about what our customers are using and what they want us to integrate with next. So please let us know what would be high priority for you.
And one additional caveat here is we showed this briefly at the end of the workflow demo. But there is a custom integration type that lets you fill in all of the like it gives you kind of full control over filling in all the information you need to hit any standard OAuth providers. So if you have something that is not supported implicitly in one of those dropdown templates that we have, that custom integration is sort of a way to integrate with anything that we don't have in that list.
Yeah, I think that bears repeating, right? Even though we have a list right now of three supported providers, we do provide a way to add custom integrations there. So you might be able to add support to a lot of things today. In the future, though, there will be better, easier ways to do it, but you should be able to do it today too.
Unity Catalog and Active Directory groups
Thank you. I see someone else asked a question that was, does the masking and filtering in Unity Catalog also work with Active Directory groups? In addition to providing a username like you showed in the demo?
That's not something we have tested. I think it would depend on how Active Directory is configured. I'm assuming Unity Catalog has some way to connect out to Active Directory, which would mean that Databricks would be aware of the user's group. If that's the case, I think you should be able to write a Databricks filter that uses the group to perform some logic during the masking process. And that would mean, so I think that would mean, yes, it would be possible to do that today. I'll try to find some associated documentation, and I'll send a link in the chat if I can find it.
Alternative approaches for user-specific data access
Yeah, it's a good question. I think the most common one we've seen, like specifically within the context of Connect, is deploying those multiple instances of the same app and setting static service account-esque credentials on that app, where the service account has the same permissions as the user that is interacting with that duplicated instance of the app that's published, which obviously is very clunky, and that's why we wanted to build this feature. That's the main one that I'm aware of that people have used as alternative approaches to try to do this type of workflow. I don't know if David or Pablo are aware of any other things, but that's the main one that I know of.
There are two others that come to mind. There's the possibility of actually implementing OAuth inside of your content. It is technically possible to handle the OAuth flow from your content, but it would mean you would have to implement it for every piece of content, and that's a lot of work. It's also really difficult to maintain, and it gets really complicated, particularly with high-availability Connect. You never know which Python or R process you're going to connect to, and so that would be weird.
The other option is using some of the metadata provided by Connect when the content is running. Connect does expose information about the connecting viewer to the content itself. You can get that information out of one of Connect's headers. You could write application-specific logic for every piece of content that says, if this viewer then access the external resource with this other set of credentials, but that doesn't scale very well either.
So we think that what we've got here is a much better approach for managing that because it's going to scale a little bit better. Rather than maintaining that access control logic inside of your content, you can maintain the access control logic at the data source where they have really robust controls for writing those access control rules.
Rather than maintaining that access control logic inside of your content, you can maintain the access control logic at the data source where they have really robust controls for writing those access control rules.
Row-level security in Databricks and RStudio
Okay, I see one more question I think that we haven't gotten to yet, and it was, at my employer, Databricks has recently been added, but I don't think anyone uses it yet. Are today's row-level security features new for Databricks in RStudio?
The row-level security features are not new for Databricks, but the ability to delegate those row-level security features to RStudio is new. Previously, Workbench had integrations with Databricks already, where the user inside of Workbench could have delegated those Databricks rules to Workbench, but that ability did not exist in Connect before today. So now that Connect can kind of inherit those row-level permissions, it gives you the ability to have that same experience that you have in Workbench in Connect.
Security considerations
Yes, for sure. There's a lot of different ways that you could think about this question and what the security considerations are. We have a doc in our admin guide that lays out a lot of those implications. The sort of high-level summary I can give here is that publishers and app writers need to consider sort of the lifecycle of their app and where in memory the OAuth credentials are and avoid using, for example, global variables to store that information. But we have a doc that helps kind of talk through what those security considerations are and the things to be aware of when writing your apps. But the publishers just need to be aware of how the feature works and where those credentials are stored. That's the main thing to be aware of when using the feature.
Feature availability and requirements
Great, thank you. All right, I want to give people another two minutes here if there are any questions that you didn't get to put in there yet or if there's anything that maybe I missed and you want to repost in YouTube as well.
One just quick confirmation question. Somebody asked, is this feature released already or is it soon to be released? It is available, yes. Yeah, if you upgrade to the August edition, August 24 edition of Kinect, it'll have the feature in it.
Awesome. I was going to try and go grab the blog post or something right now for that, but if anybody has it handy and wants to put that in the chat, that would be great.
Okay, one more question. Is there a minimum requirement for the R version? Or let me copy it over so you can read it. Okay, is there a minimum requirement for the R version or Python version to use these new OAuth features?
No, no minimum requirements for R or Python versions. Everything we've implemented to use this feature uses really standard REST APIs and is built on top of OAuth. So you might be somewhat limited by what libraries you have available, but that doesn't mean it's not possible. You would just have to write a little bit more boilerplate to make it work.
Comparison with run as current user
Great, thank you. Okay, was there anything you thought you were going to be asked about today, but we didn't cover? Or anything else you want to share?
I don't know. I thought for sure someone would ask about run as current user. I'm very hesitant to bring it up at all. But I do want to point out that this feature is similar to run as current user, but it is still a little bit different.
Run as current user allows the viewer of an application to view the application. Basically, it's a way to use Kerberos with Connect. But it does require PAM authentication. And this new feature can be used with any authentication provider. So that's one distinction. Another distinction is that with run as current user, every viewer of the application is executing the copy of the application. Another distinction is that with run as current user, every viewer of the application is executing the content in their own isolated process on the server. This OAuth integrations feature will share content processes under the hood. So every viewer of the application is not getting their own instance of a process. It does have some different security implications. We try to talk about that a little bit in the security talk that we've already shared out in the chat. But this new feature is much more lightweight. And it's going to work with any authentication provider. Those are the main distinctions.
Thank you. Anything else, Pablo or Zach?
I don't have anything in particular, but I do appreciate all the questions that were asked. So thank you, everyone. Yeah, thank you so much, everybody, for taking the time to join us today. I just wanted to add that I said this in the beginning, but we do hold these monthly workflow demos the last Wednesday of every month at 11 a.m. Eastern Time. And so if you ever have suggestions or workflows that you'd like to see, please let us know. You can add that into the comments here below in YouTube.
But I did want to give you a little sneak preview of next month's demo. Ryan Johnson is going to be joining us to talk about PDFs and talking about types and showing you how to create beautiful PDF with Posit. And so we'd love to have you join us at that one as well. This link that I showed on here adds the recurring series of the workflow demos. They are also all recorded as well. So you can see them all on the playlist there. Thank you so much, Pablo, David and Zach. Really appreciate you taking the time to join us and have a great rest of the day, everybody.