Dec 15, 2023
|
16 min
|
2.5k views
How to Use R with Databricks Connect and sparklyr
Spark Connect lets you talk to a remote Spark cluster over a network connection rather than running inside it, which means you can work with Databricks from your own laptop in RStudio or Positron instead of opening a notebook in the Databricks workspace. sparklyr supports it through pysparklyr: upgrade sparklyr, install the Python components, supply a cluster ID and a token, and you have a connection you can drive with ordinary dplyr code. With Edgar Ruiz, who maintains sparklyr.
What you will learn
What Spark Connect is, and what changed when it arrived Why the connection goes through Python, and what pysparklyr installs for you How local Spark and Spark Connect differ in practice What you need to connect to Databricks: the cluster ID and a token Why you should never put a token in your code How to preview tables and explore the data from your editor How dplyr code runs against a Spark table What the current limitations are with Spark 3.4 How reticulate gets you the rest of the API when sparklyr does not wrap it Chapters 0:00 What this talk covers 0:28 What Spark Connect is, and why it changes things 2:07 Why gRPC means Python is the bridge 3:10 What sparklyr gives you today 4:14 Upgrading sparklyr and installing pysparklyr 5:17 Local Spark and Spark Connect compared 5:52 Databricks Connect: working from your laptop 7:24 Connecting with a cluster ID and a token 8:28 Previewing tables and exploring the data 9:32 Using dplyr against a Spark table 10:35 Managing clusters from the Databricks UI 12:41 Current limitations with Spark 3.4 13:46 Reaching the full API through reticulate 15:53 Running models and getting results back
Learn more sparklyr: https://spark.posit.co pysparklyr: https://github.com/mlverse/pysparklyr Posit and Databricks: https://posit.co/solutions/databricks/ Databases with R: https://solutions.posit.co/connections/db/
Presented at posit::conf(2023). Spark Connect and sparklyr have both moved on since; check the sparklyr site for current support
dplyr
Positron
reticulate
rstudio
sparklyr
Databricks Connect
Spark R
SparklyR
Spark With R
Databricks R
Spark Connect
Pysparklyr
R Databricks
Dplyr Spark
Reticulate
Remote Spark Cluster
Big Data With R
Posit