Series of Apache Spark posts:
- Dec 01: What is Apache Spark
- Dec 02: Installing Apache Spark
- Dec 03: Getting around CLI and WEB UI in Apache Spark
- Dec 04: Spark Architecture – Local and cluster mode
- Dec 05: Setting up Spark Cluster
Let’s look into the IDE that can be used to run Spark.
Remember that Spark can be used with languages: Scala, Java, R, Python and each give you different IDE and different installations.
Start Jupyter Notebooks and create a new notebook and you can connect to Local Spark installation.
For the testing purposes you can add code like:
spark = SparkSession.builder.set_master("spark://tomazs-MacBook-Air.local:7077")
And start working with the Spark code.
In Python, you can open a PyCharm or Spyder and start working with python code:
import findspark findspark.init("/opt/spark") from pyspark import SparkContext sc = SparkContext(appName="SampleLambda") x = sc.parallelize([1, 2, 3, 4]) res = x.filter(lambda x: (x % 2 == 0)) print(res.collect()) sc.stop()
Open RStudio and install sparkly package, create a context and run a simple R script:
# install devtools::install_github("rstudio/sparklyr") spark_disconnect(sc) # install local version spark_install(version = "2.2.0") # Create a local Spark master sc <- spark_connec(master = "local") iris_tbl <- copy_to(sc, iris) iris_tbl spark_disconnect(sc)
There you go. This part was fairly short but crucial for coding.
Tomorrow we will start exploring spark code.
Compete set of code, documents, notebooks, and all of the materials will be available at the Github repository: https://github.com/tomaztk/Spark-for-data-engineers
Happy Spark Advent of 2021!