PySpark – What is SparkSession?
Since Spark 2.0 SparkSession has become an entry point to PySpark to work with RDD, and DataFrame. Prior to 2.0, SparkContext used to be an entry point. Here, I will…
1 Comment
July 22, 2020
Spark – What is SparkSession Explained
Since Apache Spark 2.0, SparkSession has become a unified entry point to Spark to work with RDD, DataFrame, and Dataset. Prior to 2.0, SparkContext used to be an entry point.…
March 28, 2020
SparkSession vs SparkContext
SparkSession vs SparkContext - Since earlier versions of Spark or Pyspark, SparkContext (JavaSparkContext for Java) is an entry point to Spark programming with RDD and to connect to Spark Cluster,…
March 28, 2020
SparkSession vs SQLContext
In Spark, SparkSession is an entry point to the Spark application and SQLContext is used to process structured data that contains rows and columns Here, I will mainly focus on…
March 23, 2020
PySpark – Create DataFrame with Examples
You can manually create a PySpark DataFrame using toDF() and createDataFrame() methods, both these function takes different signatures in order to create DataFrame from existing RDD, list, and DataFrame. You…
January 12, 2020
Spark Create DataFrame with Examples
In Spark, createDataFrame() and toDF() methods are used to create a DataFrame manually, using these methods you can create a Spark DataFrame from already existing RDD, DataFrame, Dataset, List, Seq…
February 1, 2019
Spark – Create a SparkSession and SparkContext
In Spark or PySpark SparkSession object is created programmatically using SparkSession.builder() and if you are using Spark shell SparkSession object "spark" is created by default for you as an implicit…
December 17, 2018
