Spark SparkContext is an entry point to Spark and defined in org.apache.spark package since 1.x and used to programmatically create Spark RDD , accumulators and broadcast variables on the cluster. Since Spark 2.0 most of the functionalities (methods) available in SparkContext are also available in SparkSession. Its object sc is default available in spark-shell and it can be programmatically created using SparkContext class.

SparkSession introduced in version 2.0 and and is an entry point to underlying Spark functionality in order to programmatically create Spark RDD, DataFrame and DataSet. It's object spark is default available in spark-shell and it can be created programmatically using SparkSession builder pattern.

SparkSession vs SparkContext

| *** Please Subscribe for Ad Free & Premium Content ***

Post author:Naveen Nelamali
Post category:Apache Spark / Member
Post last modified:April 24, 2024
Reading time:9 mins read

You are currently viewing SparkSession vs SparkContext

This content is for members only.
Join Now

Already a member? Log in here

Tags: SparkSession, SQLContext