How to create a SparkSession and Spark Context

  • Post Author:
  • Post Category:Apache Spark

SparkSession

SparkSession is an entry point to underlying Spark functionality. All functionality available with SparkContext are also available in SparkSession. Also, it provides APIs to work on DataFrames and Datasets.

Spark Session also includes all the APIs available in different contexts –

  • Spark Context,
  • SQL Context,
  • Streaming Context,
  • Hive Context.

SparkContext

A Spark “driver” is an application that creates a SparkContext for executing one or more jobs in the Spark cluster. It allows your Spark Application to access Spark Cluster with the help of Resource Manager.

Below is an example on how to create a SparkSession using builder pattern method and SparkContext. SparkContext will be created only once for an application; even if you try to create another SparkContext, it still return existing SparkContext.


package com.sparkbyexamples.spark
import org.apache.spark.sql.SparkSession
object SparkSessionTest {
  def main(args:Array[String]): Unit ={
    val spark = SparkSession.builder()
      .master("local[1]")
      .appName("SparkByExample")
      .getOrCreate();

    println("First SparkContext:")
    println("APP Name :"+spark.sparkContext.appName);
    println("Deploy Mode :"+spark.sparkContext.deployMode);
    println("Master :"+spark.sparkContext.master);

    val sparkSession2 = SparkSession.builder()
      .master("local[1]")
      .appName("SparkByExample-test")
      .getOrCreate();

    println("Second SparkContext:")
    println("APP Name :"+sparkSession2.sparkContext.appName);
    println("Deploy Mode :"+sparkSession2.sparkContext.deployMode);
    println("Master :"+sparkSession2.sparkContext.master);
  }
}

Above example returns below output.


First SparkContext:
APP Name :SparkByExample
Deploy Mode :client
Master :local[1]

Second SparkContext:
APP Name :SparkByExample
Deploy Mode :client
Master :local[1]

This example also available at GitHub

NNK

SparkByExamples.com is a BigData and Spark examples community page, all examples are simple and easy to understand and well tested in our development environment using Scala and Maven.

Leave a Reply