Spark Load CSV File into RDD
In this tutorial, I will explain how to load a CSV file into Spark RDD using a Scala example. Using the textFile() the method in SparkContext class we can read…
6 Comments
November 24, 2019
Spark – Read multiple text files into single RDD?
Spark core provides textFile() & wholeTextFiles() methods in SparkContext class which is used to read single and multiple text or csv files into a single Spark RDD. Using this method…
November 23, 2019
Spark – How to create an empty RDD?
We often need to create empty RDD in Spark, and empty RDD can be created in several ways, for example, with partition, without partition, and with pair RDD. In this…
November 21, 2019
Convert Spark RDD to DataFrame | Dataset
While working in Apache Spark with Scala, we often need to Convert Spark RDD to DataFrame and Dataset as these provide more advantages over RDD. For instance, DataFrame is a…
August 22, 2019
Photo by Deva Darshan on Unsplash
Different ways to create Spark RDD
Spark RDD can be created in several ways using Scala & Pyspark languages, for example, It can be created by using sparkContext.parallelize(), from text file, from another RDD, DataFrame, and…
February 4, 2019
Create a Spark RDD using Parallelize
Let's see how to create Spark RDD using parallelize with sparkContext.parallelize() method and using Spark shell and Scala example. Before we start let me explain what is RDD, Resilient Distributed…
December 4, 2018
