Convert PySpark RDD to DataFrame
In PySpark, toDF() function of the RDD is used to convert RDD to DataFrame. We would need to convert RDD to DataFrame as DataFrame provides more advantages over RDD. For…
0 Comments
August 14, 2020
PySpark parallelize() – Create RDD from a list data
PySpark parallelize() is a function in SparkContext and is used to create an RDD from a list collection. In this article, I will explain the usage of parallelize to create…
August 13, 2020
Convert Spark RDD to DataFrame | Dataset
While working in Apache Spark with Scala, we often need to Convert Spark RDD to DataFrame and Dataset as these provide more advantages over RDD. For instance, DataFrame is a…
August 22, 2019
Photo by Deva Darshan on Unsplash
Different ways to create Spark RDD
Spark RDD can be created in several ways using Scala & Pyspark languages, for example, It can be created by using sparkContext.parallelize(), from text file, from another RDD, DataFrame, and…
February 4, 2019
Create a Spark RDD using Parallelize
Let's see how to create Spark RDD using sparkContext.parallelize() method and using Spark shell and Scala example. Before we start let me explain what is RDD, Resilient Distributed Datasets (RDD)…
December 4, 2018