PySpark – Loop/Iterate Through Rows in DataFrame

PySpark provides map(), mapPartitions() to loop/iterate through rows in RDD/DataFrame to perform the complex transformations, and these two returns the same number of records as in the original DataFrame but the number of columns could be different (after add/update). PySpark also provides foreach() & foreachPartitions() actions to loop/iterate through each…

Continue Reading PySpark – Loop/Iterate Through Rows in DataFrame

Spark map() vs flatMap() with Examples

What is the difference between Spark map() vs flatMap() is a most asked interview question, if you are taking an interview on Spark (Java/Scala/PySpark), so let's understand the differences with examples? Regardless of an interview, you have to know the differences as this is also one of the most used…

Continue Reading Spark map() vs flatMap() with Examples

Spark map() Transformation

Spark map() is a transformation operation that is used to apply the transformation on every element of RDD, DataFrame, and Dataset and finally returns a new RDD/Dataset respectively. In this article, you will learn the syntax and usage of the map() transformation with an RDD & DataFrame example. Transformations like…

Continue Reading Spark map() Transformation

PySpark map() Transformation

PySpark map (map()) is an RDD transformation that is used to apply the transformation function (lambda) on every element of RDD/DataFrame and returns a new RDD. In this article, you will learn the syntax and usage of the RDD map() transformation with an example and how to use it with…

Continue Reading PySpark map() Transformation

Spark map() vs mapPartitions() with Examples

Spark map() and mapPartitions() transformations apply the function on each element/record/row of the DataFrame/Dataset and returns the new DataFrame/Dataset, In this article, I will explain the difference between map() vs mapPartitions() transformations, their syntax, and usages with Scala examples. map() - Spark map() transformation applies a function to each row…

Continue Reading Spark map() vs mapPartitions() with Examples