Spark Get Current Number of Partitions of DataFrame
While working with Spark/PySpark we often need to know the current number of partitions on…
Comments Off on Spark Get Current Number of Partitions of DataFrame
April 5, 2021
Spark Check Column Present in DataFrame
You can get all columns of a DataFrame as an Array[String] by using columns attribute…
Comments Off on Spark Check Column Present in DataFrame
April 4, 2021
PySpark Select Top N Rows From Each Group
In PySpark Find/Select Top N rows from each group can be calculated by partition the…
Comments Off on PySpark Select Top N Rows From Each Group
April 4, 2021
PySpark Find Maximum Row per Group in DataFrame
In PySpark, finding the maximum (max) row per group can be calculated using the Window.partition()…
Comments Off on PySpark Find Maximum Row per Group in DataFrame
April 3, 2021
PySpark Select First Row of Each Group?
In PySpark select/find the first row of each group within a DataFrame can be get…
April 3, 2021
PySpark Check Column Exists in DataFrame
Problem: I have a PySpark DataFrame and I would like to check if a column…
Comments Off on PySpark Check Column Exists in DataFrame
April 3, 2021
AttributeError: ‘DataFrame’ object has no attribute ‘map’ in PySpark
Problem: In PySpark I am getting error AttributeError: 'DataFrame' object has no attribute 'map' when…
Comments Off on AttributeError: ‘DataFrame’ object has no attribute ‘map’ in PySpark
April 3, 2021
PySpark Convert DataFrame to RDD
PySpark dataFrameObject.rdd is used to convert PySpark DataFrame to RDD; there are several transformations that…
Comments Off on PySpark Convert DataFrame to RDD
April 3, 2021
PySpark mapPartitions() Examples
Similar to map() PySpark mapPartitions() is a narrow transformation operation that applies a function to…
Comments Off on PySpark mapPartitions() Examples
April 3, 2021
PySpark Select Nested struct Columns
Using PySpark select() transformations one can select the nested struct columns from DataFrame. While working…
Comments Off on PySpark Select Nested struct Columns
April 2, 2021
- Go to the previous page
- 1
- …
- 83
- 84
- 85
- 86
- 87
- 88
- 89
- …
- 127
- Go to the next page