PySpark Select Nested struct Columns
Using the PySpark select() and selectExpr() transformations, one can select the nested struct columns from…
Comments Off on PySpark Select Nested struct Columns
April 2, 2021
Spark SQL – Select Columns From DataFrame
In Spark SQL, select() function is used to select one or multiple columns, nested columns,…
March 14, 2021
PySpark Select Columns From DataFrame
In PySpark, select() function is used to select single, multiple, column by index, all columns…
August 14, 2020
PySpark StructType & StructField Explained with Examples
The StructType and StructField classes in PySpark are used to specify the custom schema to…
August 13, 2020
PySpark withColumnRenamed to Rename Column on DataFrame
Use PySpark withColumnRenamed() to rename a DataFrame column, we often need to rename one column…
January 11, 2020
Spark read JSON with or without schema
By default Spark SQL infer schema while reading JSON file, but, we can ignore this…
December 8, 2019
Photo by Shane Rounce on Unsplash
Spark Convert case class to Schema
Spark SQL provides Encoders to convert case class to the spark schema (struct StructType object),…
November 9, 2019
Spark Schema – Explained with Examples
Spark Schema defines the structure of the DataFrame which you can get by calling printSchema()…
November 8, 2019
Spark – Create a DataFrame with Array of Struct column
Problem: How to create a Spark DataFrame with Array of struct column using Spark and…
October 30, 2019
Spark – explode Array of Struct to rows
Problem: How to explode Array of StructType DataFrame columns to rows using Spark. Solution: Spark…
October 20, 2019