PySpark lit() – Add Literal or Constant to DataFrame

PySpark SQL functions lit() and typedLit() are used to add a new column to DataFrame by assigning a literal or constant value. Both these functions return Column type as return type. Both of these are available in PySpark by importing pyspark.sql.functions First, let's create a DataFrame. import pyspark from pyspark.sql…

Continue Reading PySpark lit() – Add Literal or Constant to DataFrame

Spark SQL Built-in Standard Functions

Spark SQL provides several built-in standard functions org.apache.spark.sql.functions to work with DataFrame/Dataset and SQL queries. All these Spark SQL Functions return org.apache.spark.sql.Column type. In order to use these SQL Standard Functions, you need to import below packing into your application. import org.apache.spark.sql.functions._ Spark also includes more built-in functions that are…

Continue Reading Spark SQL Built-in Standard Functions