Spark Cast String Type to Integer Type (int)

In Spark SQL, in order to convert/cast String Type to Integer Type (int), you can use cast() function of Column class, use this function with withColumn(), select(), selectExpr() and SQL expression. This function takes the argument string representing the type you wanted to convert or any type that is a…

Continue Reading Spark Cast String Type to Integer Type (int)

PySpark Convert String Type to Double Type

In PySpark SQL, using the cast() function you can convert the DataFrame column from String Type to Double Type or Float Type. This function takes the argument string representing the type you wanted to convert or any type that is a subclass of DataType. Key points cast() - cast() is…

Continue Reading PySpark Convert String Type to Double Type

Spark select() vs selectExpr() with Examples

Spark SQL select() and selectExpr() are used to select the columns from DataFrame and Dataset, In this article, I will explain select() vs selectExpr() differences with examples. Both these are transformation operations and return a new DataFrame or Dataset based on the usage of UnTyped and Type columns. Spark select()…

Continue Reading Spark select() vs selectExpr() with Examples

Pyspark – Get substring() from a column

In PySpark, the substring() function is used to extract the substring from a DataFrame string column by providing the position and length of the string you wanted to extract. In this tutorial, I have explained with an example of getting substring of a column using substring() from pyspark.sql.functions and using…

Continue Reading Pyspark – Get substring() from a column

PySpark – Cast Column Type With Examples

In PySpark, you can cast or change the DataFrame column data type using cast() function of Column class, in this article, I will be using withColumn(), selectExpr(), and SQL expression to cast the from String to Int (Integer Type), String to Boolean e.t.c using PySpark examples. Note that the type…

Continue Reading PySpark – Cast Column Type With Examples

Spark – How to Change Column Type?

To change the Spark SQL DataFrame column type from one data type to another data type you should use cast() function of Column class, you can use this on withColumn(), select(), selectExpr(), and SQL expression. Note that the type which you want to convert to should be a subclass of…

Continue Reading Spark – How to Change Column Type?