You are currently viewing Spark – Sort by column in descending order?

In order to sort by descending order in Spark DataFrame, we can use desc property of the Column class or desc() sql function. In this article, I will explain the sorting dataframe by using these approaches on multiple columns.

1. Using sort() for descending order

First, let’s do the sort.

// Using sort() for descending order

Now, let’s do the sort using desc property of Column class and In order to get column class we use col() SQL function

import org.apache.spark.sql.functions.col

Finally, let’s see how desc() SQL function by importing org.apache.spark.sql.functions.desc

import org.apache.spark.sql.functions.desc

2. Using orderBy() for descending

Alternatively, we can also use orderBy() function of the DataFrame to sort by descending order. All examples explained with sort() also works here.

// Using orderBy() for descending
import org.apache.spark.sql.functions.col
import org.apache.spark.sql.functions.desc

3. Using SQL to sort

And, we can also use SQL expression to sort by descending order.

// Using SQL to sort
spark.sql(" select employee_name,desc('department'),desc('state'),salary,age,bonus from DEPT")

Happy Learning !!

Naveen Nelamali

Naveen Nelamali (NNK) is a Data Engineer with 20+ years of experience in transforming data into actionable insights. Over the years, He has honed his expertise in designing, implementing, and maintaining data pipelines with frameworks like Apache Spark, PySpark, Pandas, R, Hive and Machine Learning. Naveen journey in the field of data engineering has been a continuous learning, innovation, and a strong commitment to data integrity. In this blog, he shares his experiences with the data as he come across. Follow Naveen @ LinkedIn and Medium