You are currently viewing Spark SQL Left Outer Join with Example

Spark SQL Left Outer Join (left, left outer, left_outer) returns all rows from the left DataFrame regardless of the match found on the right Dataframe, when the join expression doesn’t match, it assigns null for that record and drops records from right where match not found.

In this Spark article, I will explain how to do Left Outer Join (left, leftouter, left_outer) on two DataFrames with Scala Example.

Before we jump into Spark Left Outer Join examples, first, let’s create an emp and dept DataFrames. here, column emp_id is unique on emp and dept_id is unique on the dept datasets and emp_dept_id from emp has a reference to dept_id on dept dataset.

import org.apache.spark.sql.SparkSession
val spark = SparkSession.builder

val emp = Seq((1,"Smith",-1,"2018","10","M",3000),
val empColumns = Seq("emp_id","name","superior_emp_id","year_joined",
import spark.sqlContext.implicits._
val empDF = emp.toDF(empColumns:_*)

val dept = Seq(("Finance",10),

val deptColumns = Seq("dept_name","dept_id")
val deptDF = dept.toDF("deptColumns")

This prints emp and dept DataFrame to the console.

# Emp Dataset
|emp_id|name    |superior_emp_id|year_joined|emp_dept_id|gender|salary|
|1     |Smith   |-1             |2018       |10         |M     |3000  |
|2     |Rose    |1              |2010       |20         |M     |4000  |
|3     |Williams|1              |2010       |10         |M     |1000  |
|4     |Jones   |2              |2005       |10         |F     |2000  |
|5     |Brown   |2              |2010       |40         |      |-1    |
|6     |Brown   |2              |2010       |50         |      |-1    |

# Dept Dataset
|Finance  |10     |
|Marketing|20     |
|Sales    |30     |
|IT       |40     |

Spark DataFrame Left Outer Join Example

Below is an example of how to use Left Outer Join (left, leftouter, left_outer) on Spark DataFrame.

empDF.join(deptDF,empDF("emp_dept_id") ===  deptDF("dept_id"),"left")
  empDF.join(deptDF,empDF("emp_dept_id") ===  deptDF("dept_id"),"leftouter")

From our dataset, emp_dept_id 6o doesn’t have a record on dept dataset hence, this record contains null on dept columns (dept_name & dept_id). and dept_id 30 from dept dataset dropped from the results. Below is the result of the above Join expression.

|emp_id|name    |superior_emp_id|year_joined|emp_dept_id|gender|salary|dept_name|dept_id|
|1     |Smith   |-1             |2018       |10         |M     |3000  |Finance  |10     |
|2     |Rose    |1              |2010       |20         |M     |4000  |Marketing|20     |
|3     |Williams|1              |2010       |10         |M     |1000  |Finance  |10     |
|4     |Jones   |2              |2005       |10         |F     |2000  |Finance  |10     |
|5     |Brown   |2              |2010       |40         |      |-1    |IT       |40     |
|6     |Brown   |2              |2010       |50         |      |-1    |null     |null   |

Using Spark SQL Left OUTER Join

Let’s see how to use Left Outer Join on Spark SQL expression, In order to do so first let’s create a temporary view for EMP and DEPT tables.


val joinDF2 = spark.sql("SELECT e.* FROM EMP e LEFT OUTER JOIN DEPT d ON e.emp_dept_id == d.dept_id")

This also returns the same output as above.


In this Spark article, you have learned Left Outer Join is used to get all rows from the left dataset regardless of the match found on the right dataset when the join expression doesn’t match.

Hope you Like it !!


Naveen Nelamali

Naveen Nelamali (NNK) is a Data Engineer with 20+ years of experience in transforming data into actionable insights. Over the years, He has honed his expertise in designing, implementing, and maintaining data pipelines with frameworks like Apache Spark, PySpark, Pandas, R, Hive and Machine Learning. Naveen journey in the field of data engineering has been a continuous learning, innovation, and a strong commitment to data integrity. In this blog, he shares his experiences with the data as he come across. Follow Naveen @ LinkedIn and Medium

Leave a Reply