PySpark DataFrame Tutorial

This PySpark DataFrame Tutorial will help you start understanding and using PySpark DataFrame API with python examples and All DataFrame examples provided in this Tutorial were tested in our development environment and are available at PySpark-Examples GitHub project for easy reference.

Examples I used in this tutorial to explain DataFrame concepts are very simple, easy to practice for beginners who are enthusiastic to learn PySpark DataFrame.

This tutorial is a work in progress, If you are looking for a specific topic that can’t find here, please don’t disappoint and I would highly recommend searching using the search option on top of the page as I’ve already covered hundreds of PySpark DataFrame concepts with real-time examples and you might get lucky finding it.

In case you still can’t find it, please send me the topic you are looking for in the comments or Q&A section and I will try my best to cover it ASAP.

Finally, subscribe by providing your e-mail to get more updates.

Table of Contents

• DataFrame Introduction
  • What is PySpark DataFrame
  • RDD vs DataFrame
  • DataFrame Advantages
• Creating PySpark DataFrame
  • Create DataFrame
  • Creating empty DataFrame
  • Convert RDD to DataFrame
• Working with DataFrame columns
  • Add column
  • Rename column
  • Update column
  • Drop column
  • Case when and when otherwise
• Filtering rows on DataFrame
  • Using filter & where methods
  • Using relation operators
  • Using conditional operators
• PySpark StructType and schema
  • Programmatically specifying schema
  • Loading schema from JSON
• DataFrame Transformations
  • Map transformations
  • Pivot & Unpivot
  • Handling nulls
  • DataFrame group by
• DataFrame Joins
  • Join Types
  • Inner join
  • Outer join
  • Left outer join
  • Right outer join
  • Cross join
  • Self join
• DataFrame Union
  • Union and Union All
• PySpark SQL Functions
  • String functions
  • Math functions
  • Date & Time Functions
  • Array & Map functions
  • Sorting Functions
  • Aggregate Functions
  • Window Functions
• PySpark Datasource API
  • Read & write CSV
  • Read & Write JSON
  • Read & write Avro
  • Read & write parquet
  • Read & write XML
  • Read & Write HBase tables