On this post we will see several examples or usages of accessing Spark Avro file format using Spark 2.3 or earlier. Spark SQL supports loading and saving DataFrames from and to a Avro data files by using spark-avro library. spark-avro originally developed by databricks as a open source library which supports reading and writing data in Avro file format.
Spark withColumn() is a DataFrame function that is used to add a new column to…
In Spark withColumnRenamed() is used to rename one column or multiple DataFrame column names. Depends…
In this tutorial, we will learn what is Apache Parquet?, It's advantages and how to…
Like SQL "case when" statement and Swith statement from popular programming languages, Spark SQL Dataframe also supports similar syntax using "when otherwise" or we can also use "case when" statement.
Spark RDD can be created in several ways, for example, It can be created by…
In Spark, createDataFrame() and toDF() methods are used to create a DataFrame manually, using these…
This article describes and provides scala example on how to Pivot Spark DataFrame ( creating Pivot tables ) and Unpivot back. Pivoting is used to rotate the data from one column into multiple columns. It is an aggregation where one of the grouping columns values transposed into individual columns with distinct data.
This post explains how to setup Apache Spark and run Spark applications on the Hadoop…
In Spark or PySpark SparkSession object is created programmatically using SparkSession.builder() and if you are…