On this post we will see several examples or usages of accessing Spark Avro file format using Spark 2.3 or earlier. Spark SQL supports loading and saving DataFrames from and to a Avro data files by using spark-avro library. spark-avro originally developed by databricks as a open source library which supports reading and writing data in Avro file format.
Spark withColumn() is a DataFrame function that is used to add a new column to…
In Spark withColumnRenamed() is used to rename one column or multiple DataFrame column names. Depends…
In this tutorial, we will learn what is Apache Parquet?, It's advantages and how to…
Like SQL "case when" statement and Swith statement from popular programming languages, Spark SQL Dataframe also supports similar syntax using "when otherwise" or we can also use "case when" statement.
Spark RDD can be created in several ways, for example, It can be created by…
In Spark, createDataFrame() and toDF() methods are used to create a DataFrame manually, using these…
While running spark jobs, you may come across java.io.IOException: org.apache.spark.SparkException: Failed to get broadcast_0_piece0 of broadcast_0 error with below stack trace. This error occurs when you try to create multiple spark contexts. In another case, when I tried to crate SparkContext and Streamingcontext from scratch I was getting this error. Below is the code how to create StreamingContext from existing Sparkcontext.
This article describes and provides scala example on how to Pivot Spark DataFrame ( creating Pivot tables ) and Unpivot back. Pivoting is used to rotate the data from one column into multiple columns. It is an aggregation where one of the grouping columns values transposed into individual columns with distinct data.
Some times you see this error in zookeeper logs, Don't worry about this error as…