Spark Read and Write Apache Parquet
In this tutorial, we will learn what is Apache Parquet?, It's advantages and how to…
6 Comments
February 6, 2019
Spark SQL “case when” and “when otherwise”
Like SQL "case when" statement and Swith statement from popular programming languages, Spark SQL Dataframe also supports similar syntax using "when otherwise" or we can also use "case when" statement.
February 5, 2019
Photo by Deva Darshan on Unsplash
Different ways to create Spark RDD
Spark RDD can be created in several ways, for example, It can be created by…
February 4, 2019
Spark Create DataFrame with Examples
In Spark, createDataFrame() and toDF() methods are used to create a DataFrame manually, using these…
February 1, 2019
java.io.IOException: org.apache.spark.SparkException: Failed to get broadcast_0_piece0 of broadcast_0
While running spark jobs, you may come across java.io.IOException: org.apache.spark.SparkException: Failed to get broadcast_0_piece0 of broadcast_0 error with below stack trace. This error occurs when you try to create multiple spark contexts. In another case, when I tried to crate SparkContext and Streamingcontext from scratch I was getting this error. Below is the code how to create StreamingContext from existing Sparkcontext.
January 31, 2019
Spark Pivot Table DataFrame
How to Pivot and Unpivot a Spark Data Frame
This article describes and provides scala example on how to Pivot Spark DataFrame ( creating Pivot tables ) and Unpivot back. Pivoting is used to rotate the data from one column into multiple columns. It is an aggregation where one of the grouping columns values transposed into individual columns with distinct data.
January 29, 2019
Zookeeper and Kafka error KeeperErrorCode=NodeExists
Some times you see this error in zookeeper logs, Don't worry about this error as…
January 27, 2019
Kafka Delete Topic and its messages
Kafka Delete Topic - Every message Apache Kafka receives stores it in log and by…
January 25, 2019
How to Setup a Kafka Cluster (step-by-step)
This article provides step by step instructions on how to install, setup, and run Apache…
January 11, 2019
Spark Step-by-Step Setup on Hadoop Yarn Cluster
This post explains how to setup Apache Spark and run Spark applications on the Hadoop…
January 10, 2019