This article describes Spark Batch Processing using Kafka Data Source. Unlike Spark structure stream processing, we may need to process batch jobs which reads the data from Kafka and writes the data to Kafka topic in batch mode. To do this we should use read instead of resdStream similarly write instead of writeStream on DataFrame
Spark provides built-in support to read from and write DataFrame to Avro file using "spark-avro"…
On this post we will see several examples or usages of accessing Spark Avro file format using Spark 2.3 or earlier. Spark SQL supports loading and saving DataFrames from and to a Avro data files by using spark-avro library. spark-avro originally developed by databricks as a open source library which supports reading and writing data in Avro file format.
Spark withColumn() is a DataFrame function that is used to add a new column to…
In Spark withColumnRenamed() is used to rename one column or multiple DataFrame column names. Depends…
In this tutorial, we will learn what is Apache Parquet?, It's advantages and how to…
Like SQL "case when" statement and Swith statement from popular programming languages, Spark SQL Dataframe also supports similar syntax using "when otherwise" or we can also use "case when" statement.
Spark RDD can be created in several ways, for example, It can be created by…
In Spark, createDataFrame() and toDF() methods are used to create a DataFrame manually, using these…
While running spark jobs, you may come across java.io.IOException: org.apache.spark.SparkException: Failed to get broadcast_0_piece0 of broadcast_0 error with below stack trace. This error occurs when you try to create multiple spark contexts. In another case, when I tried to crate SparkContext and Streamingcontext from scratch I was getting this error. Below is the code how to create StreamingContext from existing Sparkcontext.