Spark Streaming with Kafka Example
Spark Streaming with Kafka Example Using Spark Streaming we can read from Kafka topic and…
2 Comments
March 17, 2019
Spark Streaming – Different Output modes explained
This article describes usage and differences between complete, append and update output modes in Apache Spark Streaming. outputMode describes what data is written to a data sink (console, Kafka e.t.c) when there is new data available in streaming input (Kafka, Socket, e.t.c)
March 17, 2019
Spark Streaming – Reading data from TCP Socket
Using Spark streaming we will see a working example of how to read data from…
March 16, 2019
Spark Streaming files from a directory
This article describes and provides an example of how to continuously stream or read a JSON file source from a folder, process it and write the data to another source
March 16, 2019
Spark SQL Batch Processing – Produce and Consume Apache Kafka Topic
This article describes Spark Batch Processing using Kafka Data Source. Unlike Spark structure stream processing, we may need to process batch jobs which reads the data from Kafka and writes the data to Kafka topic in batch mode. To do this we should use read instead of resdStream similarly write instead of writeStream on DataFrame
March 11, 2019
How to Create and Describe a Kafka topic
This article describes how to Create Kafka topic and explains how to describe newly created and all existing topics in Kafka. All massages to and from Apache Kafka will happen via topics. Topic is a first class citizen in Kafka.
March 9, 2019
Read & Write Avro files using Spark DataFrame
Spark provides built-in support to read from and write DataFrame to Avro file using "spark-avro"…
March 7, 2019
Using Avro Data Files From Spark SQL 2.3.x or earlier
On this post we will see several examples or usages of accessing Spark Avro file format using Spark 2.3 or earlier. Spark SQL supports loading and saving DataFrames from and to a Avro data files by using spark-avro library. spark-avro originally developed by databricks as a open source library which supports reading and writing data in Avro file format.
March 7, 2019
Spark DataFrame withColumn
Spark withColumn() is a DataFrame function that is used to add a new column to…
February 28, 2019
Spark withColumnRenamed to Rename Column
In Spark withColumnRenamed() is used to rename one column or multiple DataFrame column names. Depends…
February 27, 2019