What is Spark Streaming Checkpoint?
What is the Spark or PySpark Streaming Checkpoint? As the Spark streaming application must operate…
Comments Off on What is Spark Streaming Checkpoint?
November 21, 2022
Apache Kafka Article :
Spark Streaming – Kafka messages in Avro format
This article describes Spark Structured Streaming from Kafka in Avro file format and usage of…
March 23, 2019
Spark SQL Batch Processing – Produce and Consume Apache Kafka Topic
This article describes Spark Batch Processing using Kafka Data Source. Unlike Spark structure stream processing, we may need to process batch jobs which reads the data from Kafka and writes the data to Kafka topic in batch mode. To do this we should use read instead of resdStream similarly write instead of writeStream on DataFrame
March 11, 2019
How to Create and Describe a Kafka topic
This article describes how to Create Kafka topic and explains how to describe newly created and all existing topics in Kafka. All massages to and from Apache Kafka will happen via topics. Topic is a first class citizen in Kafka.
March 9, 2019
Zookeeper and Kafka error KeeperErrorCode=NodeExists
Some times you see this error in zookeeper logs, Don't worry about this error as…
January 27, 2019
Kafka Delete Topic and its messages
Kafka Delete Topic - Every message Apache Kafka receives stores it in log and by…
January 25, 2019
How to Setup a Kafka Cluster (step-by-step)
This article provides step by step instructions on how to install, setup, and run Apache…
January 11, 2019
Kafka consumer and producer example with a custom serializer
Kafka allows us to create our own serializer and deserializer so that we can produce and consume different data types like Json, POJO e.t.c. In this post will see how to produce and consumer User pojo object. To stream pojo objects one need to create custom serializer and deserializer.
January 4, 2019
Apache Kafka Producer and Consumer in Scala
This article explains how to write Kafka Producer and Consumer example in Scala. Producer sends messages to Kafka topics in the form of records, a record is a key-value pair along with topic name and consumer receives a messages from a topic.
January 4, 2019