Kafka Delete Topic – Every message Apache Kafka receives stores it in log and by default, it keeps the messages for 168 hrs which is 7 days. To delete the topic or all its messages can be done in several ways and the rest of the article explains these.
Before you try the below examples, make sure you have at least one Kafka topic created and have some messages in it. And when you run these commands on production be extra cautious as these are permanent and you can’t retrieve the messages back.
- Delete all messages from the Kafka topic
- Delete Kafka topic and re-create
- Manually delete the topic messages from logs
1. Delete all messages from the topic
Apache Kafka distribution comes with bin/kafka-configs.sh script which provides many useful options to modify Kafka configuration. Among different options, we will use “–alter” and “–add-config retention.ms” options to temporarily change the retention policy to 1 sec which will delete all messages from a topic.
First, let’s run below kafka-configs.sh command to get the retention value.
bin/kafka-configs.sh --zookeeper localhost:2181 \
--alter --entity-type topics \
--add-config retention.ms=1000 \
--entity-name text_topic
Verify if the retention policy value changed by running the below command.
bin/kafka-configs.sh --zookeeper localhost:2181 \
--entity-type topics \
--describe \
--entity-name text_topic
Wait for a few seconds, It should have deleted all your old messages from the Kafka topic. Now remove the retention policy using "--delete-config" config which will set it back to default.
Note: This is a very important step, make sure you run this without fail. Otherwise, you would be losing all future messages.
bin/kafka-configs.sh --zookeeper localhost:2181 \
--alter --entity-type topics \
--delete-config retention.ms \
--entity-name text_topic
2. Delete Kafka topic and re-create
In the recent versions of Apache’s Kafka, deleting a topic is easy. You just need to set one property in the configuration to ‘true’, and issue a command to delete a topic.
Before we remove an existing topic, first get the partition and replica of the existing topic as you would need these to re-create with the same configuration. You can get this information by running “kafka-topics.sh“ script with option “–describe” on topic “text_topic”
bin/kafka-topics.sh --zookeeper localhost:2181 \
--topic text_topic \
--describe
Topic “text_topic” has 1 replication factor and 1 partition.
a. Remove Kafka topic
First, log in to the server where Kafka runs and go to your config directory which is located under Kafka home/installation “$KAFKA_HOME/config/”. Here, you’ll see a server.properties file, open the properties file in your favorite text editor, add “delete.topic.enable=true” line, or change the value of the property to true if the property already present:
delete.topic.enable=true
Now, got to bin directory and run the kafka-topics.sh command with “–delete” option to remove topic “text_topic”
bin/kafka-topics.sh --zookeeper localhost:2181 \
--topic text_topic \
--delete
This removes text_topic from all Kafka brokers partitions.
b. Re-create Kafka topic
Run “bin/kafka-topics.sh” command to re-create the topic with replication and partition details you got it from earlier command. Refer Creating Kafka Topic article for more detail descriptions with examples.
bin/kafka-topics.sh --zookeeper localhost:2181 \
--create \
--topic text_topic \
--replication-factor 1 --partitions 1
This creates Kafka topic text_topic with replication-factor 1 and partitions 1
3. Manually remove the data from Kafka logs.
As I specified in the beginning, Kafka stores all messages in logs in their respective nodes at the location specified in log.dir. We should remove all these messages for a topic from all nodes. Below are the steps to remove.
- Stop zookeeper and Kafka broker from all nodes.
- Clean logs from all nodes. it stores log files at /tmp/kafka-logs/MyTopic-0 where /tmp/kafka-logs is specified by the log.dir attribute
- Restart zookeeper and broker from all nodes.
Conclusion:
You have learned different ways to delete or purge a Kafka topic by using kakfa-config.sh and kafka-topics.sh also, learned how to manually remove messages from Kafka logs.
Happy Learning !!
