• SpringBoot-Kafka Tutorial

Introduction

• Apache Kafka Introduction
• Messages & Event Introduction
• Kafka Topic Introduction
• Ordering of Events
• Kafka Broker Introduction
• Kafka Server Installation

Kafka CLI

• Topic
• Producer
• Consumer

Kafka Producer

• Producer Microservice
• Acknowledgement & Retries
• Idempotent Producer

Kafka Consumer

• Consumer Microservice
• Handle Deserializer Errors
• Dead Leter Topic
• Exceptions & Retries
• Consumers in Consumer Group
• Idempotent Consumers
• Kafka Transactions

Apache Kafka CLI: Topics

From this lesson will learn how to use the CLI or terminal to create and update a topic and topic with partition.

What is an Apache Kafka Topic?

A Kafka Topic is a logical collection of messages that are produced and consumed by Kafka applications. A message is a piece of data that contains some information, such as an event, a record, or a notification.

A Kafka Topic has a unique name that identifies it in a Kafka cluster. A Kafka cluster is a group of servers, called brokers, that store and manage the topics and messages. A Kafka application can connect to a Kafka cluster and interact with the topics and messages.

What is Topic Partition?

A Kafka Topic is divided into one or more partitions, each of which stores a subset of messages in an ordered sequence. A partition is a physical unit of storage and processing in a Kafka broker. Each partition has a unique identifier, called a partition ID, that is assigned by the broker.

How to create topic?

To create a topic in a Kafka cluster, we can use the kafka-topics.sh command line tool, which is provided by Kafka. This tool allows us to perform various operations on topics, such as creating, deleting, listing, and describing topics.

To create a topic, we need to specify the following parameters:

1. --bootstrap-server: The address of one or more brokers in the Kafka cluster that we want to connect to. For example, localhost:9092.
2. --create: The flag that indicates that we want to create a topic.
3. --topic: The name of the topic that we want to create. For example, product-created-events-topic.
4. --partitions: The number of partitions that we want to create for the topic. For example, 3.
5. --replication-factor: The number of replicas that we want to create for each partition of the topic. For example, 2.

For example, the following command creates a topic named product-created-events-topic with 3 partitions and 2 replicas in a Kafka cluster with a broker at localhost:9092:
Create Command: kafka-topics.sh --bootstrap-server localhost:9092 --create --topic product-created-events-topic --partitions 3 --replication-factor 2
If the topic is created successfully, the tool will print a confirmation message, such as: Created topic product-created-events-topic.
If the topic already exists, or if there is an error, the tool will print an error message, such as: Topic 'product-created-events-topic' already exists.

How to update topic?

Sometimes, you may want to change the configuration and properties of an existing topic, such as changing the number of partitions, the replication factor, the retention policy, and other parameters. You can do this by using the kafka-topics.sh command line tool, which I introduced in the previous sections. This tool allows you to perform various operations on topics, such as creating, deleting, listing, and describing topics.

To update a topic, you need to specify the following parameters:

1. --bootstrap-server: The address of one or more brokers in the Kafka cluster that you want to connect to. For example, localhost:9092.
2. --alter: The flag that indicates that you want to update a topic.
3. --topic: The name of the topic that you want to update. For example, product-created-events-topic.
4. --config: A comma-separated list of key=value pairs, where key is the name of the parameter that you want to update, and value is the new value that you want to assign to it. For example, retention.ms=43200000.

For example, the following command updates the retention policy of the topic named product-created-events-topic to 12 hours in a Kafka cluster with a broker at localhost:9092:
Update Command: kafka-topics.sh --bootstrap-server localhost:9092 --alter --topic product-created-events-topic --config retention.ms=43200000
Update Partition Command: kafka-topics.sh --bootstrap-server localhost:9092 --alter --topic product-created-events-topic --partitions 6
Update replication Command: kafka-topics.sh --bootstrap-server localhost:9092 --alter --topic product-created-events-topic --replication-factor 3
Update retention Policy Command: kafka-topics.sh --bootstrap-server localhost:9092 --alter --topic product-created-events-topic --config retention.bytes=1073741824

How to delete topic?

Sometimes, you may want to delete a topic from a Kafka cluster, either because you no longer need it, or because you want to recreate it with different settings. You can do this by using the kafka-topics.sh command line tool, which I introduced in the previous sections. This tool allows you to perform various operations on topics, such as creating, updating, listing, and describing topics.

To delete a topic, you need to specify the following parameters:

1. --bootstrap-server: The address of one or more brokers in the Kafka cluster that you want to connect to. For example, localhost:9092.
2. --delete: The flag that indicates that you want to delete a topic.
3. --topic: The name of the topic that you want to delete. For example, product-created-events-topic.

For example, the following command deletes the topic named product-created-events-topic from a Kafka cluster with a broker at localhost:9092
kafka-topics.sh --bootstrap-server localhost:9092 --delete --topic product-created-events-topic

List & Describe kafka topics

List Command: ./bin/kafka-topics.sh --list --bootstrap-server localhost:9092
Describe Command: ./bin/kafka-topics.sh --describe --bootstrap-server localhost:9092

---