• SpringBoot-Kafka Tutorial

Introduction

• Apache Kafka Introduction
• Messages & Event Introduction
• Kafka Topic Introduction
• Ordering of Events
• Kafka Broker Introduction
• Kafka Server Installation

Kafka CLI

• Topic
• Producer
• Consumer

Kafka Producer

• Producer Microservice
• Acknowledgement & Retries
• Idempotent Producer

Kafka Consumer

• Consumer Microservice
• Handle Deserializer Errors
• Dead Leter Topic
• Exceptions & Retries
• Consumers in Consumer Group
• Idempotent Consumers
• Kafka Transactions

Apache Kafka for SpringBoot Microservices

In this course will learn how to build apache kafka event driven springboot microservices.

Defination: "Apache Kafka is distributed event streaming platform that is used to collect, process, store and integrate data at scale."

As a microservice developer Apache Kafka as a platform that can run a cluster of multiple servers, which can span across several data centers. It is a platform that enables developers to build event driven applications that can continuously exchange data with each other, and we can scale this microservices as needed to support larger traffic. If we have small number of users using our application, then we can scale down and start fewer instances of our microservices. If we have a large number of users using our application, then we can scale up and start large number of instances of our microservices.

Cluster: Cluster is nothing but a group of computers or applications that work together to achive a common goal. Clusters are built to eliminate a single point of failure, which can occur when software runs on a single computer.

Data Center: A data center is a physical location that stores computing machines and their related hardware equipment. It contains the computing infrastructure that IT systems require, such as servers, data storage drives, and network equipment. It is the physical facility that stores any company’s digital data.

Server: server is a virtual server that runs on a cloud computing platform and is accessible remotely over the internet. Cloud servers are a key component of cloud computing and allow users to access computing resources without the need to purchase and maintain physical server.

High level overview of how Kafka can be used with microservices.

Lets take, you have a Products Microservice using this as soon a new product is created, this microservice needs to inform other microservices by publishing an event. For example, Once a new product is created the Products Microservice will publish a ProductCreated Event in the form of Json payload to kafka and it is called as producer.

In microservices architecture producer is a microservice that publishes an event, which is on the right side of above diagram. I have three other microservices that want to be notified when a new product is created. And because these microservices are waiting to receive information about the newly created product in Kafka, they are called consumers.

Consumer is a second core component in application that uses Kafka in microservices architecture and it also consumes an events. The consumer microservices, they are dynamic in nature. They can scale up and down. When load on our system increases, we can start up more instances of this microservices. And when the load on our system decreases, we can shut down extra instances, IP addresses, and port numbers of this.

Microservices will keep on changing, and the products microservice that publishes events will never know the exact address of these consumer microservices and it will not know how many of them are running. They are completely location transparent to products, microservice and products.

Microservice cannot send information about newly created product to any of these microservices directly. Instead, it will send this information to Kafka Broker. You can think of Kafka Broker as a server that accepts information from Kafka Producer, and it stores it on its hard disk. To make our system more reliable, you will usually start more than one Kafka broker, and this way, if one Kafka broker is down, then another Kafka broker that is running can be used to service requests.

So broker is another very important architectural component in Kafka based application. And it has several very important responsibilities. And one of them is to accept published event and store it on its hard disk in Kafka.

Topic is another very important component in application architecture that uses Apache Kafka. You can think of Kafka topic as a very durable storage of events information inside the topic. It is stored on a hard disk and it is replicated to other servers in Kafka cluster, meaning that you will have multiple copies of your data created. This way if you lose one server, and if you lose all data that is stored on that server, you still have your data stored on other servers in Kafka cluster. And the good news is that your microservices, they will not even notice that one of the servers went down and lost all of the data. This is because you still have other servers running and all of these servers, they have copy of your data.

Now inside the topic. Publish events are stored in multiple partitions, and each of these little cells in topic partition stores an event. When a new event gets stored in a topic, it is stored inside of this little cells, and then it gets replicated across multiple Kafka servers. When a new event arrives, it gets stored into a new cell, and then it gets copied over to other Kafka servers.

So Kafka Topic is another very important element in Kafka architecture that you need to understand. You will see how topics work in details in following lessons, because we will talk more about topics and topic partitions. But for now, just remember that topics store events. If you are familiar with message queues, then Kafka topic is very similar to message queue, except that when event is consumed from Kafka topic, this event does not get deleted from topic right away. It stays in the topic for as long as needed so that other microservices can also read this event.

All right, so producer publishes an event. This event gets stored in Kafka topic. And as soon as this event gets successfully stored in Kafka topic, consumer microservices can read this event from that Kafka topic. And because events are stored in separate partitions, consumer microservices can read events from these partitions in parallel, and reading events in parallel makes our system work much faster, and it allows us to scale up our system because if needed, we can start more consumer microservices and each consumer will read events from its own partition in parallel. This way, we can process a very large amount of events much faster. But again, you will see how it works in details in different lessons.

---