How to install and run Apache Kafka

In this article, I will download and install Apache Kafka. Then I create a Kafka topic, produce messages to it, and consume them. I use the terminal to perform all the operations.

Pre-requisites

Kafka needs Java to run. If you don't have Java installed, now is the time to do so. The example commands are in Mac/Linux systems. For Windows, use the .bat extension in place of .sh.

How to install Kafka

Navigate to Apache Kafka download page and download the Kafka binary. At the time of this article, the latest release was 3.3.2. Make sure to download the binaries, not the sources. Then extract it to a folder; it is kafka in my case. I removed the version information for clarity.
That's it. Now I have Kafka installed on my machine.

How to start Kafka

Navigate to the folder where Kafka resides. In my case, that folder is kafka. To start the Kafka server, first, I need to start the Kafka zookeeper.

How to start Kafka zookeeper

Zookeeper monitors brokers belonging to the Kafka cluster. It appoints and manages leaders for partitions in case of resource failures and a lot of housekeeping.
Note that the zookeeper is being sunset from the new versions of Kafka, but those changes are still not production ready. Here is how to start zookeeper. Make sure to navigate to the kafka folder.

bin/zookeeper-server-start.sh config/zookeeper.properties

That will start the zookeeper on port 2181. I can change that port by editing the zookeeper.properties file residing in the kafka/config.

How to start the Kafka server

That is also called the broker. Before starting the Kafka server, I must specify the hostname for listeners to listen. Open the server.properties file and change the listeners property.

listeners=PLAINTEXT://localhost:9092

Then start the server.

bin/kafka-server-start.sh config/server.properties

That will start the Kafka broker on port 9092. I can change that port by editing the server.properties file residing in the kafka/config.

How to create a topic in Kafka

I need a topic to publish messages on the broker. The below command creates the topic TEST-TOPIC on the broker running on port 9092.

bin/kafka-topics.sh --create --topic TEST-TOPIC --bootstrap-server localhost:9092

Here is how to verify the topic creation.

bin/kafka-topics.sh --list --bootstrap-server localhost:9092

How to publish to a topic in Kafka

Producers publish messages to topics. I can create a simple producer by using Kafka's built-in producer script.

bin/kafka-console-producer.sh --topic TEST-TOPIC --broker-list localhost:9092

The above command enables me to publish messages to the topic TEST-TOPIC.

How to consume messages from a topic in Kafka

Kafka has a built-in script to create consumers to consume messages through the terminal.

bin/kafka-console-consumer.sh --topic TEST-TOPIC --bootstrap-server localhost:9092

If you want to consume all the messages in the topic, append --from-beginning to the above command.

bin/kafka-console-consumer.sh --topic TEST-TOPIC --bootstrap-server localhost:9092 --from-beginning