In this article, I will download and install Apache Kafka. Then I create a Kafka topic, produce messages to it, and consume them. I use the terminal to perform all the operations.
Kafka needs Java to run. If you don't have Java installed, now is the time to do so.
The example commands are in Mac/Linux systems. For Windows, use the
.bat extension in place of
How to install Kafka
Navigate to Apache Kafka download page and download the Kafka binary.
At the time of this article, the latest release was
3.3.2. Make sure to download the binaries, not the sources.
Then extract it to a folder; it is kafka in my case. I removed the version information for clarity.
That's it. Now I have Kafka installed on my machine.
How to start Kafka
Navigate to the folder where Kafka resides. In my case, that folder is
To start the Kafka server, first, I need to start the Kafka zookeeper.
How to start Kafka zookeeper
Zookeeper monitors brokers belonging to the Kafka cluster.
It appoints and manages leaders for partitions in case of resource failures and a lot of housekeeping.
Note that the zookeeper is being sunset from the new versions of Kafka, but those changes are still not production ready. Here is how to start zookeeper. Make sure to navigate to the kafka folder.
That will start the zookeeper on port 2181. I can change that port by editing the
zookeeper.properties file residing in the
How to start the Kafka server
That is also called the broker. Before starting the Kafka server, I must specify the hostname for listeners to listen. Open the
server.properties file and change the
Then start the server.
That will start the Kafka broker on port 9092. I can change that port by editing the
server.properties file residing in the
How to create a topic in Kafka
I need a topic to publish messages on the broker. The below command creates the topic
TEST-TOPIC on the broker running on port 9092.
bin/kafka-topics.sh --create --topic TEST-TOPIC --bootstrap-server localhost:9092
Here is how to verify the topic creation.
bin/kafka-topics.sh --list --bootstrap-server localhost:9092
How to publish to a topic in Kafka
Producers publish messages to topics. I can create a simple producer by using Kafka's built-in producer script.
bin/kafka-console-producer.sh --topic TEST-TOPIC --broker-list localhost:9092
The above command enables me to publish messages to the topic
How to consume messages from a topic in Kafka
Kafka has a built-in script to create consumers to consume messages through the terminal.
bin/kafka-console-consumer.sh --topic TEST-TOPIC --bootstrap-server localhost:9092
If you want to consume all the messages in the topic, append
--from-beginning to the above command.
bin/kafka-console-consumer.sh --topic TEST-TOPIC --bootstrap-server localhost:9092 --from-beginning