Apache Kafka Broker Cluster Setup on Docker

High availability and network balance of a service is now recommended an approach for a server. To work with this approach an application is run under a cluster configuration. Apache Kafka is a server which provides real-time data pipeline and messaging for the application. To Serve the high availability of the Kafka we set up the Kafka broker cluster so that there is a redundancy of the service and high availability with proper load balance. If we talk about a cluster then, a cluster must provide these three features.

• Zero downtime.
• Work as a single unit.
• Data Redundancy.

In this blog, I will tell you how to set up an Apache Kafka broker cluster on Docker so that serving of data can be continuous. Why we are setting this on docker? Because it is easier to set up a node without any problem and without depending on the base environment.
For now, we are setting up the environment with a three node cluster, as it is a basic requirement so that a cluster can choose a master node to manage the environment. So we require three servers to set up the cluster.
 
Prerequisites

• Docker ( to run containers)
• Docker-compose (to run a docker compose file)
• The network connection between the servers.

A Kafka broker cluster also consists of Zookeeper to manage the nodes master and kept metadata between them that which node is containing data and how many replicas are there. Each Node contains both Kafka and zookeeper as a pair. All zookeeper nodes will be connected to each other, also all Kafka servers will be connected to all the zookeeper nodes.

Here is the docker-compose file to set up the cluster.

version: '2'

services:

 zoo:

    image: zookeeper:3.4

    restart: always

    ports:

     - 2181:2181

     - 2888:2888

     - 3888:3888

    environment:

     ZOO_MY_ID: 1

     ZOO_SERVERS: server.1=0.0.0.0:2888:3888 server.2=<server 2 IP>:2888:3888 server.3=<server 3 IP>:2888:3888

 kafka:

    image: wurstmeister/kafka:2.11-1.0.1

    restart: always

    ports:

     - 9092:9092

    environment:

     KAFKA_ADVERTISED_HOST_NAME: <host IP>

     KAFKA_PORT_NUMBER: 9092

     KAFKA_ZOOKEEPER_CONNECT: <server 1 IP>:2181,<server 2 IP> :2181,<server 3 IP>:2181

     KAFKA_ADVERTISED_LISTENERS: INSIDE://:19092,OUTSIDE://<Host IP>:9092

     KAFKA_LISTENERS: INSIDE://:19092,OUTSIDE://:9092

     KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT

     KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE

     KAFKA_BROKER_ID: 1

The ZOO_MY_ID and KAFKA_BROKER_ID value is unique from the other server. For server1 we are using 1, for server2 we are using 2 and for server3 we are using 3 ID for both ZOO_MY_ID and KAFKA_BROKER_ID.

Change the <server IP> according to that server host IP.

After creating docker-compose.yaml file in all three servers run the service of both kafka and Zookeeper by using folloqing command.

docker-compose up -d zoo kafka

Now the kafka Broker cluster is fully setup.

To connect to the kafka cluster you have to provide all the three kafka server IPs with broker port i.e. 9092 in comma separated form so that the listener and producer can connect to all the brokers.