Steps To Connect Kafka Running In Docker

Step 1: First create a docker network .

docker network create kafka-network

Step 2: Run the Zookeeper container using the docker network created in the step 1 (kafka-network).

docker run -d --name zookeeper --network kafka-network -p 2181:2181 zookeeper:latest

Step 3: Run the Kafka container using the docker network created in step 1 .

docker run -d --name kafka --network kafka-network -p 9092:9092 \
    -e KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092 \
    -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9092 \
    -e KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 \
    -e KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181 \
    wurstmeister/kafka:latest

Step 4: Create a topic inside Kafka container . Here you have to first enter into the Kafka container and then create a topic .

Step 5: Check if your containers are running or not .

Step 6: Now create two different directory producer and consumer . Then inside producer create a file producer.go and in consumer directory consumer.go.

producer.go

package main

import (
    "bufio"
    "context"
    "fmt"
    "os"
    "strings"

    "github.com/segmentio/kafka-go"
)

func main() {
    writer := kafka.NewWriter(kafka.WriterConfig{
        Brokers: []string{"localhost:9092"},
        Topic:   "gfg-test-topic",
    })

    scanner := bufio.NewScanner(os.Stdin)

    fmt.Println("Enter message:")
    for scanner.Scan() {
        message := scanner.Text()
        if strings.ToLower(message) == "exit" {
            break
        }
        err := writer.WriteMessages(context.Background(),
            kafka.Message{Value: []byte(message)},
        )
        if err != nil {
            fmt.Println("Error :", err)
        }
    }
    writer.Close()
}

consumer.go

package main

import (
    "context"
    "fmt"
    "github.com/segmentio/kafka-go"
)

func main() {
    reader := kafka.NewReader(kafka.ReaderConfig{
        Brokers: []string{"localhost:9092"},
        Topic:   "gfg-test-topic",
        GroupID: "my-group",
    })
    defer reader.Close()

    for {
        msg, err := reader.ReadMessage(context.Background())
        if err != nil {
            panic(err)
        }
        fmt.Printf("Received message: %s\n", string(msg.Value))
    }
}

After you have written the code then you can run the below command to download all the imported packages .

go mod tidy

Step 7: Run the go files in two different terminals . Then you can write anything in the producer terminal . All the messages you will receive in the consumer terminal . You will real time data streaming .

go run producer.go

go run consumer.go

Docker and Kafka are two important tools used in today’s digital world. Docker makes an easy deployment process by encapsulating applications into containers while Kafka provides a distributed event-streaming platform for building real-time data pipelines and data streaming applications. Here in this guide, I will first discuss what is Docker. Then I will discuss Kafka and its components. Finally, I will walk you through the various steps to run Kafka using a Docker container and also perform real-time data streaming using some Golang codes.