Event-Driven Architecture (Kafka)

Build an asynchronous messaging pipeline with Apache Kafka where a producer and consumer are fully decoupled through a message broker.

Time: ~20 minutes Difficulty: Intermediate

Resources: This demo needs ~2GB RAM. Clean up other demos first: task clean:all

What You Will Learn

Running Apache Kafka and Zookeeper as StatefulSets in Kubernetes
The producer/consumer messaging pattern
How services communicate asynchronously without knowing about each other
Using persistent volumes to survive pod restarts without losing messages
Kafka topics, partitions, and consumer groups

Architecture

    +------------------+          +------------------+
    | Producer Service |          | Consumer Service |
    | (sends events    |          | (reads events    |
    |  every 5 sec)    |          |  from topic)     |
    +--------+---------+          +--------+---------+
             |                             ^
             |  produce                    |  consume
             v                             |
    +--------+-----------------------------+---------+
    |                 Kafka Broker                    |
    |              (topic: events)                    |
    +------------------------+------------------------+
                             |
                    +--------+--------+
                    |    Zookeeper    |
                    | (coordination) |
                    +-----------------+

The Producer sends timestamped event messages to a Kafka topic every 5 seconds. The Consumer reads those messages in real time. Neither service knows about the other. Kafka handles the routing, buffering, and ordering. Zookeeper manages Kafka’s cluster metadata.

Deploy

Step 1: Apply the namespace

kubectl apply -f demos/event-driven-kafka/manifests/namespace.yaml

Step 2: Deploy Zookeeper

kubectl apply -f demos/event-driven-kafka/manifests/zookeeper.yaml

Wait for Zookeeper to be ready:

kubectl get pods -n kafka-demo -w

Wait until zookeeper-0 shows Running and 1/1 ready.

Step 3: Deploy Kafka

kubectl apply -f demos/event-driven-kafka/manifests/kafka.yaml

Wait for Kafka to be ready:

kubectl get pods -n kafka-demo -w

Wait until kafka-0 shows Running and 1/1 ready. This may take 1-2 minutes as Kafka connects to Zookeeper.

Step 4: Deploy the producer

kubectl apply -f demos/event-driven-kafka/manifests/producer.yaml

Step 5: Deploy the consumer

kubectl apply -f demos/event-driven-kafka/manifests/consumer.yaml

Step 6: Wait for all pods

kubectl get pods -n kafka-demo -w

Wait until all four pods (zookeeper-0, kafka-0, producer, consumer) are Running.

Verify

# Check all pods are running
kubectl get pods -n kafka-demo

# Check persistent volume claims
kubectl get pvc -n kafka-demo

# Watch the producer sending messages
kubectl logs -f deploy/producer -n kafka-demo

# Watch the consumer receiving messages
kubectl logs -f deploy/consumer -n kafka-demo

You should see the producer logging messages like:

[producer] Sent: event-1: order-placed at 2026-04-11 14:30:05
[producer] Sent: event-2: order-placed at 2026-04-11 14:30:10

And the consumer printing those same messages as they arrive:

event-1: order-placed at 2026-04-11 14:30:05
event-2: order-placed at 2026-04-11 14:30:10

# List Kafka topics
kubectl exec -it kafka-0 -n kafka-demo -- \
  kafka-topics.sh --list --bootstrap-server localhost:9092

# Describe the events topic
kubectl exec -it kafka-0 -n kafka-demo -- \
  kafka-topics.sh --describe --topic events --bootstrap-server localhost:9092

What is Happening

manifests/
  namespace.yaml    # kafka-demo namespace
  zookeeper.yaml    # Zookeeper StatefulSet (1 replica) with 1Gi PVC
  kafka.yaml        # Kafka StatefulSet (1 replica) with 2Gi PVC
  producer.yaml     # Deployment sending events to topic every 5 seconds
  consumer.yaml     # Deployment reading events from topic

Zookeeper runs as a StatefulSet with persistent storage so cluster metadata survives restarts. Kafka also runs as a StatefulSet with its own PVC, connecting to Zookeeper for coordination. The KAFKA_CFG_ADVERTISED_LISTENERS is set to the StatefulSet’s stable DNS name (kafka-0.kafka.kafka-demo.svc.cluster.local) so clients can always reach the broker. The Producer uses kafka-console-producer.sh to write messages to the events topic. The Consumer uses kafka-console-consumer.sh with --from-beginning to read all messages from the topic.

The key insight is decoupling: the producer does not know about the consumer, and the consumer does not know about the producer. Kafka acts as a buffer between them. You can stop the consumer, let messages pile up, restart it, and it will catch up from where it left off.

Experiment

Stop the consumer and watch messages accumulate, then restart it:

kubectl scale deployment consumer --replicas=0 -n kafka-demo
# Wait 30 seconds while the producer keeps sending
kubectl scale deployment consumer --replicas=1 -n kafka-demo
kubectl logs -f deploy/consumer -n kafka-demo
# The consumer catches up on all missed messages

Scale the producer to send from multiple instances:

kubectl scale deployment producer --replicas=3 -n kafka-demo
kubectl logs -f deploy/consumer -n kafka-demo
# Messages from all three producers appear interleaved

Interact with Kafka directly from the broker pod:

kubectl exec -it kafka-0 -n kafka-demo -- \
  kafka-console-producer.sh --broker-list localhost:9092 --topic events
# Type a message and press Enter, then Ctrl+C

Create a new topic with multiple partitions:

kubectl exec -it kafka-0 -n kafka-demo -- \
  kafka-topics.sh --create --topic orders \
  --partitions 3 --replication-factor 1 \
  --bootstrap-server localhost:9092

Check consumer group offsets:

kubectl exec -it kafka-0 -n kafka-demo -- \
  kafka-consumer-groups.sh --list --bootstrap-server localhost:9092

Cleanup

kubectl delete namespace kafka-demo

Note: Deleting the namespace also deletes the PVCs. If you want to keep the data, delete deployments and statefulsets individually instead.

Next Step

Move on to EFK Logging to learn how to collect and search logs from across your cluster.