EFK Logging Stack

Centralized logging with Elasticsearch, Fluent Bit, and Kibana to collect, store, and visualize container logs.

Time: ~20 minutes Difficulty: Intermediate

Resources: This demo needs ~3GB RAM. Clean up other demos first: task clean:all

What You Will Learn

• Deploying a centralized logging stack on Kubernetes
• Using Fluent Bit as a lightweight log collector (DaemonSet pattern)
• Storing and indexing logs in Elasticsearch
• Visualizing and searching logs in Kibana
• How Kubernetes metadata enrichment works in log pipelines

Architecture

App Pods  -->  Fluent Bit (DaemonSet)  -->  Elasticsearch  -->  Kibana
                reads /var/log/containers      stores & indexes     search & dashboards

Fluent Bit runs on every node as a DaemonSet, tailing container log files from /var/log/containers. It enriches each log line with Kubernetes metadata (pod name, namespace, labels) and forwards them to Elasticsearch. Kibana provides a web UI for searching and visualizing the indexed logs.

Deploy

Step 1: Create the namespace

kubectl apply -f demos/efk-logging/manifests/namespace.yaml

Step 2: Deploy Elasticsearch

kubectl apply -f demos/efk-logging/manifests/elasticsearch.yaml

Wait for Elasticsearch to become ready:

kubectl rollout status statefulset/elasticsearch -n logging-demo --timeout=120s

Step 3: Deploy Kibana

kubectl apply -f demos/efk-logging/manifests/kibana.yaml

Step 4: Deploy Fluent Bit

kubectl apply -f demos/efk-logging/manifests/fluent-bit.yaml

Step 5: Deploy the sample application

kubectl apply -f demos/efk-logging/manifests/sample-app.yaml

Verify

# Check all pods are running
kubectl get pods -n logging-demo

# Check Elasticsearch cluster health
kubectl exec -n logging-demo statefulset/elasticsearch -- \
  curl -s http://localhost:9200/_cluster/health | python3 -m json.tool

# Check Fluent Bit is shipping logs
kubectl logs -n logging-demo daemonset/fluent-bit --tail=10

# Access Kibana
kubectl port-forward svc/kibana 5601:5601 -n logging-demo

Open http://localhost:5601 in your browser. Go to Discover, create an index pattern matching fluent-bit-*, and you should see logs flowing from the sample app and all other pods.

What is Happening

manifests/
  namespace.yaml       # logging-demo namespace
  elasticsearch.yaml   # Single-node Elasticsearch 8 StatefulSet with PVC
  kibana.yaml          # Kibana Deployment connected to Elasticsearch
  fluent-bit.yaml      # DaemonSet + ConfigMap (input, filter, output)
  sample-app.yaml      # BusyBox pod generating log lines every 2 seconds

Fluent Bit reads raw log files from the node filesystem (/var/log/containers/*.log). The Kubernetes filter calls the API server to enrich each record with the pod name, namespace, container name, and labels. The Elasticsearch output plugin sends batches of enriched records to the fluent-bit-* index.

Elasticsearch stores logs in a single-node configuration with security disabled for simplicity. Data is persisted to a PVC so logs survive pod restarts.

Kibana connects to Elasticsearch and provides full-text search, filtering by namespace or pod, and time-range histograms.

Experiment

1. Tail logs from the sample app and watch them appear in Kibana:

   kubectl logs -f deploy/log-generator -n logging-demo

2. Search in Kibana for kubernetes.namespace_name: logging-demo to filter logs from this demo only.

3. Scale the sample app and watch new pod logs appear automatically:

   kubectl scale deployment log-generator --replicas=3 -n logging-demo

4. Check Elasticsearch index size:

   kubectl exec -n logging-demo statefulset/elasticsearch -- \
     curl -s http://localhost:9200/_cat/indices?v

5. Deploy something in another namespace and confirm Fluent Bit picks up those logs too.

Cleanup

kubectl delete namespace logging-demo
kubectl delete clusterrole fluent-bit-read --ignore-not-found
kubectl delete clusterrolebinding fluent-bit-read --ignore-not-found

Further Reading

See docs/deep-dive.md for details on Fluent Bit parsers, Elasticsearch index lifecycle management, multi-node Elasticsearch clusters, log retention policies, and production considerations for the EFK stack.

Next Step

Move on to ML Model Serving to deploy a model prediction API with autoscaling.