EFK Logging Stack: Deep Dive

Overview

The EFK stack (Elasticsearch, Fluent Bit, Kibana) is a widely used centralized logging solution for Kubernetes. This document covers the architecture decisions, configuration details, and production considerations for the demo.

Why Fluent Bit Instead of Fluentd

Fluent Bit is a lightweight, high-performance log processor written in C. Compared to Fluentd (Ruby-based), it uses roughly 10x less memory and is purpose-built for container environments. Fluent Bit handles log collection and forwarding, while Fluentd is better suited as a log aggregator in larger deployments.

For a single-cluster demo, Fluent Bit alone is sufficient. In production, you might run Fluent Bit on every node forwarding to a central Fluentd aggregator that handles complex routing.

Fluent Bit Configuration Breakdown

Input: Tail Plugin

The tail input reads log files from disk, similar to tail -f. Kubernetes writes container stdout/stderr to /var/log/containers/<pod>_<namespace>_<container>-<id>.log. The DB parameter tracks file offsets so Fluent Bit resumes correctly after restarts.

Filter: Kubernetes Plugin

This filter calls the Kubernetes API to resolve pod metadata from the log file path. It adds kubernetes.pod_name, kubernetes.namespace_name, kubernetes.labels, and other fields. The Merge_Log option parses JSON log bodies into structured fields.

Output: Elasticsearch Plugin

Logstash_Format On creates daily indices like fluent-bit-2024.01.15. This makes index lifecycle management straightforward: you can delete old indices by date.

Elasticsearch Single-Node Mode

Setting discovery.type=single-node disables cluster formation. This avoids the bootstrap checks that require specific kernel parameters (vm.max_map_count). For production, run at least 3 nodes with proper discovery configuration.

Index Lifecycle Management

In production, configure ILM policies to:

1. Roll over indices when they reach a size or age threshold
2. Move old indices to cheaper storage tiers
3. Delete indices after a retention period

Security Considerations

This demo disables xpack.security for simplicity. In production:

• Enable TLS between Fluent Bit and Elasticsearch
• Use authentication for Elasticsearch and Kibana
• Restrict Fluent Bit’s RBAC to only the namespaces it needs
• Consider network policies to isolate the logging namespace

Resource Sizing

Component	Demo	Small Production
Elasticsearch	1 node, 1GB heap	3 nodes, 4GB heap each
Fluent Bit	64Mi per node	128-256Mi per node
Kibana	512Mi	1Gi

Further Resources

• Fluent Bit Documentation
• Elasticsearch Guide
• Kubernetes Logging Architecture