Prometheus & Grafana

Monitor your cluster with the industry-standard observability stack.

Time: ~20 minutes Difficulty: Intermediate

What You Will Learn

• Installing Prometheus and Grafana via Helm (kube-prometheus-stack)
• PromQL basics: querying CPU, memory, and pod metrics
• Pre-built Grafana dashboards for Kubernetes
• ServiceMonitor CRDs for application metrics
• Alert rules and how Prometheus scrapes targets

Prerequisites

• Helm v3.13+
• Minikube with at least 4 CPUs and 8GB RAM

Install the Monitoring Stack

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

helm install monitoring prometheus-community/kube-prometheus-stack \
  --namespace monitoring-demo \
  --create-namespace \
  --set grafana.adminPassword=admin \
  --set prometheus.prometheusSpec.retention=2h

Wait for all pods to be ready (takes 2-3 minutes):

kubectl get pods -n monitoring-demo -w

Deploy a Sample App

Navigate to the demo directory:

cd demos/prometheus-grafana

kubectl apply -f manifests/sample-app.yaml

Access Grafana

kubectl port-forward svc/monitoring-grafana 3000:80 -n monitoring-demo

Open http://localhost:3000. Log in with:

• Username: admin
• Password: admin

Explore Pre-Built Dashboards

Navigate to Dashboards in the left sidebar. The kube-prometheus-stack includes:

Dashboard	What It Shows
Kubernetes / Compute Resources / Namespace	CPU and memory by namespace
Kubernetes / Compute Resources / Pod	Per-pod resource usage
Kubernetes / Networking / Namespace	Network I/O by namespace
Node Exporter / Nodes	Node-level CPU, memory, disk, network

Open “Kubernetes / Compute Resources / Namespace”, select monitoring-demo from the namespace dropdown.

Access Prometheus

kubectl port-forward svc/monitoring-kube-prometheus-prometheus 9090:9090 -n monitoring-demo

Open http://localhost:9090 and try these PromQL queries:

# CPU usage per pod
rate(container_cpu_usage_seconds_total{namespace="monitoring-demo"}[5m])

# Memory usage per pod
container_memory_working_set_bytes{namespace="monitoring-demo"}

# Pod restart count
kube_pod_container_status_restarts_total{namespace="monitoring-demo"}

# Number of running pods
count(kube_pod_status_phase{phase="Running"})

What is Happening

manifests/
  namespace.yaml    # monitoring-demo namespace
  sample-app.yaml   # App to observe in dashboards

Installed via Helm:
  Prometheus        # Scrapes metrics from all pods and nodes
  Grafana           # Visualizes metrics with dashboards
  AlertManager      # Routes alerts (email, Slack, PagerDuty)
  Node Exporter     # Exports node-level metrics
  kube-state-metrics # Exports Kubernetes object state as metrics

How it works:

Pods/Nodes expose /metrics ──> Prometheus scrapes every 30s
                                    |
                              Stores time-series data
                                    |
                              Grafana queries via PromQL
                                    |
                              Dashboards + Alerts

Experiment

1. Generate load on the sample app and watch metrics update:

   kubectl run load --rm -it --image=busybox:1.36 -n monitoring-demo -- \
     sh -c 'while true; do wget -qO- http://sample-app > /dev/null; done'

2. Check what Prometheus is scraping: Open http://localhost:9090/targets to see all scrape targets and their status.

3. View active alerts:

   kubectl port-forward svc/monitoring-kube-prometheus-alertmanager 9093:9093 -n monitoring-demo

   Open http://localhost:9093.

Cleanup

helm uninstall monitoring -n monitoring-demo
kubectl delete namespace monitoring-demo

Further Reading

See docs/deep-dive.md for a detailed explanation of the Prometheus data model, PromQL functions, ServiceMonitor/PodMonitor CRDs, custom metrics for HPA, recording rules, and alerting best practices.

Next Step

Move on to Deployment Strategies to learn blue/green and canary rollouts.