kubectl Debug

Troubleshoot running and crashed pods without modifying deployments.

Time: ~10 minutes Difficulty: Intermediate

What You Will Learn

• kubectl debug: attach a debug container to a running or crashed pod
• Ephemeral containers: debug without rebuilding images
• Debugging CrashLoopBackOff pods
• Debugging distroless/minimal images that have no shell
• Node-level debugging
• Common troubleshooting workflows

Deploy

Navigate to the demo directory:

cd demos/kubectl-debug

kubectl apply -f manifests/namespace.yaml
kubectl apply -f manifests/crashing-pod.yaml
kubectl apply -f manifests/distroless-app.yaml
kubectl apply -f manifests/running-app.yaml

Scenario 1: Debug a CrashLoopBackOff Pod

kubectl get pods -n debug-demo
# crash-loop is in CrashLoopBackOff

Check logs first:

kubectl logs crash-loop -n debug-demo
# ERROR: Config file not found!

You cannot exec into a crashed container. Use kubectl debug to create a copy with a different command:

kubectl debug crash-loop -n debug-demo -it \
  --copy-to=crash-debug \
  --container=app \
  -- /bin/sh

Inside the debug copy, investigate:

# Check what files exist
ls /config/
# Directory doesn't exist - that's the problem

# Check env vars, mounts, etc.
env
mount
exit

Clean up the debug pod:

kubectl delete pod crash-debug -n debug-demo

Scenario 2: Debug a Distroless Container

The distroless-app uses the pause image, which has no shell and no tools:

# This fails - no shell available
kubectl exec -it deploy/distroless-app -n debug-demo -- /bin/sh

Add an ephemeral debug container that shares the process namespace:

kubectl debug -it deploy/distroless-app -n debug-demo \
  --image=busybox:1.36 \
  --target=app

Inside the debug container, you can see the target container’s processes:

ps aux
# You can see the pause process from the app container

# Check the network
wget -qO- http://localhost:80 2>&1 || echo "No web server (expected for pause)"

# Check filesystem
ls /proc/1/root/
exit

Scenario 3: Debug a Running Pod

Add a debug container to a running nginx pod:

POD=$(kubectl get pod -l app=web-app -n debug-demo -o jsonpath='{.items[0].metadata.name}')

kubectl debug -it "$POD" -n debug-demo \
  --image=curlimages/curl \
  --target=nginx \
  -- /bin/sh

Inside the debug container:

# Test the app from inside the pod
curl localhost:80

# Check DNS resolution
nslookup web-app.debug-demo.svc.cluster.local

# Check network connectivity
curl -s http://web-app.debug-demo.svc/

exit

Scenario 4: Debug at the Node Level

kubectl debug node/minikube -it --image=busybox:1.36

Inside the node debug pod, the host filesystem is at /host:

# Check host processes
chroot /host ps aux | head -20

# Check host disk usage
chroot /host df -h

# Check kubelet logs
chroot /host journalctl -u kubelet --no-pager | tail -20

exit

Common Troubleshooting Checklist

# 1. Check pod status
kubectl get pods -n debug-demo

# 2. Check events
kubectl describe pod <pod-name> -n debug-demo

# 3. Check logs (current and previous)
kubectl logs <pod-name> -n debug-demo
kubectl logs <pod-name> -n debug-demo --previous

# 4. Check resource usage
kubectl top pods -n debug-demo

# 5. Check endpoints (is the Service routing correctly?)
kubectl get endpoints web-app -n debug-demo

# 6. Test DNS from inside the cluster
kubectl run dns-test --rm -it --image=busybox:1.36 -n debug-demo -- nslookup web-app

What is Happening

manifests/
  namespace.yaml        # debug-demo namespace
  crashing-pod.yaml     # Pod that crashes (missing config file)
  distroless-app.yaml   # Minimal image with no shell
  running-app.yaml      # Normal nginx app for testing

kubectl debug modes:

Mode	Flag	Use Case
Copy pod	--copy-to=name	Debug crashed containers by creating a copy
Ephemeral container	(default)	Add a debug container to a running pod
Node debug	node/<name>	Debug at the node level with host access

Cleanup

kubectl delete namespace debug-demo

# If you ran the node debug scenario, clean up the debug pod in default namespace
kubectl get pods --field-selector=status.phase=Running -o name | grep node-debugger | xargs -r kubectl delete

Further Reading

See docs/deep-dive.md for a detailed explanation of ephemeral containers, process namespace sharing, debug profiles, and advanced troubleshooting patterns.

Next Step

Move on to Multi-Container to learn sidecar, adapter, and ambassador patterns.