10.51 - Observability and Debugging

Debugging order that answers 90% of problems:

1. kubectl get events          (what happened, cluster-wide view)
2. kubectl describe <obj>      (state + events for one object)
3. kubectl logs <pod> [--previous]
4. kubectl exec -it <pod> -- ...   (look inside)

Never jump to logs first for a pod that won’t start: the reason is usually in events, not in logs.

The event flow you will see

Pod phase	Meaning	Typical cause
Pending	not scheduled yet	no node fits (resources), PVC pending, taints
ContainerCreating	runtime starting it	image pull, bad mount, bad env ref
Running	containers up	healthy (probes may still fail)
CrashLoopBackOff	exits immediately, restarting	app error, bad command, bad config
ImagePullBackOff	image can’t be pulled	wrong tag, registry auth, no network
Terminating	being deleted	stuck on finalizer or unmount
Unknown	node lost	kubelet down / network partition

kubectl get pods -A -o wide
kubectl get pods -o wide -w                 # watch live
kubectl get events --sort-by=.lastTimestamp | tail -20

describe: the workhorse

kubectl describe pod <name>
# sections that matter:
#   Status / Conditions:  Ready True/False + why
#   Containers.<name>.State: Waiting (reason: ImagePullBackOff / CrashLoopBackOff)
#   Last State: Terminated with exit code + reason (OOMKilled = exit 137)
#   Events: the narrative, newest last

Exit codes worth knowing: 0 clean, 1 app error, 137 OOM-killed (SIGKILL), 143 SIGTERM (graceful shutdown started).

kubectl describe node <node>      # allocated resources, conditions, taints
kubectl describe svc <name>       # endpoints, selector, nodePort
kubectl describe pvc <name>       # status, bound PV, events (Pending reasons)

Logs

kubectl logs <pod>                    # current container
kubectl logs <pod> -c <container>     # multi-container pods
kubectl logs <pod> --previous         # last crashed container's output (gold)
kubectl logs deploy/hello --tail=50 --follow
kubectl logs deploy/hello --since=10m

--previous is the single most useful flag in k8s debugging: when a pod is CrashLoopBackOff, the current container has no logs; the crashed one does.

Exec and port-forward for investigation

kubectl exec -it <pod> -- bash          # sh if no bash (alpine/busybox)
kubectl exec -it <pod> -- ls /etc/config
kubectl exec <pod> -- curl -s http://svc:80   # test connectivity in-cluster
kubectl port-forward svc/<name> 8080:80  # reach a pod without exposing it

Probes: health as an API

• livenessProbe: is the app alive? Failure -> kubelet kills and restarts the container. Prevents serving zombies.
• readinessProbe: is the app ready to serve? Failure -> pod removed from Service endpoints (no traffic), but NOT restarted. Prevents routing to half-booted apps.
• startupProbe: for slow-booting apps, gates the other two until first success (avoids killing a JVM that needs 90s).

livenessProbe:
  httpGet: { path: /healthz, port: 8080 }
  initialDelaySeconds: 5    # wait before first probe
  periodSeconds: 10         # probe interval
  timeoutSeconds: 1
  failureThreshold: 3       # consecutive failures before action
readinessProbe:
  httpGet: { path: /ready, port: 8080 }

Types: httpGet, tcpSocket (port: 3306), exec (command: ["cat", "/tmp/ok"]).

Test the pattern (challenge C23): point a liveness probe at a dead port; kubectl get pods shows the restart count climbing and describe shows Liveness probe failed: ...; fix the port and the restarts stop.

Resource pressure

kubectl top nodes          # CPU/mem usage per node (needs metrics-server; k3s ships it)
kubectl top pods           # per pod
kubectl describe node      # "Allocated resources" = requests vs allocatable

OOM story: pod exceeds memory LIMIT -> OOMKilled (exit 137) -> restart loop. Pod requests more than the node has -> Pending forever (check events: Insufficient memory). Fix by raising limits or reducing replicas, not by praying.

Break it on purpose (day 7 drill)

Recreate these four disasters, find each cause with describe/events/logs only, then fix. Time yourself. (These are C20-C23 in the challenges.)

1. Exit-forever container (CrashLoopBackOff). Logs –previous shows exit 1.
2. Nonexistent image tag (ImagePullBackOff). Events show the pull failure.
3. Service selector matching no pods (endpoints ). Fix the selector.
4. Liveness probe on a closed port (restart storm). Fix the probe.

The 2am debugging checklist

# everything in one shot:
kubectl get events --sort-by=.lastTimestamp | tail -30
kubectl get pods -A -o wide
kubectl get nodes
kubectl get pvc -A                 # stuck Pending kills scheduling
kubectl describe pod <bad-pod>
kubectl logs <bad-pod> --previous
kubectl top nodes

Gotchas

• kubectl get events only shows recent ones; old ones are evicted. Describe the object to see its full event list.
• Events are per-namespace; -A for all. Pod events appear in the pod’s namespace, NOT kube-system.
• A pod in Running with a failing readiness probe looks fine in get pods (READY shows 0/1). Always read the READY column.
• Timestamps: events use the cluster’s clock; if your host and cluster clocks drift, “recent” looks wrong. Rare on single-node k3s.
• metrics-server takes ~30-60s to show data after install; kubectl top before that errors with “metrics not available yet”.