10.42 - Storage and Stateful Workloads

The storage stack (three layers) 

StorageClass  ->  PersistentVolume  ->  PersistentVolumeClaim  ->  pod mount
(local-path)      (actual disk dir)     (your request for disk)    (/data)
  • StorageClass: the “how disk is provisioned” template. k3s ships local-path (creates hostPath-style dirs on the node) and marks it default. kubectl get sc.
  • PersistentVolume (PV): the actual storage unit, cluster-wide. kubectl get pv. With local-path, PVs appear dynamically when you create a PVC.
  • PersistentVolumeClaim (PVC): your namespace-scoped request. Bind to a PV matching its size + access mode.

You only ever create PVCs (and StorageClasses for special cases). PVs are provisioned for you.

kubectl get sc,pv,pvc -A
kubectl get sc local-path -o yaml   # provisioner, reclaimPolicy, volumeBindingMode

Access modes

  • ReadWriteOnce (RWO): one node can mount it read-write. Default; right for most single-pod apps.
  • ReadOnlyMany (ROX) / ReadWriteMany (RWX): multiple nodes. local-path does NOT support RWX; it is per-node hostPath. RWX needs a real shared filesystem (NFS, CephFS). For a single-node k3s, RWO is all you need.

Reclaim policies

  • Delete (local-path default): delete the PVC, the PV and its data are removed. Good for scratch.
  • Retain: PV survives PVC deletion; you must delete it manually. Good for backups.

The “my data survives” recipe (challenge C13) 

kubectl apply -f - <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mydata
spec:
  accessModes: [ReadWriteOnce]
  resources: { requests: { storage: 1Gi } }
---
apiVersion: apps/v1
kind: Deployment
metadata: { name: writer }
spec:
  replicas: 1
  selector: { matchLabels: { app: writer } }
  template:
    metadata: { labels: { app: writer } }
    spec:
      volumes:
        - name: data
          persistentVolumeClaim: { claimName: mydata }
      containers:
        - name: writer
          image: busybox
          command: ["sh", "-c", "echo survive > /data/marker; sleep 3600"]
          volumeMounts: [ { name: data, mountPath: /data } ]
EOF
kubectl exec deploy/writer -- sh -c "cat /data/marker"   # survive
kubectl delete pod -l app=writer                          # kill the pod
kubectl exec deploy/writer -- sh -c "cat /data/marker"   # STILL survive

Deleting the pod does nothing to the PVC. Deleting the PVC with local-path deletes the PV and the directory. This is why: keep PVCs, delete pods.

Volume binding mode

local-path is WaitForFirstConsumer: the PV is created only when a pod actually schedules and needs it, and it lands on that pod’s node. That matters for multi-node clusters (a PVC created “eagerly” could bind to a PV on the wrong node, and RWO then blocks scheduling). On single-node k3s this is invisible, but know the term for interviews and for when you add agents.

StatefulSets: identity + per-pod storage

Use when pods need to be individually addressable and have their own disk (databases, kafka, anything with “node id”).

kubectl apply -f - <<'EOF'
apiVersion: v1
kind: Service
metadata: { name: db }
spec:
  clusterIP: None               # headless: no VIP, DNS returns pod IPs
  selector: { app: db }
  ports: [ { port: 3306 } ]
---
apiVersion: apps/v1
kind: StatefulSet
metadata: { name: db }
spec:
  serviceName: db
  replicas: 3
  selector: { matchLabels: { app: db } }
  template:
    metadata: { labels: { app: db } }
    spec:
      containers:
        - name: db
          image: mysql:8
          env:
            - name: MYSQL_ROOT_PASSWORD
              value: rootpw
          volumeMounts: [ { name: data, mountPath: /var/lib/mysql } ]
  volumeClaimTemplates:
    - metadata: { name: data }
      spec:
        accessModes: [ReadWriteOnce]
        resources: { requests: { storage: 1Gi } }
EOF
kubectl get pods -o wide        # db-0, db-1, db-2
kubectl get pvc                 # data-db-0, data-db-1, data-db-2 (per pod!)
kubectl get endpoints db        # three pod IPs, no VIP
kubectl scale statefulset db --replicas=0   # scale down: pods gone, PVCs stay
kubectl scale statefulset db --replicas=3   # scale up: db-0/1/2 return, same identity

What’s different from a Deployment:

  • Stable names db-0..db-N (Deployment pods get random suffixes).
  • Each pod gets its OWN PVC from the template (a Deployment shares one PVC among all replicas, which breaks RWO).
  • Pods start/stop in order: db-0 first, then db-1… (depends on podManagementPolicy, default OrderedReady).
  • DNS: db-0.db.default.svc.cluster.local works via the headless service.

Storage on k3s: local-path details

  • Default storage class: local-path (rancher local-path-provisioner).
  • Data lands in /var/lib/rancher/k3s/storage/<pvc-uid>_default_mydata on the node (check with sudo find /var/lib/rancher/k3s/storage).
  • It’s hostPath under the hood: single-node fine, multi-node means data lives on the node that owns the pod. For HA storage you’d add Longhorn or NFS; note it, don’t build it this week.

Backups (day 7 final project hook)

Backups of PVC data = back up the hostPath directory. k3s has k3s etcd snapshot save (or sqlite backup) for cluster state, but app data you back up yourself. A CronJob that tars /data to another PVC (or offsite) is the pattern:

apiVersion: batch/v1
kind: CronJob
metadata: { name: backup }
spec:
  schedule: "0 2 * * *"
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: Never
          volumes:
            - name: data
              persistentVolumeClaim: { claimName: mydata }
          containers:
            - name: backup
              image: busybox
              command: ["sh", "-c", "tar czf /backup/$(date +%F).tgz -C /data ."]
              volumeMounts: [ { name: data, mountPath: /data, readOnly: true } ]

Gotchas

  • RWO + multiple replicas = pods stuck Pending (volume can’t mount on two nodes). If your app is stateless, don’t give it a PVC at all.
  • Deleting a StatefulSet does NOT delete its PVCs (by design). To wipe state: delete the StatefulSet, then kubectl delete pvc -l app=db.
  • local-path + Retain: change reclaimPolicy on a custom StorageClass, not the shared default.
  • kubectl get pv shows Bound; a PVC stuck in Pending with no events = no StorageClass matches (check kubectl get sc and the class name).
This site uses Just the Docs, a documentation theme for Jekyll.