10.11 - Install and First Cluster

Three options. Pick ONE as your main cluster.

• Option A: native k3s on NixOS (recommended). A real systemd-managed cluster, uses its own containerd (no Docker), most reliable on your box.
• Option B: k3d, k3s in Docker. Best for simulating a MULTI-NODE cluster (server + agents) on one machine. Has caveats under rootless Docker.
• Option C: k3sup, for a remote VPS. Mentioned for completeness, not needed for this week.

Option A: native k3s (recommended)

Add to your NixOS system config (configuration.nix, not home-manager):

{ pkgs, ... }:
{
  services.k3s = {
    enable = true;
    role = "server";            # "server" = control plane + worker in one
    # token = "changeme";       # set this when you later add agent nodes
    # Write kubeconfig world-readable so you can use it without sudo:
    extraFlags = ''
      --write-kubeconfig-mode 644
      --disable traefik
      --disable servicelb
    '';
  };

  environment.systemPackages = with pkgs; [ kubectl k9s ];
  # Add k3d too if you also want multi-node simulation (Option B):
  # environment.systemPackages = with pkgs; [ kubectl k3d ];
}

Why --disable traefik --disable servicelb: k3s’s Traefik ingress controller and its built-in LoadBalancer (servicelb) both want to bind host ports 80/443, which Caddy already owns. For week 1 you use kubectl port-forward and NodePort (30000-32767 range) instead, which never touch 80/443. On the networking day you can re-enable Traefik on alternate ports (10.41 has the recipe).

Apply and verify:

sudo nixos-rebuild switch --flake ~/nix#<your-host>

systemctl status k3s                 # should be active (running)
sudo journalctl -u k3s -f            # watch it boot; Ctrl-C to stop following

Make kubectl usable without sudo (NixOS k3s already passes –write-kubeconfig-mode 644 from above, so this is a fallback):

sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config 2>/dev/null
# if ~/.kube doesn't exist yet: mkdir -p ~/.kube && sudo cp ...
sudo chown "$USER" ~/.kube/config
chmod 600 ~/.kube/config

Sanity checks (run all three; all must succeed):

kubectl get nodes -o wide          # one node, STATUS=Ready
kubectl get pods -A                # coredns + metrics-server + local-path
kubectl get svc -A                 # kubernetes service (ClusterIP)
kubectl cluster-info               # control plane is reachable

Fish niceties:

alias k kubectl
kubectl completion fish | source

Option B: k3d (multi-node simulation in Docker)

Useful later in the week and for any “cluster of 3 nodes” experiments. k3s runs inside Docker containers, so your host system stays untouched and you can destroy/recreate clusters freely.

nix shell nixpkgs#k3d -c k3d --help   # or add k3d to systemPackages

k3d cluster create lab --servers 1 --agents 2
# waits ~60s, then:
kubectl get nodes -o wide              # 3 nodes: k3d-lab-server-0, -agent-0, -agent-1
kubectl get pods -A

k3d cluster delete lab                 # full teardown, instant reset

For ingress experiments later you can publish a host port to Traefik:

k3d cluster create lab --servers 1 --agents 2 --port "8080:80@loadbalancer"

Rootless Docker caveat: k3d’s k3s containers want privileged mode and certain kernel features. Under rootless Docker this sometimes works and sometimes hangs (pods stuck in ContainerCreating). If k3d cluster create succeeds but pods never become Ready, don’t fight it: use Option A as your main cluster. k3d is a nice-to-have for simulating multi-node, not a requirement.

Option C: k3sup (remote VPS, skip for now)

nix shell nixpkgs#k3sup -c k3sup install --ip <vps-ip> --user root
# sets up k3s over SSH and drops a kubeconfig on your machine

First pod, first deployment (Day 1 exercise)

# 1. Imperative quick test (fine for a smoke test)
kubectl run hello --image=nginx:alpine
kubectl get pods
kubectl logs hello
kubectl delete pod hello

# 2. Declarative, the way you will actually work
kubectl create deployment hello --image=nginx:alpine --replicas=2
kubectl get deployments,replicasets,pods
kubectl get pods -o wide
kubectl port-forward deployment/hello 8080:80   # open http://localhost:8080
kubectl exec -it deployment/hello -- bash
kubectl logs deployment/hello --tail=20
kubectl delete deployment hello

Troubleshooting the install

Symptom	Likely cause	Fix
kubectl says connection refused	API not up yet, or wrong kubeconfig	journalctl -u k3s -f; check kubectl config current-context
get nodes works but pods stuck	image pull on first run is slow	wait 60-120s, kubectl get pods -A again
Traefik pod crashlooping	port 80/443 already taken (Caddy)	you disabled traefik above; if not, sudo k3s kubectl -n kube-system delete deploy traefik or reinstall with the flag
kubeconfig permission denied	file owned by root	run the chown/chmod block above
Can’t reach NodePort from host	firewall	k3s opens 30000-32767 via firewalld only on some distros; on NixOS add networking.firewall.allowedTCPPortRanges = [{ from = 30000; to = 32767; }]

Next: 10.12 - Core Concepts for the mental model, then 10.21 - Week 1 Plan for the day-by-day schedule.