10.41 - Networking and Ingress

Three questions define cluster networking: how do pods reach each other, how does the outside reach pods, and where does HTTP routing happen?

Pod-to-pod (the CNI, Flannel in k3s)

Every pod gets its own IP on a flat overlay network (Flannel VXLAN). Any pod can reach any other pod directly by IP, no NAT, regardless of node. Check it:

kubectl get pods -o wide          # note pod IPs (10.42.x.x)
kubectl exec -it <pod-a> -- sh -c "curl -s http://<pod-b-ip>"   # works

Rule of thumb: think of pods as processes on one giant machine. They talk to each other by IP or by Service DNS name, never by node/host IP.

Service DNS (the thing you will use daily)

Services get stable DNS names via CoreDNS:

<service>.<namespace>.svc.cluster.local

Inside the same namespace, curl http://<service>:<port> is enough. curl http://<service> uses port 80 by default; for other ports always specify them. Everything resolves cluster-wide; a Service in lab is api.lab.svc.cluster.local from anywhere.

kubectl exec -it deploy/web -- sh -c "curl -s http://api:80"    # same ns
kubectl exec -it deploy/web -- sh -c "curl -s http://api.lab.svc.cluster.local"

Why this matters: pod IPs change on every restart; Service names never do. Your apps should always use Service DNS, never pod IPs, never other pods’ names.

Service types

Type	Exposed	Use when
ClusterIP (default)	cluster-internal only	inter-app calls, backend APIs
NodePort	nodeIP:30000-32767 on every node	quick manual testing from host
LoadBalancer	external IP/LB (k3s: servicelb)	real external exposure
Headless (clusterIP: None)	pod IPs, no VIP	StatefulSets, direct pod discovery

Port logic: port is what the Service serves (and what DNS targets); targetPort is the container port. They can differ; port: 80, targetPort: 8080 is common when images listen on 8080.

Reaching the cluster from your host

Three ways, in increasing order of “production”:

1. kubectl port-forward - a tunnel to ONE pod/service. Debug only.

   kubectl port-forward svc/web 8080:80      # localhost:8080 -> svc:80
   kubectl port-forward pod/db-0 3306:3306   # reach a specific pod
   

2. NodePort - every node listens on 30000-32767 for the Service.

   kubectl get svc web                        # find the assigned nodePort
   curl http://localhost:<nodePort>
   

   Single-node k3s: localhost works directly. This is why the install disables servicelb; NodePort + port-forward cover all of week 1.

3. Ingress (Traefik) - HTTP-aware routing in the cluster. See below.

Ingress and the Caddy conflict

Ingress = hostname/path rules -> Services. In a normal k3s, Traefik (the default ingress controller) binds host ports 80/443 and routes by Host header. On your box, Caddy owns 80/443 already, so week 1 disables Traefik and servicelb (see 10.11).

Enabling Traefik on alternate ports (Day 3, optional but recommended)

Two ways; the cleaner one is a second k3s install flag at setup time, but you can also edit the Traefik Deployment’s ports:

# Option 1: reinstall/rebuild k3s with alternate ports (cleanest)
#   add to services.k3s.extraFlags:
#     --disable traefik  (already there)
#   and when you DO want traefik, instead of --disable traefik use:
#     --kube-apiserver-arg=...  (not needed) and edit the helm chart below

# Option 2: edit the running Traefik deployment (no reinstall)
kubectl -n kube-system get deploy traefik -o yaml > /tmp/traefik.yaml
#   change the two containerPorts AND the service ports:
kubectl -n kube-system get svc traefik -o yaml
#   simplest: change the traefik Service from 80/443 to 8081/8443:
kubectl -n kube-system patch svc traefik \
  --type=merge -p '{"spec":{"ports":[{"name":"web","port":8081,"targetPort":"web"},{"name":"websecure","port":8443,"targetPort":"websecure"}]}}'

Then Ingress objects route through Traefik on 8081. Test with a Host header so you don’t need DNS:

kubectl apply -f - <<'EOF'
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: web
spec:
  rules:
    - host: lab.example
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service: { name: web, port: { number: 80 } }
EOF
curl -H "Host: lab.example" http://localhost:8081   # -> web service

Path routing: pathType: Prefix with /api sends lab.example/api/* to one service, / to another. This is exactly what your Caddy handle blocks do, except it lives in the cluster.

The clean endgame for your homelab

Caddy on the host does TLS termination and sends k8s-bound hostnames to Traefik: lab.example -> localhost:8081. One Caddy site block, everything else lives in the cluster. (10.53 - Next Steps has the Caddyfile sketch.)

CoreDNS debugging

kubectl -n kube-system get deploy coredns
kubectl -n kube-system get pod -l k8s-app=kube-dns
kubectl exec -it deploy/web -- sh -c "cat /etc/resolv.conf"   # nameserver 10.43.0.10
kubectl exec -it deploy/web -- sh -c "nslookup api.default.svc.cluster.local"

If DNS fails: check coredns pod is Running, check /etc/resolv.conf inside a pod points at the kube-dns ClusterIP, check the Service kube-dns exists in kube-system.

Network policies (mention, not required this week)

By default pods can talk to everything. NetworkPolicy objects restrict that, but Flannel (the k3s CNI) does NOT enforce them; you would need Canal or Cilium. Note it and move on.

Common failure modes

Symptom	Cause	Check
curl http://api:80 times out	service selector matches nothing	kubectl get endpoints api
curl works in-cluster, fails from host	you are using ClusterIP DNS	use NodePort/port-forward/Ingress
NodePort refuses from host	firewall	allowedTCPPortRanges 30000-32767 (10.11)
Can’t resolve api in another namespace	need FQDN	api.<ns>.svc.cluster.local
Two services same name in same ns	DNS ambiguity / apply conflict	names unique per kind+ns