Migrating my home infrastructure to Kubernetes

Moving from Ceph custom containers to Kubernetes sounds like a complete rewrite, but the basic idea stayed the same: run an image on one of several machines and attach storage that survives the machine. The big difference is that Kubernetes actually cares about applications.

I kept Ceph and replaced only the improvised scheduler around it. The cluster was bootstrapped with kubeadm on Arch Linux, CRI-O runs the containers and Cilium provides networking with kube-proxy replacement.

Ceph became normal Kubernetes storage

Rook manages Ceph and its CSI drivers. I did not want an innocent spare disk on the control-plane machine to become an OSD, so discovery is restricted to the six Odroid storage nodes and NVMe device names:

storage:
  useAllNodes: false
  useAllDevices: false
  deviceFilter: "^nvme[0-9]+n[0-9]+$"
  nodes:
    - name: odroid-h3-00
    - name: odroid-h3-01
    - name: odroid-h3-02
    - name: odroid-h3-03
    - name: odroid-h3-04
    - name: odroid-h3-05

Applications can now request an RBD block volume, a shared CephFS volume or use the Rados Gateway through S3. I also added erasure-coded variants for the larger data. The exact changes live in my-rook-config.

One service at a time

I did not migrate everything during one exciting weekend. Traefik temporarily routed to both Kubernetes services and old services outside the cluster. For each application I identified its data, created the PVC and workload, added a Service and Ingress, tested it, and only then removed the old route.

The strange Ceph deployment specs were actually helpful documentation. They already told me the required paths, users, ports and entrypoints. Those values moved into Deployments and StatefulSets instead of being rediscovered.

The first visible improvement was maintenance. I can drain a worker and watch the applications appear on other nodes. Readiness checks prevent traffic from arriving too early. Rollouts have an actual status instead of me refreshing a web page and hoping.

The less visible improvement is Git. More of the current cluster configuration is split into small repositories on code.brunner.ninja. Changes get validated before a restricted CI user applies them.

Kubernetes did not remove complexity; it gave all my existing complexity names and YAML schemas. That is still an improvement. The remaining problem was how a LoadBalancer Service gets an address on a home network, which gets port forwarded by my mikrotik core router.


Beitrag veröffentlicht

in

,

von

Schlagwörter: