For several years, most of my homelab ran as Docker containers on a single host. By the time I started this migration, that host was running roughly 25 containers covering GitHub Actions runners, website/cert monitoring, photo management, 3D printing, budgeting, a media server with OTA live TV, and the services behind my local voice assistant.
The environment worked, but it had three architectural problems.
First, the Docker host was a single failure domain. Application data was backed up, but a hardware failure would still take most of the environment offline until the host could be repaired or replaced.
Second, the configuration itself wasn’t particularly reproducible. Most application configuration lived in bind mounts, containers were updated automatically from mutable tags, and much of the configuration required to reconstruct the host wasn’t stored in Git.
Third, I was running my family’s password vault, this blog, and a Unifi OS Server each on their own dedicated VM, wasting resources and making patching harder than it needed to be.
The migration started with four requirements:
- tolerate the loss of a compute host;
- make GPUs on multiple physical hosts available to workloads;
- move persistent state onto shared storage;
- make the environment reproducible enough that adding or rebuilding a node was a deployment process rather than a manual rebuild.
The resulting environment uses three Proxmox hosts running three Ubuntu VMs as a k3s cluster, with TrueNAS providing shared NFS and NVMe-oF storage.
Cluster architecture
Each Proxmox host runs an Ubuntu VM created from the same cloud-init template. All three VMs are k3s server nodes and therefore participate in the embedded etcd cluster.
Two of the Proxmox hosts also have NVIDIA GPUs passed through to their respective k3s VMs.
The basic layout is:
pfSense
|
HAProxy
|
Traefik VIP
(MetalLB)
|
+--------------+--------------+
| | |
k3s-server-1 k3s-server-2 k3s-server-3
Ubuntu VM Ubuntu VM Ubuntu VM
NVIDIA GPU NVIDIA GPU
| | |
+--------------+--------------+
|
TrueNAS
+-------+-------+
| |
NFS NVMe-oF
shared/bulk block storage
MetalLB provides addresses on the physical network for Kubernetes LoadBalancer services. Traefik uses one of those addresses as the primary ingress VIP and performs hostname-based routing to applications inside the cluster.
For externally accessible applications, HAProxy on pfSense remains in front of Traefik. That kept the existing external routing model separate from the Kubernetes migration.
Storage is split according to workload.
NFS is used where applications need shared or bulk filesystem storage. Workloads that are better suited to block storage, particularly databases, use persistent volumes backed by NVMe-oF from TrueNAS.
This separation also means the Kubernetes nodes are intended to be disposable compute. A workload can be rescheduled without its persistent data being tied to the VM on which it previously ran.
Moving from one k3s server to three
An early version of the environment wasn’t actually a three-node control plane.
It consisted of one k3s server and additional agent nodes. That allowed workloads to run across several machines, but the Kubernetes API and datastore still depended on the server node.
Rebooting that VM demonstrated the problem immediately. The other nodes continued running their existing containers, but the control plane was unavailable. Kubernetes couldn’t schedule replacements or otherwise reconcile the cluster while the server was down.
The final design, therefore, uses three k3s server nodes with embedded etcd.
Three is important here. etcd requires a majority of members for quorum:
| etcd members | Quorum | Members that can fail |
|---|---|---|
| 1 | 1 | 0 |
| 2 | 2 | 0 |
| 3 | 2 | 1 |
| 4 | 3 | 1 |
| 5 | 3 | 2 |
A two-member etcd cluster provides no additional failure tolerance over a single member. Both members must be available to maintain quorum.
With three server nodes, any one can be stopped while the remaining two continue providing the Kubernetes control plane.
This was the first place where I found it useful to separate several things that tend to get grouped together under “high availability.”
Data availability
Persistent application data needs to survive the loss of the node running the application.
Moving persistent volumes onto TrueNAS handles that part of the problem. A pod can disappear from one Kubernetes node and be recreated on another while mounting the same underlying storage.
For NFS-backed volumes, this is relatively straightforward because multiple nodes can mount the filesystem.
RWO block volumes over NVMe-oF are more complicated. Kubernetes and the CSI driver need to stop using the volume on the failed node and stage it on the replacement node.
That distinction became important later when I started testing abrupt node failures rather than clean drains. I’ll cover that separately because the behavior of RWO workloads during an ungraceful failure turned out to be less deterministic than I expected.
Control-plane availability
Shared storage doesn’t help if Kubernetes itself cannot respond to a failure.
Moving to three k3s server nodes addressed this separately from application storage. The Kubernetes API and embedded etcd datastore can now tolerate one server disappearing.
I tested this initially with rolling maintenance: cordon and drain a node, stop the VM, start it again, wait for it to become healthy, and repeat on the next node.
With three etcd members, the control plane remains available throughout that process.
That still doesn’t mean every application is highly available. A single-replica application will necessarily disappear temporarily while Kubernetes moves it. Applications with multiple appropriately distributed replicas can remain available.
The important distinction is that the cluster is capable of performing the recovery without depending on the failed node.
Container images are another form of locality
There was another dependency on individual nodes that I hadn’t accounted for initially: container images.
Kubernetes doesn’t distribute images between nodes.
If a workload has only ever run on node 1, its image may not exist on nodes 2 or 3. When Kubernetes reschedules it, containerd has to pull the image before the replacement pod can start.
For ordinary application images, this isn’t particularly noticeable. It became very noticeable with the ML workloads.
The combined image data for those workloads is roughly 31 GB. A failover to a node that didn’t already have the required images could therefore spend around ten minutes downloading images before the workload even began starting.
I already had a nightly image-pruning job on the nodes since containers are auto-updated using Keel, which made the problem worse. Pre-pulling every image onto every node wasn’t a durable solution because the pruning process would eventually remove unused images again.
Adding a registry pull-through cache
Instead of trying to keep every image resident on every node, I added a local registry cache.
The cache provides mirrors for five upstream registries used by the cluster. Image blobs are stored centrally on the NAS, and the mirrors are exposed through one MetalLB address with a separate port for each upstream registry.
Each k3s node has the mirrors configured in registries.yaml, while retaining the original registry as a fallback.
Conceptually:
k3s nodes
|
| container image pull
v
local registry mirrors
|
+-- Docker Hub cache
+-- GHCR cache
+-- registry.k8s.io cache
+-- ...
|
v
NAS storage
The first pull still has to retrieve the image from the upstream registry. Subsequent pulls by any node are served from the local cache.
This lets the node-local containerd cache remain disposable. The nightly pruning process can continue removing unused images without throwing away the expensive part of the cache.
It also changed node replacement behavior. A newly provisioned node can start with an empty containerd image store and populate it from the LAN rather than downloading tens of gigabytes from several public registries.
For Docker Hub specifically, authenticating the mirror also means upstream pulls use the configured account rather than each node independently consuming the anonymous pull allowance.
Reproducibility was the harder problem
Moving from Docker to Kubernetes solved some of the original architectural problems, but Kubernetes itself doesn’t make an environment reproducible.
I learned that during the migration from the initial single-server k3s installation to the three-server etcd cluster.
That migration requires rebuilding the datastore. I exported the existing Kubernetes resources, prepared a dependency-ordered restore bundle containing 531 objects, and validated it with a server-side dry run.
The dry run completed without errors, but as I would find out later, it wasn’t perfect, and the backup was also incomplete.
One namespace’s workload controllers—roughly 20 Deployments—and a storage credential weren’t in the bundle. The validation succeeded because it could determine whether the objects present in the bundle were valid against the running cluster. It couldn’t tell me which objects I had failed to back up.
After the old cluster was removed, some of those missing manifests had to be reconstructed from previous development session transcripts because they had originally been applied through shell heredocs and had never existed as files.
That deserves its own post, but it changed the priority of the project.
The original goal was to make the infrastructure highly available.
The more important goal became making the infrastructure reconstructable.
A three-node cluster with redundant storage is useful when a machine fails. It doesn’t help nearly as much when the only correct copy of a Deployment is the one currently stored inside etcd.
Where the environment ended up
The current architecture separates several things that had previously existed on one Docker host:
- Proxmox provides the physical compute and VM layer.
- k3s provides scheduling and a three-member control plane.
- TrueNAS holds a persistent application state.
- NFS provides shared filesystem storage.
- NVMe-oF provides block-backed persistent volumes.
- MetalLB provides addresses on the physical networks.
- Traefik handles Kubernetes ingress.
- HAProxy remains the external reverse proxy.
- GPUs are schedulable resources on two cluster nodes.
- Registry mirrors provide a shared image cache.
- Kubernetes manifests and supporting configuration are moving into Git rather than existing only as live cluster state.
There are still deliberate single points of failure in the homelab, particularly TrueNAS. The objective wasn’t to remove every possible failure domain. It was to make failures explicit and decide which ones were worth engineering around.
The migration also exposed several assumptions that didn’t survive actual failure testing: shared storage doesn’t automatically make a stateful application highly available; GPU resources aren’t interchangeable just because Kubernetes labels both of them nvidia.com/gpu; and a healthy three-node cluster can still produce user-visible failures when the storage underneath etcd has poor latency.
Those are the subjects of the rest of this series.
The main thing I’d change if I started again is the order.
I would put the configuration in Git before rebuilding the cluster, not afterward.
High availability protects the infrastructure you have. Reproducibility determines whether you can get it back.
