Both before and after moving most of my homelab workloads onto k3s, I started seeing intermittent outages in Uptime Kuma. Moving to k3s helped, but didn’t fully resolve it.
There wasn’t an obvious pattern. Different services would occasionally show approximately 60 seconds of downtime, then recover without intervention. The affected applications weren’t necessarily related, and checking them afterward showed nothing obviously wrong.
The cluster itself also appeared healthy.
The eventual problem wasn’t an application, Traefik, Kubernetes networking, or Uptime Kuma. The underlying issue was storage latency on the virtual disks hosting the k3s nodes, which was causing etcd operations to exceed their expected latency.
Getting from a red bar in Uptime Kuma to the Proxmox disk configuration required working backward through several layers of the stack.
The first mistake: assuming the outage lasted 60 seconds
The first useful observation was that the apparent duration wasn’t actually measuring the duration of the failure.
Uptime Kuma was probing these services every 60 seconds.
A service would look like this:
12:00:00 200 OK
12:01:00 503 Service Unavailable
12:02:00 200 OK
The dashboard renders that as roughly a minute of downtime.
But all I actually knew was that one probe failed.
The service might have been unavailable for 50 seconds. It might also have been unavailable for 500 milliseconds at exactly the wrong time.
That distinction matters when debugging intermittent failures. The monitoring interval determines the resolution of the observation; it doesn’t necessarily tell you the duration of the event.
So the question changed from:
Why are these applications going down for a minute?
to:
Why are individual HTTP requests occasionally failing?
The status codes narrowed it considerably
The failed probes were overwhelmingly returning HTTP 503.
That was useful because a 503 is different from a timeout, DNS failure, refused connection, or TLS error.
The request was reaching something capable of returning HTTP.
More importantly, the affected applications didn’t share an application stack. There wasn’t a common database, runtime, or application dependency that explained why unrelated services would fail at approximately the same time.
They did share ingress.
External requests followed roughly this path:
Internet
|
Cloudflare
|
pfSense / HAProxy
|
Traefik
|
Kubernetes Service
|
Pod
A 503 from the ingress path suggested that Traefik was reachable but temporarily unable to route the request to a healthy backend.
That moved the investigation into Kubernetes.
Why would Traefik temporarily have no backend?
The pods themselves weren’t repeatedly crashing.
There was also no evidence of nodes dropping out of the cluster for long periods.
What stood out in the logs were messages from etcd:
apply request took too long
Some operations were taking between roughly 100 and 475 ms.
That is a problem for etcd.
etcd sits underneath the Kubernetes control plane and is extremely sensitive to disk latency. Kubernetes continually writes state through it: leases, EndpointSlices, controller state, object updates, and other coordination data.
The disk didn’t need to be completely unavailable to cause problems. It just needed to stall often enough that operations which normally complete quickly began exceeding their timing expectations.
Looking at Linux pressure stall information on the k3s VMs provided another clue:
/proc/pressure/io
was reporting full I/O pressure with avg10 around 3.
In other words, there were periods where runnable work was stalled waiting for storage.
At that point, the chain started to make sense:
Proxmox VM disk latency
|
v
etcd apply/fsync latency
|
v
Kubernetes API stalls
|
v
EndpointSlice / lease churn
|
v
Traefik temporarily loses usable backend state
|
v
HTTP 503
|
v
Uptime Kuma catches one failed probe
|
v
"60 second outage"
The red bar was several layers removed from the actual problem.
Leader election provided another signal
The etcd warnings weren’t the only evidence.
Several Kubernetes controllers that use leader election had accumulated surprisingly high restart counts.
I found counts including:
| Component | Restarts |
|---|---|
| CSI driver | 139 |
| NFS | 69 |
| ARC | 80 |
| cert-manager | 30 |
These weren’t four independent unstable applications.
They had a common dependency: Kubernetes leases.
When a controller holds leadership, it periodically renews a Lease object through the Kubernetes API. If the control plane stalls long enough that it can’t renew the lease, the process can lose leadership and terminate.
In this case, the affected controllers were repeatedly exiting with code 255 after losing their leases.
That was much stronger evidence than the original HTTP failures.
The cluster wasn’t simply experiencing an ingress problem. Components throughout the control plane were having trouble completing time-sensitive operations.
The problem was below Kubernetes
At this point, it would have been easy to start tuning etcd.
That would have addressed the wrong layer.
The k3s nodes are Ubuntu VMs running on Proxmox. etcd, therefore, doesn’t write directly to physical storage. Its writes pass through the guest filesystem, the virtual block device, QEMU, and eventually the host storage stack.
The relevant Proxmox virtual disks weren’t configured with an I/O thread.
I changed the VM disk configuration to use:
iothread=1
cache=none
Using an I/O thread allows QEMU to process disk I/O separately rather than handling it through the main emulator thread. cache=none avoids the QEMU host page cache and uses direct I/O semantics.
There was one important operational detail here that cost me some time.
I changed the disk configuration and rebooted the VM.
Nothing improved.
I rebooted it again through Proxmox.
Still nothing.
The reason was that the new QEMU disk configuration was pending. Restarting the guest operating system doesn’t recreate the QEMU process, and qm reboot doesn’t necessarily do so either.
The VM had to be completely stopped and started:
qm stop <vmid>
qm start <vmid>
Only then was the new virtual disk configuration actually active.
That distinction is easy to miss when changing virtual hardware settings in Proxmox.
Measuring the result
Before the change, etcd was producing a steady stream of operations exceeding 100 ms even when the cluster wasn’t doing anything particularly interesting.
After changing the disk configuration and fully restarting the VMs, the distribution changed substantially.
Median latency dropped to approximately 1 ms, with roughly 93% of observations below 8 ms.
The long tail didn’t disappear completely.
Approximately 2% of the observations were still above 64 ms, which suggests there is still occasional contention lower in the host storage stack. That’s worth monitoring, but it is very different from routinely seeing 100–475 ms operations while the cluster is effectively idle.
More importantly, the intermittent 503s stopped appearing with the previous frequency.
The access log that wouldn’t turn on
During the investigation, I also wanted better evidence about where the 503 responses originated.
Traefik access logging seemed like the obvious answer.
Enabling it should have been a straightforward Helm configuration change.
Instead, the upgrade failed with an ownership conflict similar to:
helm upgrade failed: conflict ... "kubectl-client-side-apply"
After addressing that, another conflict appeared involving:
before-first-apply
This turned out to be unrelated to the etcd problem, but it exposed another issue in the cluster.
A resource originally managed by Helm had previously been modified using kubectl apply or kubectl edit. Those operations had introduced other field managers onto the Deployment.
With newer Helm behavior using server-side apply, Helm was no longer willing to silently take ownership of fields another manager claimed to own.
So an attempt to enable an access log turned into a second debugging exercise involving Kubernetes-managed fields.
The practical lesson was simple: manually modifying Helm-owned resources can leave behind ownership state that matters much later.
The access logs were useful once enabled, but by then, the etcd and controller evidence had already identified the more important problem.
Fixing the monitoring as well
There was one additional change worth making.
Even with the underlying storage problem corrected, a single failed probe shouldn’t necessarily produce an alert.
Pods restart. Deployments roll. Images get updated. There are legitimate cases where a service can briefly fail a request without representing an incident worth notifying me about.
I increased Uptime Kuma’s retry count to at least two for these monitors.
That doesn’t hide sustained outages. It prevents one missed request from being interpreted as an actionable failure.
The storage fix and the monitoring change solve two different problems:
Fix the infrastructure so requests don’t fail because etcd is waiting hundreds of milliseconds for disk I/O.
Configure the monitor so that one transient request failure isn’t treated as a minute-long outage.
What the 60-second outage actually was
There wasn’t a 60-second outage.
There was an intermittent storage-latency problem on the virtual disks underneath the Kubernetes control plane.
That latency caused etcd operations to stall. Those stalls affected Kubernetes API operations and lease renewals. Controllers lost leader elections, and ingress occasionally found itself without usable backend state long enough to return a 503.
Uptime Kuma happened to probe during one of those events.
Because the monitor ran every 60 seconds, one failed request became a 60-second red bar on the dashboard.
The final path from symptom to cause looked like this:
Uptime Kuma
↓
HTTP 503
↓
Traefik / Kubernetes backend state
↓
Kubernetes API + controller leases
↓
etcd apply/fsync latency
↓
QEMU virtual disk configuration
↓
Proxmox storage
The useful part of this incident wasn’t the specific Proxmox setting.
It was that every layer initially presented a plausible place to investigate. The monitoring system appeared to show minute-long outages. HTTP pointed toward ingress. Kubernetes exposed controller instability. etcd finally showed that the control plane was waiting on storage.
The fix ended up being below all of them.
