When I moved my homelab from a collection of Docker workloads to a three-node k3s cluster, getting the applications running was only part of the migration.
I also wanted the dashboard I use to get into everything to understand the cluster underneath it.
A traditional homelab dashboard is mostly a collection of links. Give it a name, an icon, a URL, maybe point a widget at an application’s API, and you’re done.
That becomes less useful once the infrastructure underneath those links is dynamic.
In Kubernetes, knowing that https://some-service.example.com responds doesn’t tell me much about what is actually happening inside the cluster. I wanted to be able to glance at the dashboard and see whether the pods behind each application were healthy, which nodes were carrying the workloads, and what CPU, memory, and GPU resources were being used across the cluster.
The result turned the dashboard from a prettier bookmarks page into an actual view of the homelab.
It also exposed a surprising number of edge cases along the way.
A dashboard should tell me more than where an application lives
The basic job of the dashboard hasn’t changed.
I still want a tile I can click to open an application.
But once the applications are running in Kubernetes, there’s considerably more information available than a URL and an icon.
A service can be reachable while one of its pods is unhealthy. A deployment can have fewer replicas than expected. A node can be under significantly more load than the others. A GPU workload can be consuming resources without anything being obviously wrong from the application’s external endpoint.
What I wanted was closer to this:
Homelab Dashboard (Homepage)
|
+----------------+----------------+
| | |
v v v
Applications Pod health Node health
| | |
v v v
URLs/API Kubernetes CPU / Memory
widgets API / GPU
The dashboard is still the front door.
It just knows considerably more about what’s behind that door.
Making individual service tiles Kubernetes-aware
The first thing I wanted was Kubernetes pod health on the application tiles themselves.
This seemed like it should be simple.
The dashboard already knew which application a tile represented. Kubernetes knew which pods belonged to that application. I just needed to connect the two.
Instead, I found a slightly unintuitive dependency between two configuration values.
Per-tile pod health needed both an app field and a podSelector override.
My Kubernetes Deployments label their pods using:
app=<name>
while the dashboard assumed a selector based on:
app.kubernetes.io/name=<name>
So I needed to override the pod selector to match the labels I actually use.
The obvious configuration was therefore the equivalent of:
podSelector: app=my-service
Except that didn’t work by itself.
Nothing happened.
No request was made.
The reason was that the dashboard’s app field wasn’t merely descriptive metadata. It was also used to construct the status request. Without it, the pod selector existed, but the dashboard never actually queried anything.
So the service needed both pieces of information:
app: my-service
podSelector: app=my-service
Using only podSelector resulted in no request being made.
Using only app caused the request to happen, but with the dashboard’s assumed Kubernetes selector. The result was effectively:
no pods found
Two configuration options, two different silent failure modes, and neither option worked alone.
Once both were configured correctly, the dashboard could finally associate each application tile with the pods actually providing that application.
That immediately made the tiles more useful.
Instead of:
┌──────────────────────┐
│ Application │
│ │
│ Open application │
└──────────────────────┘
I now had something conceptually closer to:
┌──────────────────────┐
│ Application │
│ │
│ Pods: ● ● ● │
│ │
│ Open application │
└──────────────────────┘
The dashboard wasn’t replacing Kubernetes observability tooling.
It was answering the much simpler question I actually care about most of the time:
Is the thing I’m about to click on healthy?
Application health wasn’t enough
Once the service tiles knew about Kubernetes, the obvious next step was making the dashboard understand the nodes, too.
My k3s cluster has three nodes, and two of them have NVIDIA GPUs.
I wanted a section showing resource utilization for each node:
┌─────────────────────┐
│ k3s-01 │
│ CPU 18% │
│ Memory 47% │
│ GPU 12% │
└─────────────────────┘
┌─────────────────────┐
│ k3s-02 │
│ CPU 24% │
│ Memory 51% │
│ GPU 0% │
└─────────────────────┘
┌─────────────────────┐
│ k3s-03 │
│ CPU 11% │
│ Memory 39% │
│ GPU — │
└─────────────────────┘
For that, I deployed a metrics collector as a DaemonSet.
That gives every Kubernetes node its own metrics endpoint:
Dashboard
|
+-------------+-------------+
| | |
v v v
k3s-01 k3s-02 k3s-03
| | |
v v v
metrics metrics metrics
| | |
CPU/MEM/GPU CPU/MEM/GPU CPU/MEM
Running it as a DaemonSet is a natural fit because the data I care about is explicitly node-local.
Then the GPU metrics didn’t work.
The GPU was there. The library was there. NVML still wasn’t there.
The GPU-enabled metrics pods had access to the NVIDIA devices.
The NVIDIA container runtime was working.
libnvidia-ml.so was present.
And yet the GPU endpoint returned:
[]
The underlying error was:
NVMLError_LibraryNotFound
At first glance, that’s a straightforward error: NVML isn’t installed.
Except it was.
The problem wasn’t that the library couldn’t be found on disk.
It was that the process couldn’t load it.
The stock metrics image was based on Alpine Linux.
Alpine uses musl libc.
The NVIDIA container runtime was injecting a glibc libnvidia-ml.so into the container.
That left me with:
Metrics container
|
+--> Alpine
| |
| +--> musl
|
+--> NVIDIA runtime
|
+--> libnvidia-ml.so
|
+--> built for glibc
The file existed.
The device nodes existed.
The runtime had mounted everything where it was supposed to be.
But musl couldn’t load the glibc library.
So the application surfaced the situation as NVMLError_LibraryNotFound.
That’s a much more interesting failure than an actually missing library because inspecting the filesystem makes everything look correct.
The fix was to run the metrics service on a glibc-based image instead.
Once I did that, NVML could load, and the GPU metrics appeared.
This is also one of those bugs where containerization can obscure the actual compatibility boundary.
The host NVIDIA driver worked.
GPU passthrough worked.
The NVIDIA container runtime worked.
The container could see the GPU.
The failure was inside the userspace ABI of the tiny metrics container sitting at the very end of the chain.
Then came the smaller dashboard bugs
With the major pieces working, I ran into several smaller problems that individually weren’t particularly interesting but collectively consumed a surprising amount of time.
The first was an API version mismatch.
One of the dashboard widgets defaulted to an older API version. The endpoint simply returned a 404, and the widget failed without giving me much indication of why.
Once the correct API version was specified, it worked.
Then there was the GPU selector.
I initially configured the metric as:
gpu:0
That seemed reasonable. I wanted GPU zero.
The tile showed dashes.
The selector wasn’t actually looking for a numeric GPU index. It was matching the device ID string.
What worked was:
gpu:nvidia0
Again, the difference is tiny once you know it.
Before you know it, you spend time questioning the metrics endpoint, the NVIDIA runtime, and the dashboard configuration when the actual issue is one string.
My CPUs were apparently colder than absolute zero
The CPU temperature widget produced a much more entertaining result:
-273 °C
That would certainly solve my homelab cooling requirements.
The problem was that these Kubernetes nodes are VMs.
There isn’t any host hardware-monitoring data exposed inside the guests for the widget to consume.
So the CPU temperature value wasn’t useful.
GPU temperature, on the other hand, worked because that data comes through NVML.
That distinction is easy to overlook when building a dashboard because the UI presents both values as if they’re equivalent:
CPU temperature ---> host hwmon ---> unavailable in VM
GPU temperature ---> NVML -------> available through GPU passthrough
Rather than pretending the CPU temperature meant something, the correct answer was simply not to treat that metric as valid for these virtualized nodes.
A dashboard displaying a number isn’t the same thing as the number representing reality.
HTTP status isn’t always application health
I hit a similar problem when adding one of my local LLM services to the dashboard’s site monitoring.
The monitor queried:
/
and reported:
415
That made the service look unhealthy.
It wasn’t.
The LLM server simply doesn’t accept a plain GET against its root endpoint.
The correct monitoring target was:
/health
Pointing the monitor at the actual health endpoint fixed it.
It’s a small distinction, but an important one.
A site monitor can answer:
Does this URL return the HTTP status I expect?
That isn’t automatically the same question as:
Is this application healthy?
For a traditional website, / may be a perfectly good health check.
For an API, inference server, or other specialized service, it may tell you almost nothing.
Then I made the dashboard configuration declarative
Originally, the dashboard configuration lived as a hand-edited file on a network share.
That was convenient.
Need to change a tile?
Edit the file.
Refresh the dashboard.
Done.
As the rest of the homelab became more Kubernetes-native, though, leaving an important piece of configuration as a manually maintained file didn’t fit very well.
I moved the dashboard configuration into a ConfigMap.
An initContainer then syncs that configuration into the volume used by the application.
Conceptually:
Git / Kubernetes manifest
|
v
ConfigMap
|
v
initContainer
|
v
dashboard volume
|
v
application
Now the configuration deployed with the application is the source of truth.
That solved one problem and created an easy way to confuse myself later.
The file you’re editing may not be the source of truth anymore
Once the ConfigMap became authoritative, manually editing the copy on the network share still worked.
Temporarily.
The application could read the changed file, so everything looked normal.
Then the pod restarted.
The initContainer ran again.
The ConfigMap was copied back into the volume.
My manual changes disappeared.
Nothing had malfunctioned.
That was exactly what I’d configured Kubernetes to do.
The ownership model had changed:
BEFORE
Network-share config
|
v
Dashboard
Edit file ---> persistent change
AFTER
ConfigMap
|
v
initContainer
|
v
Network-share config
|
v
Dashboard
Edit file ---> temporary change
Restart ---> ConfigMap wins
That’s the kind of change that absolutely needs to be documented.
Otherwise, six months later, I’ll edit the file I’ve edited dozens of times before, see the change work, and then wonder why Kubernetes “deleted” it after a restart.
It didn’t.
I changed which copy was authoritative and forgot about it.
A dashboard can lie in several different ways
The interesting part of this project wasn’t really making the dashboard prettier.
It was making the information on it trustworthy.
Every problem I hit represented a different way a dashboard can give the wrong impression.
A pod selector can silently query the wrong Kubernetes labels and tell me no pods exist.
An NVML loader failure can turn a functioning GPU into an empty metric.
A device selector can turn valid GPU data into dashes.
A VM can turn a CPU temperature widget into physically impossible nonsense.
A perfectly healthy LLM server can appear down because the monitor queried an endpoint that doesn’t support GET.
A manually edited configuration file can appear authoritative right until the next pod restart overwrites it.
None of those is fundamentally a UI problem.
They’re data-model and integration problems.
The dashboard is just where they become visible.
The dashboard is the product
There’s a tendency in homelabs to treat dashboards as decoration.
Build the infrastructure first, then put a nice homepage in front of it.
I think that’s backwards, at least for the infrastructure I interact with every day.
The dashboard is the part I actually see.
I don’t normally open Kubernetes tooling just to find out whether everything is working. I don’t SSH into each node to check CPU utilization. I don’t query NVML manually to see whether a GPU is busy.
I look at the dashboard.
That means the quality of the dashboard determines how understandable the infrastructure feels.
The underlying cluster can be beautifully automated, highly available, and meticulously monitored, but if the interface I use every day gives me stale, incomplete, or misleading information, the system still feels unreliable.
Conversely, a dashboard that understands the infrastructure underneath it can collapse a surprising amount of complexity into one screen:
┌──────────────────────────────────────────────┐
│ HOMELAB │
├──────────────────────────────────────────────┤
│ │
│ Applications │
│ │
│ Service A ●●● Service B ●● │
│ Service C ● Service D ●●● │
│ │
├──────────────────────────────────────────────┤
│ │
│ Cluster │
│ │
│ k3s-01 k3s-02 k3s-03 │
│ CPU 18% CPU 24% CPU 11% │
│ MEM 47% MEM 51% MEM 39% │
│ GPU 12% GPU 0% GPU — │
│ │
└──────────────────────────────────────────────┘
That’s not a replacement for Prometheus, Kubernetes events, application logs, or proper monitoring.
It doesn’t need to be.
Its job is to give me enough information to answer the first question:
Does anything need my attention?
If the answer is yes, then I can open the specialized tooling and investigate.
Cluster-aware is more useful than cluster-themed
Moving the dashboard into Kubernetes would have been easy.
Making it actually understand Kubernetes was the useful part.
There’s a meaningful difference between running a dashboard on a cluster and building a dashboard for the cluster.
The first is deployment.
The second is integration.
For me, that meant tying application tiles to their actual pods, collecting node-level resource metrics, exposing GPU utilization through NVML, monitoring services through endpoints that actually represent their health, and making the configuration itself part of the declarative deployment.
It also meant removing metrics that looked impressive but weren’t valid in my environment.
The result isn’t dramatically more complicated to look at.
That’s the point.
The infrastructure underneath it became more complicated when I moved from Docker to a three-node k3s cluster.
The interface I use to understand it didn’t have to.
If anything, it got simpler.
Because once the dashboard actually understands the cluster, I don’t have to.
