When I moved my homelab workloads from a single Docker host into a three-node k3s cluster, persistent storage was one of the architectural decisions I wanted to get right early.
Most of the persistent data already lived on TrueNAS, so Kubernetes didn’t need to become a storage platform itself. What I needed was a way for workloads to dynamically request storage from TrueNAS and have that storage follow them between nodes.
For shared files, NFS is an obvious answer.
For databases and other workloads that expect local block storage, I wanted something different.
I ended up using NVMe over Fabrics, specifically NVMe/TCP, to present dynamically provisioned TrueNAS zvols to Kubernetes.
The basic architecture is:
Kubernetes PVC
|
v
CSI driver
|
| provision
v
TrueNAS
|
v
ZFS zvol
|
| NVMe/TCP
v
k3s node
|
v
filesystem
|
v
pod
Getting that working turned out to involve more than defining a StorageClass.
Partway through the build, the TrueNAS API that the CSI driver I originally planned to use depended on simply wasn’t there anymore.
That sent the storage side of the migration down a different path.
Why block storage instead of putting everything on NFS?
I already use NFS extensively in the homelab.
It remains the right choice for workloads that genuinely need shared filesystem access. Kubernetes applications that need ReadWriteMany storage can mount the same NFS-backed volume from multiple nodes without pretending the underlying storage is something it isn’t.
Databases are different.
For databases and other applications designed around a local filesystem, I wanted Kubernetes to see an ordinary block device rather than place the application’s filesystem on top of NFS.
That gives the application the storage model it expects:
Application
|
v
Filesystem
|
v
Block device
|
v
NVMe/TCP
|
v
TrueNAS zvol
rather than:
Application
|
v
Filesystem
|
v
NFS client
|
v
Remote filesystem
The important distinction isn’t that NFS is inherently bad storage. I still use it where its semantics make sense.
I wanted different storage classes for different workload requirements.
Conceptually, my cluster has two main persistent-storage paths:
Kubernetes
/ \
/ \
v v
Block storage Shared storage
| |
v v
NVMe/TCP NFS
| |
+--------+---------+
|
v
TrueNAS
That lets the workload determine the storage model rather than forcing everything onto one protocol.
Why NVMe/TCP instead of iSCSI?
Once I decided I wanted block storage from TrueNAS, the next obvious choice was between iSCSI and NVMe-oF.
I’ve used iSCSI plenty. It would have worked.
But this was a new deployment, my network already supports the bandwidth I need, and TrueNAS supports NVMe-oF. There wasn’t much reason to build a new block-storage architecture around iSCSI if I could use NVMe/TCP instead.
NVMe/TCP also fits nicely into the rest of the homelab.
I’m already using NVMe-oF-backed storage with Proxmox, so using the same underlying protocol for Kubernetes keeps the storage architecture relatively consistent.
The Kubernetes side still sees a normal block device.
The transport between the node and TrueNAS is what changes:
Pod
|
v
Filesystem
|
v
/dev/nvme...
|
v
Linux NVMe initiator
|
| TCP/IP
v
TrueNAS NVMe target
|
v
ZFS zvol
The application doesn’t need to know that this is happening.
That’s exactly what I wanted from the CSI layer.
Then the TrueNAS REST API disappeared
The original plan depended on a CSI driver that communicated with TrueNAS through its REST API.
Then I hit a fairly fundamental problem:
/api/v2.0/...
returned a 404.
TrueNAS 26 has removed the REST v2.0 API. The middleware API had moved to WebSocket-only communication.
This wasn’t a case of an endpoint changing names or an authentication method changing.
The API model the driver expected was gone.
That forced me to switch approaches in the middle of building the storage layer.
It’s also a good example of why “supports TrueNAS” isn’t quite enough information when evaluating an integration. TrueNAS has changed its management interfaces significantly over time, and a project designed around an older API can be perfectly functional while still being incompatible with a current release.
Finding the actual API endpoint was more confusing than expected
Once I was working with the newer middleware API, I needed to establish the correct WebSocket connection.
That produced several answers that looked plausible but were wrong in different ways.
The first attempt was essentially:
https://<truenas-ip>
That immediately ran into TLS validation because the certificate didn’t contain the IP address as a SAN.
That was useful information, but it wasn’t the actual protocol problem.
Then there was:
/websocket
That looks extremely promising when you’re looking for the WebSocket API.
It is also the legacy DDP endpoint.
Wrong protocol.
The bare HTTPS root wasn’t much more useful. It returned successfully, but a 200 OK from the TrueNAS web server doesn’t tell you that you’ve found the middleware API.
What I actually needed was the JSON-RPC WebSocket endpoint.
The distinction matters because “I can connect to the TrueNAS web server” and “I can speak the current TrueNAS middleware protocol” are two entirely different tests.
The resulting control path is roughly:
CSI controller
|
| JSON-RPC over WebSocket
v
TrueNAS middleware
|
v
Create/manage zvol
|
v
Expose through NVMe-oF
The storage data itself doesn’t travel through that API.
The API is the control plane.
The actual I/O path is directly between the Kubernetes node and TrueNAS over NVMe/TCP.
CONTROL PLANE
CSI driver --------> TrueNAS middleware
|
v
create/configure zvol
DATA PLANE
Pod
|
v
k3s node ============== TrueNAS
NVMe/TCP
Keeping those two paths mentally separate makes the architecture much easier to reason about.
The parent dataset isn’t created for you
Once the driver could actually communicate with TrueNAS, the next surprise was that the configured parent dataset had to already exist.
The CSI deployment wasn’t going to create the entire ZFS hierarchy from scratch.
If the StorageClass expects its dynamically provisioned volumes underneath something like:
tank/kubernetes
then:
tank/kubernetes
needs to exist first.
The CSI driver can then provision individual volumes underneath it.
That’s reasonable once you think about ownership boundaries. I don’t necessarily want a Kubernetes controller deciding how the top-level datasets on my NAS should be structured.
But it’s an easy prerequisite to miss when the goal is “dynamic provisioning.”
Dynamic provisioning doesn’t mean zero preparation on the storage system.
The Kubernetes nodes need NVMe support too
The TrueNAS side is only half of the configuration.
Every Kubernetes node that might mount one of these volumes needs to be capable of acting as an NVMe/TCP initiator.
That means the relevant kernel support needs to be available:
nvme-tcp
nvme-fabrics
and each node needs a host NQN. The installation also requires the NVMe tooling needed by the storage path.
This is one of the prerequisites that showed up in the previous post about my Proxmox golden template.
If the modules aren’t available, Kubernetes can successfully schedule a pod to a node and provision the volume, only to leave the pod sitting in:
ContainerCreating
because the node can’t actually connect to the storage.
From Kubernetes, that looks like a storage problem.
From TrueNAS, the zvol may already exist.
The missing piece is the host underneath Kubernetes.
That’s why I eventually persisted the necessary modules in the node provisioning process instead of relying on manually loading them after a mount failed.
A couple of TrueNAS UI details weren’t obvious either
There were also smaller configuration details that cost more time than they should have.
In the current TrueNAS UI, what I was looking for when creating the parent dataset was represented as a dataset preset, not a dataset “type.”
That’s minor, but it matters when documentation or examples use terminology that no longer matches the UI in front of you.
Quota fields presented another small trap: they rejected decimal values.
So rather than trying to specify fractional GiB values, the configuration needed integer GiB quantities.
Neither of these is architecturally interesting.
Both are exactly the kind of detail that makes an otherwise straightforward deployment feel broken when you’re doing it for the first time.
What happens when a PVC is created?
Once everything is connected, the workflow is what I wanted from the beginning.
An application requests storage through a PersistentVolumeClaim:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: database
spec:
accessModes:
- ReadWriteOnce
storageClassName: truenas-nvme
resources:
requests:
storage: 20Gi
The exact StorageClass name and driver configuration depend on the deployment, but conceptually, Kubernetes handles the rest:
PVC created
|
v
CSI controller
|
v
TrueNAS middleware API
|
v
Create zvol
|
v
Expose via NVMe/TCP
|
v
Node connects to namespace
|
v
Filesystem mounted
|
v
Pod starts
To the application, it looks like locally attached storage.
To me, the storage remains centralized on TrueNAS.
And because the storage isn’t physically tied to the Kubernetes VM, the workload can move.
attachRequired: false changes an important Kubernetes safety mechanism
One detail in the CSI deployment deserves more attention than it initially appears to.
The CSIDriver object is configured with:
attachRequired: false
That’s deliberate.
The node plugin mounts the volume directly rather than relying on Kubernetes’ controller attach/detach path. As a result, Kubernetes doesn’t create VolumeAttachment objects for these volumes.
That keeps the storage path simple.
It also has a consequence:
Kubernetes isn’t providing multi-attach fencing for you.
For a ReadWriteOnce filesystem backed by a block device, that distinction matters.
The normal expectation is that one node owns the volume at a time.
With this design, the safety of moving the volume depends in part on the old node genuinely no longer being able to write to it.
A clean migration is straightforward:
Node A
|
| unmount
X
Volume
|
| mount
v
Node B
A hard failure raises a more interesting question:
Node A
|
| ???
|
Volume
|
| mount?
v
Node B
If Node A is truly stopped, it can’t write.
If it isn’t, Kubernetes doesn’t have a VolumeAttachment object providing another layer of attach/detach coordination.
That detail eventually became important enough to deserve its own failure-testing exercise, but the important point for the storage design is that attachRequired: false isn’t just an implementation detail.
It changes where the fencing responsibility lives.
Don’t restore the CSIDriver object blindly
That same field creates a particularly nasty restore scenario.
If you manually restore or recreate the CSIDriver object with:
attachRequired: true
Kubernetes switches behavior.
Now it expects the controller attach path to exist and starts creating attachment operations, which the driver wasn’t intended to handle that way.
The resulting failure can be confusing, including errors such as:
node X not found
The storage itself isn’t necessarily broken.
The restored Kubernetes object changed how Kubernetes thinks the driver works.
The correct recovery is to reinstall the driver from its chart so the CSIDriver object is recreated with the values the driver actually expects, rather than treating a rendered cluster object as an authoritative backup configuration.
That’s a broader Kubernetes lesson too: not every object in the cluster should be restored verbatim simply because it existed before.
Some are outputs of another deployment system.
The chart is the source of truth.
Creating a PVC isn’t enough validation
The first successful PVC is satisfying:
STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS
Bound ... 20Gi RWO truenas-nvme
But a Bound PVC only proves part of the system works.
For block storage that’s supposed to support a multi-node cluster, I wanted to verify the entire lifecycle.
That means:
- Create the PVC.
- Mount it to a pod.
- Write data.
- Read the data back.
- Move the workload to another node.
- Mount the same volume there.
- Verify that the existing filesystem and data are recognized.
The critical behavior on the second node is that the CSI path recognizes an existing filesystem rather than treating the device as new.
The log message I wanted to see was:
Existing filesystem, skipping format
That proves considerably more than seeing Bound in kubectl get pvc.
The actual test is:
TrueNAS zvol
|
v
Node A
|
v
write test data
|
X
workload moves
|
v
Node B
|
v
recognize existing filesystem
|
v
read same test data
If that works, the storage is behaving like cluster storage rather than storage that happens to work on the first node that mounted it.
The storage layer now has clear roles
After working through all of this, the persistent storage architecture ended up being fairly simple.
NFS handles workloads that need shared filesystem semantics.
NVMe/TCP-backed zvols handle workloads where I want a block device and ReadWriteOnce semantics.
TrueNAS owns the actual persistent storage.
Kubernetes owns the lifecycle of dynamically provisioned application volumes through CSI.
And the Kubernetes nodes remain compute:
TrueNAS
/ \
/ \
NVMe/TCP NFS
| |
v v
RWO PVCs RWX PVCs
\ /
\ /
Kubernetes
|
v
application
That separation is important to the larger design of the cluster.
A Kubernetes node can disappear without taking its application’s persistent data with it.
The API was the most fragile dependency
The part of this build I expected to spend time on was NVMe/TCP.
That wasn’t really the difficult part.
Linux already knows how to speak NVMe/TCP. TrueNAS knows how to expose NVMe-oF storage. Kubernetes knows how to consume block devices through CSI.
The fragile part was the management interface connecting Kubernetes provisioning to TrueNAS.
The REST API the original integration expected disappeared, and suddenly an otherwise reasonable storage design couldn’t provision anything.
The replacement path required understanding that the current TrueNAS middleware API was WebSocket-based, finding the correct JSON-RPC endpoint, and switching the integration accordingly.
That’s a useful distinction when thinking about infrastructure integrations.
The data plane can be based on an established protocol like NVMe/TCP and remain completely functional.
The control plane can still break because an API has changed.
In this case:
Data plane:
Kubernetes node <==== NVMe/TCP ====> TrueNAS
stable
Control plane:
CSI driver <==== API ====> TrueNAS
^
|
changed
Once the control plane was working again, NVMe/TCP itself was almost boring.
That’s probably the best outcome for storage.
The interesting part should be provisioning it.
The actual block I/O should just work.
