Skip to content

Storage: democratic-csi + TrueNAS Core (NFS)

NFS-backed storage via democratic-csi. Enforces hard size limits per PVC using ZFS refquota on TrueNAS. Default storage class for the entire cluster.

TrueNAS Setup

  • Pool: main
  • Volumes dataset: main/vols — democratic-csi creates one child dataset per PVC here
  • Snapshots dataset: main/vols-snap — used if VolumeSnapshot CRDs are ever enabled
  • NFS share: /mnt exported with "All dirs" enabled (covers all child datasets)
  • SSH: enabled, root login allowed
  • SSH key: RSA 4096, PEM format (-m PEM flag required)
  • API key: stored in the Helm values

Gotchas

  • freenas-api-nfs driver is TrueNAS Scale only — use freenas-nfs for Core
  • SSH key must be RSA in PEM format: ssh-keygen -t rsa -b 4096 -f key -N "" -m PEM
  • ed25519 keys crash the driver with "Unsupported key format"
  • datasetParentName and detachedSnapshotsDatasetParentName must be different paths
  • The driver needs both httpConnection (API key) and sshConnection (ZFS commands)
  • KUBECONFIG=/etc/rancher/k3s/k3s.yaml must be set for Helm to work on k3s nodes

Install

helm repo add democratic-csi https://democratic-csi.github.io/charts/
helm repo update
helm install democratic-csi democratic-csi/democratic-csi \
  --namespace democratic-csi \
  --create-namespace \
  -f setup/storage/values.yaml

Set truenas-nfs as default and disable local-path:

kubectl patch storageclass local-path \
  -p '{"metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'

kubectl patch storageclass truenas-nfs \
  -p '{"metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

PVCs with no storageClassName will automatically use truenas-nfs.

Upgrade

helm upgrade democratic-csi democratic-csi/democratic-csi \
  --namespace democratic-csi \
  -f setup/storage/values.yaml

How it works

  • Each PVC → one ZFS dataset under main/vols/ (e.g. main/vols/pvc-<uuid>)
  • ZFS refquota is set to the PVC requested size — hard limit, writes fail when full
  • Snapshots don't count against the quota (refquota vs quota)
  • Each dataset gets its own NFS share exported to the cluster subnet
  • natOutgoing + shareMaprootUser: root lets pods write as any UID

Storage class

Property Value
Name truenas-nfs (default class)
Access modes RWO and RWX both supported
Reclaim policy Delete (dataset + NFS share removed when PVC is deleted)
Volume expansion Enabled