Thumbnail image

Debugging a Silent SELinux Denial in a Netdata Kubernetes DaemonSet

  • Wed, Aug 19, 2026
  • 7-minute read

I spent an evening this week wiring the Netdata Cloud agent into a small single-node k3s cluster - the usual parent/child/k8s-state trio from the official netdata/netdata Helm chart. The parent (the metrics store and Cloud relay) and the k8s-state collector (cluster object state - pods, deployments, nodes) both came up clean on the first helm install.

The child DaemonSet - the one that actually reads host metrics off the node - didn’t.

It crash-looped in its init container, with an error that looked like it should take thirty seconds to explain:

chmod: /persistencevarlibdir: Permission denied

That init container is nothing exotic. It’s a plain chmod 777 against a hostPath volume, straight from the chart’s own templates/child/daemonset.yaml. The pod runs as root. The target directory was already 0777 root:root. Real DAC permissions, real root, a mode bit that already allows everyone to write - that combination should never produce Permission denied. It did anyway, and the reason turned out to be more interesting than a typo’d securityContext.

The Setup

  • Host: AlmaLinux 10, SELinux enforcing
  • Kubernetes: single-node k3s, containerd as the container runtime
  • Chart: netdata/netdata (Helm), version 3.7.171, appVersion v2.11.0
  • The offending mount: a hostPath at /var/lib/netdata-k8s-child/var/lib/netdata (the chart’s own default for child.persistence.hostPath), mounted into the init container at /persistencevarlibdir

The Investigation

First thoughts: just a DAC problem?

The obvious first move is to check ownership and mode:

$ ls -ld /var/lib/netdata-k8s-child/var/lib/netdata
drwxrwxrwx. 2 root root 6 Aug 19 05:52 /var/lib/netdata-k8s-child/var/lib/netdata

Already 777, already root:root. And the process trying to chmod it is real root, not remapped by a user namespace - confirmed with crictl inspect against the init container:

"user": {
  "uid": 0,
  "gid": 0,
  "additionalGids": [0, 1, 2, 3, 4, 6, 10, 11, 20, 26, 27]
},
"capabilities": {
  "effective": [
    "CAP_CHOWN", "CAP_DAC_OVERRIDE", "CAP_FSETID", "CAP_FOWNER",
    "CAP_MKNOD", "CAP_SETGID", "CAP_SETUID", "CAP_SETFCAP",
    "CAP_SETPCAP", "CAP_SYS_CHROOT", "CAP_KILL", "CAP_AUDIT_WRITE"
  ]
}

CAP_DAC_OVERRIDE and CAP_FOWNER are both present. By ordinary Unix rules, this process can chmod anything on the filesystem regardless of who owns it. Standard DAC permissions were never going to explain this denial - which meant something else was in the way.

Chasing a ghost in the audit log

On a SELinux-enforcing host, Permission denied from a process that should clearly have DAC rights is the textbook symptom of an AVC denial. So:

$ sudo ausearch -m avc -ts recent
<no matches>

$ sudo journalctl -k --since "-10min" | grep -i avc
$ sudo dmesg | grep -i avc

Nothing. Not one line, in any of the three usual places, with auditd confirmed active the whole time:

$ sudo systemctl is-active auditd
active

A clean audit log usually means SELinux isn’t the cause. Here, it wasn’t telling the truth - or rather, it was telling a very specific, deliberate lie. The container-selinux policy ships dontaudit rules for a number of container-vs-host access patterns that are common enough to be noisy. When one of those rules matches, the kernel denies the access and generates no AVC record at all - not filtered by the audit daemon, never written in the first place. semodule -DB will disable every dontaudit rule on the system and let you see what’s actually being blocked, but it’s a blunt, host-wide instrument you don’t want to leave on - useful for one debugging session, not something to run and forget.

Reproducing it live

Rather than keep guessing, I reproduced the failure interactively. A throwaway pod, mounting the exact same hostPath:

$ kubectl exec -it netdata-debug -- sh
/ # id
uid=0(root) gid=0(root) groups=0(root),1(bin),2(daemon),...

/ # ls -ld /persistencevarlibdir
drwxrwxrwx    2 root     root             6 Aug 19 05:52 /persistencevarlibdir

/ # touch /persistencevarlibdir/testfile
touch: /persistencevarlibdir/testfile: Permission denied

/ # mkdir /persistencevarlibdir/testdir
mkdir: can't create directory '/persistencevarlibdir/testdir': Permission denied

/ # chmod 777 /persistencevarlibdir
chmod: /persistencevarlibdir: Permission denied

Same failure, reproduced on demand, for every write operation - not just chmod. That ruled out anything specific to the init container’s exact command and pointed squarely at the mount itself. Two more commands, run against the host and the running container, gave the actual answer:

$ ps -eZ | grep sleep
system_u:system_r:container_t:s0:c396,c806   1770237 ?   00:00:00 sleep

$ stat -c '%C' /var/lib/netdata-k8s-child/var/lib/netdata
system_u:object_r:container_var_lib_t:s0

There it is. The container process runs under container_t - the confined SELinux domain every container gets on this host, tagged with its own per-instance MCS category pair. The directory kubelet auto-created (DirectoryOrCreate on first pod start) got labeled container_var_lib_t - the type meant for the container runtime’s own storage, like /var/lib/containers, not for an arbitrary directory an application bind-mounts and writes to. container_t has no create/write/setattr access to container_var_lib_t, full stop - and that specific denial happens to be one of the ones dontaudit hides.

The Fix

The correct type for a host directory a confined container needs to write to is container_file_t. Relabel it, make the relabel persistent, then let Kubernetes retry the pod:

# immediate fix
sudo chcon -Rt container_file_t /var/lib/netdata-k8s-child

# persist it - survives a future restorecon -R /
sudo dnf install -y policycoreutils-python-utils
sudo semanage fcontext -a -t container_file_t '/var/lib/netdata-k8s-child(/.*)?'
sudo restorecon -Rv /var/lib/netdata-k8s-child

Then delete the crash-looping pod so the DaemonSet recreates it:

$ kubectl delete pod -n netdata -l role=child
pod "netdata-child-mgjqj" deleted

$ kubectl get pods -n netdata
NAME                   READY   STATUS    RESTARTS   AGE
netdata-child-hc988    1/1     Running   0          22s
netdata-k8s-state-...  1/1     Running   0          19m
netdata-parent-...     1/1     Running   0          21m

Clean on the first restart after the relabel - no more crash loop.

Why Relabel Instead of the Easier Fixes

There are three shortcuts here that would also make the crash loop go away, and all three are worse than the fix above.

The laziest one is chmod -R 777 on the whole persistent volume - and it’s not hypothetical, it’s the actual accepted workaround in netdata/helmchart#123, an open issue against this exact chart hitting this exact hostPath. It works, once, on the files that exist at the moment you run it. It does nothing about the label, so a future restorecon -R / or a freshly recreated directory puts you right back where you started - and the issue thread itself calls the approach out as unsafe for production.

setenforce 0 is worse. It doesn’t fix this one directory’s label - it turns SELinux off for the entire host, for every process, indefinitely. Whatever SELinux was protecting against, on every other service running on that box, stops being protected the moment you run it, to solve a problem that’s scoped to one directory.

Running the pod --privileged, or dropping Pod Security enforcement for the namespace, is the same category of mistake wearing a Kubernetes costume: it removes the control that caught the misconfiguration instead of fixing the misconfiguration.

The relabel is the same number of keystrokes as any of the above. It fixes the actual defect - a directory labeled for the wrong purpose - rather than working around it, and once it’s persisted through semanage (or, better, through whatever configuration management already owns the host), it survives a rebuild the same way the rest of the system’s state does.

Worth Remembering

  1. A clean ausearch/journalctl is not proof SELinux isn’t the cause. container-selinux ships dontaudit rules for exactly this class of denial.
  2. container_var_lib_t is for the container runtime’s own storage, not for application bind mounts. container_file_t is the type you want for a hostPath a pod legitimately writes to.
  3. Reproduce with a throwaway debug pod before trusting the logs. ps -eZ for the process’s domain and MCS categories, stat for the target’s own context - that’s the actual evidence, not whatever the audit log did or didn’t capture.

Closing Thoughts

The general mechanism here isn’t new - confined container domains colliding with a host directory labeled for something else is a well-documented class of problem (see this Red Hat bugzilla and this containerd/Fedora CoreOS issue for the general shape of it). What’s a little more interesting is that the netdata Helm chart itself has a multi-year-old open issue - netdata/helmchart#123 - describing this exact hostPath producing this exact error, and the thread never gets past “chmod 777 the PV, but that’s not safe for production.” If you found this post by searching that error string, the actual cause is a silenced SELinux denial, and the actual fix is a container_file_t relabel, not a permissions workaround.

If you enjoyed a good dontaudit mystery, I’ve run into SELinux quietly blocking something that looked fine on paper before - see the GNOME Boxes ISO permission story and a systemd unit failing 203/EXEC because of a mislabeled binary.

Hope this helps!

infrastructurelinux server kubernetesk3sselinuxnetdatacontainerddevops