One command tells you if your next Kubernetes upgrade will brick a node

Linux has two versions of cgroups, the kernel feature that the kubelet and your container runtime use to put CPU and memory limits on pods. Kubernetes has supported cgroup v2 as GA since v1.25, and it is now phasing v1 out.

Since v1.35, the kubelet setting failCgroupV1 defaults to true. On a node that still runs cgroup v1, the kubelet refuses to start, so the node goes NotReady and stays there. On a control-plane node it is worse: the kubelet is what starts the API server and etcd as static pods, so after a reboot they do not come back either.

kubeadm warns you first. Its SystemVerification preflight check turns from a warning into an error on init, join and upgrade once the kubelet is v1.35 or later. Treat that error as the early warning it is, and do not skip it with --ignore-preflight-errors.

Finding those nodes takes one command.

#Check one node

On the node itself:

$ stat -fc %T /sys/fs/cgroup/
cgroup2fs

There are only two answers:

• cgroup2fs: the node runs cgroup v2. Nothing to do.
• tmpfs: the node runs cgroup v1. After an upgrade to v1.35 or later, its kubelet will not start.

#Check every node without SSH

You do not need SSH access. kubectl debug starts a pod on the node with the host's root filesystem mounted at /host, and chroot runs the host's own stat:

$ for n in $(kubectl get nodes -o name); do
    printf "%s " "$n"
    kubectl debug "$n" -n default -q -i --profile=general --image=busybox \
      -- chroot /host stat -fc %T /sys/fs/cgroup
  done
node/worker-1 cgroup2fs
node/worker-2 tmpfs

Two things I hit while testing this on a v1.35 cluster:

• Keep the chroot /host. Without it the command runs busybox's own stat, which does not know the cgroup v2 filesystem type and prints UNKNOWN instead of cgroup2fs. The chroot uses the node's GNU stat, which answers correctly.
• Clean up afterwards. kubectl debug leaves one node-debugger-* pod behind per node:

$ kubectl get pods -n default -o name | grep node-debugger | xargs kubectl delete -n default

#Which nodes are likely on cgroup v1

Anything on an older distribution. These default to cgroup v1:

• CentOS 7 and RHEL 7
• RHEL 8
• Ubuntu 18.04 and 20.04
• Amazon Linux 2

These default to cgroup v2: Ubuntu 21.10 and later, Debian 11 and later, RHEL 9, Fedora 31 and later, and Container-Optimized OS from M97.

Self-managed kubeadm clusters on long-lived VMs are where this bites, along with local labs built on old images. Managed node images on recent versions mostly use cgroup v2 already, but the check takes seconds, so run it on every cluster.

#What a node needs for cgroup v2

From the Kubernetes documentation:

• Linux kernel 5.8 or later
• containerd 1.4 or later, or CRI-O 1.20 or later
• The systemd cgroup driver, on both the kubelet and the container runtime

Check the drivers on a kubeadm node:

$ grep cgroupDriver /var/lib/kubelet/config.yaml
cgroupDriver: systemd
$ grep SystemdCgroup /etc/containerd/config.toml
            SystemdCgroup = true

If the two disagree, fix that first. A kubelet and runtime on different cgroup drivers is its own outage. On containerd 2.0 or later the kubelet reads the driver from containerd and ignores its own cgroupDriver, so the containerd setting is the one that counts.

#Moving a node to cgroup v2

The kernel decides how. Check it first with uname -r.

Kernel too old, or v2 unsupported: replace the node. Most cgroup v1 nodes are here:

• CentOS 7 and RHEL 7: kernel 3.10.
• RHEL 8: kernel 4.18. Red Hat supports cgroup v2 from 8.2, but the kernel is still below the documented 5.8. Plan the move to RHEL 9.
• Amazon Linux 2: AWS does not support cgroup v2 on it at all. Move to Amazon Linux 2023.

Bring up new nodes on a cgroup v2 image (Ubuntu 22.04 or 24.04, RHEL 9, AL2023), then drain and remove the old ones. On a managed service that is a new node pool or node group.

Kernel 5.8 or later: switch in place. For example Ubuntu 20.04 after installing the 5.15 HWE kernel with sudo apt install linux-generic-hwe-20.04. One node at a time:

1. kubectl drain <node> --ignore-daemonsets --delete-emptydir-data
2. Add the kernel argument systemd.unified_cgroup_hierarchy=1 to GRUB_CMDLINE_LINUX in /etc/default/grub, then run sudo update-grub.
3. Reboot, then run the stat check again. It should print cgroup2fs.
4. kubectl uncordon <node>

#The escape hatch, and its expiry date

If you have to upgrade before every node is moved, you can let the kubelet run on cgroup v1 by setting this in its configuration:

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
failCgroupV1: false

It buys time and fixes nothing. Per KEP-5573, cgroup v1 support is slated for removal no earlier than v1.38, whose release is scheduled for 16 December 2026. The same KEP notes that systemd 258 drops cgroup v1 support too, so the operating system is moving in the same direction.

#In short

1. Run stat -fc %T /sys/fs/cgroup/ on every node, or the kubectl debug loop above.
2. tmpfs means cgroup v1: that node will not run a v1.35+ kubelet with default settings.
3. Move those nodes to cgroup v2 before your next upgrade, or replace the OS where the kernel is too old.
4. Use failCgroupV1: false only as a bridge.

Sources: the Kubernetes blog post The Shift to cgroup v2 in Kubernetes, the cgroup v2 documentation, and KEP-5573.