Turning on metrics-server's /metrics made its health checks call the apiserver
Sai Pisey4 min readKubernetes · metrics-server · Helm
In short
- The metrics-server Helm chart sets --authorization-always-allow-paths=/metrics when metrics.enabled is true. That flag replaces the default list (/healthz, /readyz, /livez) instead of adding to it, so liveness and readiness probes are authorized with SubjectAccessReviews to the apiserver.
- On a normal cluster the review is allowed and nothing looks wrong. If metrics-server can't reach the apiserver for longer than about 25 seconds, its probes return 500, the kubelet kills it, and in my runs it came back 69 to 76 seconds after the apiserver did.
- Charts 3.8.0 to 3.14.0 are affected and the fix isn't in a chart release yet. Add --authorization-always-allow-paths=/healthz,/readyz,/livez to args until it is.
If you run metrics-server from its Helm chart and scrape it with Prometheus, you've probably set metrics.enabled: true. The chart needs it before it will create a ServiceMonitor. The README describes it as "allow unauthenticated access to /metrics", which is what it does.
It also does something the README doesn't mention. It makes every health check metrics-server answers depend on the Kubernetes apiserver.
The person who noticed this in October, and had already worked around it in their own clusters, opened kubernetes-sigs/metrics-server#1903. They traced it to a chart change from January 2022 and suggested the one-line fix. I sent the PR, #1904, with before and after numbers from a kind cluster, and the chart maintainer merged it on 12 November. It isn't in a chart release yet, so this post is mostly about the workaround.
One flag, one value
metrics-server is built on the same library as the apiserver, k8s.io/apiserver. Requests to its HTTPS port are authorized by asking the real apiserver, with a SubjectAccessReview, whether the caller may do this. A short list of paths skips that check. The library's default for that list is:
AlwaysAllowPaths: []string{"/healthz", "/readyz", "/livez"},The chart wants /metrics on the list too, so with metrics.enabled: true it renders this:
- --authorization-always-allow-paths=/metricsThe flag is a string slice. The first time a string slice flag is set, it throws the default away and keeps only what you gave it. So the list is now just /metrics, and the three health paths go back to needing a review.
What the probes do now
The chart's probes hit /livez and /readyz every 10 seconds. The kubelet sends them without credentials, so metrics-server sees system:anonymous and asks the apiserver whether anonymous may get /livez.
On a stock cluster the answer is yes. The built-in system:public-info-viewer ClusterRole lets unauthenticated users read /healthz, /livez, /readyz and /version. The probe passes, the pod is Ready, and nothing in kubectl get pods looks different.
default list · metrics.enabled: false
chart 3.8.0 to 3.14.0 · metrics.enabled: true
I measured the extra traffic with an apiserver audit log that records every SubjectAccessReview. In 90 seconds, chart 3.14.0 with metrics.enabled: true sent 5 reviews for /livez and 5 for /readyz. With the setting off, or with the fix, it sent none. Allowed answers are cached for 10 seconds, the same as the probe interval, so about every other probe is answered from the cache and the rest go to the apiserver.
Seven requests a minute won't hurt anyone's apiserver. The cost is somewhere else.
When the apiserver goes away
A health check that calls another service now fails when that service does. To see what that looks like, I stopped the apiserver on the kind node for 60 seconds by moving its static pod manifest out of /etc/kubernetes/manifests, and polled /livez on metrics-server the whole time.
With chart 3.14.0 and metrics.enabled: true, /livez started failing within 5 seconds:
Internal Server Error: "/livez": Post "https://10.96.0.1:443/apis/authorization.k8s.io/v1/subjectaccessreviews?timeout=10s": dial tcp 10.96.0.1:443: connect: connection refusedThe probe's failureThreshold is 3 at a 10 second period, so after about 25 seconds of 500s the kubelet killed the container. The new container couldn't start at all. metrics-server reads the extension-apiserver-authentication ConfigMap at startup, and with no apiserver it panics:
panic: unable to load configmap based request-header-client-ca-file: Get "https://10.96.0.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication": dial tcp 10.96.0.1:443: connect: connection refusedSo it went into CrashLoopBackOff, and each crash made the next back-off longer. When the apiserver came back, metrics-server was still waiting out its back-off. It was Ready 69 to 76 seconds later, with 5 restarts on the counter. I ran this four times on a fresh install and got 69, 70, 72 and 76 seconds.
With the fixed chart, /livez returned 200 for the whole minute, the container never restarted, and it was Ready 1 second after the apiserver was.
500, killed at ~25s, CrashLoopBackOff, Ready ~70s after the apiserver
200 every poll, 0 restarts, Ready 1s after
That's more than a minute when kubectl top says "Metrics API not available" and the HPA controller gets no resource metrics, on top of the outage itself.
Short blips are survivable. With a 30 second stop, /livez returned 500 for most of it, but the kubelet hadn't collected enough failures to act, and the container didn't restart. The trouble starts when metrics-server can't reach the apiserver for longer than that, which can happen during a slow control plane upgrade or a network partition. Without the flag, none of that touches metrics-server's own health.
There's one more case. If your cluster has removed or tightened system:public-info-viewer as part of hardening, the review is denied, the probes get 403, and metrics-server crash-loops even with a healthy apiserver. I checked that too.
Am I affected?
Only the Helm chart is affected, from 3.8.0 (February 2022) to 3.14.0, the latest release. The plain components.yaml manifest doesn't set the flag.
Check what your Deployment is actually running:
kubectl -n kube-system get deploy -l app.kubernetes.io/name=metrics-server \
-o yaml | grep always-allowIf the only match is --authorization-always-allow-paths=/metrics, you have it. If you see /healthz,/readyz,/livez in there too, you're already covered.
The workaround
Add the default paths back yourself through the chart's args:
metrics:
enabled: true
args:
- --authorization-always-allow-paths=/healthz,/readyz,/livezThe chart still renders its own /metrics flag, so the container ends up with the flag twice. That's fine. The first value replaces the default and every later one is appended, so the result is all four paths. I ran this against chart 3.14.0. It sent zero reviews in 90 seconds, /livez and /readyz both returned 200 when I checked them 30 seconds into the same 60 second apiserver stop, and the container didn't restart.
configuration
reviews / 90s
apiserver down
restarts
metrics offchart 3.14.0 default0not runn/ametrics onchart 3.14.0105005metrics on + argchart 3.14.0, workaround0200 at 30s0metrics onfixed chart (#1904)02000When a chart release with the fix ships, remove the extra arg. Leaving it in does no harm either, since it repeats paths that are already on the list.
The fix itself is one line in templates/deployment.yaml, listing the defaults next to /metrics, which is exactly what the issue suggested:
- - --authorization-always-allow-paths=/metrics
+ - --authorization-always-allow-paths=/healthz,/readyz,/livez,/metricsThe same trap is waiting in any component built on k8s.io/apiserver. If you set --authorization-always-allow-paths to add one path, list the defaults with it.