Skip to content
← cd ../blog

Turning on metrics-server's /metrics made its health checks call the apiserver

Sai Pisey4 min readKubernetes · metrics-server · Helm

In short

  • The metrics-server Helm chart sets --authorization-always-allow-paths=/metrics when metrics.enabled is true. That flag replaces the default list (/healthz, /readyz, /livez) instead of adding to it, so liveness and readiness probes are authorized with SubjectAccessReviews to the apiserver.
  • On a normal cluster the review is allowed and nothing looks wrong. If metrics-server can't reach the apiserver for longer than about 25 seconds, its probes return 500, the kubelet kills it, and in my runs it came back 69 to 76 seconds after the apiserver did.
  • Charts 3.8.0 to 3.14.0 are affected and the fix isn't in a chart release yet. Add --authorization-always-allow-paths=/healthz,/readyz,/livez to args until it is.

If you run metrics-server from its Helm chart and scrape it with Prometheus, you've probably set metrics.enabled: true. The chart needs it before it will create a ServiceMonitor. The README describes it as "allow unauthenticated access to /metrics", which is what it does.

It also does something the README doesn't mention. It makes every health check metrics-server answers depend on the Kubernetes apiserver.

The person who noticed this in October, and had already worked around it in their own clusters, opened kubernetes-sigs/metrics-server#1903. They traced it to a chart change from January 2022 and suggested the one-line fix. I sent the PR, #1904, with before and after numbers from a kind cluster, and the chart maintainer merged it on 12 November. It isn't in a chart release yet, so this post is mostly about the workaround.

One flag, one value

metrics-server is built on the same library as the apiserver, k8s.io/apiserver. Requests to its HTTPS port are authorized by asking the real apiserver, with a SubjectAccessReview, whether the caller may do this. A short list of paths skips that check. The library's default for that list is:

go
AlwaysAllowPaths: []string{"/healthz", "/readyz", "/livez"},

The chart wants /metrics on the list too, so with metrics.enabled: true it renders this:

yaml
- --authorization-always-allow-paths=/metrics

The flag is a string slice. The first time a string slice flag is set, it throws the default away and keeps only what you gave it. So the list is now just /metrics, and the three health paths go back to needing a review.

library defaultk8s.io/apiserver
/healthz/readyz/livez
chart passesmetrics.enabled: true
/metrics
flag holdsfirst value replaces the default
/healthz/readyz/livez/metrics
with the fixdefaults listed too
/healthz/readyz/livez/metrics
A string slice flag drops its default the first time it is set. The chart only meant to add /metrics, but the three health paths fall off the list and go back to needing a SubjectAccessReview.

What the probes do now

The chart's probes hit /livez and /readyz every 10 seconds. The kubelet sends them without credentials, so metrics-server sees system:anonymous and asks the apiserver whether anonymous may get /livez.

On a stock cluster the answer is yes. The built-in system:public-info-viewer ClusterRole lets unauthenticated users read /healthz, /livez, /readyz and /version. The probe passes, the pod is Ready, and nothing in kubectl get pods looks different.

default list · metrics.enabled: false

kubelet
GET /livez
metrics-server
always allowed
200 ok

chart 3.8.0 to 3.14.0 · metrics.enabled: true

kubelet
GET /livez
metrics-server
SubjectAccessReview
apiserverpublic-info-viewer
allowed
200 ok
With the default list, metrics-server answers /livez on its own. With the chart's flag, it first asks the apiserver whether system:anonymous may read /livez. On a stock cluster system:public-info-viewer says yes, so the probe passes and the extra hop is invisible.

I measured the extra traffic with an apiserver audit log that records every SubjectAccessReview. In 90 seconds, chart 3.14.0 with metrics.enabled: true sent 5 reviews for /livez and 5 for /readyz. With the setting off, or with the fix, it sent none. Allowed answers are cached for 10 seconds, the same as the probe interval, so about every other probe is answered from the cache and the rest go to the apiserver.

Seven requests a minute won't hurt anyone's apiserver. The cost is somewhere else.

When the apiserver goes away

A health check that calls another service now fails when that service does. To see what that looks like, I stopped the apiserver on the kind node for 60 seconds by moving its static pod manifest out of /etc/kubernetes/manifests, and polled /livez on metrics-server the whole time.

With chart 3.14.0 and metrics.enabled: true, /livez started failing within 5 seconds:

terminal
Internal Server Error: "/livez": Post "https://10.96.0.1:443/apis/authorization.k8s.io/v1/subjectaccessreviews?timeout=10s": dial tcp 10.96.0.1:443: connect: connection refused

The probe's failureThreshold is 3 at a 10 second period, so after about 25 seconds of 500s the kubelet killed the container. The new container couldn't start at all. metrics-server reads the extension-apiserver-authentication ConfigMap at startup, and with no apiserver it panics:

terminal
panic: unable to load configmap based request-header-client-ca-file: Get "https://10.96.0.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication": dial tcp 10.96.0.1:443: connect: connection refused

So it went into CrashLoopBackOff, and each crash made the next back-off longer. When the apiserver came back, metrics-server was still waiting out its back-off. It was Ready 69 to 76 seconds later, with 5 restarts on the counter. I ran this four times on a fresh install and got 69, 70, 72 and 76 seconds.

With the fixed chart, /livez returned 200 for the whole minute, the container never restarted, and it was Ready 1 second after the apiserver was.

apiserver stopped · 60sapiserver back · next 70s
chart 3.14.0

500, killed at ~25s, CrashLoopBackOff, Ready ~70s after the apiserver

fixed chart

200 every poll, 0 restarts, Ready 1s after

200 / Ready500 from the reviewkilled, restart panickednot Ready, back-off
Each cell is 5 seconds, from one of four runs. While the apiserver was stopped I polled /livez. On chart 3.14.0 the first five polls got a 500 from the failed review, then the kubelet killed the container, and every restart panicked at startup because it couldn't reach the apiserver either. After the apiserver came back I tracked Ready, which took 69 to 76 seconds of back-off, with 5 restarts. With the fix, every poll was 200 and it was Ready 1 second after the apiserver.

That's more than a minute when kubectl top says "Metrics API not available" and the HPA controller gets no resource metrics, on top of the outage itself.

Short blips are survivable. With a 30 second stop, /livez returned 500 for most of it, but the kubelet hadn't collected enough failures to act, and the container didn't restart. The trouble starts when metrics-server can't reach the apiserver for longer than that, which can happen during a slow control plane upgrade or a network partition. Without the flag, none of that touches metrics-server's own health.

There's one more case. If your cluster has removed or tightened system:public-info-viewer as part of hardening, the review is denied, the probes get 403, and metrics-server crash-loops even with a healthy apiserver. I checked that too.

Am I affected?

Only the Helm chart is affected, from 3.8.0 (February 2022) to 3.14.0, the latest release. The plain components.yaml manifest doesn't set the flag.

Helm chart, metrics.enabled: falsenot affectedthe default, flag not rendered
Helm chart 3.8.0 to 3.14.0, metrics.enabled: trueaffectedrequired for serviceMonitor.enabled
Helm chart with #1904fixedmerged 12 Nov 2026, not yet released
components.yaml manifestnot affecteddoesn't set the flag
The chart's ServiceMonitor requires metrics.enabled, so any install that turns on serviceMonitor.enabled has the flag. Chart releases before 3.8.0 didn't render it.

Check what your Deployment is actually running:

bash
kubectl -n kube-system get deploy -l app.kubernetes.io/name=metrics-server \
  -o yaml | grep always-allow

If the only match is --authorization-always-allow-paths=/metrics, you have it. If you see /healthz,/readyz,/livez in there too, you're already covered.

The workaround

Add the default paths back yourself through the chart's args:

yaml
metrics:
  enabled: true
args:
  - --authorization-always-allow-paths=/healthz,/readyz,/livez

The chart still renders its own /metrics flag, so the container ends up with the flag twice. That's fine. The first value replaces the default and every later one is appended, so the result is all four paths. I ran this against chart 3.14.0. It sent zero reviews in 90 seconds, /livez and /readyz both returned 200 when I checked them 30 seconds into the same 60 second apiserver stop, and the container didn't restart.

configuration

reviews / 90s

apiserver down

restarts

metrics offchart 3.14.0 default0not runn/ametrics onchart 3.14.0105005metrics on + argchart 3.14.0, workaround0200 at 30s0metrics onfixed chart (#1904)02000
kind v1.36.1, one node. Reviews are SubjectAccessReviews for /livez and /readyz sent by metrics-server in 90 seconds, from the apiserver audit log. The other two columns come from the 60 second apiserver stop. With the fix, /metrics still answered 200 without credentials and kubectl top nodes worked.

When a chart release with the fix ships, remove the extra arg. Leaving it in does no harm either, since it repeats paths that are already on the list.

The fix itself is one line in templates/deployment.yaml, listing the defaults next to /metrics, which is exactly what the issue suggested:

diff
-            - --authorization-always-allow-paths=/metrics
+            - --authorization-always-allow-paths=/healthz,/readyz,/livez,/metrics

The same trap is waiting in any component built on k8s.io/apiserver. If you set --authorization-always-allow-paths to add one path, list the defaults with it.

Ready to Build Something Resilient?

Whether it's hardening your platform, untangling a messy CI/CD setup, or just talking infrastructure over a call, let's connect. I'm always up for a good systems conversation.

Book a call

$ ./contact.sh

Get in Touch

I only use these details to reply to you. Ask and I'll delete them.