Skip to content

$ ls ./blog

Blog

Field notes from production infrastructure and from fixes merged upstream: real output, real numbers, what broke and why.

RSS feed
  1. metrics.enabled: true/healthz/readyz/livez/metricsevery probe asks the apiserverkubeletmetrics-serverapiserver200apiserver gone 1 min → metrics-server crash-loopschart 3.8.0 to 3.14.0fix merged · #1904

    Turning on metrics-server's /metrics made its health checks call the apiserver

    In metrics-server Helm chart 3.8.0 to 3.14.0, metrics.enabled=true replaces the apiserver's default always-allow paths, so /livez and /readyz probes are authorized by the apiserver. Lose the apiserver for a minute and metrics-server crash-loops.

    • Kubernetes
    • metrics-server
    • Helm

    · 4 min read

  2. docker → proxy (http://) → alloyv1.20.1 · cut every 60sv1.21.0 · one streamrefresh_interval was the HTTP timeoutv1.0.0 to v1.20.1fixed in v1.21.0

    The setting called refresh_interval that hung up on Docker every minute

    Before v1.21.0, Grafana Alloy used refresh_interval as an HTTP timeout for Docker hosts on http:// and https://, so every log stream was cut every minute and lines were duplicated.

    • Grafana Alloy
    • Loki
    • Docker

    · 4 min read

  3. http_headers: X-Api-Key (once)req 1x1req 2x2req 3x3…2h laterx~1,000431v0.69 to v0.72.0fixed in v0.72.1

    One header in the config. A thousand on the wire.

    prometheus/common v0.69.0 to v0.72.0 append every configured http_headers value again on each reused request. How it happened, who is exposed, and how to check a binary.

    • Prometheus
    • Go
    • Observability

    · 3 min read

  4. $ stat -fc %T /sys/fs/cgroup/worker-1cgroup2fs/kubepodspod-akubelet v1.35Readyworker-2tmpfs/cpupod/mempod/pidspodkubelet v1.35NotReadyv2 · finev1 · kubelet down

    One command tells you if your next Kubernetes upgrade will brick a node

    Since Kubernetes v1.35 the kubelet refuses to start on cgroup v1 nodes. One command shows which of your nodes are affected, and how to move them to cgroup v2.

    • Kubernetes
    • Linux
    • Upgrades

    · 4 min read

  5. us-east-1aus-east-1bus-east-1cZONE DOWNcheckout-api LOST 3/3fix: topology spread ✓

    Your cluster says 3/3. It will still go dark with one zone.

    Deployments say 3/3, yet one zone outage can take them down. kubectl survive-zone shows which Kubernetes workloads stop serving, from real placement.

    • Kubernetes
    • Scheduling
    • Reliability

    · 4 min read