One header in the config. A thousand on the wire.

In late August someone opened an issue on Grafana Alloy. They had one header configured on a pyroscope.scrape component, an X-Api-Key. Profiling ran fine for about two hours and then stopped. Their packet capture showed a single GET carrying roughly 1,000 copies of that X-Api-Key line, and the target answering 431 Request Header Fields Too Large before the request reached any handler.

They'd already done the hard part. The issue pointed at cloneRequest in prometheus/common, the library that turns an http_config block into a Go HTTP client for Prometheus, Alertmanager, the exporters and Grafana Alloy, and named the commit that broke it. I wrote the upstream fix, #982, which shipped in v0.72.1 on 11 October 2026. Then I went looking for who else was exposed, which took longer than the fix.

#Reproducing it

Build a client from a config with one header, send the same request four times, and count what the server sees:

$ go run .   # prometheus/common v0.71.0
server saw X-Api-Key x1
server saw X-Api-Key x2
server saw X-Api-Key x3
server saw X-Api-Key x4

$ go run .   # prometheus/common v0.72.1
server saw X-Api-Key x1
server saw X-Api-Key x1
server saw X-Api-Key x1
server saw X-Api-Key x1

It keeps climbing until the request crosses whatever header limit the server has. Go's default is 1 MB in total. Proxies are usually much stricter, so how long it takes to break depends on what sits in front of your target.

#The copy that copied nothing

Go's http.RoundTripper docs say a round tripper "should not modify the request". So every round tripper in the library that adds something to a request clones it first. From v0.69.0 the clone looked like this:

func cloneRequest(r *http.Request) *http.Request {
	// Shallow copy of the struct.
	r2 := new(http.Request)
	*r2 = *r
	// Deep copy of the Header.
	maps.Copy(r.Header, r2.Header)
	return r2
}

The comment says deep copy. It isn't one. *r2 = *r copies the struct, and the Header field inside it is a map, so both requests now point at the same map. maps.Copy then copies that map into itself. Whatever the round tripper adds next goes straight into the caller's request.

It slipped in during a linter-driven "Modernize Go" cleanup that swapped a hand-written loop for maps.Copy. The old loop opened with r2.Header = make(http.Header). That line didn't survive the rewrite, and nothing in the tests noticed.

The fix gives the clone its own map:

	r2.Header = r.Header.Clone()
	if r2.Header == nil {
		r2.Header = make(http.Header)
	}

#Which settings make it grow

On an affected version, every round tripper that calls cloneRequest writes into the caller's request. Only one of them piles up:

http_headers uses Header.Add, so each send appends the configured values again, whether they're literal values, secrets or read from files. authorization, basic_auth, the user agent and the host header use Set. They still write into your request, but they overwrite the same value, so you'd never see a duplicate. oauth2 doesn't use cloneRequest at all.

#Why Prometheus never hit it

Prometheus 3.15 pins the affected v0.71.0, and its scraper keeps one cached request per target and sends it every scrape, the same pattern as the Alloy component. I ran a real 3.15.0 binary with http_headers set anyway. Across 18 scrapes the target saw the header once each time.

The answer is one layer up. Prometheus wraps its scrape transport in OpenTelemetry's otelhttp.NewTransport, and that wrapper calls r.Clone(ctx) (a proper deep copy) before prometheus/common ever sees the request. So every scrape starts from a fresh copy and the cached one stays clean. Alloy's prometheus.scrape uses the same wrapper.

Alloy's pyroscope.scrape caches its request too, but sends it without the wrapper. That's the path from the original report.

#Checking your own binaries

The library version is compiled into every Go binary, so you don't need the source:

$ go version -m ./alloy | grep prometheus/common
	dep	github.com/prometheus/common	v0.71.0	h1:...

• v0.69.0 to v0.72.0: affected.
• v0.72.1 or later: fixed. Anything older than v0.69.0 never had the bug.

You need three things to actually hit it: an affected version, an http_headers block, and code that sends the same *http.Request more than once without cloning it. Here's where the tools I checked stood on 24 October 2026:

If you run pyroscope.scrape with http_headers, keep an eye out for an Alloy release on prometheus/common v0.72.1, and watch your profiling targets for 431s until then. The Alloy issue was still open when I wrote this.

#If you write Go HTTP clients

One test that sends the same request twice would have caught this. A few habits for anyone writing a RoundTripper:

1. Don't touch the incoming request. Start with req = req.Clone(req.Context()), which copies the headers for real, and skip the hand-rolled struct copy.
2. Treat *a = *b as sharing: the copy still points at the original's maps, slices and pointers, so give each one its own allocation.
3. Reuse one *http.Request across several calls in your tests and count the headers on the server side. A test that builds a fresh request every time can't see this bug.

That last one is the regression test that went in with #982.