The setting called refresh_interval that hung up on Docker every minute
Sai Pisey4 min readGrafana Alloy · Loki · Docker
In short
- From v1.0.0 to v1.20.1, loki.source.docker set Go's http.Client.Timeout to refresh_interval (60s by default) for Docker hosts reached over http:// or https://. That timeout covers reading the body, so every followed log stream was cut once a minute.
- Each reconnect asks Docker for logs since a whole second, and Docker's filter is inclusive, so the last second of lines comes back twice.
- It is fixed in Alloy v1.21.0. On older versions, check your logs for 'Client.Timeout or context cancellation while reading body'. Promtail has the same line and still ships it.
In early October someone opened an issue on Grafana Alloy. They ship container logs to Loki with loki.source.docker, and they'd put a docker-socket-proxy in front of the Docker socket, the way hardening guides usually suggest. Every minute, every log stream dropped. The proxy's own logs showed each follow request ending at 60,018 ms and 60,002 ms. They tried refresh_interval = "65s" and the cut moved to 65 seconds.
That was the whole clue. A setting that sounds like it controls how often to look for new containers was really deciding how long a log stream was allowed to live.
I wrote the fix, #7297. It shipped in Alloy v1.21.0 on 19 October 2026.
The symptom
If you're on an affected version and reach Docker over http:// or https://, your Alloy logs have a line like this once a minute for every container:
level=warn msg="could not transfer logs" component_id=loki.source.docker.default
err="context deadline exceeded (Client.Timeout or context cancellation while reading body)"Some runs show net/http: request canceled (Client.Timeout or context cancellation while reading body) instead. Same cause.
Nothing else looks wrong. Logs keep arriving in Loki, because Alloy reconnects a few seconds later. The only other trace is a handful of duplicate lines.
cut at 60s, 125s, 190s · warning logged each time
one stream, no timeout
one stream, no timeout
Where the timeout came from
In v1.20.1 the component builds its own HTTP client when the Docker host is http:// or https://:
if hostURL.Scheme == "http" || hostURL.Scheme == "https" {
// ...
client.WithHTTPClient(&http.Client{
Transport: rt,
Timeout: args.RefreshInterval, // 60s by default
})
}Go's http.Client.Timeout is easy to read as a connect timeout, but its docs say it covers "connection time, any redirects, and reading the response body". A followed log stream is a response body that never ends. So with a 60 second Timeout, every GET /containers/{id}/logs?follow=1 gets killed at 60 seconds, every time.
unix:// and tcp:// hosts skip that branch and use the Docker client's default HTTP client, which has no timeout. That's why most people never saw it. If you mount /var/run/docker.sock straight into Alloy, you were fine. The bug only hit people who did the safer thing and put a proxy in front.
host
HTTP client used
v1.0.0 to v1.20.1
The line wasn't new. It came over from Promtail in 2023, when Grafana Agent ported its Docker target, and stayed through every Alloy release from v1.0.0 to v1.20.1.
Why the lines come back twice
After a cut, Alloy restarts the tailer about five seconds later and asks Docker for logs since the last timestamp it saw. It stores that timestamp in whole seconds. Docker's since filter is inclusive, so it sends back everything from the start of that second, including lines Alloy already shipped.
what Loki receives
the reconnect request
since has no fraction of a second, and Docker returns lines with a timestamp at or after it. The half second already shipped comes back.
One reconnect a minute means a burst of duplicates a minute, per container. On a chatty service that's a steady trickle of doubled lines in Loki.
What changed in the fix
My first version dropped the client timeout and gave the short container-inspect calls their own deadline. The reviewer, who maintains the component, pushed for something cleaner in the first round: using refresh_interval as a timeout had been a mistake, so deprecate it, make it do nothing, and let every request use the context the tailer already runs with. That's what merged. Two smaller rounds tidied the context handling.
So in v1.21.0:
- No timeout on the Docker HTTP client. Streams live until the container stops or Alloy shuts down.
refresh_intervalstill parses, but the docs now say "Deprecated. This argument has no effect."- Every Docker request follows the tailer's own context, so shutdown still cancels it cleanly.
Before sending it, I ran the old build and the fixed build side by side against the same container through a socket proxy. The container printed one numbered line a second, and refresh_interval was 10s to make the cuts come faster:
build
lines received
duplicates
timeout warnings
pre-fix build9856fixed build9500The old build logged six timeouts in about 95 seconds and delivered five lines twice, one per reconnect. Its sixth cut came near the end, so it was still reconnecting when the run stopped at line-92. The fixed build delivered all 95 lines once, in order.
What to check
- Find the warning. Search Alloy's logs for
Client.Timeout or context cancellation while reading body. If it repeats on a steady interval for every container, you have this bug. - Check your host scheme. Only
http://andhttps://Docker hosts are affected.unix://andtcp://aren't. - Upgrade to Alloy v1.21.0 or later. Then remove
refresh_intervalfrom your config; it does nothing now.
If you can't upgrade yet, there are two workarounds, each with a catch:
- Switch the host to
tcp://. Streams stop being cut, buthttp_client_configis only applied on thehttp/httpspath, so any TLS or basic auth you configured there is silently dropped. - Set a very large
refresh_interval, like"8760h". In v1.20.x it has no other runtime effect, but it has to stay above zero.
Promtail still has it
The same line exists in Promtail's Docker target, Timeout: time.Duration(tg.refreshInterval), and it's still there in Loki v3.6.20, released on 6 November 2026. Promtail's code has since been removed from Loki's main branch, so a fix there is unlikely. If you still run Promtail with docker_sd_configs against an http:// host, expect the same cut every refresh interval. The way out is to move to Alloy v1.21.0 or later.
The bug report that started this is grafana/alloy#7290, and the fix is #7297.