Steady ~15 MiB/day memory growth on an idle gateway (2026.2.4, Linux container)

[alexander-zimmermann]

Summary

devolutions-gateway 2026.2.4 grows its RSS by a constant ~15.3 MiB/day while completely idle. The growth is linear, does not depend on traffic, and does not level off. In a container with a 256 MiB memory limit this ends in an OOM kill after about 16 days; on a host without a limit it would just keep climbing.

Environment

• Image: docker.io/devolutions/devolutions-gateway:2026.2.4 (sha256:f9ea0f4d…54887), amd64
• Runtime: Kubernetes 1.34 / containerd, Talos Linux, cgroup v2, memory limit 256 MiB
• Started with the packaged binary directly rather than the pwsh entrypoint:
  /opt/devolutions/gateway/devolutions-gateway --config-path /var/lib/devolutions-gateway
• Single instance, one HTTP listener, web app enabled with "Authentication": "None" (an external reverse proxy does the authentication)

{
  "Listeners": [{ "InternalUrl": "http://*:7171", "ExternalUrl": "https://*:443" }],
  "WebApp": { "Enabled": true, "Authentication": "None" },
  "ProvisionerPublicKeyFile": "…/provisioner.pub",
  "ProvisionerPrivateKeyFile": "…/provisioner.key",
  "JrlFile": "…/jrl.json",
  "TlsVerifyStrict": true
}

Observed behaviour

container_memory_rss, one sample per day, over a single 15.7-day process lifetime:

08-22  38.4 MiB
08-23  53.5 MiB   (+15.1)
08-24  68.6 MiB   (+15.1)
08-25  83.8 MiB   (+15.2)
08-26  99.1 MiB   (+15.3)
08-27 114.5 MiB   (+15.4)
08-28 129.9 MiB   (+15.4)
08-29 145.3 MiB   (+15.4)
08-30 160.5 MiB   (+15.2)
08-31 175.8 MiB   (+15.3)
09-01 191.6 MiB   (+15.8)
09-02 207.7 MiB   (+16.1)
09-03 223.5 MiB   (+15.9)
09-04 240.1 MiB   (+16.5)
09-05          -> OOMKilled at 253.9 MiB against a 256 MiB limit

Page cache is not involved: container_memory_cache was 0.2 MiB at the time of the kill, so all of this is anonymous memory in the process.

Why it looks usage-independent

• Between process start and the first user request there were 12 days with zero traffic — the slope over that stretch is identical to the rest.
• The last log line before the OOM was written ~40 h earlier (the end of an RDP session). Nothing was logged in those 40 h, and the curve keeps rising at the same rate.
• The replacement process, which nobody has used at all, is leaking at 15.26 MiB/day — measured as deriv(container_memory_rss[90m]).

So something on a timer or on a periodically-repeated request path accumulates memory.

Hypothesis

The only recurring input to the idle process is the health endpoint: a readiness probe every 20 s and a liveness probe every 60 s, i.e. ~4320 requests/day to GET /jet/health. That works out to roughly 3.7 KB retained per health request, which is in the right ballpark for a per-request leak (a retained span, connection record, or similar).

This is a hypothesis, not a proof — I have not yet run the process with the probes disabled to confirm. One detail that may matter: requests to /jet/health do not appear in the devolutions_gateway::middleware::log output, while every other request does, so they seem to take a different path through the stack.

Steps to reproduce (expected)

1. Run 2026.2.4 in a container with the config above and no user traffic.
2. Poll GET /jet/health every 20 s.
3. Sample RSS hourly for 48 h.

Expected: a straight line at roughly +0.64 MiB/h.

Questions

• Is a background task (job queue, traffic audit, JRL refresh, mDNS/network-scanner daemon) known to retain memory when idle?
• Is there a build or an env var that would let me hand you a heap profile from the running container?

I can run any diagnostic build you point me at — the instance is easy to redeploy.