Local Kong keeps idle upstream connections longer than PostgREST, causing sporadic 502 on POST/PATCH

[Take888a]

Affected area

Local development (supabase start)

Describe the bug

Kong keeps idle upstream connections to PostgREST longer than PostgREST keeps them open. When Kong reuses a connection PostgREST has just closed, nginx retries idempotent methods but not POST/PATCH, so those fail with a sporadic 502:

{"message":"An invalid response was received from the upstream server"}

This shows up as flaky integration tests: a different test file fails on each run, and every failing file passes when run alone.

Measurements

On a stack started by CLI 2.98.0 (public.ecr.aws/supabase/kong:2.8.1, PostgREST v14.10, macOS + Docker):

What	Result
Kong's upstream_keepalive_idle_timeout (/usr/local/kong/.kong_env)	60s (also upstream_keepalive_pool_size = 60, KONG_NGINX_WORKER_PROCESSES=1)
Time until PostgREST closes an idle connection (measured from inside the Kong container, 3 runs)	35.1s / 42.1s / 58.1s (Warp's idle timeout, coarse ~30s timer)
Kong error log over ~3 days	68 upstream errors: upstream prematurely closed connection ×47, recv() failed (104: Connection reset by peer) ×21 — GET 38 / HEAD 4 / DELETE 2 / PATCH 7 / POST 17
502s in the access log over the same window	24 — all of them POST (17) or PATCH (7). No GET/HEAD/DELETE became a 502
503 / 504 in the same window	0. PostgREST logged no PGRST003 (pool timeout) either
PostgREST log at the exact minutes of the 502s	nothing — no reconnect, no schema cache reload, no DB disconnect

So the pool is not exhausted and PostgREST is not restarting; Kong is simply handing a request to a connection the upstream has already closed. Non-idempotent methods are not retried, which is why only POST/PATCH surface.

Expected behavior

Kong's idle timeout for the PostgREST upstream should be shorter than PostgREST's, so Kong drops the connection before the upstream does.

Workaround

Setting the value below PostgREST's minimum (30s) fixes it:

docker exec supabase_kong_<project> sh -c \
  "sed -i 's/^upstream_keepalive_idle_timeout = .*/upstream_keepalive_idle_timeout = 20/' /usr/local/kong/.kong_env && kong reload"

Why this needs a fix in the CLI

There is no way to change this from config.toml. The CLI embeds Kong's environment variables with fixed values, and PostgREST has no server-side idle timeout setting (postgrest --example only exposes server-host / server-port / server-cors-allowed-origins / server-timing-enabled / server-unix-socket). Patching .kong_env from outside works but silently stops working whenever the CLI changes how the container is built.

Could the CLI either ship a shorter upstream_keepalive_idle_timeout for the PostgREST upstream, or expose it in config.toml?

System information

• Supabase CLI: 2.98.0
• Kong image: public.ecr.aws/supabase/kong:2.8.1
• PostgREST: v14.10
• OS: macOS (Docker Desktop)

[LR2studio]

We hit the same sporadic POST/PATCH 502 (An invalid response was received from the upstream server) in CI and traced it to a different trigger from the idle timeout: HEAD requests. A shorter upstream_keepalive_idle_timeout would not fix this case.

Mechanism

1. supabase-js sends an HTTP HEAD for every { count: 'exact', head: true }.
2. PostgREST answers the HEAD with no Content-Length (and no Connection: close), then closes the TCP connection immediately. Its server (Warp) only keeps a HEAD's connection alive when the response has a Content-Length.
3. HTTP/1.1 is persistent by default and PostgREST never said otherwise, so Kong puts the connection back in its keepalive pool, next in line.
4. If the next request reaches Kong before Kong has processed PostgREST's FIN, Kong writes into the dead connection and logs upstream prematurely closed connection while reading response header from upstream (or recv() failed (104: Connection reset by peer)). nginx silently retries GET/HEAD/PUT/DELETE, but not POST/PATCH, so those return the 502.

The window is sub-millisecond and opens right after a HEAD, not after an idle period, so the idle-timeout workaround never gets a chance to evict the connection.

Evidence

Raw-socket probe from inside the Docker network:

PostgREST	Request	Content-Length	Socket after response
v14.5	HEAD, with or without Prefer: count=exact	absent	closed
v14.5	GET	present	open
v14.10 (CLI 2.100.0)	HEAD, with or without Prefer: count=exact	absent	closed
v14.10	GET	present	open

Repro against a private kong:2.8.1 on the stack's Docker network (same image and PostgREST, shared gateway untouched), with 8 concurrent in-network clients each doing a HEAD count and then a zero-row PATCH:

Gateway	Sequence	PATCH 502s
stock Kong 2.8.1	HEAD → PATCH	2,195 / 40,000 (two runs, ~5.5%)
stock Kong 2.8.1	GET → PATCH	0 / 20,000
KONG_UPSTREAM_KEEPALIVE_POOL_SIZE=0	HEAD → PATCH	0 / 20,000

The same harness run from the macOS host never hits it (0 / 6,000): Docker Desktop's round trip is long enough that Kong always processes the FIN first. That's why it shows up on Linux CI runners much more than locally. In two weeks of our local Kong log (about 1.23M PostgREST requests), all 6 prematurely closed errors came 0.0 s after a HEAD, and all 6 were GETs that nginx retried to a 200.

Your table lists 4 HEADs among the upstream errors. It may be worth checking whether your POST/PATCH 502s also sit right after a HEAD access line.

Workaround we use in CI

Turn off upstream keepalive in the running Kong container. Pass the CLI's nginx template again, because a bare kong reload falls back to Kong's default template and silently drops the :8088 email-templates server:

docker exec supabase_kong_<project> sh -c \
  'KONG_UPSTREAM_KEEPALIVE_POOL_SIZE=0 kong reload --nginx-conf /home/kong/custom_nginx.template'

The cost is one TCP connect per request inside the bridge network, plus a TIME_WAIT port per connection Kong closes. On a GitHub-hosted runner running our full integration suite, the peak was about 7,000 TIME_WAIT sockets, well under the ~28k-per-minute ephemeral port budget.

Possible fixes

• CLI: ship KONG_UPSTREAM_KEEPALIVE_POOL_SIZE=0 for the local stack, or expose Kong's keepalive settings in config.toml.
• Root cause: a PostgREST HEAD response that carried Content-Length (or Connection: close) would stop the silent close entirely.

[LR2studio]

A correction to my comment above: PostgREST has already fixed the silent close, so the fix on the CLI side is just a newer PostgREST.

• Since PostgREST v14.15 (warp 3.4.14, commit 032ceb6), a HEAD response that is about to close says Connection: close (warp 3.4.12, warp: Send Connection: close when closing the connection yesodweb/wai#958). nginx doesn't pool an upstream connection whose response says that, so Kong can't hand the dead connection to the next POST/PATCH.

• Supabase CLI 2.110.0 and later bundle PostgREST v14.15 or newer (2.110/2.111: v14.15, 2.112/2.113: v14.16, 2.114+: v16.x). Upgrading the CLI should be enough. The keepalive-off reload I posted is only needed on older CLIs like 2.98.0 (this report) or 2.100.0 (ours).

• If you can't upgrade the CLI, you can pin only PostgREST. The CLI reads supabase/.temp/rest-version (the file supabase link writes) and swaps just that image:

  mkdir -p supabase/.temp && printf 'v14.15' > supabase/.temp/rest-version   # no trailing newline
  supabase start

Verified in CI on a GitHub-hosted runner with CLI 2.100.0 plus that pin. A raw HEAD to PostgREST from inside the stack's network returned:

HTTP/1.1 200 OK | connection: close | content-length: nil   (image public.ecr.aws/supabase/postgrest:v14.15)

Against PostgREST v14.5, the same probe gets 200 OK with no Connection header, and the socket is already closed when a second HEAD is sent (5 of 5 runs).

[7ttp]

thanks @LR2studio 💚 that was it, can confirm it's fixed now

@Take888a you're on 2.98.0 which bundles postgrest v14.10, so upgrading to 2.110.0 or later should clear the POST/PATCH 502s

closing this up as it's resolved, if you still find anything feel free to reopen!