PG::Connection#exec/#sync_exec: query buffer not pinned across GVL release — GC (compaction or ordinary) can relocate/free it mid-send, corrupting or crashing on Ruby 3.2+

[colins]

Summary

pg_cstr_enc() in ext/pg_connection.c returns a raw const char * pointing directly into a Ruby String's own memory. pgconn_send_query (#exec/#async_exec) and pgconn_sync_exec (#sync_exec) pass that pointer to gvl_PQsendQuery/gvl_PQexec and release the GVL for the duration of the libpq call. Nothing pins the source String for the life of the call.

Under Ruby 3.2+'s variable-width allocation, a String short enough to be embedded lives directly inside its object slot. If GC (compaction, or in some cases even an ordinary collection freeing an unreferenced transcoding temporary) relocates or frees that memory while the GVL is released, libpq is left holding a dangling pointer. Two observed outcomes, both confirmed to be the same underlying mechanism:

• SQL corruption: whatever now occupies that memory (frequently a same-size-class string allocated concurrently) is sent to Postgres instead of the intended query.
• SIGSEGV: if the slot is unmapped, pg crashes inside PG::Connection#exec.

Environment

• pg 1.6.3 (also present on git master as of investigation date — see "Affected code paths" below)
• Ruby 3.3.9, 3.3.11, 3.4.10, 4.0.6 — all reproduce (Linux x86_64 only; does not reproduce on macOS, see "Why it needs Linux" below)
• Ubuntu 22.04 (Jammy), libpq via libpq-dev
• Postgres 14 (server version not load-bearing to the bug)

Minimal reproducer (no ORM involved)

# gem install pg
# createdb pg_segv_test
# DATABASE_URL=postgres://localhost/pg_segv_test ruby pg_gc_compact_segv.rb

require 'pg'

URL         = ENV.fetch('DATABASE_URL', 'postgres://localhost/pg_segv_test')
PROCESSES   = Integer(ENV.fetch('PROCESSES', '16'))
THREADS     = Integer(ENV.fetch('THREADS', '32'))
RUN_SECONDS = Integer(ENV.fetch('RUN_SECONDS', '540'))
WORKAROUND  = ENV['PG_WORKAROUND'] == '1'

# Heap-held, not locals: locals get pinned by the conservative machine-stack
# scan, which masks the bug. ~200 bytes stays inside an embedded slot, so
# compaction moves the bytes.
SQL = 20.times.map { |i| "SELECT pg_sleep(0.01) -- #{i} " + ('x' * 180) }

deadline = Time.now + RUN_SECONDS

pids = PROCESSES.times.map do
  fork do
    THREADS.times.map do |i|
      Thread.new do
        conn = PG::Connection.new(URL)
        while Time.now < deadline
          sql = SQL.sample
          churn = "(conn: #{conn.backend_pid}) #{sql}" # refills the vacated slot
          conn.exec(WORKAROUND ? String.new(sql) : sql)
          churn.clear
          GC.compact if i % 4 == 2
        end
        conn.close
      end
    end.each(&:join)
  end
end

crashed = pids.map { |pid| Process.wait2(pid)[1] }.select { |st| st.signaled? }
puts crashed.empty? ? "not reproduced this run (race; retry)" : "REPRODUCED: #{crashed.size} crashed"

Expected: one or more forked children die with [BUG] Segmentation fault at 0x... inside PG::Connection#exec. PG_WORKAROUND=1 (which passes String.new(sql) instead of the shared string) survives.

Why multiple processes, not just threads

Thread count alone does not reproduce it — this was bisected explicitly:

Config	Result
16 procs × 32 threads, 540s	reproduces ~50% of runs
1 proc × 32 threads, 1800s	never
1 proc × 512 threads, 900s	never

One process has one GVL, so only one thread executes Ruby (including GC.compact) at a time. N processes give N independent GVLs/heaps and therefore roughly N× the real compaction throughput. The bug needs a high rate of actual heap relocation that single-process Ruby can't reach at any thread count.

Two further details, both load-bearing for reproduction:

• SQL strings must be held in a heap-reachable structure (e.g. an Array), not locals — a local is conservatively pinned by the machine-stack scan.
• Each iteration should allocate a same-size-class string with matching text, which is what refills the vacated slot and is what libpq ends up reading when the corruption manifests as text rather than a crash.

Why it needs Linux

Not reproducible on macOS, even under Docker/VM — the macOS hypervisor doesn't replicate Linux CFS thread-preemption frequency closely enough to hit the race window at practical run lengths.

Affected code paths

// pg_cstr_enc: returns a raw pointer into Ruby String memory on the fast path
static const char *pg_cstr_enc(VALUE str, int enc_idx, VALUE *transcoded_str){
    const char *ptr = StringValueCStr(str);
    if( ENCODING_GET(str) == enc_idx ){
      return ptr;                    // no copy, no guard
    ...
}

// pgconn_send_query (#exec / #async_exec):
gvl_PQsendQuery(this->pgconn, pg_cstr_enc(argv[0], this->enc_idx, &transcoded_str));
// GVL released here — nothing pins argv[0]'s bytes

// pgconn_sync_exec (#sync_exec):
result = gvl_PQexec(this->pgconn, pg_cstr_enc(query_str, this->enc_idx));
// same issue

RB_GC_GUARD on git master covers only the transcoded-string branch (where pg_cstr_enc allocates a new String), not the fast path where the original String's bytes are used directly. On the fast path — same-encoding SQL, the common case — nothing prevents GC from moving or freeing the source String while libpq (with the GVL released) is still reading from it.

A second, GC.compact-independent path was also isolated: on released 1.6.3, pg_cstr_enc can return a pointer into a transcoding temporary that becomes unreachable the instant the function returns (encoding-mismatch branch). An ordinary GC — no explicit compaction needed — can free it while the GVL is released inside PQsendQuery. This is arguably the better fit for typical production crashes, since most applications never call GC.compact explicitly, yet the corruption/crash was observed in production without it. This path appears to already be addressed on git master by the transcoded_str out-parameter, so for that specific branch the ask is a release, not a new patch — but the fast-path issue above is present on both 1.6.3 and current master.

Proposed fix

Copy the query bytes into a buffer pg owns before releasing the GVL, in both pgconn_send_query and pgconn_sync_exec:

// pgconn_send_query -> gvl_PQsendQuery  (illustrative diff)
const char *query = pg_cstr_enc(argv[0], this->enc_idx, &transcoded_str);
size_t query_len = strlen(query);
char *query_copy = ALLOC_N(char, query_len + 1);

/* Own this buffer across the GVL release: libpq keeps reading the pointer
 * while the GVL is dropped, and GC (compaction, or an ordinary collection
 * freeing a transcoding temp) can relocate/free the Ruby-owned bytes. */
memcpy(query_copy, query, query_len + 1);
int send_rc = gvl_PQsendQuery(this->pgconn, query_copy);
xfree(query_copy);

if (send_rc == 0) { ... }

Same pattern for pgconn_sync_exec / gvl_PQexec. I have a working patch (and a script that applies it idempotently to either the released-1.6.3 or git-master variant of the source, verifying no unguarded gvl_PQ(sendQuery|exec)(this->pgconn, pg_cstr_enc(...)) call sites remain) and can open a PR if that's useful.

Verification: built pg 1.6.3 from source with this patch and re-ran the exact concurrent workload that reliably reproduced SQL corruption on stock 1.6.3 (16 procs × 32 threads × 540s, Ruby 4.0.6) — 0 corruptions, 0 crashes, vs. reliable reproduction on the unpatched build under identical conditions.

How we got here (production trigger, for context)

We first hit this via Sequel + Postgres in a Rails-adjacent service, seeing:

PG::SyntaxError: ERROR:  syntax error at or near "conn"
LINE 1: (conn: 20600) SELECT "templates".* FROM "templates" INNER JOIN ...

Sequel's Dataset#select_sql caches and returns the same unfrozen String object to every caller across threads, and Database#log_connection_yield builds a "(conn: N) <sql>" string for debug logging concurrently with execute_query. That combination is not itself the bug — we built a Sequel-free reproducer (pg_only_test.rb, require "sequel" asserted to fail) that hits the identical corruption/crash with only pg involved — but it's a very efficient trigger: a long-lived shared mutable String plus a same-shape concurrently-allocated string is exactly what maximizes the odds of the freed/relocated slot being refilled with corrupting text instead of silently reused for something unrelated.

Related issues

This looks like the same family as:

• set_notice_receiver/processor: raw VALUE handed to libpq breaks under GC compaction (SEGV) #734 (set_notice_receiver raw VALUE, fixed in Fix broken set_notice_(receiver|processor) callback after GC.compact #735)
• Segfault in PG::Coder#encode after GC.compact #327 (PG::Coder#encode after GC.compact)
• Segfault in TypeMapByClass after GC.compact #328 (TypeMapByClass after GC.compact)

i.e. a recurring pattern of C-level pointers into Ruby object memory not surviving GC across a GVL release, this time on the primary query-send path.

Happy to provide the full reproducer suite (Sequel-based and Sequel-free variants, plus the patch-verification harness) if that helps triage.

[larskanis]

Thank you for this detailed bug report! I can reproduce it. It is very similar to #721 and the proposed fix in #723 fixes this issue as well.

Regarding your context: Is it right that your service runs GC.compact from time to time? Otherwise the cached sql string shouldn't be relocated.

[colins]

I can reproduce it.

That's a relief. These issues can be tricky to reliably reproduce.

the proposed fix in #723 fixes this issue as well.

I'll watch out for this coming through. Thanks

Regarding your context: Is it right that your service runs GC.compact from time to time? Otherwise the cached sql string shouldn't be relocated.

The reproduction case has an explicit call to GC.compact. In our prod environment we don't explicitly call GC.compact but it will be running periodically.

[larskanis]

Regarding the production context: Original issue description and workaround with Sequel is here: cloudfoundry/bosh#2739 And the re-applied workaround after investigation and the above bug report is here: cloudfoundry/bosh#2809