Skip to content

fix: serialize SQLite operations and protect connection lifetime - #326

Merged
chrispader merged 2 commits into
codex/configurable-sqlite-performancefrom
codex/sqlite-operation-serialization
Sep 10, 2026
Merged

fix: serialize SQLite operations and protect connection lifetime#326
chrispader merged 2 commits into
codex/configurable-sqlite-performancefrom
codex/sqlite-operation-serialization

Conversation

@chrispader

@chrispader chrispader commented Sep 10, 2026

Copy link
Copy Markdown
Member

Async statements could run concurrently against the same SQLite handle, and ordinary queries were not coordinated with the transaction queue. That allowed an external write to become part of another caller's transaction and roll back with it. The native connection registry also exposed raw handles to close/reopen races, including result metadata being read after another statement changed the connection state.

This PR is stacked on #328, which adds independent performance-mode configuration, and #328 is stacked on #325, which restores SQLite mutexes by default. Merge the stack in that order. This addresses the operation scheduling and connection lifetime problems in #303, #304, and #312.

Solution

This change routes session async operations through one FIFO queue per database and rejects conflicting sync operations with the existing busy error. Transaction callbacks keep explicit ownership through their tx object, so their internal statements bypass the public queue without recursively waiting on themselves. Session close now refuses pending work, while attach, detach, batch, file import, and delete follow the same admission rules.

The native registry now stores shared connection state behind lifecycle and registry locks. Each connection lock covers the full statement, row conversion, metadata, change count, and insert ID sequence. Async jobs capture that connection before entering the worker pool, so close/reopen cannot redirect old work to a replacement handle. Batch and file import hold the same lock for their whole transaction.

This branch preserves Tr岷 膼矛nh Huy's original-author commit from #319, then adds synchronization around its duplicate-open guard and failed-open cleanup. If #319 merges separately first, this branch can be rebased to omit that commit.

Inside a transaction callback, callers must use the passed tx object, including from helper functions. Awaiting a same-database session/global queued operation from that callback creates a circular wait because the operation waits for the transaction and the transaction waits for the operation. This was already possible with executeBatchAsync; ordinary async execution now follows the same ownership rule.

Native calls are serialized per connection, not across all database handles. SQLITE_THREADSAFE=0 therefore remains unsafe when separate databases run concurrently unless the caller globally serializes every SQLite call.

Reproduction

On iOS with SQLITE_THREADSAFE=1 and #319 applied, the new rollback-isolation regression fails before this change: the external row disappears with the transaction. It passes after the queue fix. All 80 committed Harness tests passed on iOS and Android, along with a temporary runtime probe that checks THREADSAFE=1 and completes 100 parallel reads.

  • Run parallel public and TypeORM reads against one connection.
  • Start concurrent value-tagged inserts and verify every reported insert ID resolves to the value from that operation.
  • Hold a transaction open, submit an external async write, roll the transaction back, and verify the external write remains.
  • Start async work and verify sync execution and close report the database as busy; close and reopen after the work settles.
  • Close/reopen a native connection under the same name at another location and verify the old async job either finishes on the original handle or reports it closed, never the replacement.
  • Hold a transaction on one database while an async query completes on another.

@chrispader
chrispader added this pull request to stack #327 September 10, 2026 13:41
@chrispader
chrispader force-pushed the codex/sqlite-operation-serialization branch from 523fdfd to 6c12737 Compare September 10, 2026 14:06
@chrispader
chrispader removed this pull request from stack #327 September 10, 2026 14:09
@chrispader
chrispader changed the base branch from codex/configurable-sqlite-threading to codex/configurable-sqlite-performance September 10, 2026 14:10
@chrispader
chrispader added this pull request to stack #329 September 10, 2026 14:10
@chrispader
chrispader force-pushed the codex/sqlite-operation-serialization branch from 6c12737 to 0da6c3d Compare September 10, 2026 16:04
@chrispader
chrispader removed this pull request from stack #329 September 10, 2026 16:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants