Skip to content

fix(runtime): use process-owned credential-home locks - #613

Open
mldangelo-oai wants to merge 5 commits into
mainfrom
mdangelo/codex/credential-lock-ownership
Open

fix(runtime): use process-owned credential-home locks#613
mldangelo-oai wants to merge 5 commits into
mainfrom
mdangelo/codex/credential-lock-ownership

Conversation

@mldangelo-oai

@mldangelo-oai mldangelo-oai commented Aug 22, 2026

Copy link
Copy Markdown
Collaborator

Summary

A stale heartbeat does not prove that a credential-home owner has exited: a paused process may still hold credentials, and PID reuse can prevent safe crash recovery. Hold ownership in an exclusive SQLite transaction while preserving compatibility with released 0.1.18 clients that still use the directory heartbeat.

Changes

  • Hold an exclusive SQLite transaction in the process preparing the scan runtime, using the SQLite implementation already bundled with supported Node and Bun runtimes.
  • Preserve the existing owner directory and refresh its heartbeat while the SQLite transaction is held so released 0.1.18 clients cannot replace an active newer owner's lock.
  • Mark native-lock owner records so orphaned records can be recovered without trusting a reused PID, while continuing to respect live legacy owners.
  • Keep the SQLite guard file across operations, validate its path and permissions, and close the connection on cancellation or acquisition and release failures.
  • Extend the real-process credential-lock fixture to reproduce the released 0.1.18 stale-directory takeover before the fix and verify mixed-version exclusion afterward; retain the existing paused-owner and crash-with-reused-PID coverage in both Bun and installed-package tests.
  • Synchronize the mixed-version fixture on an observed heartbeat instead of racing separate process timers, including explicit suspended-holder controls.
  • Document the compatibility heartbeat, the remaining paused-owner limitation with older clients, and the required upgrade precautions in the shipped README.

Testing

Validated final commit e51e9dbc9b177d3db0fca4aeea8f0602260cd725:

  • Failing-first mixed-version subprocess regression: the released-protocol contender replaced the active native owner's directory before the fix; the fixed version kept the original owner and blocked the contender.
  • Controlled suspended-holder subprocess probe: the previous fixture incorrectly reported takeover; the synchronized fixture waited for the observed heartbeat and preserved the active owner.
  • bun test --timeout 30000 tests-ts/runtime.test.ts --seed 12345: 129 passed, 11 platform-specific skips, 0 failed.
  • Focused mixed-version, paused-owner, and crash-recovery regression: 1 passed, 0 failed.
  • pnpm run types, pnpm run format, pnpm run build, and git diff --check: passed.
  • Built Node 24 runtime credential-lock subprocess fixture: passed.
  • pnpm pack --pack-destination ../../dist followed by pnpm run check:package <packed tarball>: passed, including 255 archive entries, public imports, NodeNext types, the installed Node credential-lock regression, 111 bundled plugin files, the bundled Codex executable, and a nested worker.
  • Full Bun suite at parent commit 4c69e65db56b0034eddbd64c28071bfff7ec0dab with seed 12345: 1,593 passed, 30 skipped, 1 failed; the failed unrelated deep-scan reducer-deadline test exceeded its 30-second limit under concurrent local load. The identical reducer test passed in 11.28 seconds when rerun separately with the same 30-second limit; the final follow-up changes only the credential-lock test fixture.

Risk and rollout

Released 0.1.18 clients do not participate in the native SQLite protocol. The compatibility heartbeat prevents those clients from taking over an active newer owner's lock, but a paused newer owner cannot refresh its heartbeat and an older client can still reclaim its directory. Finish operations using older versions before upgrading; do not rely on mixed-version protection when an owner may pause.

Legacy PID-only records cannot distinguish a live owner from a reused PID, so ambiguous records still require manual recovery after stopping all operations using the credential home. Never remove the persistent SQLite guard while an operation is running. Supported Node versions and the public CLI remain unchanged; Node may emit its existing built-in SQLite experimental warning. Cross-platform CI remains required before landing.

Public disclosure review

  • No customer, partner, prospect, or user identities, data, or identifying details are included.
  • No credentials, personal data, private source, scan findings, or nonpublic links or tickets are included.
  • I reviewed the branch name, title, description, commits, changes, comments, logs, screenshots, attachments, and links for public disclosure.

@mldangelo-oai

Copy link
Copy Markdown
Collaborator Author

@codex review

Please review current head 379a59e3f83106c4d05c479d5571086072e63a11, especially native lock lifetime, paused/crashed owners, conservative legacy PID handling, cancellation/error cleanup, and Node/Bun behavior on Windows and Unix. This is stacked on #611; the regression fixture also runs against the installed npm package.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Aug 22, 2026

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review Completed 2026-08-25T04:26:27.117384Z e51e9db Manual request
🔒 Security Review Completed 2026-08-25T04:28:33.068330Z e51e9db Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 379a59e3f8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread sdk/typescript/src/runtime.ts
@mldangelo-oai

Copy link
Copy Markdown
Collaborator Author

@codex review

Please re-review current head 379a59e3f83106c4d05c479d5571086072e63a11 with the clarified rollout requirements. The code is unchanged and all 27 CI checks passed. #611 is an unmerged review base, not a release to deploy before this PR. The latest released tag (npm-v0.1.16) has PID-only locking; two-process probes with that pre-heartbeat protocol and this head waited correctly in both directions.

The PR now explicitly requires landing the corrected combined tree and stopping any heartbeat-only preview processes first. Please verify whether the mixed-version finding remains a blocker under that rollout, and check the corrected tree for any other issues. The reasoning and compatibility evidence are in the existing review thread.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Bravo.

Reviewed commit: 379a59e3f8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@mldangelo-oai
mldangelo-oai changed the base branch from mdangelo/codex/pr611-base to main August 22, 2026 07:57
@github-actions github-actions Bot added the bug Something isn't working label Aug 22, 2026
@mldangelo-oai
mldangelo-oai force-pushed the mdangelo/codex/credential-lock-ownership branch from 379a59e to 5ab8e68 Compare August 22, 2026 07:57
@mldangelo-oai
mldangelo-oai marked this pull request as ready for review August 22, 2026 07:58
@mldangelo-oai

Copy link
Copy Markdown
Collaborator Author

@codex review

Please review rebased head 5ab8e682a90633a1731c99cd58a580f43a7c1431 against main. #611 is now merged, so this PR is no longer based on the temporary mirror. git range-diff confirms the follow-up patch is unchanged.

The runtime suite (129 passed, 11 platform skips), types, formatting, build, and the built Node 22.13 paused-owner/crash-recovery regression passed again. Please check integration with the current base and the documented requirement to stop heartbeat-only operations before upgrading.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Nice work!

Reviewed commit: 5ab8e682a9

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Let the stalled holder exit if its controller dies. Shorten credential-lock documentation and remove the outdated release-failure explanation.
@mldangelo-oai

Copy link
Copy Markdown
Collaborator Author

@codex review

Please review current head 170d1d1f5826f5807ecdc75c6a240d1a9ac0be5c, including the bounded stalled-child lifetime and shortened credential-lock documentation. Production locking behavior is unchanged.

The runtime suite passed (129 tests, 11 platform-specific skips), as did types, formatting, and build. The normal paused-owner/crash-recovery fixture passed on Node 22.13 and Node 24.15. Parent-exit probes passed on Node 22, Node 24, and Bun; a before-change control still reproduced the orphaned holder.

The broader seeded and random-order test suites and fresh CI are still running.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Keep it up!

Reviewed commit: 170d1d1f58

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@mldangelo-oai

Copy link
Copy Markdown
Collaborator Author

@codex review

Please review current head 0557442c1d425b45229b506cc88cc3ba2d21ae4d.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Breezy!

Reviewed commit: 0557442c1d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@kmbroai kmbroai left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Critical review

Reviewed 0557442c1d425b45229b506cc88cc3ba2d21ae4d. A process-owned lock addresses a real correctness defect in heartbeat-based ownership. No additional blocking code defect found, but the mixed-version rollout limitation must remain explicit.

Necessity and correctness

A paused process can still own and mutate the credential home. A stale heartbeat is therefore not proof that admitting another writer is safe. Holding an exclusive SQLite transaction in the process performing the protected work removes that expiry inference and releases ownership when the process exits.

The acquisition and release paths retain home identity checks, validate the persistent guard file, handle contention without a production expiry timeout, and close the database after acquisition errors and release failures. The protocol marker allows orphaned new-format owner records to be reclaimed only after acquiring the native lock. Conservative treatment of live legacy PID-only records is appropriate; a reused PID cannot safely be distinguished by age alone.

Simplification and compatibility

SQLite may look heavy for a mutex, but using the SQLite implementation already built into both supported runtimes avoids another dependency or helper process and provides the needed crash semantics. Keep the implementation limited to lock ownership; it should not become another state database or heartbeat service. The legacy directory remains justified only as a compatibility boundary, not a second independent source of ownership for new clients.

The old heartbeat-only client still cannot be made safe by this new client's lock: it does not participate in the SQLite protocol. A compatibility heartbeat would also fail when the new owner pauses. The PR's instruction to stop heartbeat-only operations before upgrading is therefore a real requirement, not an optional caution. Please put that upgrade requirement alongside the shipped README lock guidance as well; package users will not necessarily read the PR description. I verified that #611 is merged and the npm-v0.1.18/current-main snapshot contains the heartbeat implementation, so the older thread's characterization of it as only an unreleased preview is no longer current.

Verification

Ran the credential-lock subset with seed 12345: 9 passed, 0 failed, including the real stalled-owner/crash-with-reused-PID fixture, Bun 1.3.14/Linux with locally available dependencies and a normal 022 umask outside the sandbox. I did not rerun the built Node package smoke, native Windows locking, or a mixed-version live deployment. No credential lock files were manually removed outside synthetic fixtures.

@faizan-oai faizan-oai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking on the released-client compatibility issue in the existing credential-lock thread. The mixed-version probe reproduces concurrent ownership with an active new holder; the matched old/old control remains exclusive. The new-only lock tests and CI pass but do not cover this regression.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4c69e65db5

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread sdk/typescript/scripts/fixtures/credential-lock.mjs Outdated
@mldangelo-oai

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Breezy!

Reviewed commit: 4c69e65db5

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@mldangelo-oai

Copy link
Copy Markdown
Collaborator Author

@codex review

@mldangelo-oai

Copy link
Copy Markdown
Collaborator Author

@codex security review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. 👍

Reviewed commit: e51e9dbc9b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector

Copy link
Copy Markdown

Security review completed. No security issues were found in this pull request.

Reviewed commit: e51e9dbc9b

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@faizan-oai faizan-oai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed e51e9dbc9b177d3db0fca4aeea8f0602260cd725 against current main. The fixture concern is addressed: under the reproduced 1.1-second holder suspension, the updated fixture observes the heartbeat and preserves ownership; the previous fixture acquires incorrectly. The updated fixture still fails against the unmodified runtime without the compatibility heartbeat, so it retains regression coverage.

Production locking code is unchanged from the already-verified released-client fix. All 11 focused lock tests pass, and full current-head CI passes all 25 jobs, including the three required checks. No remaining blocking findings.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants