[AGE-4063] fix(runner): rebuild the Codex subscription auth symlink on every durable mount - #5696
Merged
mmabrouk merged 3 commits intoAug 3, 2026
Conversation
…able mount The durable session cwd is a geesefs mount over object storage, which has no symlinks: `<cwd>/.codex/auth.json` flushes to the store as a 0-byte object and comes back as an ordinary empty file. `symlinkCodexSubscriptionAuthFile` guarded with `existsSync`, read that degraded entry as "already linked", and returned early, so every turn after the first authenticated from an empty token file. Extract the lstat/readlink/replace logic `linkAgentFiles` already used for the same hazard into `ensureDurableSymlink`, use it for both links, and attach the Codex link to the durable-mount lifecycle so a mid-session remount re-materializes it. A subscription run that still fails auth now says the mounted login is empty or unreadable instead of asking for a project key. Closes #5692
dosubot
Bot
added
size:L
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
📝 WalkthroughSummary by CodeRabbit
WalkthroughThe runner now uses a shared durable symlink helper for agent mounts and Codex subscription authentication. It repairs links after durable-cwd remounts and reports specific errors for missing, unreadable, empty, or dangling subscription login files. ChangesCodex authentication repair
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant Environment
participant DurableMount
participant symlinkCodexSubscriptionAuthFile
participant ensureDurableSymlink
participant Codex
DurableMount->>Environment: report successful local durable-cwd mount
Environment->>symlinkCodexSubscriptionAuthFile: recreate subscription auth link
symlinkCodexSubscriptionAuthFile->>ensureDurableSymlink: inspect and repair auth.json
ensureDurableSymlink-->>symlinkCodexSubscriptionAuthFile: return link outcome
Codex->>Environment: report authentication result
Environment->>Codex: return specialized auth fault when login is unusable
Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: e2857e88-94ad-44aa-8a92-2731240f23c5
📒 Files selected for processing (10)
services/runner/src/engines/sandbox_agent/agent-mount.tsservices/runner/src/engines/sandbox_agent/codex-assets.tsservices/runner/src/engines/sandbox_agent/durable-symlink.tsservices/runner/src/engines/sandbox_agent/environment.tsservices/runner/src/engines/sandbox_agent/errors.tsservices/runner/src/engines/sandbox_agent/run-turn.tsservices/runner/tests/unit/durable-symlink.test.tsservices/runner/tests/unit/sandbox-agent-codex-assets.test.tsservices/runner/tests/unit/sandbox-agent-errors.test.tsservices/runner/tests/unit/sandbox-agent-orchestration.test.ts
Contributor
Railway Preview Environment
Updated at 2026-08-03T19:30:03.010Z |
…k-5692 The release branch regrouped RunPlan by concern and replaced the flat top-level model credential fields with a resolved `modelConnection` object. Three conflicts, all mechanical once both intents are kept: - codex-assets.ts: keep this branch's async `ensureDurableSymlink` repair (no `existsSync` early return, which is the bug) and port it onto the grouped plan, so `symlinkCodexSubscriptionAuthFile` and `describeCodexSubscriptionAuthFault` take `CodexHomePlan` and read `plan.workspace.cwd`. - environment.ts / run-turn.ts: keep the `authFault` callback this branch adds to `conciseError`, with the release branch's `request.modelConnection?.provider` as the provider argument. The two codex tests this branch added still built requests and plans in the old flat shape, which the release branch now rejects outright, so they move to `modelConnection` and to `credentials`/`workspace`. Runner suite green: 1559 tests, typecheck clean.
`ensureDurableSymlink` only logged a non-ENOENT unlink failure and carried on. If the following symlink then threw EEXIST, it returned "linked" on the assumption that a concurrent creator had won the race. But the same EEXIST is what a degraded entry that could not be removed produces, so a transient EBUSY, EACCES, or EIO on the FUSE mount left the 0-byte auth.json in place while the caller logged a successful repair and the run kept failing authentication. Track whether the unlink failed and treat EEXIST as benign only when it did not. The existing unlink-failure test could not catch this: its injected symlink stub always succeeds, so it never reaches the EEXIST a failed unlink would really produce. Two tests added for the two EEXIST paths. Refs #5692
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
Running a Codex agent with an operator ChatGPT subscription works on the first turn of a session and fails on every turn after that, with
codex: model authentication failed — add the project's OpenAI key to the project vault, or log in (OAuth).The login on the host is healthy the whole time.The Codex home lives at
<cwd>/.codex, on the durable session working directory. Whenever the object store is configured (the default self-hosted setup) that directory is a geesefs FUSE mount over S3, and S3 has no symlinks. Theauth.jsonsymlink the first turn creates flushes to the store as a 0-byte object and comes back from the next mount as an ordinary empty file.symlinkCodexSubscriptionAuthFileguarded withexistsSync, read that degraded entry as "the link is already there", and returned early. Codex then read an empty token file, and the harness reported it as an auth failure. Once a working directory held that object it stayed broken forever.Changes
The repo already had the right pattern for exactly this hazard.
linkAgentFileskeeps<cwd>/agent-filespointing at the agent mount bylstat-ing the path, keeping only a symlink whosereadlinkmatches the wanted target (including a dangling one), and replacing anything else. Its comment says geesefs silently degrades symlinks to empty files across remounts.That logic now lives once, in
ensureDurableSymlink(durable-symlink.ts).linkAgentFilesdelegates to it, so its behavior and log lines are unchanged, andsymlinkCodexSubscriptionAuthFileuses it in place ofexistsSync.Before, the guard was:
Now the entry has to actually be a symlink to the operator's login, or it gets replaced. Sessions that already hold the degraded file self-heal on their next turn, with no manual cleanup on upgrade.
Two smaller things ride along.
The link is now attached to the durable-mount lifecycle instead of only to the acquire path.
mountLocalAgentCwdalready re-ranlinkAgentFileson every mount; its siblingmountLocalDurableCwddid not re-run the Codex link, so a mid-session remount (the ENOTCONN recovery path) left the session authenticating from whatever the store handed back. The link is created after the mount is active, never before, and stays a no-op for Daytona runs, managed-key runs, and non-Codex runs. The acquire-path call remains for the run that has no durable mount at all.The error message no longer misdirects. When a subscription run hits the auth branch,
conciseErrorconsults an optionalauthFaultcallback, and the Codex path checks the token file the run actually reads. If it is empty or unreadable the operator now sees:codex: the mounted ChatGPT login is empty or unreadable — this run authenticates from the CODEX_HOME mount, not a project key. Sign in again on the host, then retry.The check is lazy, so it only runs on the error path it explains, and the message carries no credential material.The Codex home stays on the durable cwd, so native rollouts still persist and durable resume and warm daemon reuse are untouched. The token is never copied into the working directory, and nothing writes to or deletes the operator's mounted login. Teardown still only ever removes the session-local symlink.
Tests / notes
tests/unit/durable-symlink.test.tscovers the shared helper against the real filesystem (0-byte file replaced, wrong target replaced, correct link kept including while dangling) plus the injected failure paths.sandbox-agent-codex-assets.test.tscovers the same three cases throughsymlinkCodexSubscriptionAuthFile, asserts the operator's own login file is untouched while self-healing, and covers the auth-fault diagnosis (empty, unreadable, healthy, and the modes it must stay silent for). The old test that asserted a pre-existing plainauth.jsonis left alone encoded the bug and is replaced.sandbox-agent-orchestration.test.tsdrives a full subscription Codex run: one test simulates the store handing back a 0-byte file on a mid-prompt remount and asserts the link is a symlink again afterwards, another asserts a 401 with an empty mounted login surfaces the new message and never mentions the vault. The remount test fails against the pre-fix mount path, so it pins the gap it closes.pnpm test(1496 passed, 97 files) andpnpm run typecheckboth pass fromservices/runner.What to QA
codex subscription auth.json symlinkedline on each mount rather than only on the first turn..codex/auth.jsonis a 0-byte object) should recover on its next turn without touching the store by hand.auth.jsonshould appear under<cwd>/.codexfor them. Claude and Pi subscription runs point their config var straight at the mount and should be unaffected.Closes #5692