Skip to content

fix: Retry after partial file reads. - #218

Open
kinyoklion wants to merge 52 commits into
mainfrom
rlamb/file-data-source-json-retry
Open

fix: Retry after partial file reads.#218
kinyoklion wants to merge 52 commits into
mainfrom
rlamb/file-data-source-json-retry

Conversation

@kinyoklion

@kinyoklion kinyoklion commented Jan 9, 2026

Copy link
Copy Markdown
Member

When we detect a file has been written there isn't any promise that the file content itself will actually be complete, or that a subsequent notification for that file will be received when it is.

Currently sometimes the file update will trigger, and the JSON parsing will fail, and then it won't get re-triggered. Which causes some tests to fail.

This introduces a retry for any failed JSON parsing up to a maximum number of times.

Ideally we would extract the file watcher and abstract the file system API so we could simulate these conditions.


Note

Overview
When auto-update is enabled, FileDataSource now retries loads after parse failures (e.g. reading a file mid-write), up to 5 attempts with a 600 ms delay between tries, instead of stopping until another file-change event.

Retries are scoped to per-episode failure budgets: a new Start() or watcher reload clears counts and starts fresh. Auto-update off keeps the documented one-shot load—no background retries. Multi-path loads share a single retry chain; if one path exhausts its budget, other paths that were waiting on retries get terminal error logs and wait for the next change.

Dispose cancels pending retries; a successful external reload before a scheduled retry avoids redundant Init and spurious change events. Transient read errors during a retry episode are charged to the same per-path budget as parse failures.

Builder docs for AutoUpdate describe the bounded retry behavior. Tests use scripted file readers and relaxed reload assertions where watcher timing makes versions non-deterministic.

Reviewed by Cursor Bugbot for commit 64cb514. Bugbot is set up for automated code reviews on this repo. Configure here.

@kinyoklion
kinyoklion force-pushed the rlamb/file-data-source-json-retry branch from f3aaa08 to d1f90f1 Compare January 9, 2026 23:00
The two reload tests asserted an exact ExpectedDataSetForSegmentOnlyFile(2),
which encoded an assumption of exactly two LoadAll attempts. With the file
watcher firing on truncate-then-write plus the new JSON-parse retry, the
successful attempt's version can be 3+ and the strict JSON comparison times
out waiting for a version-2 event that never comes.

Switch both tests to ExpectPredicate with a shared structural matcher and
refactor the existing predicate in ModifiedFileIsReloadedIfAutoUpdateIsOn
to use the same helper.

Also tighten the retry path in FileDataSource: drop the trailing TODO,
short-circuit LoadAll once Dispose() has been called, and skip the post-
delay LoadAll if disposal raced the retry.
kinyoklion added 22 commits May 26, 2026 16:03
… autoUpdate.

Review round-1 fixes:
- Reset the retry budget when a failure is observed on an externally
  triggered load (Start or file-change notification), so a later
  partial-write episode still gets retries after an earlier episode
  exhausted the counter.
- Treat any parser failure as retryable, not only JsonException, so
  alternate parsers (e.g. YAML) get the same partial-read handling.
- Keep at most one pending retry, and skip it if an intervening load
  already succeeded, to avoid redundant Inits and spurious change events.
- Do not retry when autoUpdate is off, preserving the documented
  load-once semantics.
- Log the parse exception summary on retry and full detail on give-up;
  catch exceptions escaping the retry task so they are not unobserved.
- Add deterministic IFileReader-driven tests for retry recovery, the
  attempt cap, episode reset, YAML parsing, autoUpdate(false), and
  dispose cancellation; episode reset, YAML, and autoUpdate(false)
  tests fail on the previous implementation.
… external loads.

Review round-2 fixes:
- Externally triggered loads clear the per-path failure counts up front,
  and give-up clears them as well, so a stale count from a dead retry
  chain can never be charged to a later episode (which previously caused
  an immediate give-up with zero retries on multi-path configs).
- When one path's give-up ends the chain, other paths that were promised
  retries get a terminal error log naming them.
- Remove a path's failure count as soon as it parses, before the merge
  step, so a duplicate-key configuration error cannot strand it.
- Log the exception stack trace at debug level on each retry.
- Correct the AutoUpdate doc: retries are bounded per detected change,
  not unconditional.
- Tests: deterministic multi-path episode tests (fresh budget after
  another path's give-up; terminal log for abandoned paths) that fail on
  the previous implementation; a redundant-reload guard test; waits now
  fail loudly on timeout; the dispose test no longer depends on running
  within the retry delay.
@kinyoklion
kinyoklion marked this pull request as ready for review August 17, 2026 23:51
@kinyoklion
kinyoklion requested a review from a team as a code owner August 17, 2026 23:51
Comment thread pkgs/sdk/server/src/Internal/DataSources/FileDataSource.cs
Comment thread pkgs/sdk/server/src/Internal/DataSources/FileDataSource.cs
…sing.

Addresses Bugbot review: a transient read error (for example, a writer
replacing the file) during a retry previously ended the episode with
budget remaining and no terminal log, recreating the stuck-stale-data
state this PR targets. Such failures are now charged to the same
per-path budget: the chain schedules the next retry until the budget is
exhausted, then ends the episode with terminal logs. Externally
triggered loads keep the existing non-retrying behavior for non-parse
errors. New test fails on the previous implementation.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 64cb514. Configure here.

// file-change notification is not guaranteed. Charge the failure to
// the same per-path budget and continue the chain.
HandleRetryLoadFailure(path);
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Error logged before retriable failure

Low Severity

LogHelpers.LogException always runs before HandleRetryLoadFailure, so every retriable read failure during a retry episode is logged at Error even though a Warn retry follows. That diverges from HandleParseFailure, which keeps retriable failures at Warn and reserves Error for giving up, and can falsely look like a terminal load failure to operators and monitors.

Additional Locations (1)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 64cb514. Configure here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant