fix: download dataset atomically so an interrupted download can't block retries#8
Open
steps-re wants to merge 1 commit into
Open
fix: download dataset atomically so an interrupted download can't block retries#8steps-re wants to merge 1 commit into
steps-re wants to merge 1 commit into
Conversation
…ck retries The dataset downloader wrote directly to spectrum_sensing_dataset.hdf5. If the download failed partway (network stall, HTTP error, or Ctrl+C), a corrupt partial file was left at the target path. On the next run the "already exists" guard saw it and exited without any integrity check, permanently trapping the user with a corrupt file they had to manually locate and delete. - Download to a temporary ".part" file and os.replace() into place only after the SHA-256 checksum verifies. - Remove the partial file on any error or interruption (KeyboardInterrupt). - Validate the checksum of a pre-existing target instead of blindly aborting; exit(0) when it is already valid, re-download when it is corrupt. - Add r.raise_for_status() so an HTTP error body is never written as the dataset. - Raise the streaming timeout from 3s to 30s (3s aborts on any brief stall during the ~6 GB download). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Mike German <mike@stepsventures.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
An interrupted dataset download leaves a corrupt file that permanently blocks all future download attempts.
Details
download_dataset.pystreams the ~6 GB dataset straight into the final path. If the download is interrupted (network stall, HTTP error, Ctrl+C), a corrupt partial file is left there. On the next run theos.path.exists()guard sees it, prints "already exists", andexit(1)s with no integrity check — the user is stuck with a corrupt file they must manually find and delete. Contributing factors:timeout=3aborts on any brief stall during a multi-GB download, and there was noraise_for_status(), so an HTTP error body would be written into the.hdf5file.Fix
Download to a temporary
.partfile, verify the SHA-256 checksum, thenos.replace()atomically into place; remove partials on any failure. A pre-existing target is now checksum-validated (skip if valid, re-download if corrupt). Timeout raised to 30s; addedraise_for_status().Verification
.part), retry-after-interrupt, corrupt-existing re-download, HTTP 404 raises and leaves nothing behind.(Verified against a local server rather than the real 6 GB URL; the control flow is identical.)