sd2b2

Copy files from an SD card (or any local directory) to a Backblaze B2 bucket.

This tool's reason to exist is --rename-from-exif: grouping RAW+JPEG shots and renaming them into day folders by capture time, for camera SD cards. For a large, plain "copy everything, no renaming" backup — e.g. cloning a whole external drive with hundreds of thousands of files — see Large, non-photo backups: use rclone instead.

1. Set up Backblaze B2

1. Create (or reuse) a B2 account at https://www.backblaze.com/b2 and create a bucket to receive the files, e.g. my-photos-backup.
2. Create a bucket-scoped application key, not the master key: B2 dashboard → App Keys → Add a New Application Key
   • Name it something like sd2b2-uploader.
   • Restrict it to the specific bucket.
   • Capabilities needed: listFiles, writeFiles, readFiles (read is only needed if you use --skip-existing, which is on by default).
   • Leave file name prefix set if you want the key restricted to a specific folder inside the bucket.
3. Copy the resulting keyID and applicationKey — the secret is shown only once.

Scoping the key to one bucket/prefix means a leaked key from a laptop or SD card workflow can't touch your other buckets or account settings.

2. Configure credentials

The script never accepts secrets as CLI flags (they'd end up in shell history and ps). Instead it reads them from the environment:

B2_APPLICATION_KEY_ID   required
B2_APPLICATION_KEY      required
B2_BUCKET_NAME          optional default for --bucket
B2_REALM                optional, default "production"

Set them either:

• directly in your shell:

  export B2_APPLICATION_KEY_ID=xxxx
  export B2_APPLICATION_KEY=xxxx

• or via a local .env file (copy .env.example to .env; it's already gitignored) — the script loads it automatically but never overrides a variable that's already set in the real environment, so a secret manager or CI-injected env var always wins over .env.

3. Install dependencies

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Use a Python built against OpenSSL (3.11+ from python.org, Homebrew, or pyenv) rather than macOS's system /usr/bin/python3. The system Python links against LibreSSL, which makes urllib3 print a NotOpenSSLWarning on every run — harmless, but easy to avoid by picking the venv's interpreter deliberately.

4. Run it

python3 sd2b2.py --source /Volumes/SDCARD/DCIM --bucket my-photos-backup --prefix imports/2026-08-01

Useful flags:

Flag	Purpose
--engine {native,rclone}	native (default) is this script's own uploader; rclone shells out to rclone for large, non-EXIF backups (see below)
--prefix PATH	Folder inside the bucket to upload into
--flatten	Ignore source subfolders, upload everything directly under --prefix
--ext jpg,cr2,mp4	Only upload matching extensions
--rename-from-exif	Rename into day folders using EXIF capture time (see below)
--workers N	Parallel uploads (default 4)
--no-skip-existing	Re-upload even if a same-named file already exists at the destination
--delete-source	Delete each file from the SD card only after it uploads successfully (off by default)
--dry-run	List what would be uploaded without transferring anything
--dry-run-sample N	With --dry-run, only scan/print an evenly-spread sample of N files instead of the whole source (default: 20, 0 = scan everything)

Renaming from EXIF

--rename-from-exif reorganizes uploads into day folders and renames each file from its EXIF capture time, instead of preserving the SD card's original folder/file names:

python3 sd2b2.py --source /Volumes/SDCARD/DCIM --bucket my-photos-backup --prefix imports --rename-from-exif

• Files are grouped into "shots" by folder + filename stem, so IMG_0262.JPG and IMG_0262.NEF (a RAW+JPEG pair from one shutter press) are treated as one shot and get matching names:

  imports/2026-06-10/20260610184003_1.jpg
  imports/2026-06-10/20260610184003_1.nef
  

  20260610184003 is the capture timestamp (YYYYMMDDHHMMSS) and _1 is that shot's sequence number for the day, in capture order.
• The day folder and sequence number reset for each new capture date found on the card.
• If a file has no readable EXIF DateTimeOriginal (e.g. most video, or a corrupt/stripped file), its file modification time is used instead and a warning is logged — mtime is a good proxy straight off an SD card, but less trustworthy than EXIF.
• Numbering is deterministic for a given file set, so re-running the script on the same (partially uploaded) card lines up with --skip-existing instead of renumbering everything. Numbering does not persist across separate runs — each run starts every day's counter at --seq-start (default 1). If you're importing a second card that adds more shots to a day you already uploaded, either pass --seq-start set past the last number you used for that day, or give the second run its own --prefix to keep the two imports apart.
• Tune the format with --day-folder-format, --timestamp-format, --seq-width (zero-pad, e.g. 3 → _001), and --seq-start.

This mode requires the exifread package (already in requirements.txt).

Example dry run first, then a real import that clears the card once verified uploaded:

python3 sd2b2.py --source /Volumes/SDCARD --bucket my-photos-backup --dry-run
python3 sd2b2.py --source /Volumes/SDCARD --bucket my-photos-backup --delete-source

On a large card, --dry-run stops early once it has collected --dry-run-sample files (20 by default) spread across a few folders, instead of scanning the whole source — good enough to spot-check naming and format without waiting on a full walk. Pass --dry-run-sample 0 to scan and list every file.

Large, non-photo backups: use rclone instead

For a plain "copy everything from this drive" backup with no EXIF renaming — especially at very large file counts (hundreds of thousands+) — rclone is a better fit than this script's own uploader: it handles bucket listing and rate-limit/retry behavior more gracefully at that scale than a small hand-rolled script can.

--engine rclone is a thin wrapper, not a reimplementation. Passing it makes sd2b2.py build an rclone copy (or rclone move, if --delete-source is set) command from your --source/--bucket/--prefix/ --dry-run flags and hand off to the real rclone binary — install it first with brew install rclone. Credentials are passed to the rclone subprocess as environment variables (never as command-line arguments, same rule this script follows for its own credentials), so there's no need to run rclone config — sd2b2.py configures an in-memory remote for you on each run.

# preview
python3 sd2b2.py --source /Volumes/SOURCE --bucket my-bucket --prefix my-prefix --engine rclone --dry-run

# real run
python3 sd2b2.py --source /Volumes/SOURCE --bucket my-bucket --prefix my-prefix --engine rclone

Flags specific to this engine:

Flag	Purpose
--rclone-transfers N	Parallel transfers (default: 8)
--rclone-tpslimit N	Cap on B2 transactions per second, to stay under B2's throttling threshold (default: 100)

Every --engine rclone run always adds:

• --fast-list: cheap bucket listing at large object counts.
• --checksum: compares B2's stored SHA1 rather than size + modification time — important because rclone's default modtime-based check trusts a metadata field it sets on its own uploads, which files uploaded by the native sd2b2.py engine never had set; SHA1 is a plain B2 file attribute regardless of which tool uploaded a file, so this correctly recognizes already-uploaded files either way and lets you switch engines mid-backup without re-uploading everything.
• --copy-links: follows symlinks and uploads the real file they point to, matching the native engine's behavior (Python's path.is_file() follows symlinks transparently). Without this, rclone's default is to silently skip symlinks entirely — a real gap for sources like /Applications, where .app bundles are full of framework symlinks.

Notes and limitations of the wrapper:

• Native-engine-only flags (--rename-from-exif, --flatten, --ext, --workers, --no-skip-existing, --dry-run-sample) have no effect here; --rename-from-exif combined with --engine rclone is rejected outright, since rclone has no EXIF-renaming concept — use the default --engine native for camera imports.
• The wrapper never runs rclone sync's destination-mirroring mode (which deletes remote files absent from the source) — only copy/move — to match this script's "never delete data beyond what --delete-source says" design. If you want mirroring, invoke rclone directly yourself.
• Only the production B2 endpoint is configured; B2_REALM test realms aren't supported by this engine.
• Don't run --engine rclone at the same time as an in-progress --engine native upload to the same bucket/source — stop one before starting the other.

Design notes / best practices baked in

• Least-privilege credentials: designed around a bucket-scoped application key rather than the account master key.
• No secrets on the command line: keys are only ever read from the environment / .env, never from argv, so they don't leak into shell history or ps aux.
• Nothing persisted to disk: uses b2sdk's InMemoryAccountInfo instead of its default SQLite cache, so no auth token or key material is written to disk between runs.
• Integrity: b2sdk computes a SHA1 of each file while uploading and B2 verifies it server-side, so a corrupted transfer is rejected rather than silently stored.
• Idempotent by default: --skip-existing (default on) lets you re-run the script safely, e.g. if an import is interrupted halfway through.
• Safe deletes: source files are only ever removed with the explicit --delete-source flag, and only after a successful, verified upload.