Skip to content

fix: make legacy dataset provenance explicit#723

Open
shawcharles wants to merge 2 commits into
igerber:mainfrom
shawcharles:fix/dataset-loader-provenance
Open

fix: make legacy dataset provenance explicit#723
shawcharles wants to merge 2 commits into
igerber:mainfrom
shawcharles:fix/dataset-loader-provenance

Conversation

@shawcharles

Copy link
Copy Markdown
Contributor

Fixes #722.

Summary

  • replace the dead Card-Krueger, Castle Doctrine, and mpdta URLs with immutable commit-pinned sources and verify every downloaded or cached byte sequence against a pinned SHA-256
  • validate source schemas, categories, panel keys, treatment/cohort invariants, dimensions, missingness, and numeric domains before marking data canonical
  • make load_divorce_laws() explicitly warn and return its existing synthetic fallback because no verified source reproduces its composite public schema without additional analytical choices
  • mark all successful frames with a stable df.attrs["source"]; expected source failures emit one UserWarning containing SYNTHETIC and use source="synthetic_fallback"
  • bound cache reads/downloads to 50 MiB and replace cache files atomically, preserving a valid cached copy if replacement fails

The verified artefacts are:

  • Card-Krueger public.dat at commit 07bc929f1d6552db117bd27a7cf0d881d16e9494, SHA-256 04bde0cad5540980f32ce099c6dad369e2f05494698071d8a65b3e1cbe9ca53a
  • Cheng-Hoekstra castle.dta at commit ca4279a87a6f0759f6b6f02841a53bdd68e27d3c, SHA-256 804633c161827b6c0824f86f239046386d1a8266a866f83bf5ddb2aa762a5f29
  • Callaway-Sant'Anna mpdta.csv at commit 7ad707385354cb3924b8da94ef7e62a76bf55a4d, SHA-256 2283bea1221a152420f98dfa20f633c5d054ea51d881115c8cd702a97bcd3167

mpdta is now the genuine packaged example data. The Wooldridge/Callaway estimator cross-check remains on the pre-existing controlled synthetic DGP so that it tests estimators rather than source availability or a different finite-sample estimand.

Validation

  • python -m pytest tests/test_datasets.py -q (Python 3.12): 69 passed
  • python3.9 -m pytest tests/test_datasets.py -q in an isolated Python 3.9.25 environment: 69 passed
  • affected estimator, results, event-study, methodology, and documentation suites: 543 passed, 13 skipped, 2 deselected
  • python -m ruff check on all touched Python files
  • python -m black --check on all touched Python files
  • python -m mypy --follow-imports=skip diff_diff/datasets.py
  • live smoke test of all four loaders with an empty temporary cache, followed by cache reload equality checks

All tests are deterministic and required CI does not depend on live network access.

@shawcharles

Copy link
Copy Markdown
Contributor Author

Thanks. I pushed a small review-response follow-up in b611147f.

Changes since the initial PR:

  • verified fresh downloads are now returned even if best-effort cache persistence fails;
  • _load_verified_dataset() warnings now use caller-facing stacklevel attribution;
  • list_datasets() now marks divorce_laws as synthetic fallback only;
  • added regression tests for the cache-write failure path, warning attribution, and catalogue wording;
  • updated the changelog entry.

Validation:

  • python -m pytest tests/test_datasets.py -q -> 71 passed
  • Python 3.9 throwaway venv: tests/test_datasets.py -> 71 passed
  • python -m ruff check diff_diff/datasets.py tests/test_datasets.py -> passed
  • python -m black --check diff_diff/datasets.py tests/test_datasets.py -> passed
  • python -m mypy --follow-imports=skip diff_diff/datasets.py -> passed

From my side this is ready for maintainer review / CI.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Make legacy dataset-loader synthetic fallbacks explicit and source-verified

2 participants