Four public dataset loaders currently point at URLs that return HTTP 404 and then silently substitute generated synthetic observations:
load_card_krueger()
load_castle_doctrine()
load_divorce_laws()
load_mpdta()
On current main (0c618857), all four force_download=True calls returned a plausible DataFrame without a warning and without df.attrs["source"]. The resulting shapes were (310, 9), (539, 8), (861, 8), and (2500, 7) respectively.
Minimal reproducer:
import warnings
from diff_diff.datasets import (
load_card_krueger,
load_castle_doctrine,
load_divorce_laws,
load_mpdta,
)
for loader in (
load_card_krueger,
load_castle_doctrine,
load_divorce_laws,
load_mpdta,
):
with warnings.catch_warnings(record=True) as caught:
warnings.simplefilter("always")
data = loader(force_download=True)
print(loader.__name__, data.shape, data.attrs.get("source"), len(caught))
Observed:
load_card_krueger (310, 9) None 0
load_castle_doctrine (539, 8) None 0
load_divorce_laws (861, 8) None 0
load_mpdta (2500, 7) None 0
This can make a replication appear to use the named empirical dataset when it actually uses seeded synthetic data. load_prop99() and load_walmart() already provide the safer precedent: they warn on fallback and expose provenance through df.attrs["source"].
Would you accept a PR with this boundary?
Proposed scope
- Preserve the four public loader signatures and existing fallback schemas.
- On every successful canonical-data path, set a stable, dataset-specific
df.attrs["source"] value.
- On download, checksum, parsing, or source-validation failure, emit exactly one
UserWarning that clearly says SYNTHETIC and set df.attrs["source"] = "synthetic_fallback".
- Replace dead URLs only where an authoritative, redistributable source can be established. Use immutable source URLs and pinned SHA-256 checksums; do not replace a dead URL with an unpinned mutable mirror merely to make the download succeed.
- Verify expected columns and key structural invariants before returning downloaded data. Unverified bytes must never be represented as canonical data.
- Extend the cache helper as needed so CSV content is checksum-verified before it is trusted or cached, following the existing binary-download integrity pattern.
- Add deterministic, mocked tests for canonical success, network failure, checksum mismatch, malformed/schema-invalid content, stale cache replacement, warning count, and both provenance markers.
- Update the loader docstrings and add a brief
CHANGELOG.md entry.
If no trustworthy source can be established for one of the datasets, retaining its generated fallback is preferable to adopting an unofficial source, provided the fallback is loud and provenance-marked.
Explicitly out of scope
- No changes to the generated fallback values or random seeds.
- No estimator, inference, or public dataset-dispatch API changes.
- No required live-network CI canary in this PR; external-source availability can be considered separately without making required CI network-dependent.
- No broad dataset-module redesign or new runtime dependency solely to read an upstream format.
Four public dataset loaders currently point at URLs that return HTTP 404 and then silently substitute generated synthetic observations:
load_card_krueger()load_castle_doctrine()load_divorce_laws()load_mpdta()On current
main(0c618857), all fourforce_download=Truecalls returned a plausibleDataFramewithout a warning and withoutdf.attrs["source"]. The resulting shapes were(310, 9),(539, 8),(861, 8), and(2500, 7)respectively.Minimal reproducer:
Observed:
This can make a replication appear to use the named empirical dataset when it actually uses seeded synthetic data.
load_prop99()andload_walmart()already provide the safer precedent: they warn on fallback and expose provenance throughdf.attrs["source"].Would you accept a PR with this boundary?
Proposed scope
df.attrs["source"]value.UserWarningthat clearly says SYNTHETIC and setdf.attrs["source"] = "synthetic_fallback".CHANGELOG.mdentry.If no trustworthy source can be established for one of the datasets, retaining its generated fallback is preferable to adopting an unofficial source, provided the fallback is loud and provenance-marked.
Explicitly out of scope