Skip to content

Make legacy dataset-loader synthetic fallbacks explicit and source-verified #722

Description

@shawcharles

Four public dataset loaders currently point at URLs that return HTTP 404 and then silently substitute generated synthetic observations:

  • load_card_krueger()
  • load_castle_doctrine()
  • load_divorce_laws()
  • load_mpdta()

On current main (0c618857), all four force_download=True calls returned a plausible DataFrame without a warning and without df.attrs["source"]. The resulting shapes were (310, 9), (539, 8), (861, 8), and (2500, 7) respectively.

Minimal reproducer:

import warnings

from diff_diff.datasets import (
    load_card_krueger,
    load_castle_doctrine,
    load_divorce_laws,
    load_mpdta,
)

for loader in (
    load_card_krueger,
    load_castle_doctrine,
    load_divorce_laws,
    load_mpdta,
):
    with warnings.catch_warnings(record=True) as caught:
        warnings.simplefilter("always")
        data = loader(force_download=True)
    print(loader.__name__, data.shape, data.attrs.get("source"), len(caught))

Observed:

load_card_krueger (310, 9) None 0
load_castle_doctrine (539, 8) None 0
load_divorce_laws (861, 8) None 0
load_mpdta (2500, 7) None 0

This can make a replication appear to use the named empirical dataset when it actually uses seeded synthetic data. load_prop99() and load_walmart() already provide the safer precedent: they warn on fallback and expose provenance through df.attrs["source"].

Would you accept a PR with this boundary?

Proposed scope

  • Preserve the four public loader signatures and existing fallback schemas.
  • On every successful canonical-data path, set a stable, dataset-specific df.attrs["source"] value.
  • On download, checksum, parsing, or source-validation failure, emit exactly one UserWarning that clearly says SYNTHETIC and set df.attrs["source"] = "synthetic_fallback".
  • Replace dead URLs only where an authoritative, redistributable source can be established. Use immutable source URLs and pinned SHA-256 checksums; do not replace a dead URL with an unpinned mutable mirror merely to make the download succeed.
  • Verify expected columns and key structural invariants before returning downloaded data. Unverified bytes must never be represented as canonical data.
  • Extend the cache helper as needed so CSV content is checksum-verified before it is trusted or cached, following the existing binary-download integrity pattern.
  • Add deterministic, mocked tests for canonical success, network failure, checksum mismatch, malformed/schema-invalid content, stale cache replacement, warning count, and both provenance markers.
  • Update the loader docstrings and add a brief CHANGELOG.md entry.

If no trustworthy source can be established for one of the datasets, retaining its generated fallback is preferable to adopting an unofficial source, provided the fallback is loud and provenance-marked.

Explicitly out of scope

  • No changes to the generated fallback values or random seeds.
  • No estimator, inference, or public dataset-dispatch API changes.
  • No required live-network CI canary in this PR; external-source availability can be considered separately without making required CI network-dependent.
  • No broad dataset-module redesign or new runtime dependency solely to read an upstream format.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions