Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,8 @@ Both are cheap to follow and expensive to discover. `PLAN.md` carries the reason
- **Never delete a file a sibling repo reads.** `lecture-wasm` fetches `lecture-python-intro`'s *committed blobs* by URL, so deleting intro's copy in a repoint PR 404s the wasm build immediately. "Delete the lecture repo's own copy in the same repoint PR" applies only where no sibling reads it; where one does, the sibling's repoint lands first or in the same set.
- **Repoint every consumer of a dataset together.** The strict audit has no green state for a partially-repointed dataset — `pending`/`landed` fails once any consumer reads data-lectures, and `repointed`/`final` fails while any consumer still does not. Land the lecture repoints first, then flip `migration.yml`; that flip is the push that re-runs the audit, so reality and the tracker agree by the time it runs.

- **A migration moves bytes; it does not update them.** Land the copy the lectures already consume, validated byte-identical — that is what makes a repoint provably unable to change a figure. If the committed file differs from what upstream publishes today, migrate it unchanged anyway, record the delta in `integrity.upstream` **and** in the register at [#39](https://github.com/QuantEcon/data-lectures/issues/39), and leave the decision for after the migration. Adopting a newer vintage changes lecture output and is an author's call, not an infrastructure one — and per "Corrections vs vintages" below it gets a **new filename**, never a silent replacement.

Cross-repo repoints are worked from [`QuantEcon/workspace-lectures`](https://github.com/QuantEcon/workspace-lectures) — same branch name in each repo, one PR per repo, no aggregate PR.

### Corrections vs vintages
Expand Down
14 changes: 11 additions & 3 deletions CATALOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,23 +4,31 @@

# Dataset catalog — `QuantEcon/data-lectures`

The migrated-dataset registry, **auto-generated** from the sidecar manifests (`lectures/*.yml`). Do not edit by hand — run `python scripts/build_catalog.py`. A dataset appears here once it has a manifest; files not yet migrated are tracked in [PLAN.md](PLAN.md) Phase 9.
The dataset registry, **auto-generated** from the sidecar manifests (`lectures/*.yml`). Do not edit by hand — run `python scripts/build_catalog.py`. A dataset appears here once it has a manifest, which may be before its consuming lectures are repointed — an empty **Used by** column means the file is here and documented but no lecture reads it from this repo yet. Files still to migrate are tracked in [PLAN.md](PLAN.md).

**10 datasets migrated** · 2.4 MB total · 9 permitted / 1 restricted redistribution
**18 datasets** · 10 read by lectures today, 8 awaiting repoint · 5.3 MB total · 17 permitted / 1 restricted redistribution

| Dataset | Class | Source | Licence | Redist. | Integrity | Builder | Size | Used by |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| [**ames_house_prices.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/ames_house_prices.csv)<br><sub>Ames, Iowa — residential house sales, 2006-2010</sub> | constructed | [Ames Housing data (De Cock 2011), Journal of Statistics Education](http://jse.amstat.org/v19n3/decock.pdf) | | ✅ permitted | ✅ verified | ✅ committed | 75.2 KB | [lecture-python-intro · observed_distributions.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/observed_distributions.md)<br>[lecture-python-intro · fitting_distributions.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/fitting_distributions.md) |
| [**assignat.xlsx**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/assignat.xlsx)<br><sub>French Revolution — assignat issues, budgets and seigniorage (Sargent-Velde)</sub> | verbatim | [Sargent and Velde, "Macroeconomic Features of the French Revolution" — supporting spreadsheets](https://www.journals.uchicago.edu/doi/10.1086/261992) | | ✅ permitted | ⚠️ unverifiable | n/a (verbatim) | 204.6 KB | — |
| [**caron.npy**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/caron.npy)<br><sub>French Revolution — monthly specie value of the assignat, 1791-1796</sub> | constructed | unrecorded | | ✅ permitted | ⚠️ unverifiable | ⚠️ unrecovered | 1.1 KB | — |
| [**chapter_3.xlsx**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/chapter_3.xlsx)<br><sub>The Ends of Four Big Inflations — appendix tables, transcribed</sub> | constructed | [Sargent, "Rational Expectations and Inflation", chapter 3 appendix tables](https://press.princeton.edu/books/paperback/9780691158709/rational-expectations-and-inflation) | | ✅ permitted | ⚠️ unverifiable | ⚠️ unrecovered | 71.6 KB | — |
| [**countries.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/countries.csv)<br><sub>WorldData.info country reference table</sub> | verbatim | [WorldData.info — country data downloads](https://www.worlddata.info/downloads/) | Proprietary — © WorldData.info, all rights reserved | ⚠️ restricted | ⚠️ unverifiable | n/a (verbatim) | 48.4 KB | [lecture-python-programming · pandas_panel.md](https://github.com/QuantEcon/lecture-python-programming/blob/main/lectures/pandas_panel.md)<br>[lecture-python.myst · pandas_panel.md](https://github.com/QuantEcon/lecture-python.myst/blob/main/lectures/pandas_panel.md) |
| [**dette.xlsx**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/dette.xlsx)<br><sub>French Revolution — public debt, military spending and revenues (Sargent-Velde)</sub> | verbatim | [Sargent and Velde, "Macroeconomic Features of the French Revolution" — supporting spreadsheets](https://www.journals.uchicago.edu/doi/10.1086/261992) | | ✅ permitted | ⚠️ unverifiable | n/a (verbatim) | 617.2 KB | — |
| [**employ.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/employ.csv)<br><sub>Eurostat employment in Europe — by age and sex, 2007–2016</sub> | constructed | [Eurostat — Employment database](https://ec.europa.eu/eurostat/data/database) | Eurostat reuse (Commission Decision 2011/833/EU) | ✅ permitted | ⚠️ unverifiable | ⚠️ unrecovered | 1.6 MB | [lecture-python-programming · pandas_panel.md](https://github.com/QuantEcon/lecture-python-programming/blob/main/lectures/pandas_panel.md)<br>[lecture-python.myst · pandas_panel.md](https://github.com/QuantEcon/lecture-python.myst/blob/main/lectures/pandas_panel.md) |
| [**epl_match_goals.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/epl_match_goals.csv)<br><sub>English Premier League — full-time scores, 2015-16 to 2024-25</sub> | constructed | [openfootball / football.json](https://github.com/openfootball/football.json) | Public domain | ✅ permitted | ✅ verified | ✅ committed | 203.2 KB | [lecture-python-intro · fitting_distributions.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/fitting_distributions.md) |
| [**fig_3.xlsx**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/fig_3.xlsx)<br><sub>French Revolution — figure 3 series (Sargent-Velde)</sub> | verbatim | [Sargent and Velde, "Macroeconomic Features of the French Revolution" — supporting spreadsheets](https://www.journals.uchicago.edu/doi/10.1086/261992) | | ✅ permitted | ⚠️ unverifiable | n/a (verbatim) | 9.2 KB | — |
| [**japan_deaths_by_age.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/japan_deaths_by_age.csv)<br><sub>Japan — deaths by single year of age, 2023</sub> | constructed | [United Nations, Department of Economic and Social Affairs, Population Division — World Population Prospects 2024](https://population.un.org/wpp/downloads) | CC BY 3.0 IGO | ✅ permitted | ✅ verified | ✅ committed | 1.7 KB | [lecture-python-intro · observed_distributions.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/observed_distributions.md)<br>[lecture-python-intro · fitting_distributions.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/fitting_distributions.md) |
| [**japan_earthquakes.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/japan_earthquakes.csv)<br><sub>Japan region — earthquakes of magnitude 5 and above, 2000-2024</sub> | constructed | [Advanced National Seismic System (ANSS) Comprehensive Earthquake Catalog (ComCat), US Geological Survey](https://earthquake.usgs.gov/earthquakes/search/) | US Government work — public domain | ✅ permitted | ✅ verified | ✅ committed | 172.8 KB | [lecture-python-intro · fitting_distributions.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/fitting_distributions.md) |
| [**japan_population_by_age.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/japan_population_by_age.csv)<br><sub>Japan — population by single year of age, 2024</sub> | constructed | [Population Estimates, Statistics Bureau of Japan, Ministry of Internal Affairs and Communications](https://www.stat.go.jp/english/data/jinsui/index.html) | Japan Statistics Bureau terms of use | ✅ permitted | ✅ verified | ✅ committed | 1.3 KB | [lecture-python-intro · prob_dist.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/prob_dist.md) |
| [**lingcod_msy_recovery.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/lingcod_msy_recovery.csv)<br><sub>Pacific Coast lingcod — biomass and fishing pressure relative to MSY</sub> | constructed | [RAM Legacy Stock Assessment Database](https://www.ramlegacy.org/) | CC BY 4.0 | ✅ permitted | ⚠️ unverifiable | ⚠️ unrecovered | 2.3 KB | [lecture-python-intro · msy_fishery.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/msy_fishery.md) |
| [**longprices.xls**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/longprices.xls)<br><sub>Price levels in four hard-currency countries, 1600-2000</sub> | verbatim | [Sargent and Velde, "The Big Problem of Small Change" — supporting data](https://press.princeton.edu/books/hardcover/9780691116358/the-big-problem-of-small-change) | | ✅ permitted | ⚠️ unverifiable | n/a (verbatim) | 379.5 KB | — |
| [**mpd2020.xlsx**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/mpd2020.xlsx)<br><sub>Maddison Project Database 2020 — GDP per capita and population, 1 CE to 2018</sub> | constructed | [Maddison Project Database, version 2020](https://www.rug.nl/ggdc/historicaldevelopment/maddison/releases/maddison-project-database-2020) | CC BY 4.0 | ✅ permitted | ⇄ diverged | ⚠️ unrecovered | 1.7 MB | — |
| [**nom_balances.npy**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/nom_balances.npy)<br><sub>French Revolution — monthly nominal assignat balances, 1789-1796</sub> | constructed | unrecorded | | ✅ permitted | ⚠️ unverifiable | ⚠️ unrecovered | 1.4 KB | — |
| [**realwage.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/realwage.csv)<br><sub>OECD real minimum wages — 32 countries, 2006–2016</sub> | constructed | [OECD — Real minimum wages (RMW)](https://stats.oecd.org/Index.aspx?DataSetCode=RMW) | CC BY 4.0 | ✅ permitted | ⚠️ unverifiable | ⚠️ unrecovered | 118.7 KB | [lecture-python-programming · pandas_panel.md](https://github.com/QuantEcon/lecture-python-programming/blob/main/lectures/pandas_panel.md)<br>[lecture-python.myst · pandas_panel.md](https://github.com/QuantEcon/lecture-python.myst/blob/main/lectures/pandas_panel.md) |
| [**us_adult_heights.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/us_adult_heights.csv)<br><sub>United States — adult standing height by sex, NHANES 2015-2018</sub> | constructed | [National Health and Nutrition Examination Survey (NHANES), US Centers for Disease Control and Prevention, National Center for Health Statistics](https://www.cdc.gov/nchs/nhanes/index.htm) | US Government work — public domain | ✅ permitted | ✅ verified | ✅ committed | 123.1 KB | [lecture-python-intro · prob_dist.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/prob_dist.md)<br>[lecture-python-intro · observed_distributions.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/observed_distributions.md)<br>[lecture-python-intro · fitting_distributions.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/fitting_distributions.md) |

---

**Legend** — *Integrity* is the `integrity.upstream.status` (is this what the source says?): ✅ verified · ◑ spot-checked · ⚠️ unverifiable · … unverified · ❌ failing. *Redist.* ⚠️ restricted files are cached as inherited exposures and tracked for licence review ([workspace-lectures#20](https://github.com/QuantEcon/workspace-lectures/issues/20)). *Builder* ⚠️ unrecovered marks a constructed dataset whose builder was never committed (PLAN Phase 9).
**Legend** — *Integrity* is the `integrity.upstream.status` (is this what the source says?): ✅ verified · ◑ spot-checked · ⇄ diverged (checked, differs, delta known and tracked) · ⚠️ unverifiable · … unverified · ❌ failing. *Redist.* ⚠️ restricted files are cached as inherited exposures and tracked for licence review ([workspace-lectures#20](https://github.com/QuantEcon/workspace-lectures/issues/20)). *Builder* ⚠️ unrecovered marks a constructed dataset whose builder was never committed (PLAN Phase 9).
18 changes: 17 additions & 1 deletion PLAN.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ This repository is being shaped into the **single canonical repository for data

## Repoint rules

Two rules learned the hard way. Both are about *ordering*, both are cheap to follow and expensive to discover, and neither is enforced by CI — the strict audit catches the second only after the fact.
Three rules learned the hard way. The first two are about *ordering*, both cheap to follow and expensive to discover, and neither is enforced by CI — the strict audit catches the second only after the fact. The third is about *scope*.

### 1. Repoint a sibling reader before deleting the file it reads

Expand All @@ -60,6 +60,22 @@ The strict audit has **no green state for a partially-repointed dataset**. `scri

Practically: one branch name across data-lectures + every consuming repo, PRs opened together, lecture repoints merged first, then the `migration.yml` flip to `repointed` — that last push is what re-runs the audit, and by then reality and the tracker agree.

### 3. A migration moves bytes; it does not update them

The copy that lands here is the copy the lectures **already consume**, validated byte-identical in the repoint PR. That is what makes a repoint safe to merge: it provably cannot change a single figure.

Adopting a newer upstream vintage is a *different change* with a different risk profile — it does change lecture output, it needs figures re-reviewed, and it is an author-facing decision rather than an infrastructure one. Conflating the two turns every repoint into a content review and stalls the programme.

So when a migration finds that the committed file differs from what upstream publishes today:

1. **Migrate what the lectures use**, unchanged, with the byte-compare gate as normal.
2. **Record the delta** in the dataset's manifest (`integrity.upstream`) *and* in the register at [#39](https://github.com/QuantEcon/data-lectures/issues/39) — the manifest makes it visible in the catalog from day one, the register is where it gets reasoned about.
3. **Review the register once the migration completes**, and decide each case on its merits.

Two deltas look alike and need opposite responses. *Upstream moved* — a newer vintage exists; adopting it means a **new filename**, per "Corrections vs vintages" in `AGENTS.md`, so consumers opt in. *Our copy diverges* — upstream is unchanged but our file was modified; resolving means reconciling the edit. `mpd2020.xlsx` is the first recorded instance of the second kind, and it is instructive: the local edits are load-bearing for the consuming lecture, so the file and the lecture have to move together.

Detecting these automatically rather than by accident is proposed in [#40](https://github.com/QuantEcon/data-lectures/issues/40).

## Migration tracks

The remaining work decomposes by **consuming series** rather than by hosting pattern, because — apart from the `intro`/`wasm` pairing — each series now owns its own data. This is the execution view; the phases below remain the machinery each track passes through.
Expand Down
Loading
Loading