From 0338a4ac28c71c387d41f9bf0db8ec3461a8cd47 Mon Sep 17 00:00:00 2001 From: Matt McKay Date: Thu, 6 Aug 2026 14:10:43 +1000 Subject: [PATCH 1/5] Write manifests for the eight intro statics, and check them against upstream These files landed in the Feb 2025 migration (#5) and have sat in lectures/ without metadata ever since. They are Track A's repoint targets, and a repoint needs a manifest, so the metadata research lands here as one PR and leaves the three repoint sets purely mechanical. Status is `landed`, consumers empty: the files are here and documented, but no lecture reads them from this repo yet. The strict audit is green in that state -- verified before writing, not assumed. Two findings came out of actually checking rather than transcribing. mpd2020.xlsx is NOT a pristine upstream copy. Re-fetched from the Maddison release and compared cell by cell across all six sheets: every data value matches, including all 21,683 rows of `Full data` -- but three header labels on `Regional data` are local edits ("GDP pc 2011 prices" -> "gdppc_2011", "Population" -> "pop", and a "gdppc_2011" added where upstream is empty). The lecture reads that sheet with header=(0,1,2), so those renames are load-bearing: replacing the file with a clean upstream copy would silently break long_run_growth. Classified `constructed` rather than `verbatim` as a result, and the delta is enumerated in the manifest so recovery is cheap. caron.npy and nom_balances.npy have no recorded provenance anywhere -- the lecture's own Data Sources section names three spreadsheets and omits both. Checked whether either is a dump of a column in the committed Sargent-Velde workbooks: neither is. caron's terminal values appear nowhere in them, and while nom_balances' final value does occur in several assignat.xlsx sheets, no column holds the 81-value series contiguously. So they cannot be re-derived from what we hold, and the manifests say so instead of guessing a source. The filename "caron" is recorded as a lead, explicitly flagged as inference. The three French Revolution workbooks and longprices.xls are the authors' own files republished unmodified, so `verbatim` with `unverifiable` upstream status -- there is no publisher distribution point to re-fetch from. chapter_3.xlsx is `constructed`: the lecture states it was transcribed by hand from printed appendix tables, which is our processing and cannot ship a builder. Two schema notes worth carrying into #14. These are multi-sheet workbooks, so the manifests record a `sheets:` list -- the sheets and cell ranges lectures actually read -- rather than one flat column list. And the French Revolution files are read POSITIONALLY (header=None with explicit usecols/skiprows/nrows), so a row or column inserted above or left of a range silently changes what the lecture plots. The manifests flag that with `positional_reads: true`. Also fixes the catalog headline, which would otherwise have claimed "18 datasets migrated" when 8 of them are landed but not yet repointed. It now counts what lectures actually read, derived from `consumers`. Co-Authored-By: Claude Opus 5 (1M context) --- CATALOG.md | 12 +++- lectures/assignat.xlsx.yml | 109 ++++++++++++++++++++++++++++++ lectures/caron.npy.yml | 104 ++++++++++++++++++++++++++++ lectures/chapter_3.xlsx.yml | 98 +++++++++++++++++++++++++++ lectures/dette.xlsx.yml | 114 +++++++++++++++++++++++++++++++ lectures/fig_3.xlsx.yml | 89 ++++++++++++++++++++++++ lectures/longprices.xls.yml | 94 ++++++++++++++++++++++++++ lectures/mpd2020.xlsx.yml | 123 ++++++++++++++++++++++++++++++++++ lectures/nom_balances.npy.yml | 97 +++++++++++++++++++++++++++ migration.yml | 91 +++++++++++++++++++++++++ scripts/build_catalog.py | 19 ++++-- 11 files changed, 943 insertions(+), 7 deletions(-) create mode 100644 lectures/assignat.xlsx.yml create mode 100644 lectures/caron.npy.yml create mode 100644 lectures/chapter_3.xlsx.yml create mode 100644 lectures/dette.xlsx.yml create mode 100644 lectures/fig_3.xlsx.yml create mode 100644 lectures/longprices.xls.yml create mode 100644 lectures/mpd2020.xlsx.yml create mode 100644 lectures/nom_balances.npy.yml diff --git a/CATALOG.md b/CATALOG.md index 7ebd110..6e1fd07 100644 --- a/CATALOG.md +++ b/CATALOG.md @@ -4,20 +4,28 @@ # Dataset catalog — `QuantEcon/data-lectures` -The migrated-dataset registry, **auto-generated** from the sidecar manifests (`lectures/*.yml`). Do not edit by hand — run `python scripts/build_catalog.py`. A dataset appears here once it has a manifest; files not yet migrated are tracked in [PLAN.md](PLAN.md) Phase 9. +The dataset registry, **auto-generated** from the sidecar manifests (`lectures/*.yml`). Do not edit by hand — run `python scripts/build_catalog.py`. A dataset appears here once it has a manifest, which may be before its consuming lectures are repointed — an empty **Used by** column means the file is here and documented but no lecture reads it from this repo yet. Files still to migrate are tracked in [PLAN.md](PLAN.md). -**10 datasets migrated** · 2.4 MB total · 9 permitted / 1 restricted redistribution +**18 datasets** · 10 read by lectures today, 8 awaiting repoint · 5.3 MB total · 17 permitted / 1 restricted redistribution | Dataset | Class | Source | Licence | Redist. | Integrity | Builder | Size | Used by | | --- | --- | --- | --- | --- | --- | --- | --- | --- | | [**ames_house_prices.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/ames_house_prices.csv)
Ames, Iowa — residential house sales, 2006-2010 | constructed | [Ames Housing data (De Cock 2011), Journal of Statistics Education](http://jse.amstat.org/v19n3/decock.pdf) | | ✅ permitted | ✅ verified | ✅ committed | 75.2 KB | [lecture-python-intro · observed_distributions.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/observed_distributions.md)
[lecture-python-intro · fitting_distributions.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/fitting_distributions.md) | +| [**assignat.xlsx**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/assignat.xlsx)
French Revolution — assignat issues, budgets and seigniorage (Sargent-Velde) | verbatim | [Sargent and Velde, "Macroeconomic Features of the French Revolution" — supporting spreadsheets](https://www.journals.uchicago.edu/doi/10.1086/261992) | | ✅ permitted | ⚠️ unverifiable | n/a (verbatim) | 204.6 KB | — | +| [**caron.npy**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/caron.npy)
French Revolution — monthly specie value of the assignat, 1791-1796 | constructed | unrecorded | | ✅ permitted | ⚠️ unverifiable | ⚠️ unrecovered | 1.1 KB | — | +| [**chapter_3.xlsx**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/chapter_3.xlsx)
The Ends of Four Big Inflations — appendix tables, transcribed | constructed | [Sargent, "Rational Expectations and Inflation", chapter 3 appendix tables](https://press.princeton.edu/books/paperback/9780691158709/rational-expectations-and-inflation) | | ✅ permitted | ⚠️ unverifiable | ⚠️ unrecovered | 71.6 KB | — | | [**countries.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/countries.csv)
WorldData.info country reference table | verbatim | [WorldData.info — country data downloads](https://www.worlddata.info/downloads/) | Proprietary — © WorldData.info, all rights reserved | ⚠️ restricted | ⚠️ unverifiable | n/a (verbatim) | 48.4 KB | [lecture-python-programming · pandas_panel.md](https://github.com/QuantEcon/lecture-python-programming/blob/main/lectures/pandas_panel.md)
[lecture-python.myst · pandas_panel.md](https://github.com/QuantEcon/lecture-python.myst/blob/main/lectures/pandas_panel.md) | +| [**dette.xlsx**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/dette.xlsx)
French Revolution — public debt, military spending and revenues (Sargent-Velde) | verbatim | [Sargent and Velde, "Macroeconomic Features of the French Revolution" — supporting spreadsheets](https://www.journals.uchicago.edu/doi/10.1086/261992) | | ✅ permitted | ⚠️ unverifiable | n/a (verbatim) | 617.2 KB | — | | [**employ.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/employ.csv)
Eurostat employment in Europe — by age and sex, 2007–2016 | constructed | [Eurostat — Employment database](https://ec.europa.eu/eurostat/data/database) | Eurostat reuse (Commission Decision 2011/833/EU) | ✅ permitted | ⚠️ unverifiable | ⚠️ unrecovered | 1.6 MB | [lecture-python-programming · pandas_panel.md](https://github.com/QuantEcon/lecture-python-programming/blob/main/lectures/pandas_panel.md)
[lecture-python.myst · pandas_panel.md](https://github.com/QuantEcon/lecture-python.myst/blob/main/lectures/pandas_panel.md) | | [**epl_match_goals.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/epl_match_goals.csv)
English Premier League — full-time scores, 2015-16 to 2024-25 | constructed | [openfootball / football.json](https://github.com/openfootball/football.json) | Public domain | ✅ permitted | ✅ verified | ✅ committed | 203.2 KB | [lecture-python-intro · fitting_distributions.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/fitting_distributions.md) | +| [**fig_3.xlsx**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/fig_3.xlsx)
French Revolution — figure 3 series (Sargent-Velde) | verbatim | [Sargent and Velde, "Macroeconomic Features of the French Revolution" — supporting spreadsheets](https://www.journals.uchicago.edu/doi/10.1086/261992) | | ✅ permitted | ⚠️ unverifiable | n/a (verbatim) | 9.2 KB | — | | [**japan_deaths_by_age.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/japan_deaths_by_age.csv)
Japan — deaths by single year of age, 2023 | constructed | [United Nations, Department of Economic and Social Affairs, Population Division — World Population Prospects 2024](https://population.un.org/wpp/downloads) | CC BY 3.0 IGO | ✅ permitted | ✅ verified | ✅ committed | 1.7 KB | [lecture-python-intro · observed_distributions.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/observed_distributions.md)
[lecture-python-intro · fitting_distributions.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/fitting_distributions.md) | | [**japan_earthquakes.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/japan_earthquakes.csv)
Japan region — earthquakes of magnitude 5 and above, 2000-2024 | constructed | [Advanced National Seismic System (ANSS) Comprehensive Earthquake Catalog (ComCat), US Geological Survey](https://earthquake.usgs.gov/earthquakes/search/) | US Government work — public domain | ✅ permitted | ✅ verified | ✅ committed | 172.8 KB | [lecture-python-intro · fitting_distributions.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/fitting_distributions.md) | | [**japan_population_by_age.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/japan_population_by_age.csv)
Japan — population by single year of age, 2024 | constructed | [Population Estimates, Statistics Bureau of Japan, Ministry of Internal Affairs and Communications](https://www.stat.go.jp/english/data/jinsui/index.html) | Japan Statistics Bureau terms of use | ✅ permitted | ✅ verified | ✅ committed | 1.3 KB | [lecture-python-intro · prob_dist.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/prob_dist.md) | | [**lingcod_msy_recovery.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/lingcod_msy_recovery.csv)
Pacific Coast lingcod — biomass and fishing pressure relative to MSY | constructed | [RAM Legacy Stock Assessment Database](https://www.ramlegacy.org/) | CC BY 4.0 | ✅ permitted | ⚠️ unverifiable | ⚠️ unrecovered | 2.3 KB | [lecture-python-intro · msy_fishery.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/msy_fishery.md) | +| [**longprices.xls**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/longprices.xls)
Price levels in four hard-currency countries, 1600-2000 | verbatim | [Sargent and Velde, "The Big Problem of Small Change" — supporting data](https://press.princeton.edu/books/hardcover/9780691116358/the-big-problem-of-small-change) | | ✅ permitted | ⚠️ unverifiable | n/a (verbatim) | 379.5 KB | — | +| [**mpd2020.xlsx**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/mpd2020.xlsx)
Maddison Project Database 2020 — GDP per capita and population, 1 CE to 2018 | constructed | [Maddison Project Database, version 2020](https://www.rug.nl/ggdc/historicaldevelopment/maddison/releases/maddison-project-database-2020) | CC BY 4.0 | ✅ permitted | ✅ verified | ⚠️ unrecovered | 1.7 MB | — | +| [**nom_balances.npy**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/nom_balances.npy)
French Revolution — monthly nominal assignat balances, 1789-1796 | constructed | unrecorded | | ✅ permitted | ⚠️ unverifiable | ⚠️ unrecovered | 1.4 KB | — | | [**realwage.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/realwage.csv)
OECD real minimum wages — 32 countries, 2006–2016 | constructed | [OECD — Real minimum wages (RMW)](https://stats.oecd.org/Index.aspx?DataSetCode=RMW) | CC BY 4.0 | ✅ permitted | ⚠️ unverifiable | ⚠️ unrecovered | 118.7 KB | [lecture-python-programming · pandas_panel.md](https://github.com/QuantEcon/lecture-python-programming/blob/main/lectures/pandas_panel.md)
[lecture-python.myst · pandas_panel.md](https://github.com/QuantEcon/lecture-python.myst/blob/main/lectures/pandas_panel.md) | | [**us_adult_heights.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/us_adult_heights.csv)
United States — adult standing height by sex, NHANES 2015-2018 | constructed | [National Health and Nutrition Examination Survey (NHANES), US Centers for Disease Control and Prevention, National Center for Health Statistics](https://www.cdc.gov/nchs/nhanes/index.htm) | US Government work — public domain | ✅ permitted | ✅ verified | ✅ committed | 123.1 KB | [lecture-python-intro · prob_dist.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/prob_dist.md)
[lecture-python-intro · observed_distributions.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/observed_distributions.md)
[lecture-python-intro · fitting_distributions.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/fitting_distributions.md) | diff --git a/lectures/assignat.xlsx.yml b/lectures/assignat.xlsx.yml new file mode 100644 index 0000000..078e326 --- /dev/null +++ b/lectures/assignat.xlsx.yml @@ -0,0 +1,109 @@ +# Manifest for assignat.xlsx — one of three research spreadsheets assembled by +# Sargent and Velde for "Macroeconomic Features of the French Revolution". + +filename: assignat.xlsx +title: French Revolution — assignat issues, budgets and seigniorage (Sargent-Velde) +description: > + Working spreadsheet behind the monetary side of the French Revolution + analysis: assignat circulation and denominations, revolutionary budgets, + seigniorage, exchange rates and price indices, roughly 1789-1796. + +# The authors' own research workbook, republished as they assembled it. We +# apply no processing to the file itself -- the lecture slices ranges out of it +# at read time. +class: verbatim + +source: + name: Sargent and Velde, "Macroeconomic Features of the French Revolution" — supporting spreadsheets + url: https://www.journals.uchicago.edu/doi/10.1086/261992 + doi: 10.1086/261992 + citation: > + Sargent, Thomas J. and François R. Velde (1995). Macroeconomic Features of + the French Revolution. Journal of Political Economy, 103(3), 474-518. + note: > + The consuming lecture names the provenance directly: "This lecture uses + data from three spreadsheets assembled by [Sargent and Velde 1995]" -- + this file, dette.xlsx and fig_3.xlsx. The spreadsheets are the authors' + working files and are not published alongside the article. + +license: + name: null + url: null + redistribution: permitted + verified: 2026-08-06 + # No formal licence stated; the authors' own research data, served publicly + # from lecture-python-intro for years. Republished here with attribution + # under the record-and-track policy (#35). + +retrieved: null # unrecorded -- inherited bytes +maintainer: QuantEcon + +# --------------------------------------------------------------------------- +# Integrity (PLAN Phase 7) +# --------------------------------------------------------------------------- +# Migration check (repoint gate): verified 2026-08-06 by git blob hash against +# lecture-python-intro's lectures/datasets/assignat.xlsx -- identical. + +integrity: + sha256: 72213ff18a994f63520e9fc79d0c2117c451697cb29419da07c4d0c554503bc9 + upstream: + status: unverifiable + date: 2026-08-06 + against: null + note: > + Author-assembled research workbook with prose-only provenance. Not + distributed by the journal, no internal source list, no builder. Values + can be spot-checked against the figures in the published article but not + re-derived. + +# --------------------------------------------------------------------------- +# Shape +# --------------------------------------------------------------------------- +# READ POSITIONALLY. Every consuming read uses header=None with explicit +# usecols/skiprows/nrows, so the contract is a set of CELL RANGES, not named +# columns. Inserting or deleting a row or column anywhere above or left of a +# range silently shifts what the lecture plots, with no error. Treat the +# ranges below as the file's public interface. + +schema: + format: xlsx + sheet_count: 18 + positional_reads: true + sheets: + - name: Budgets + read_as: {usecols: 'J:K', skiprows: 22, nrows: 52, header: null} + shape: [52, 2] + dtypes: [float64, float64] + known_nulls_total: 2 + description: revolutionary government receipts and expenditures + - name: seignor + read_as: {usecols: 'F', skiprows: 6, nrows: 75, header: null} + shape: [75, 1] + dtypes: [float64] + known_nulls_total: 0 + description: monthly seigniorage from assignat issues + - name: Data + read_as: {usecols: 'P:Q', skiprows: 4, nrows: 80, header: null} + shape: [80, 2] + dtypes: [float64, float64] + known_nulls_total: 17 + description: assignat circulation and price level + - name: Data + read_as: {usecols: 'L', skiprows: 4, nrows: 80, header: null} + shape: [80, 1] + dtypes: [float64] + known_nulls_total: 0 + description: monthly index aligned with the P:Q block above + # 14 further sheets ship with the workbook and are not read by any lecture: + # Intro, Fig6, Data2, Denomina, Dom-nat, Exchge, Exchge2, Exchge3, + # Inflatio, Metal, Mint, Ramel, Ramel2, Revoluti, Post-95. + # Kept as the authors assembled them -- they are the working provenance for + # the ranges above. + date_range: {start: 1789, end: 1796} + row_count_floor: null # per-range, declared above + known_nulls: {} + +consumers: [] # empty until the Track A repoint lands + +builder: null +builder_status: not-applicable # verbatim republication -- nothing to build diff --git a/lectures/caron.npy.yml b/lectures/caron.npy.yml new file mode 100644 index 0000000..9df23d4 --- /dev/null +++ b/lectures/caron.npy.yml @@ -0,0 +1,104 @@ +# Manifest for caron.npy — assignat specie value, French Revolution. +# +# One of two files in this migration with NO recorded provenance anywhere. The +# consuming lecture's "Data Sources" section names three spreadsheets and does +# not mention this file or its twin, nom_balances.npy. Recorded honestly rather +# than given a plausible-sounding source. + +filename: caron.npy +title: French Revolution — monthly specie value of the assignat, 1791-1796 +description: > + 63 monthly observations of the assignat's value against specie, falling from + 96.7 at the start of 1791 to 0.431 in early 1796. The consuming lecture + differences its logarithm to obtain the monthly inflation rate used in its + real-balances regressions. + +# Constructed: a two-column numeric array is not a third-party file +# republished as distributed -- somebody produced it. What that process was is +# not recorded, which is precisely what builder_status captures. +class: constructed + +source: + name: unrecorded + url: null + citation: > + Consumed by analysis reproducing Sargent, Thomas J. and François R. Velde + (1995). Macroeconomic Features of the French Revolution. Journal of + Political Economy, 103(3), 474-518. + note: > + The filename is the only provenance signal. It most likely points to Pierre + Caron's published depreciation tables for the assignat, the standard source + for this series -- but that is an INFERENCE FROM THE NAME, stated nowhere in + the lecture or the file, and it should not be promoted to a source claim + without evidence. Recorded as a lead for whoever recovers this, not as fact. + +license: + name: null + url: null + redistribution: permitted + verified: 2026-08-06 + # Source unestablished, so no licence can be established either. These bytes + # have been served publicly from lecture-python-intro for years; migrated + # under the record-and-track policy (#35) and logged there, since a licence + # question does not hold up data we already serve. This one needs its source + # settled before its licence can be. + +retrieved: null # unrecorded -- inherited bytes +maintainer: QuantEcon + +# --------------------------------------------------------------------------- +# Integrity (PLAN Phase 7) +# --------------------------------------------------------------------------- +# Migration check (repoint gate): verified 2026-08-06 by git blob hash against +# lecture-python-intro's lectures/datasets/caron.npy -- identical. +# +# This file is one of the two LOCAL-PATH reads in lecture-python-intro +# (`np.load('datasets/caron.npy')`), which is what breaks downloaded and Colab +# notebooks. Repointing it at a public URL is the whole point of the migration +# -- the same failure mode pilot P1 fixed for msy_fishery. + +integrity: + sha256: 7e0bcc9a42fd05bb4a062041df029ca986ba70ddbc2611a7cbaf4d8d5d820885 + upstream: + status: unverifiable + date: 2026-08-06 + against: null + note: > + No source, no builder, no internal metadata -- a bare .npy carries no + provenance at all. Actively checked 2026-08-06 whether the series is a + dump of a column in the committed Sargent-Velde workbooks: it is not. + Neither terminal value (96.696, 0.431) appears anywhere in assignat.xlsx, + dette.xlsx or fig_3.xlsx, and no contiguous column matches. So it was not + derived from the data we hold, and cannot be re-derived from it. + +# --------------------------------------------------------------------------- +# Shape +# --------------------------------------------------------------------------- + +schema: + format: npy + dtype: float64 + shape: [63, 2] + columns: + - {index: 0, name: date, dtype: float64, description: decimal year, monthly steps of 1/12} + - {index: 1, name: specie_value, dtype: float64, description: assignat value against specie, index units} + row_count_floor: 63 # frozen historical extract: exact by design + date_range: {start: 1791.0, end: 1796.25} + known_nulls: {} # no NaNs present + note: > + Row alignment is load-bearing across files: the lecture pairs all 63 rows + of this array with rows 14:77 of nom_balances.npy. Changing the length of + either silently misaligns the other. + +consumers: [] # empty until the Track A repoint lands + +# --------------------------------------------------------------------------- +# Builder +# --------------------------------------------------------------------------- +# AGENTS.md: a constructed dataset without its builder is a bug -- inherited +# here rather than introduced. PLAN Phase 9 tracks recovery. Unlike the other +# unrecovered files in this migration, this one lacks even a named source, so +# recovery starts with identifying where the series came from. + +builder: null +builder_status: unrecovered diff --git a/lectures/chapter_3.xlsx.yml b/lectures/chapter_3.xlsx.yml new file mode 100644 index 0000000..146d668 --- /dev/null +++ b/lectures/chapter_3.xlsx.yml @@ -0,0 +1,98 @@ +# Manifest for chapter_3.xlsx — QuantEcon's transcription of the appendix +# tables to chapter 3 of Sargent's "Rational Expectations and Inflation". + +filename: chapter_3.xlsx +title: The Ends of Four Big Inflations — appendix tables, transcribed +description: > + Twenty-seven tables underlying the four post-World-War-I hyperinflation + figures for Austria, Hungary, Poland and Germany: budget receipts and + expenditures, note circulation, retail and wholesale price indices, and + exchange rates against the US dollar, 1919-1925. + +# Constructed, and unambiguously so: the numbers were transcribed by hand from +# printed appendix tables into this workbook. That transcription is our +# processing, and it is the step a builder would have to reproduce. +class: constructed + +source: + name: Sargent, "Rational Expectations and Inflation", chapter 3 appendix tables + url: https://press.princeton.edu/books/paperback/9780691158709/rational-expectations-and-inflation + citation: > + Sargent, Thomas J. (2013). Rational Expectations and Inflation, third + edition. Princeton, New Jersey: Princeton University Press. Chapter 3, + "The Ends of Four Big Inflations", and its appendix tables. + note: > + The consuming lecture is explicit about the provenance: "Data underlying + our graphs appear in tables in an appendix to chapter 3 ... We have + transcribed all of these data into a spreadsheet." The transcription is + QuantEcon's own work; the underlying figures are the book's. + +license: + name: null + url: null + redistribution: permitted + verified: 2026-08-06 + # The transcription is QuantEcon's own work. The underlying figures are + # historical macroeconomic statistics reproduced in a published book -- data, + # not expression. Served publicly from lecture-python-intro for years and + # republished here with attribution under the record-and-track policy (#35). + +retrieved: null # unrecorded -- inherited bytes +maintainer: QuantEcon + +# --------------------------------------------------------------------------- +# Integrity (PLAN Phase 7) +# --------------------------------------------------------------------------- +# Migration check (repoint gate): verified 2026-08-06 by git blob hash against +# lecture-python-intro's lectures/datasets/chapter_3.xlsx -- identical. + +integrity: + sha256: ee17d0031f17c663c4b96765dad85dc4d873b2c6cd00636a728b093da1d30923 + upstream: + status: unverifiable + date: 2026-08-06 + against: null + note: > + A hand transcription from a printed source. Verification means checking + cells against the book's appendix, which is possible for spot-checks but + cannot be automated or re-derived, and no record exists of who + transcribed it or when. Recorded honestly rather than being labelled + verbatim, which would imply an upstream file that does not exist. + +# --------------------------------------------------------------------------- +# Shape +# --------------------------------------------------------------------------- +# 27 sheets named Table3.1 .. Table3.27. The lecture loops over a subset, +# reading each with header=1 and truncating to a per-table row count held in +# the lecture rather than here. Table shapes vary by design -- these are +# distinct historical tables, not one series split up. + +schema: + format: xlsx + sheets_pattern: 'Table3\.\d+' + sheet_count: 27 + read_as: {header: 1} + sheets_sample: + - {name: Table3.1, shape: [6, 6], columns: [Budget Beginning, Budget Ending, Receipts, Expenditures, Deficit]} + - {name: Table3.2, shape: [78, 3], columns: [Year, Month, Total Note Circulation]} + - {name: Table3.3, shape: [44, 3], columns: [Year, Month, "Retail price index, 52 commodities"]} + - {name: Table3.4, shape: [74, 3], columns: [Year, Month, Exchange Rate]} + date_range: {start: 1919, end: 1925} + row_count_floor: null # varies per sheet; the invariant is sheet_count + known_nulls: {} + note: > + Entries are strings needing cleanup (the lecture applies a `process_entry` + map to strip whitespace and handle footnote markers), so dtypes are not + stable across sheets and are deliberately not asserted here. + +consumers: [] # empty until the Track A repoint lands + +# --------------------------------------------------------------------------- +# Builder +# --------------------------------------------------------------------------- +# A constructed dataset should ship its builder. This one cannot: the +# construction was manual transcription from print. Recorded as `unrecovered` +# per AGENTS.md rather than misclassified as verbatim (PLAN Phase 9). + +builder: null +builder_status: unrecovered diff --git a/lectures/dette.xlsx.yml b/lectures/dette.xlsx.yml new file mode 100644 index 0000000..c9421f5 --- /dev/null +++ b/lectures/dette.xlsx.yml @@ -0,0 +1,114 @@ +# Manifest for dette.xlsx — one of three research spreadsheets assembled by +# Sargent and Velde for "Macroeconomic Features of the French Revolution". + +filename: dette.xlsx +title: French Revolution — public debt, military spending and revenues (Sargent-Velde) +description: > + Working spreadsheet behind the fiscal side of the French Revolution + analysis: French and British military expenditure through the eighteenth + century wars, the structure and servicing of the French royal debt, tax + revenues, annuities and interest rates, roughly 1685-1800. + +class: verbatim + +source: + name: Sargent and Velde, "Macroeconomic Features of the French Revolution" — supporting spreadsheets + url: https://www.journals.uchicago.edu/doi/10.1086/261992 + doi: 10.1086/261992 + citation: > + Sargent, Thomas J. and François R. Velde (1995). Macroeconomic Features of + the French Revolution. Journal of Political Economy, 103(3), 474-518. + note: > + One of the three spreadsheets the consuming lecture attributes to the + authors of the 1995 article. Working files, not journal-distributed. + +license: + name: null + url: null + redistribution: permitted + verified: 2026-08-06 + # No formal licence stated; the authors' own research data, served publicly + # from lecture-python-intro for years. Republished with attribution under the + # record-and-track policy (#35). + +retrieved: null # unrecorded -- inherited bytes +maintainer: QuantEcon + +# --------------------------------------------------------------------------- +# Integrity (PLAN Phase 7) +# --------------------------------------------------------------------------- +# Migration check (repoint gate): verified 2026-08-06 by git blob hash against +# lecture-python-intro's lectures/datasets/dette.xlsx -- identical. + +integrity: + sha256: 94f766090a000d25509baea2cae8a8adc6f89b22dda068ec041cec2d88a827ae + upstream: + status: unverifiable + date: 2026-08-06 + against: null + note: > + Author-assembled research workbook with prose-only provenance. Not + distributed by the journal, no internal source list, no builder. Values + can be spot-checked against the figures in the published article but not + re-derived. + +# --------------------------------------------------------------------------- +# Shape +# --------------------------------------------------------------------------- +# READ POSITIONALLY -- see the warning in assignat.xlsx.yml, which applies +# identically here. Every read is header=None with explicit usecols/skiprows/ +# nrows, so a row or column inserted above or left of a range silently changes +# what the lecture plots. These ranges are the file's public interface. + +schema: + format: xlsx + sheet_count: 27 + positional_reads: true + sheets: + - name: Militspe + read_as: {usecols: 'M:X', skiprows: 7, nrows: 102, header: null} + shape: [102, 12] + dtypes: [float64] + known_nulls_total: 0 + description: British military spending, 1689-1790; column index 4 is plotted + - name: Militspe + read_as: {usecols: 'D', skiprows: 3, nrows: 105, header: null} + shape: [105, 1] + dtypes: [float64] + known_nulls_total: 32 + description: French military spending 1685-1789, in 1726 livres + - name: Debt + read_as: {usecols: 'R:S', skiprows: 5, nrows: 99, header: null} + shape: [99, 2] + dtypes: [float64, float64] + known_nulls_total: 78 + description: French debt service and revenue series + - name: Debt + read_as: {usecols: 'P', skiprows: 89, nrows: 15, header: null} + shape: [15, 1] + dtypes: [float64] + known_nulls_total: 0 + description: terminal-period continuation of the block above + - name: Debt + read_as: {usecols: 'K', skiprows: 41, nrows: 120, header: null} + shape: [120, 1] + dtypes: [float64] + known_nulls_total: 8 + description: debt stock series + # 25 further sheets ship with the workbook and are not read by any lecture: + # Amortiss, Britain, Cambon, Divers, Experim, Indes, Interest, Irr-matr, + # Macro, Mortalit, P-Etat, Path2, Policy, Popul, Rate-mon, Rentier1, + # Return, Revenues, Studies, Studies2, Units, Viager, White, Viagere, + # Yearly. Kept as the authors assembled them. + date_range: {start: 1685, end: 1800} + row_count_floor: null # per-range, declared above + known_nulls: {} + note: > + High null counts in the Debt ranges are structural -- the blocks are + sparse layouts in a working spreadsheet, and the lecture indexes specific + columns out of them rather than consuming the block whole. + +consumers: [] # empty until the Track A repoint lands + +builder: null +builder_status: not-applicable # verbatim republication -- nothing to build diff --git a/lectures/fig_3.xlsx.yml b/lectures/fig_3.xlsx.yml new file mode 100644 index 0000000..87702f7 --- /dev/null +++ b/lectures/fig_3.xlsx.yml @@ -0,0 +1,89 @@ +# Manifest for fig_3.xlsx — one of three research spreadsheets assembled by +# Sargent and Velde for "Macroeconomic Features of the French Revolution". +# +# Note for anyone reading git history: a source-format twin, fig_3.ods, was +# dropped in #10 after being verified a pure format duplicate (identical single +# sheet, identical 34x6 frame, DataFrame.equals true). It remains recoverable +# from history. The full-filename sidecar convention exists because of exactly +# this pair -- see manifest-schema.yml. + +filename: fig_3.xlsx +title: French Revolution — figure 3 series (Sargent-Velde) +description: > + Small working sheet holding the series behind one figure of the French + Revolution analysis. Four numeric columns over thirty rows, read as a block. + +class: verbatim + +source: + name: Sargent and Velde, "Macroeconomic Features of the French Revolution" — supporting spreadsheets + url: https://www.journals.uchicago.edu/doi/10.1086/261992 + doi: 10.1086/261992 + citation: > + Sargent, Thomas J. and François R. Velde (1995). Macroeconomic Features of + the French Revolution. Journal of Political Economy, 103(3), 474-518. + note: > + One of the three spreadsheets the consuming lecture attributes to the + authors of the 1995 article. Working files, not journal-distributed. + +license: + name: null + url: null + redistribution: permitted + verified: 2026-08-06 + # No formal licence stated; the authors' own research data, served publicly + # from lecture-python-intro for years. Republished with attribution under the + # record-and-track policy (#35). + +retrieved: null # unrecorded -- inherited bytes +maintainer: QuantEcon + +# --------------------------------------------------------------------------- +# Integrity (PLAN Phase 7) +# --------------------------------------------------------------------------- +# Migration check (repoint gate): verified 2026-08-06 by git blob hash against +# lecture-python-intro's lectures/datasets/fig_3.xlsx -- identical. + +integrity: + sha256: bf66aa332439c8a5146d96a71dc4b4b0201bb2ed381480c79e8ed7ce055e3ecf + upstream: + status: unverifiable + date: 2026-08-06 + against: null + note: > + Author-assembled research sheet with prose-only provenance and no + internal labelling -- the columns are unheaded, so even their meaning + comes from how the lecture plots them rather than from the file. Values + can be spot-checked against the published figure but not re-derived. + +# --------------------------------------------------------------------------- +# Shape +# --------------------------------------------------------------------------- +# READ POSITIONALLY -- see the warning in assignat.xlsx.yml. The single sheet +# is 35x6; the lecture takes a 30x4 block out of the middle of it, so the five +# rows and two columns outside that block are load-bearing padding: deleting +# them shifts the read. + +schema: + format: xlsx + sheet_count: 1 + positional_reads: true + sheets: + - name: Sheet1 + full_shape: [35, 6] + read_as: {usecols: 'C:F', skiprows: 5, nrows: 30, header: null} + shape: [30, 4] + dtypes: [float64, float64, float64, float64] + known_nulls_per_column: [8, 8, 4, 9] + description: > + Four unheaded series read as a block. Nulls are ragged series ends + within the block, not damage -- the lecture plots each column over its + own valid span. + date_range: {start: 1789, end: 1796} + row_count_floor: 30 # exact by design -- a frozen 30-row block + known_nulls: {} # declared per column above + +consumers: [] # empty until the Track A repoint lands + +builder: null +builder_status: not-applicable # verbatim republication -- nothing to build diff --git a/lectures/longprices.xls.yml b/lectures/longprices.xls.yml new file mode 100644 index 0000000..9c81a1e --- /dev/null +++ b/lectures/longprices.xls.yml @@ -0,0 +1,94 @@ +# Manifest for longprices.xls — four centuries of price levels, assembled by +# Sargent and Velde for "The Big Problem of Small Change". + +filename: longprices.xls +title: Price levels in four hard-currency countries, 1600-2000 +description: > + Annual price-level index for the United Kingdom, United States, France and + Castile (Spain). The consuming lecture plots 1600-1914, the period in which + all four were on commodity-money standards, to contrast with twentieth + century paper-money inflation. + +# The authors' own assembly of historical price series, republished here as +# they distributed it. We apply no processing, so this is not `constructed` -- +# but it also cannot be re-fetched from a publisher, which is what makes the +# integrity status below `unverifiable` rather than a clean re-download. +class: verbatim + +source: + name: Sargent and Velde, "The Big Problem of Small Change" — supporting data + url: https://press.princeton.edu/books/hardcover/9780691116358/the-big-problem-of-small-change + citation: > + Sargent, Thomas J. and François R. Velde (2002). The Big Problem of Small + Change. Princeton, New Jersey: Princeton University Press. + note: > + The consuming lecture states these data "originally appeared on page 35" of + the book. The spreadsheet is the authors' working assembly and is not + published at a citable location of its own; it reached QuantEcon directly. + +license: + name: null + url: null + redistribution: permitted + verified: 2026-08-06 + # No formal licence is stated. These bytes have been served publicly from + # lecture-python-intro for years and are republished here with attribution + # under the record-and-track policy (#35): a migration is not held up for a + # licence question on data we already serve. Logged there for review before + # data.quantecon.org is promoted as a public open-data host. + +retrieved: null # unrecorded -- inherited bytes +maintainer: QuantEcon + +# --------------------------------------------------------------------------- +# Integrity (PLAN Phase 7) +# --------------------------------------------------------------------------- +# Migration check (repoint gate, not a manifest field): verified 2026-08-06 by +# git blob hash against lecture-python-intro's lectures/datasets/longprices.xls +# -- identical. + +integrity: + sha256: 1160f9d3ac42cc225731a5446912acbb948e07753981e0cdc1b1da5dc2b344f7 + upstream: + status: unverifiable + date: 2026-08-06 + against: null + note: > + Author-assembled with prose-only provenance: no publisher distribution + point, no per-series source list inside the workbook, and no builder. + The figures can be spot-checked against page 35 of the cited book but + not re-derived. Trust rests on the authors' provenance. + +# --------------------------------------------------------------------------- +# Shape +# --------------------------------------------------------------------------- + +schema: + format: xls + sheets: + - name: all + read_as: {header: 2, index_col: 0, then: "iloc[1:]"} + shape: [401, 4] + columns: + - {name: UK, dtype: float64, description: United Kingdom price index} + - {name: US, dtype: float64, description: United States price index} + - {name: France, dtype: float64, description: France price index} + - {name: Castile, dtype: float64, description: Castile (Spain) price index} + known_nulls: {UK: 0, US: 152, France: 126, Castile: 0} + date_range: {start: 1600, end: 2000} + note: > + Nulls are series that do not span the full 1600-2000 index -- the US + series begins in the eighteenth century and France has gaps. Expected, + not damage. + # Nine further per-country sheets ship with the workbook and are not read: + # England 1264-1800, UK 1661-1913, UK 1947-2000, UK 1750-1998, + # US 1750-1932, US 1860-1939, US 1913-2000, Castile, France 1726-2000. + # They are the sources the `all` sheet is compiled from -- useful provenance, + # so they are kept rather than stripped. + row_count_floor: 401 # frozen historical extract: exact by design + known_nulls: {} # declared per sheet above + +consumers: [] # empty until the Track A repoint lands + +builder: null +builder_status: not-applicable # verbatim republication -- nothing to build diff --git a/lectures/mpd2020.xlsx.yml b/lectures/mpd2020.xlsx.yml new file mode 100644 index 0000000..e3ae520 --- /dev/null +++ b/lectures/mpd2020.xlsx.yml @@ -0,0 +1,123 @@ +# Manifest for mpd2020.xlsx — Maddison Project Database 2020. +# +# Landed in the Feb 2025 migration (data#5-#7) and byte-identical to the copy +# lecture-python-intro consumes today; the repoint is Track A, PR set 1. + +filename: mpd2020.xlsx +title: Maddison Project Database 2020 — GDP per capita and population, 1 CE to 2018 +description: > + Historical estimates of real GDP per capita (2011 US$) and mid-year + population for 169 countries, some series reaching back to the first + century. The reference dataset for long-run comparative growth. + +# NOT verbatim, despite being an upstream release. Three header labels on the +# `Regional data` sheet were edited locally -- see integrity.upstream below. +# A file we have modified is not "republished as distributed", so it is +# classified `constructed` rather than misreporting it as untouched. +class: constructed + +source: + name: Maddison Project Database, version 2020 + url: https://www.rug.nl/ggdc/historicaldevelopment/maddison/releases/maddison-project-database-2020 + data_url: https://www.rug.nl/ggdc/historicaldevelopment/maddison/data/mpd2020.xlsx + version: "2020" + citation: > + Bolt, Jutta and Jan Luiten van Zanden (2020). Maddison style estimates of + the evolution of the world economy. A new 2020 update. Maddison Project + Database, version 2020. + # The workbook's own `Notes` sheet states a citation policy beyond the + # licence: the ORIGINAL papers (listed on the `Sources` sheet) must be cited + # when the data is shown in any graphical form, or when a subset of fewer + # than twelve countries is used. The consuming lecture does both, so this is + # a live obligation rather than a formality. + citation_policy: > + Original source papers must be cited when data is shown graphically or + when fewer than 12 countries are used; see the `Sources` sheet. + +license: + name: CC BY 4.0 + url: https://creativecommons.org/licenses/by/4.0/ + redistribution: permitted + verified: 2026-08-06 + # Established against the release page itself, which states the database + # "is licensed under a Creative Commons Attribution 4.0 International + # License". Attribution is the sole condition; the citation above and the + # citation_policy note carry it. + +retrieved: null # unrecorded -- inherited bytes, see AGENTS.md +maintainer: QuantEcon + +# --------------------------------------------------------------------------- +# Integrity (PLAN Phase 7) +# --------------------------------------------------------------------------- +# Migration check (a repoint gate, not a manifest field): verified 2026-08-06 +# by git blob hash against lecture-python-intro's copy at +# lectures/datasets/mpd2020.xlsx -- identical, so the repoint cannot change +# lecture output. + +integrity: + sha256: f67af0fd599fcebd9b92a0ad65472998f4548ea006e6b7c0018d4ade76e3bceb + upstream: + status: verified + date: 2026-08-06 + against: https://www.rug.nl/ggdc/historicaldevelopment/maddison/data/mpd2020.xlsx + note: > + Re-fetched from the Maddison release and compared cell by cell across all + six sheets. Every data value matches, including all 21,683 rows of `Full + data`. THREE header labels on `Regional data` differ and are local edits: + row0 col1 "GDP pc 2011 prices" -> "gdppc_2011", row0 col9 "Population" -> + "pop", and row0 col18 "gdppc_2011" added where upstream is empty. The + lecture reads that sheet with header=(0,1,2), so these renames are + load-bearing -- replacing this file with a pristine upstream copy would + silently break long_run_growth. Byte hashes differ (1,765,204 vs + 1,764,793) consistent with a re-save plus these edits. + +# --------------------------------------------------------------------------- +# Shape +# --------------------------------------------------------------------------- +# Multi-sheet workbook: the contract is the sheets the lectures actually read, +# not a single flat table. Sheets not listed here are shipped as part of the +# upstream release and carry provenance (`Notes`, `Sources`) but are not read. + +schema: + format: xlsx + sheets: + - name: Full data + read_as: {header: 0} + shape: [21682, 5] + columns: + - {name: countrycode, dtype: str, description: ISO3 country code} + - {name: country, dtype: str, description: country name} + - {name: year, dtype: int64, description: calendar year} + - {name: gdppc, dtype: float64, description: real GDP per capita, 2011 US$} + - {name: pop, dtype: float64, description: mid-year population, thousands} + known_nulls: {gdppc: 1976, pop: 4483} + date_range: {start: 1, end: 2018} + - name: Regional data + read_as: {header: [0, 1, 2], index_col: 0} + shape: [23, 18] + date_range: {start: 1820, end: 2018} + note: column labels are locally edited -- see integrity.upstream + # Not read by any lecture, but shipped with the release and kept: + # Notes, Sources (provenance), GDP pc, Population (wide reshapes of Full data) + row_count_floor: 21682 # frozen release: exact by design + known_nulls: {} # declared per sheet above + +# --------------------------------------------------------------------------- +# Consumers +# --------------------------------------------------------------------------- +# Empty until the Track A repoint lands. lecture-python-intro and lecture-wasm +# both read this file today, but from intro's own copy, not from here. + +consumers: [] + +# --------------------------------------------------------------------------- +# Builder +# --------------------------------------------------------------------------- +# The transformation is exactly three header renames, enumerated in +# integrity.upstream. No script was committed for them, so the status is +# `unrecovered` -- but unusually for that state, recovery here is cheap and +# fully specified rather than lost. + +builder: null +builder_status: unrecovered diff --git a/lectures/nom_balances.npy.yml b/lectures/nom_balances.npy.yml new file mode 100644 index 0000000..e42158b --- /dev/null +++ b/lectures/nom_balances.npy.yml @@ -0,0 +1,97 @@ +# Manifest for nom_balances.npy — nominal assignat balances, French Revolution. +# +# Twin of caron.npy, and the second of the two files in this migration with NO +# recorded provenance. The consuming lecture's "Data Sources" section names +# three spreadsheets and does not mention either .npy file. + +filename: nom_balances.npy +title: French Revolution — monthly nominal assignat balances, 1789-1796 +description: > + 81 monthly observations of assignats in circulation, rising from 90 in late + 1789 to a peak of 37,540.9 and standing at 33,555.59 in mid-1796. The + consuming lecture multiplies a 63-row slice of this series by the assignat's + specie value to obtain real balances. + +class: constructed + +source: + name: unrecorded + url: null + citation: > + Consumed by analysis reproducing Sargent, Thomas J. and François R. Velde + (1995). Macroeconomic Features of the French Revolution. Journal of + Political Economy, 103(3), 474-518. + note: > + No source recorded anywhere. Unlike its twin, the filename offers no lead + beyond the quantity it holds. The natural candidate is the assignat + circulation series in the authors' own workbooks -- see integrity.upstream + for why that was tested and rejected. + +license: + name: null + url: null + redistribution: permitted + verified: 2026-08-06 + # Source unestablished, so no licence can be established either. Served + # publicly from lecture-python-intro for years; migrated under the + # record-and-track policy (#35) and logged there. Needs its source settled + # before its licence can be. + +retrieved: null # unrecorded -- inherited bytes +maintainer: QuantEcon + +# --------------------------------------------------------------------------- +# Integrity (PLAN Phase 7) +# --------------------------------------------------------------------------- +# Migration check (repoint gate): verified 2026-08-06 by git blob hash against +# lecture-python-intro's lectures/datasets/nom_balances.npy -- identical. +# +# The second of the two LOCAL-PATH reads in lecture-python-intro +# (`np.load('datasets/nom_balances.npy')`) that break downloaded and Colab +# notebooks. Repointing it is the point of the migration. + +integrity: + sha256: 8790a75e023f552dd9b444ffe4b490a0abbccf4ffc745bc34c76facf847b972a + upstream: + status: unverifiable + date: 2026-08-06 + against: null + note: > + No source, no builder, no internal metadata. Actively checked 2026-08-06 + against the committed Sargent-Velde workbooks: the terminal value + 33,555.59 does occur in several assignat.xlsx sheets (Fig6, Data, Data2, + Ramel, Ramel2, seignor), but no column in any of them contains this + 81-value series contiguously, and the series maximum 37,540.9 appears + nowhere. So the array overlaps the workbooks without being extractable + from them -- a partial lead, not a recovery. + +# --------------------------------------------------------------------------- +# Shape +# --------------------------------------------------------------------------- + +schema: + format: npy + dtype: float64 + shape: [81, 2] + columns: + - {index: 0, name: date, dtype: float64, description: decimal year, monthly steps of 1/12} + - {index: 1, name: nominal_balances, dtype: float64, description: assignats in circulation, millions of livres} + row_count_floor: 81 # frozen historical extract: exact by design + date_range: {start: 1789.833, end: 1796.5} + known_nulls: {} # no NaNs present + note: > + Row alignment is load-bearing across files: the lecture takes rows 14:77 + of this array to pair with all 63 rows of caron.npy. The leading 14 rows + (late 1789 through 1790) exist to extend the circulation series before the + price series begins, and must not be trimmed. + +consumers: [] # empty until the Track A repoint lands + +# --------------------------------------------------------------------------- +# Builder +# --------------------------------------------------------------------------- +# Inherited constructed-without-builder, tracked for recovery in PLAN Phase 9. +# Recovery starts with identifying the source, as for caron.npy. + +builder: null +builder_status: unrecovered diff --git a/migration.yml b/migration.yml index a8e45f5..885704a 100644 --- a/migration.yml +++ b/migration.yml @@ -172,6 +172,97 @@ datasets: date: 2026-08-04 cutover: null + # --------------------------------------------------------------------------- + # Track A, the intro static batch -- landed in the Feb 2025 migration + # (data#5-#7) but never repointed, which is the gap #4 recorded. Manifests + # written 2026-08-06; each flips to `repointed` when its intro + wasm PR set + # merges. Every one of these has TWO consuming repos, so all consumers move + # together (PLAN, "Repoint rules"). + # --------------------------------------------------------------------------- + + mpd2020.xlsx: + pilot: A1 + status: landed + prior_pattern: own-repo + landed: + pr: QuantEcon/data-lectures#5 + date: 2025-02-14 + repoints: [] + cutover: null + + longprices.xls: + pilot: A2 + status: landed + prior_pattern: own-repo + landed: + pr: QuantEcon/data-lectures#5 + date: 2025-02-14 + repoints: [] + cutover: null + + chapter_3.xlsx: + pilot: A2 + status: landed + prior_pattern: own-repo + landed: + pr: QuantEcon/data-lectures#5 + date: 2025-02-14 + repoints: [] + cutover: null + + assignat.xlsx: + pilot: A3 + status: landed + prior_pattern: own-repo + landed: + pr: QuantEcon/data-lectures#5 + date: 2025-02-14 + repoints: [] + cutover: null + + dette.xlsx: + pilot: A3 + status: landed + prior_pattern: own-repo + landed: + pr: QuantEcon/data-lectures#5 + date: 2025-02-14 + repoints: [] + cutover: null + + fig_3.xlsx: + pilot: A3 + status: landed + prior_pattern: own-repo + landed: + pr: QuantEcon/data-lectures#5 + date: 2025-02-14 + repoints: [] + cutover: null + + caron.npy: + pilot: A3 + status: landed + # Local-path read in intro (np.load('datasets/caron.npy')) -- one of the + # two Colab-breaking reads this batch retires, the same failure mode P1 + # fixed for msy_fishery. + prior_pattern: local-path + landed: + pr: QuantEcon/data-lectures#5 + date: 2025-02-14 + repoints: [] + cutover: null + + nom_balances.npy: + pilot: A3 + status: landed + prior_pattern: local-path + landed: + pr: QuantEcon/data-lectures#5 + date: 2025-02-14 + repoints: [] + cutover: null + # Planned waves that have not landed anything here yet. `datasets` names the # files as the audit sees them today, so the dashboard can join the two views. # `title` is the reader-facing milestone name (the dashboard is read by people diff --git a/scripts/build_catalog.py b/scripts/build_catalog.py index 468b5a2..2601144 100644 --- a/scripts/build_catalog.py +++ b/scripts/build_catalog.py @@ -124,15 +124,24 @@ def build(manifests) -> str: lines.append("# Dataset catalog — `QuantEcon/data-lectures`") lines.append("") lines.append( - "The migrated-dataset registry, **auto-generated** from the sidecar " - "manifests (`lectures/*.yml`). Do not edit by hand — run " + "The dataset registry, **auto-generated** from the sidecar manifests " + "(`lectures/*.yml`). Do not edit by hand — run " "`python scripts/build_catalog.py`. A dataset appears here once it has a " - "manifest; files not yet migrated are tracked in " - "[PLAN.md](PLAN.md) Phase 9." + "manifest, which may be before its consuming lectures are repointed — " + "an empty **Used by** column means the file is here and documented but " + "no lecture reads it from this repo yet. Files still to migrate are " + "tracked in [PLAN.md](PLAN.md)." ) lines.append("") + # `consumers` is the honest test of "in use": a manifest can land ahead of + # its repoint, so counting manifests would overstate what lectures read. + in_use = sum(1 for m in manifests if m.get("consumers")) + awaiting = len(manifests) - in_use + headline = f"**{len(manifests)} datasets** · {in_use} read by lectures today" + if awaiting: + headline += f", {awaiting} awaiting repoint" lines.append( - f"**{len(manifests)} datasets migrated** · {human_size(total)} total · " + f"{headline} · {human_size(total)} total · " f"{permitted} permitted / {restricted} restricted redistribution" ) lines.append("") From 54fa3ebb2c6b237971e69cab84ae3ea7a150f829 Mon Sep 17 00:00:00 2001 From: Matt McKay Date: Thu, 6 Aug 2026 14:30:28 +1000 Subject: [PATCH 2/5] mpd2020: prove the header edits are ours, and record the provenance chain The manifest asserted the three Regional-data header differences were local edits. That was an inference, and it had an untested alternative: Maddison could have renamed its own headers after we took the file, which would make our copy a faithful older vintage rather than a modified one -- and would flip the class back to verbatim. Ruled out. The Internet Archive holds the upstream file with one unchanging content digest (4OWZWHTE5HXBBF4XLCCQTGDXY4HXOGOK) across every snapshot from 2021-01-10 to 2026-01-02, and the copy fetched on 2026-08-06 matches that digest exactly. Upstream was byte-stable two years before these bytes were committed to lecture-python-intro on 2023-03-23, so the edits are ours. Also records the chain these bytes actually travelled, which the manifest did not make explicit: they did not come here from Maddison. They were downloaded, edited, and committed to lecture-python-intro in #120 (2023-03-23, blob unchanged since), then copied into this repo by #5 in Feb 2025. The immediate source is the lecture series; Maddison is the ultimate source. One useful corollary, now recorded: because upstream never moved, the data held here corresponds to the CURRENT Maddison release. There is no vintage ambiguity and the three header labels are the only delta. Co-Authored-By: Claude Opus 5 (1M context) --- lectures/mpd2020.xlsx.yml | 36 +++++++++++++++++++++++++++++------- 1 file changed, 29 insertions(+), 7 deletions(-) diff --git a/lectures/mpd2020.xlsx.yml b/lectures/mpd2020.xlsx.yml index e3ae520..a765717 100644 --- a/lectures/mpd2020.xlsx.yml +++ b/lectures/mpd2020.xlsx.yml @@ -33,6 +33,13 @@ source: citation_policy: > Original source papers must be cited when data is shown graphically or when fewer than 12 countries are used; see the `Sources` sheet. + # Provenance chain of THESE bytes: they did not come here from Maddison + # directly. The file was downloaded, locally edited (see integrity.upstream) + # and committed to lecture-python-intro on 2023-03-23 in + # QuantEcon/lecture-python-intro#120, where the blob has never changed. It + # was copied into this repo by #5 in Feb 2025 and is byte-identical to + # intro's copy. So the immediate source is the lecture series; Maddison is + # the ultimate source. license: name: CC BY 4.0 @@ -64,13 +71,28 @@ integrity: note: > Re-fetched from the Maddison release and compared cell by cell across all six sheets. Every data value matches, including all 21,683 rows of `Full - data`. THREE header labels on `Regional data` differ and are local edits: - row0 col1 "GDP pc 2011 prices" -> "gdppc_2011", row0 col9 "Population" -> - "pop", and row0 col18 "gdppc_2011" added where upstream is empty. The - lecture reads that sheet with header=(0,1,2), so these renames are - load-bearing -- replacing this file with a pristine upstream copy would - silently break long_run_growth. Byte hashes differ (1,765,204 vs - 1,764,793) consistent with a re-save plus these edits. + data`. THREE header labels on `Regional data` differ: row0 col1 "GDP pc + 2011 prices" -> "gdppc_2011", row0 col9 "Population" -> "pop", and row0 + col18 "gdppc_2011" added where upstream is empty. The lecture reads that + sheet with header=(0,1,2), so these renames are load-bearing -- replacing + this file with a pristine upstream copy would silently break + long_run_growth. Byte sizes differ (1,765,204 vs 1,764,793), consistent + with a re-save plus these edits. + edits_are_local: true + edits_evidence: > + Established, not assumed. The alternative -- that Maddison renamed its + own headers after we took the file, which would make our copy a faithful + older vintage -- is ruled out: the Internet Archive holds the upstream + file with one unchanging content digest + (4OWZWHTE5HXBBF4XLCCQTGDXY4HXOGOK) across every snapshot from 2021-01-10 + to 2026-01-02, and the copy fetched on 2026-08-06 matches that digest + exactly. Upstream was already byte-stable two years before this file was + committed to lecture-python-intro on 2023-03-23. The three edits are + therefore ours. + upstream_vintage_note: > + Because upstream has been byte-stable throughout, the data held here + corresponds to the CURRENT Maddison release -- there is no vintage + ambiguity, and the three header labels are the only delta. # --------------------------------------------------------------------------- # Shape From 3c8513b3da6f05cb2d1692f177d69faaadebcdc2 Mon Sep 17 00:00:00 2001 From: Matt McKay Date: Thu, 6 Aug 2026 14:48:28 +1000 Subject: [PATCH 3/5] Record the migration-scope rule: move bytes, don't update them The mpd2020 delta raised a question the plan never answered: what happens when a migration discovers that the file the lectures use is not what upstream publishes today? Answering it per-file, mid-repoint, would turn every migration into a content review. The rule is that a migration moves the copy the lectures ALREADY consume, validated byte-identical, and nothing else. That property is what makes a repoint safe to merge -- it provably cannot change a figure. Adopting a newer upstream vintage is a different change: it does alter lecture output, needs figures re-reviewed, and is an author's decision rather than an infrastructure one. So a discovered delta is recorded, not resolved -- in the dataset's manifest so it shows in the catalog immediately, and in the register (#39) where it gets reasoned about once the migration completes. Two kinds of delta look alike and need opposite responses, so both are named: upstream moved (adopt via a NEW filename, so consumers opt in) versus our copy diverges (reconcile the edit). mpd2020 is the first recorded instance of the second kind, and shows why it cannot be a mechanical fix -- its local edits are load-bearing for long_run_growth, so file and lecture must move together. Detecting these automatically rather than by accident is proposed in #40. Co-Authored-By: Claude Opus 5 (1M context) --- AGENTS.md | 2 ++ PLAN.md | 18 +++++++++++++++++- 2 files changed, 19 insertions(+), 1 deletion(-) diff --git a/AGENTS.md b/AGENTS.md index 2dc751f..ed172f0 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -43,6 +43,8 @@ Both are cheap to follow and expensive to discover. `PLAN.md` carries the reason - **Never delete a file a sibling repo reads.** `lecture-wasm` fetches `lecture-python-intro`'s *committed blobs* by URL, so deleting intro's copy in a repoint PR 404s the wasm build immediately. "Delete the lecture repo's own copy in the same repoint PR" applies only where no sibling reads it; where one does, the sibling's repoint lands first or in the same set. - **Repoint every consumer of a dataset together.** The strict audit has no green state for a partially-repointed dataset — `pending`/`landed` fails once any consumer reads data-lectures, and `repointed`/`final` fails while any consumer still does not. Land the lecture repoints first, then flip `migration.yml`; that flip is the push that re-runs the audit, so reality and the tracker agree by the time it runs. +- **A migration moves bytes; it does not update them.** Land the copy the lectures already consume, validated byte-identical — that is what makes a repoint provably unable to change a figure. If the committed file differs from what upstream publishes today, migrate it unchanged anyway, record the delta in `integrity.upstream` **and** in the register at [#39](https://github.com/QuantEcon/data-lectures/issues/39), and leave the decision for after the migration. Adopting a newer vintage changes lecture output and is an author's call, not an infrastructure one — and per "Corrections vs vintages" below it gets a **new filename**, never a silent replacement. + Cross-repo repoints are worked from [`QuantEcon/workspace-lectures`](https://github.com/QuantEcon/workspace-lectures) — same branch name in each repo, one PR per repo, no aggregate PR. ### Corrections vs vintages diff --git a/PLAN.md b/PLAN.md index a9b6cbb..33516a2 100644 --- a/PLAN.md +++ b/PLAN.md @@ -42,7 +42,7 @@ This repository is being shaped into the **single canonical repository for data ## Repoint rules -Two rules learned the hard way. Both are about *ordering*, both are cheap to follow and expensive to discover, and neither is enforced by CI — the strict audit catches the second only after the fact. +Three rules learned the hard way. The first two are about *ordering*, both cheap to follow and expensive to discover, and neither is enforced by CI — the strict audit catches the second only after the fact. The third is about *scope*. ### 1. Repoint a sibling reader before deleting the file it reads @@ -60,6 +60,22 @@ The strict audit has **no green state for a partially-repointed dataset**. `scri Practically: one branch name across data-lectures + every consuming repo, PRs opened together, lecture repoints merged first, then the `migration.yml` flip to `repointed` — that last push is what re-runs the audit, and by then reality and the tracker agree. +### 3. A migration moves bytes; it does not update them + +The copy that lands here is the copy the lectures **already consume**, validated byte-identical in the repoint PR. That is what makes a repoint safe to merge: it provably cannot change a single figure. + +Adopting a newer upstream vintage is a *different change* with a different risk profile — it does change lecture output, it needs figures re-reviewed, and it is an author-facing decision rather than an infrastructure one. Conflating the two turns every repoint into a content review and stalls the programme. + +So when a migration finds that the committed file differs from what upstream publishes today: + +1. **Migrate what the lectures use**, unchanged, with the byte-compare gate as normal. +2. **Record the delta** in the dataset's manifest (`integrity.upstream`) *and* in the register at [#39](https://github.com/QuantEcon/data-lectures/issues/39) — the manifest makes it visible in the catalog from day one, the register is where it gets reasoned about. +3. **Review the register once the migration completes**, and decide each case on its merits. + +Two deltas look alike and need opposite responses. *Upstream moved* — a newer vintage exists; adopting it means a **new filename**, per "Corrections vs vintages" in `AGENTS.md`, so consumers opt in. *Our copy diverges* — upstream is unchanged but our file was modified; resolving means reconciling the edit. `mpd2020.xlsx` is the first recorded instance of the second kind, and it is instructive: the local edits are load-bearing for the consuming lecture, so the file and the lecture have to move together. + +Detecting these automatically rather than by accident is proposed in [#40](https://github.com/QuantEcon/data-lectures/issues/40). + ## Migration tracks The remaining work decomposes by **consuming series** rather than by hosting pattern, because — apart from the `intro`/`wasm` pairing — each series now owns its own data. This is the execution view; the phases below remain the machinery each track passes through. From 16ea8da9eba9f6da7f93a0220fdfd0271d8cb24d Mon Sep 17 00:00:00 2001 From: Matt McKay Date: Thu, 6 Aug 2026 15:06:20 +1000 Subject: [PATCH 4/5] Address Copilot review: row-count consistency, a truncated column list, and a silent YAML truncation bug it did not catch Two of Copilot's three comments were correct and are fixed. The mpd2020 manifest cited 21,683 rows in its integrity note while its schema said 21,682. Both numbers were right and measured differently -- raw sheet including the header row, versus data rows as the lecture reads it -- but a manifest that asserts both without explaining is useless as a contract. Now 21,682 throughout, matching schema. chapter_3's Table3.1 sample declared shape [6, 6] and listed five columns. That was my error: the inspection script printed list(columns)[:5] and I transcribed the truncated output. The sixth column is "Percentage of expenditures covered by new issues of paper money". Added, with a note that these lists are complete rather than illustrative, since a sample that does not match its own shape is worse than no sample. Copilot's third comment -- that `dtype: str` should be `string`/`object` for consistency -- is declined on the merits and answered in the thread. Under pandas 3, which is the anaconda=2026.07 pin the lectures run on, str(df[col].dtype) for a text column IS `str`; a validator comparing against `string` or `object` is what would break. The repo-wide inconsistency it correctly notices is real and is #14's to settle. Two bugs Copilot missed, found while checking its claims: `source.data_url` was a name I invented for a field the repo already has as `source.file_url`, used by six existing manifests. Renamed -- straight schema drift, caught before it set a precedent. More seriously, unquoted descriptions containing a comma inside a YAML flow mapping silently parse as a new key: {description: mid-year population, thousands} yields a `thousands` key and a description truncated at the comma. This affected 15 columns across 7 manifests, four of them predating this PR (epl_match_goals, japan_earthquakes, japan_population_by_age, employ, realwage). All quoted, and verified: 72 column entries now parse with no spurious keys and no dropped descriptions. Co-Authored-By: Claude Opus 5 (1M context) --- lectures/caron.npy.yml | 4 ++-- lectures/chapter_3.xlsx.yml | 7 ++++++- lectures/employ.csv.yml | 2 +- lectures/epl_match_goals.csv.yml | 2 +- lectures/japan_earthquakes.csv.yml | 8 ++++---- lectures/japan_population_by_age.csv.yml | 4 ++-- lectures/mpd2020.xlsx.yml | 8 ++++---- lectures/nom_balances.npy.yml | 4 ++-- lectures/realwage.csv.yml | 2 +- 9 files changed, 23 insertions(+), 18 deletions(-) diff --git a/lectures/caron.npy.yml b/lectures/caron.npy.yml index 9df23d4..6534684 100644 --- a/lectures/caron.npy.yml +++ b/lectures/caron.npy.yml @@ -80,8 +80,8 @@ schema: dtype: float64 shape: [63, 2] columns: - - {index: 0, name: date, dtype: float64, description: decimal year, monthly steps of 1/12} - - {index: 1, name: specie_value, dtype: float64, description: assignat value against specie, index units} + - {index: 0, name: date, dtype: float64, description: "decimal year, monthly steps of 1/12"} + - {index: 1, name: specie_value, dtype: float64, description: "assignat value against specie, index units"} row_count_floor: 63 # frozen historical extract: exact by design date_range: {start: 1791.0, end: 1796.25} known_nulls: {} # no NaNs present diff --git a/lectures/chapter_3.xlsx.yml b/lectures/chapter_3.xlsx.yml index 146d668..f7d0a0c 100644 --- a/lectures/chapter_3.xlsx.yml +++ b/lectures/chapter_3.xlsx.yml @@ -72,8 +72,13 @@ schema: sheets_pattern: 'Table3\.\d+' sheet_count: 27 read_as: {header: 1} + # Column lists are complete for the sheets shown, not truncated -- the count + # must equal the second element of `shape`, or the sample is not a contract. sheets_sample: - - {name: Table3.1, shape: [6, 6], columns: [Budget Beginning, Budget Ending, Receipts, Expenditures, Deficit]} + - name: Table3.1 + shape: [6, 6] + columns: [Budget Beginning, Budget Ending, Receipts, Expenditures, Deficit, + "Percentage of expenditures covered by new issues of paper money"] - {name: Table3.2, shape: [78, 3], columns: [Year, Month, Total Note Circulation]} - {name: Table3.3, shape: [44, 3], columns: [Year, Month, "Retail price index, 52 commodities"]} - {name: Table3.4, shape: [74, 3], columns: [Year, Month, Exchange Rate]} diff --git a/lectures/employ.csv.yml b/lectures/employ.csv.yml index 797d14d..26f1668 100644 --- a/lectures/employ.csv.yml +++ b/lectures/employ.csv.yml @@ -48,7 +48,7 @@ integrity: schema: format: csv columns: - - {name: "", dtype: int64, description: pandas row index (0-based), not used by the lecture} + - {name: "", dtype: int64, description: "pandas row index (0-based), not used by the lecture"} - {name: GEO, dtype: string, description: geography (country or EU aggregate)} - {name: AGE, dtype: string, description: age band (e.g. "From 15 to 24 years")} - {name: UNIT, dtype: string, description: unit of measure (e.g. "Thousand persons")} diff --git a/lectures/epl_match_goals.csv.yml b/lectures/epl_match_goals.csv.yml index f613e55..72c5860 100644 --- a/lectures/epl_match_goals.csv.yml +++ b/lectures/epl_match_goals.csv.yml @@ -50,7 +50,7 @@ schema: format: csv columns: - {name: season, dtype: object, description: "season label, e.g. 2023-24"} - - {name: date, dtype: object, description: match date, ISO 8601} + - {name: date, dtype: object, description: "match date, ISO 8601"} - {name: home_team, dtype: object, description: home team name} - {name: away_team, dtype: object, description: away team name} - {name: home_goals, dtype: int64, description: full-time goals scored by the home team} diff --git a/lectures/japan_earthquakes.csv.yml b/lectures/japan_earthquakes.csv.yml index 4fbe7d4..a6a86f0 100644 --- a/lectures/japan_earthquakes.csv.yml +++ b/lectures/japan_earthquakes.csv.yml @@ -59,10 +59,10 @@ integrity: schema: format: csv columns: - - {name: time, dtype: object, description: event origin time, ISO 8601 UTC} - - {name: magnitude, dtype: float64, description: reported magnitude (mixed scales, as ComCat supplies)} - - {name: latitude, dtype: float64, description: epicentre latitude, degrees north} - - {name: longitude, dtype: float64, description: epicentre longitude, degrees east} + - {name: time, dtype: object, description: "event origin time, ISO 8601 UTC"} + - {name: magnitude, dtype: float64, description: "reported magnitude (mixed scales, as ComCat supplies)"} + - {name: latitude, dtype: float64, description: "epicentre latitude, degrees north"} + - {name: longitude, dtype: float64, description: "epicentre longitude, degrees east"} - {name: depth_km, dtype: float64, description: hypocentre depth in kilometres} row_count_floor: 3557 date_range: {start: 2000, end: 2024} diff --git a/lectures/japan_population_by_age.csv.yml b/lectures/japan_population_by_age.csv.yml index 789514e..10f369b 100644 --- a/lectures/japan_population_by_age.csv.yml +++ b/lectures/japan_population_by_age.csv.yml @@ -71,8 +71,8 @@ schema: format: csv columns: - {name: age, dtype: int64, description: single year of age; 100 means "100 and over"} - - {name: total_population, dtype: int64, description: total population at this age, in thousands} - - {name: japanese_population, dtype: int64, description: Japanese-national population at this age, in thousands} + - {name: total_population, dtype: int64, description: "total population at this age, in thousands"} + - {name: japanese_population, dtype: int64, description: "Japanese-national population at this age, in thousands"} row_count_floor: 101 date_range: {start: 2024, end: 2024} known_nulls: {} diff --git a/lectures/mpd2020.xlsx.yml b/lectures/mpd2020.xlsx.yml index a765717..5bbb472 100644 --- a/lectures/mpd2020.xlsx.yml +++ b/lectures/mpd2020.xlsx.yml @@ -19,7 +19,7 @@ class: constructed source: name: Maddison Project Database, version 2020 url: https://www.rug.nl/ggdc/historicaldevelopment/maddison/releases/maddison-project-database-2020 - data_url: https://www.rug.nl/ggdc/historicaldevelopment/maddison/data/mpd2020.xlsx + file_url: https://www.rug.nl/ggdc/historicaldevelopment/maddison/data/mpd2020.xlsx version: "2020" citation: > Bolt, Jutta and Jan Luiten van Zanden (2020). Maddison style estimates of @@ -70,7 +70,7 @@ integrity: against: https://www.rug.nl/ggdc/historicaldevelopment/maddison/data/mpd2020.xlsx note: > Re-fetched from the Maddison release and compared cell by cell across all - six sheets. Every data value matches, including all 21,683 rows of `Full + six sheets. Every data value matches, including all 21,682 data rows of `Full data`. THREE header labels on `Regional data` differ: row0 col1 "GDP pc 2011 prices" -> "gdppc_2011", row0 col9 "Population" -> "pop", and row0 col18 "gdppc_2011" added where upstream is empty. The lecture reads that @@ -111,8 +111,8 @@ schema: - {name: countrycode, dtype: str, description: ISO3 country code} - {name: country, dtype: str, description: country name} - {name: year, dtype: int64, description: calendar year} - - {name: gdppc, dtype: float64, description: real GDP per capita, 2011 US$} - - {name: pop, dtype: float64, description: mid-year population, thousands} + - {name: gdppc, dtype: float64, description: "real GDP per capita, 2011 US$"} + - {name: pop, dtype: float64, description: "mid-year population, thousands"} known_nulls: {gdppc: 1976, pop: 4483} date_range: {start: 1, end: 2018} - name: Regional data diff --git a/lectures/nom_balances.npy.yml b/lectures/nom_balances.npy.yml index e42158b..8d2ad73 100644 --- a/lectures/nom_balances.npy.yml +++ b/lectures/nom_balances.npy.yml @@ -74,8 +74,8 @@ schema: dtype: float64 shape: [81, 2] columns: - - {index: 0, name: date, dtype: float64, description: decimal year, monthly steps of 1/12} - - {index: 1, name: nominal_balances, dtype: float64, description: assignats in circulation, millions of livres} + - {index: 0, name: date, dtype: float64, description: "decimal year, monthly steps of 1/12"} + - {index: 1, name: nominal_balances, dtype: float64, description: "assignats in circulation, millions of livres"} row_count_floor: 81 # frozen historical extract: exact by design date_range: {start: 1789.833, end: 1796.5} known_nulls: {} # no NaNs present diff --git a/lectures/realwage.csv.yml b/lectures/realwage.csv.yml index 6f358f7..3b1379a 100644 --- a/lectures/realwage.csv.yml +++ b/lectures/realwage.csv.yml @@ -51,7 +51,7 @@ integrity: schema: format: csv columns: - - {name: "", dtype: int64, description: pandas row index (0-based), not used by the lecture} + - {name: "", dtype: int64, description: "pandas row index (0-based), not used by the lecture"} - {name: Time, dtype: string, description: observation year as ISO date (YYYY-01-01)} - {name: Country, dtype: string, description: OECD country name} - {name: Series, dtype: string, description: price basis (e.g. "In 2015 constant prices at 2015 USD PPPs")} From 048488b8f3bf7007f44abc37936a6c8d3e38c8ca Mon Sep 17 00:00:00 2001 From: Matt McKay Date: Thu, 6 Aug 2026 15:19:02 +1000 Subject: [PATCH 5/5] =?UTF-8?q?Add=20a=20`diverged`=20upstream=20status=20?= =?UTF-8?q?=E2=80=94=20"checked,=20differs,=20known"=20had=20no=20honest?= =?UTF-8?q?=20home?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit mpd2020 recorded `integrity.upstream.status: verified` while differing from upstream in three header labels. That is a claim a reader would act on: anyone refreshing the file from the source would find it silently changes lecture output. The status vocabulary simply had no value for the outcome that actually occurred -- the check ran in full and the file did not match. `failing` implies something is broken, and it is not. So `diverged` joins the vocabulary, with four fields the register (#39) needs: delta_kind local-edit | upstream-moved -- the two look alike and need opposite responses, so the distinction is machine-readable rather than buried in prose delta what differs, precisely enough that resolving it later needs no re-investigation delta_evidence how delta_kind was ESTABLISHED rather than assumed. Ruling out the other kind is the whole work: for mpd2020 it is what separates "our copy was edited" (constructed) from "we hold an older vintage" (verbatim), so it determines the file's class, not just its status register the issue tracking resolution This replaces three fields invented ad hoc for mpd2020 in an earlier commit (edits_are_local, edits_evidence, upstream_vintage_note) -- undocumented precedent set by one file, which is how a schema rots. Its upstream block now uses exactly the documented field set and nothing else. `diverged` is deliberately not a defect state. A migration moves the copy the lectures already consume (PLAN "Repoint rules" 3), so a known delta against today's upstream is an expected result to be recorded, not a problem to solve mid-repoint. The catalog gives it its own mark (⇄) so it cannot be misread as either a clean pass or a failure. Scope note: the schema sketch is behind the manifests in other ways that predate this PR -- consumers[].repo/.file is used by 10 manifests and documented in none, as are source.doi, source.note, source.version and source.file_url -- plus the multi-sheet vocabulary added here. That sweep belongs with the schema decisions in #14, against a decided convention, rather than piecemeal. Co-Authored-By: Claude Opus 5 (1M context) --- CATALOG.md | 4 +-- lectures/mpd2020.xlsx.yml | 54 ++++++++++++++++++++++----------------- manifest-schema.yml | 34 +++++++++++++++++++++++- scripts/build_catalog.py | 10 +++++--- 4 files changed, 72 insertions(+), 30 deletions(-) diff --git a/CATALOG.md b/CATALOG.md index 6e1fd07..8d6f9bc 100644 --- a/CATALOG.md +++ b/CATALOG.md @@ -24,11 +24,11 @@ The dataset registry, **auto-generated** from the sidecar manifests (`lectures/* | [**japan_population_by_age.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/japan_population_by_age.csv)
Japan — population by single year of age, 2024 | constructed | [Population Estimates, Statistics Bureau of Japan, Ministry of Internal Affairs and Communications](https://www.stat.go.jp/english/data/jinsui/index.html) | Japan Statistics Bureau terms of use | ✅ permitted | ✅ verified | ✅ committed | 1.3 KB | [lecture-python-intro · prob_dist.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/prob_dist.md) | | [**lingcod_msy_recovery.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/lingcod_msy_recovery.csv)
Pacific Coast lingcod — biomass and fishing pressure relative to MSY | constructed | [RAM Legacy Stock Assessment Database](https://www.ramlegacy.org/) | CC BY 4.0 | ✅ permitted | ⚠️ unverifiable | ⚠️ unrecovered | 2.3 KB | [lecture-python-intro · msy_fishery.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/msy_fishery.md) | | [**longprices.xls**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/longprices.xls)
Price levels in four hard-currency countries, 1600-2000 | verbatim | [Sargent and Velde, "The Big Problem of Small Change" — supporting data](https://press.princeton.edu/books/hardcover/9780691116358/the-big-problem-of-small-change) | | ✅ permitted | ⚠️ unverifiable | n/a (verbatim) | 379.5 KB | — | -| [**mpd2020.xlsx**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/mpd2020.xlsx)
Maddison Project Database 2020 — GDP per capita and population, 1 CE to 2018 | constructed | [Maddison Project Database, version 2020](https://www.rug.nl/ggdc/historicaldevelopment/maddison/releases/maddison-project-database-2020) | CC BY 4.0 | ✅ permitted | ✅ verified | ⚠️ unrecovered | 1.7 MB | — | +| [**mpd2020.xlsx**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/mpd2020.xlsx)
Maddison Project Database 2020 — GDP per capita and population, 1 CE to 2018 | constructed | [Maddison Project Database, version 2020](https://www.rug.nl/ggdc/historicaldevelopment/maddison/releases/maddison-project-database-2020) | CC BY 4.0 | ✅ permitted | ⇄ diverged | ⚠️ unrecovered | 1.7 MB | — | | [**nom_balances.npy**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/nom_balances.npy)
French Revolution — monthly nominal assignat balances, 1789-1796 | constructed | unrecorded | | ✅ permitted | ⚠️ unverifiable | ⚠️ unrecovered | 1.4 KB | — | | [**realwage.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/realwage.csv)
OECD real minimum wages — 32 countries, 2006–2016 | constructed | [OECD — Real minimum wages (RMW)](https://stats.oecd.org/Index.aspx?DataSetCode=RMW) | CC BY 4.0 | ✅ permitted | ⚠️ unverifiable | ⚠️ unrecovered | 118.7 KB | [lecture-python-programming · pandas_panel.md](https://github.com/QuantEcon/lecture-python-programming/blob/main/lectures/pandas_panel.md)
[lecture-python.myst · pandas_panel.md](https://github.com/QuantEcon/lecture-python.myst/blob/main/lectures/pandas_panel.md) | | [**us_adult_heights.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/us_adult_heights.csv)
United States — adult standing height by sex, NHANES 2015-2018 | constructed | [National Health and Nutrition Examination Survey (NHANES), US Centers for Disease Control and Prevention, National Center for Health Statistics](https://www.cdc.gov/nchs/nhanes/index.htm) | US Government work — public domain | ✅ permitted | ✅ verified | ✅ committed | 123.1 KB | [lecture-python-intro · prob_dist.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/prob_dist.md)
[lecture-python-intro · observed_distributions.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/observed_distributions.md)
[lecture-python-intro · fitting_distributions.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/fitting_distributions.md) | --- -**Legend** — *Integrity* is the `integrity.upstream.status` (is this what the source says?): ✅ verified · ◑ spot-checked · ⚠️ unverifiable · … unverified · ❌ failing. *Redist.* ⚠️ restricted files are cached as inherited exposures and tracked for licence review ([workspace-lectures#20](https://github.com/QuantEcon/workspace-lectures/issues/20)). *Builder* ⚠️ unrecovered marks a constructed dataset whose builder was never committed (PLAN Phase 9). +**Legend** — *Integrity* is the `integrity.upstream.status` (is this what the source says?): ✅ verified · ◑ spot-checked · ⇄ diverged (checked, differs, delta known and tracked) · ⚠️ unverifiable · … unverified · ❌ failing. *Redist.* ⚠️ restricted files are cached as inherited exposures and tracked for licence review ([workspace-lectures#20](https://github.com/QuantEcon/workspace-lectures/issues/20)). *Builder* ⚠️ unrecovered marks a constructed dataset whose builder was never committed (PLAN Phase 9). diff --git a/lectures/mpd2020.xlsx.yml b/lectures/mpd2020.xlsx.yml index 5bbb472..14f4ec5 100644 --- a/lectures/mpd2020.xlsx.yml +++ b/lectures/mpd2020.xlsx.yml @@ -65,34 +65,40 @@ maintainer: QuantEcon integrity: sha256: f67af0fd599fcebd9b92a0ad65472998f4548ea006e6b7c0018d4ade76e3bceb upstream: - status: verified + # `diverged`, not `verified`: the check was performed in full and the file + # does NOT match upstream. Recording `verified` here would be a claim a + # reader would act on -- someone refreshing this file from the source would + # find it silently changes lecture output. + status: diverged date: 2026-08-06 against: https://www.rug.nl/ggdc/historicaldevelopment/maddison/data/mpd2020.xlsx note: > + Checked in full against upstream and found to differ in three header + labels; every data value matches. See `delta`. + delta_kind: local-edit + delta: > Re-fetched from the Maddison release and compared cell by cell across all - six sheets. Every data value matches, including all 21,682 data rows of `Full - data`. THREE header labels on `Regional data` differ: row0 col1 "GDP pc - 2011 prices" -> "gdppc_2011", row0 col9 "Population" -> "pop", and row0 - col18 "gdppc_2011" added where upstream is empty. The lecture reads that - sheet with header=(0,1,2), so these renames are load-bearing -- replacing - this file with a pristine upstream copy would silently break - long_run_growth. Byte sizes differ (1,765,204 vs 1,764,793), consistent - with a re-save plus these edits. - edits_are_local: true - edits_evidence: > - Established, not assumed. The alternative -- that Maddison renamed its - own headers after we took the file, which would make our copy a faithful - older vintage -- is ruled out: the Internet Archive holds the upstream - file with one unchanging content digest - (4OWZWHTE5HXBBF4XLCCQTGDXY4HXOGOK) across every snapshot from 2021-01-10 - to 2026-01-02, and the copy fetched on 2026-08-06 matches that digest - exactly. Upstream was already byte-stable two years before this file was - committed to lecture-python-intro on 2023-03-23. The three edits are - therefore ours. - upstream_vintage_note: > - Because upstream has been byte-stable throughout, the data held here - corresponds to the CURRENT Maddison release -- there is no vintage - ambiguity, and the three header labels are the only delta. + six sheets. Every data value matches, including all 21,682 data rows of + `Full data`. THREE header labels on `Regional data` differ: row0 col1 + "GDP pc 2011 prices" -> "gdppc_2011", row0 col9 "Population" -> "pop", + and row0 col18 "gdppc_2011" added where upstream is empty. Byte sizes + differ (1,765,204 vs 1,764,793), consistent with a re-save plus these + edits. The lecture reads that sheet with header=(0,1,2), so the renames + are LOAD-BEARING: replacing this file with a pristine upstream copy would + silently break long_run_growth, which is why this cannot be resolved by + swapping the file alone -- file and lecture must move together. + delta_evidence: > + `local-edit` is established, not assumed. The alternative -- that Maddison + renamed its own headers after we took the file, which would make our copy + a faithful older vintage and the class `verbatim` -- is ruled out: the + Internet Archive holds the upstream file with one unchanging content + digest (4OWZWHTE5HXBBF4XLCCQTGDXY4HXOGOK) across every snapshot from + 2021-01-10 to 2026-01-02, and the copy fetched on 2026-08-06 matches that + digest exactly. Upstream was already byte-stable two years before this + file was committed to lecture-python-intro on 2023-03-23. Corollary: the + data held here corresponds to the CURRENT Maddison release, so there is no + vintage ambiguity and these three labels are the only delta. + register: QuantEcon/data-lectures#39 # --------------------------------------------------------------------------- # Shape diff --git a/manifest-schema.yml b/manifest-schema.yml index 06cd8e7..ba522c9 100644 --- a/manifest-schema.yml +++ b/manifest-schema.yml @@ -86,7 +86,8 @@ maintainer: QuantEcon # who fixes this when it breaks integrity: sha256: null # of the committed file upstream: - status: unverified # verified | spot-checked | unverifiable | unverified | failing + status: unverified # verified | spot-checked | diverged | + # unverifiable | unverified | failing date: null # ISO date of the last check against: null # what a verified check compared to (URL, # DOI, or repo path) — makes "verified" @@ -97,6 +98,37 @@ integrity: # with a note is a known status the # catalog can show; silence is not. + # --- `diverged` only ------------------------------------------------ + # `verified` means "checked, and it matches". A file that was checked and + # found to DIFFER has no honest home in the rest of this vocabulary: + # `failing` implies something is broken, and `verified` is a lie a reader + # would act on. `diverged` is the third outcome — checked, differs, and the + # difference is known and accepted (PLAN "Repoint rules" 3, register #39). + # + # It is not a defect state. A migration deliberately moves the copy the + # lectures already consume, so a known delta against today's upstream is an + # expected result, recorded rather than resolved mid-migration. + delta_kind: null # local-edit | upstream-moved + # local-edit our copy was modified; + # upstream is unchanged + # upstream-moved a newer vintage exists; + # adopting it means a NEW + # filename (see AGENTS.md + # "Corrections vs vintages") + # The two look alike and need opposite + # responses, so the register needs them + # distinguished machine-readably. + delta: null # what differs, precisely enough that + # resolving it later needs no + # re-investigation + delta_evidence: null # how delta_kind was ESTABLISHED rather + # than assumed — ruling out the other + # kind is the whole work, and an + # unevidenced claim here misclassifies + # the file's class as well as its status + register: null # the issue tracking the resolution, + # e.g. QuantEcon/data-lectures#39 + # --------------------------------------------------------------------------- # Shape — what a consumer can rely on; also the basis for PR validation tests # --------------------------------------------------------------------------- diff --git a/scripts/build_catalog.py b/scripts/build_catalog.py index 2601144..b7e6469 100644 --- a/scripts/build_catalog.py +++ b/scripts/build_catalog.py @@ -97,8 +97,11 @@ def fmt_integrity(integrity) -> str: return "?" up = integrity.get("upstream") or {} status = up.get("status", "?") - mark = {"verified": "✅", "spot-checked": "◑", "unverifiable": "⚠️", - "unverified": "…", "failing": "❌"}.get(status, "") + # `diverged` is checked-and-differs: not a clean pass, but not a defect + # either — a known, accepted delta tracked in the register (#39). It needs + # its own mark so it cannot be misread as either ✅ or ❌. + mark = {"verified": "✅", "spot-checked": "◑", "diverged": "⇄", + "unverifiable": "⚠️", "unverified": "…", "failing": "❌"}.get(status, "") return f"{mark} {status}".strip() @@ -171,7 +174,8 @@ def build(manifests) -> str: lines.append("") lines.append( "**Legend** — *Integrity* is the `integrity.upstream.status` (is this " - "what the source says?): ✅ verified · ◑ spot-checked · ⚠️ unverifiable · " + "what the source says?): ✅ verified · ◑ spot-checked · ⇄ diverged " + "(checked, differs, delta known and tracked) · ⚠️ unverifiable · " "… unverified · ❌ failing. *Redist.* ⚠️ restricted files are cached as " "inherited exposures and tracked for licence review " "([workspace-lectures#20](https://github.com/QuantEcon/workspace-lectures/issues/20)). "