diff --git a/AGENTS.md b/AGENTS.md index 2dc751f..ed172f0 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -43,6 +43,8 @@ Both are cheap to follow and expensive to discover. `PLAN.md` carries the reason - **Never delete a file a sibling repo reads.** `lecture-wasm` fetches `lecture-python-intro`'s *committed blobs* by URL, so deleting intro's copy in a repoint PR 404s the wasm build immediately. "Delete the lecture repo's own copy in the same repoint PR" applies only where no sibling reads it; where one does, the sibling's repoint lands first or in the same set. - **Repoint every consumer of a dataset together.** The strict audit has no green state for a partially-repointed dataset — `pending`/`landed` fails once any consumer reads data-lectures, and `repointed`/`final` fails while any consumer still does not. Land the lecture repoints first, then flip `migration.yml`; that flip is the push that re-runs the audit, so reality and the tracker agree by the time it runs. +- **A migration moves bytes; it does not update them.** Land the copy the lectures already consume, validated byte-identical — that is what makes a repoint provably unable to change a figure. If the committed file differs from what upstream publishes today, migrate it unchanged anyway, record the delta in `integrity.upstream` **and** in the register at [#39](https://github.com/QuantEcon/data-lectures/issues/39), and leave the decision for after the migration. Adopting a newer vintage changes lecture output and is an author's call, not an infrastructure one — and per "Corrections vs vintages" below it gets a **new filename**, never a silent replacement. + Cross-repo repoints are worked from [`QuantEcon/workspace-lectures`](https://github.com/QuantEcon/workspace-lectures) — same branch name in each repo, one PR per repo, no aggregate PR. ### Corrections vs vintages diff --git a/CATALOG.md b/CATALOG.md index 7ebd110..8d6f9bc 100644 --- a/CATALOG.md +++ b/CATALOG.md @@ -4,23 +4,31 @@ # Dataset catalog — `QuantEcon/data-lectures` -The migrated-dataset registry, **auto-generated** from the sidecar manifests (`lectures/*.yml`). Do not edit by hand — run `python scripts/build_catalog.py`. A dataset appears here once it has a manifest; files not yet migrated are tracked in [PLAN.md](PLAN.md) Phase 9. +The dataset registry, **auto-generated** from the sidecar manifests (`lectures/*.yml`). Do not edit by hand — run `python scripts/build_catalog.py`. A dataset appears here once it has a manifest, which may be before its consuming lectures are repointed — an empty **Used by** column means the file is here and documented but no lecture reads it from this repo yet. Files still to migrate are tracked in [PLAN.md](PLAN.md). -**10 datasets migrated** · 2.4 MB total · 9 permitted / 1 restricted redistribution +**18 datasets** · 10 read by lectures today, 8 awaiting repoint · 5.3 MB total · 17 permitted / 1 restricted redistribution | Dataset | Class | Source | Licence | Redist. | Integrity | Builder | Size | Used by | | --- | --- | --- | --- | --- | --- | --- | --- | --- | | [**ames_house_prices.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/ames_house_prices.csv)
Ames, Iowa — residential house sales, 2006-2010 | constructed | [Ames Housing data (De Cock 2011), Journal of Statistics Education](http://jse.amstat.org/v19n3/decock.pdf) | | ✅ permitted | ✅ verified | ✅ committed | 75.2 KB | [lecture-python-intro · observed_distributions.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/observed_distributions.md)
[lecture-python-intro · fitting_distributions.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/fitting_distributions.md) | +| [**assignat.xlsx**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/assignat.xlsx)
French Revolution — assignat issues, budgets and seigniorage (Sargent-Velde) | verbatim | [Sargent and Velde, "Macroeconomic Features of the French Revolution" — supporting spreadsheets](https://www.journals.uchicago.edu/doi/10.1086/261992) | | ✅ permitted | ⚠️ unverifiable | n/a (verbatim) | 204.6 KB | — | +| [**caron.npy**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/caron.npy)
French Revolution — monthly specie value of the assignat, 1791-1796 | constructed | unrecorded | | ✅ permitted | ⚠️ unverifiable | ⚠️ unrecovered | 1.1 KB | — | +| [**chapter_3.xlsx**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/chapter_3.xlsx)
The Ends of Four Big Inflations — appendix tables, transcribed | constructed | [Sargent, "Rational Expectations and Inflation", chapter 3 appendix tables](https://press.princeton.edu/books/paperback/9780691158709/rational-expectations-and-inflation) | | ✅ permitted | ⚠️ unverifiable | ⚠️ unrecovered | 71.6 KB | — | | [**countries.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/countries.csv)
WorldData.info country reference table | verbatim | [WorldData.info — country data downloads](https://www.worlddata.info/downloads/) | Proprietary — © WorldData.info, all rights reserved | ⚠️ restricted | ⚠️ unverifiable | n/a (verbatim) | 48.4 KB | [lecture-python-programming · pandas_panel.md](https://github.com/QuantEcon/lecture-python-programming/blob/main/lectures/pandas_panel.md)
[lecture-python.myst · pandas_panel.md](https://github.com/QuantEcon/lecture-python.myst/blob/main/lectures/pandas_panel.md) | +| [**dette.xlsx**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/dette.xlsx)
French Revolution — public debt, military spending and revenues (Sargent-Velde) | verbatim | [Sargent and Velde, "Macroeconomic Features of the French Revolution" — supporting spreadsheets](https://www.journals.uchicago.edu/doi/10.1086/261992) | | ✅ permitted | ⚠️ unverifiable | n/a (verbatim) | 617.2 KB | — | | [**employ.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/employ.csv)
Eurostat employment in Europe — by age and sex, 2007–2016 | constructed | [Eurostat — Employment database](https://ec.europa.eu/eurostat/data/database) | Eurostat reuse (Commission Decision 2011/833/EU) | ✅ permitted | ⚠️ unverifiable | ⚠️ unrecovered | 1.6 MB | [lecture-python-programming · pandas_panel.md](https://github.com/QuantEcon/lecture-python-programming/blob/main/lectures/pandas_panel.md)
[lecture-python.myst · pandas_panel.md](https://github.com/QuantEcon/lecture-python.myst/blob/main/lectures/pandas_panel.md) | | [**epl_match_goals.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/epl_match_goals.csv)
English Premier League — full-time scores, 2015-16 to 2024-25 | constructed | [openfootball / football.json](https://github.com/openfootball/football.json) | Public domain | ✅ permitted | ✅ verified | ✅ committed | 203.2 KB | [lecture-python-intro · fitting_distributions.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/fitting_distributions.md) | +| [**fig_3.xlsx**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/fig_3.xlsx)
French Revolution — figure 3 series (Sargent-Velde) | verbatim | [Sargent and Velde, "Macroeconomic Features of the French Revolution" — supporting spreadsheets](https://www.journals.uchicago.edu/doi/10.1086/261992) | | ✅ permitted | ⚠️ unverifiable | n/a (verbatim) | 9.2 KB | — | | [**japan_deaths_by_age.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/japan_deaths_by_age.csv)
Japan — deaths by single year of age, 2023 | constructed | [United Nations, Department of Economic and Social Affairs, Population Division — World Population Prospects 2024](https://population.un.org/wpp/downloads) | CC BY 3.0 IGO | ✅ permitted | ✅ verified | ✅ committed | 1.7 KB | [lecture-python-intro · observed_distributions.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/observed_distributions.md)
[lecture-python-intro · fitting_distributions.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/fitting_distributions.md) | | [**japan_earthquakes.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/japan_earthquakes.csv)
Japan region — earthquakes of magnitude 5 and above, 2000-2024 | constructed | [Advanced National Seismic System (ANSS) Comprehensive Earthquake Catalog (ComCat), US Geological Survey](https://earthquake.usgs.gov/earthquakes/search/) | US Government work — public domain | ✅ permitted | ✅ verified | ✅ committed | 172.8 KB | [lecture-python-intro · fitting_distributions.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/fitting_distributions.md) | | [**japan_population_by_age.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/japan_population_by_age.csv)
Japan — population by single year of age, 2024 | constructed | [Population Estimates, Statistics Bureau of Japan, Ministry of Internal Affairs and Communications](https://www.stat.go.jp/english/data/jinsui/index.html) | Japan Statistics Bureau terms of use | ✅ permitted | ✅ verified | ✅ committed | 1.3 KB | [lecture-python-intro · prob_dist.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/prob_dist.md) | | [**lingcod_msy_recovery.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/lingcod_msy_recovery.csv)
Pacific Coast lingcod — biomass and fishing pressure relative to MSY | constructed | [RAM Legacy Stock Assessment Database](https://www.ramlegacy.org/) | CC BY 4.0 | ✅ permitted | ⚠️ unverifiable | ⚠️ unrecovered | 2.3 KB | [lecture-python-intro · msy_fishery.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/msy_fishery.md) | +| [**longprices.xls**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/longprices.xls)
Price levels in four hard-currency countries, 1600-2000 | verbatim | [Sargent and Velde, "The Big Problem of Small Change" — supporting data](https://press.princeton.edu/books/hardcover/9780691116358/the-big-problem-of-small-change) | | ✅ permitted | ⚠️ unverifiable | n/a (verbatim) | 379.5 KB | — | +| [**mpd2020.xlsx**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/mpd2020.xlsx)
Maddison Project Database 2020 — GDP per capita and population, 1 CE to 2018 | constructed | [Maddison Project Database, version 2020](https://www.rug.nl/ggdc/historicaldevelopment/maddison/releases/maddison-project-database-2020) | CC BY 4.0 | ✅ permitted | ⇄ diverged | ⚠️ unrecovered | 1.7 MB | — | +| [**nom_balances.npy**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/nom_balances.npy)
French Revolution — monthly nominal assignat balances, 1789-1796 | constructed | unrecorded | | ✅ permitted | ⚠️ unverifiable | ⚠️ unrecovered | 1.4 KB | — | | [**realwage.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/realwage.csv)
OECD real minimum wages — 32 countries, 2006–2016 | constructed | [OECD — Real minimum wages (RMW)](https://stats.oecd.org/Index.aspx?DataSetCode=RMW) | CC BY 4.0 | ✅ permitted | ⚠️ unverifiable | ⚠️ unrecovered | 118.7 KB | [lecture-python-programming · pandas_panel.md](https://github.com/QuantEcon/lecture-python-programming/blob/main/lectures/pandas_panel.md)
[lecture-python.myst · pandas_panel.md](https://github.com/QuantEcon/lecture-python.myst/blob/main/lectures/pandas_panel.md) | | [**us_adult_heights.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/us_adult_heights.csv)
United States — adult standing height by sex, NHANES 2015-2018 | constructed | [National Health and Nutrition Examination Survey (NHANES), US Centers for Disease Control and Prevention, National Center for Health Statistics](https://www.cdc.gov/nchs/nhanes/index.htm) | US Government work — public domain | ✅ permitted | ✅ verified | ✅ committed | 123.1 KB | [lecture-python-intro · prob_dist.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/prob_dist.md)
[lecture-python-intro · observed_distributions.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/observed_distributions.md)
[lecture-python-intro · fitting_distributions.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/fitting_distributions.md) | --- -**Legend** — *Integrity* is the `integrity.upstream.status` (is this what the source says?): ✅ verified · ◑ spot-checked · ⚠️ unverifiable · … unverified · ❌ failing. *Redist.* ⚠️ restricted files are cached as inherited exposures and tracked for licence review ([workspace-lectures#20](https://github.com/QuantEcon/workspace-lectures/issues/20)). *Builder* ⚠️ unrecovered marks a constructed dataset whose builder was never committed (PLAN Phase 9). +**Legend** — *Integrity* is the `integrity.upstream.status` (is this what the source says?): ✅ verified · ◑ spot-checked · ⇄ diverged (checked, differs, delta known and tracked) · ⚠️ unverifiable · … unverified · ❌ failing. *Redist.* ⚠️ restricted files are cached as inherited exposures and tracked for licence review ([workspace-lectures#20](https://github.com/QuantEcon/workspace-lectures/issues/20)). *Builder* ⚠️ unrecovered marks a constructed dataset whose builder was never committed (PLAN Phase 9). diff --git a/PLAN.md b/PLAN.md index a9b6cbb..33516a2 100644 --- a/PLAN.md +++ b/PLAN.md @@ -42,7 +42,7 @@ This repository is being shaped into the **single canonical repository for data ## Repoint rules -Two rules learned the hard way. Both are about *ordering*, both are cheap to follow and expensive to discover, and neither is enforced by CI — the strict audit catches the second only after the fact. +Three rules learned the hard way. The first two are about *ordering*, both cheap to follow and expensive to discover, and neither is enforced by CI — the strict audit catches the second only after the fact. The third is about *scope*. ### 1. Repoint a sibling reader before deleting the file it reads @@ -60,6 +60,22 @@ The strict audit has **no green state for a partially-repointed dataset**. `scri Practically: one branch name across data-lectures + every consuming repo, PRs opened together, lecture repoints merged first, then the `migration.yml` flip to `repointed` — that last push is what re-runs the audit, and by then reality and the tracker agree. +### 3. A migration moves bytes; it does not update them + +The copy that lands here is the copy the lectures **already consume**, validated byte-identical in the repoint PR. That is what makes a repoint safe to merge: it provably cannot change a single figure. + +Adopting a newer upstream vintage is a *different change* with a different risk profile — it does change lecture output, it needs figures re-reviewed, and it is an author-facing decision rather than an infrastructure one. Conflating the two turns every repoint into a content review and stalls the programme. + +So when a migration finds that the committed file differs from what upstream publishes today: + +1. **Migrate what the lectures use**, unchanged, with the byte-compare gate as normal. +2. **Record the delta** in the dataset's manifest (`integrity.upstream`) *and* in the register at [#39](https://github.com/QuantEcon/data-lectures/issues/39) — the manifest makes it visible in the catalog from day one, the register is where it gets reasoned about. +3. **Review the register once the migration completes**, and decide each case on its merits. + +Two deltas look alike and need opposite responses. *Upstream moved* — a newer vintage exists; adopting it means a **new filename**, per "Corrections vs vintages" in `AGENTS.md`, so consumers opt in. *Our copy diverges* — upstream is unchanged but our file was modified; resolving means reconciling the edit. `mpd2020.xlsx` is the first recorded instance of the second kind, and it is instructive: the local edits are load-bearing for the consuming lecture, so the file and the lecture have to move together. + +Detecting these automatically rather than by accident is proposed in [#40](https://github.com/QuantEcon/data-lectures/issues/40). + ## Migration tracks The remaining work decomposes by **consuming series** rather than by hosting pattern, because — apart from the `intro`/`wasm` pairing — each series now owns its own data. This is the execution view; the phases below remain the machinery each track passes through. diff --git a/lectures/assignat.xlsx.yml b/lectures/assignat.xlsx.yml new file mode 100644 index 0000000..078e326 --- /dev/null +++ b/lectures/assignat.xlsx.yml @@ -0,0 +1,109 @@ +# Manifest for assignat.xlsx — one of three research spreadsheets assembled by +# Sargent and Velde for "Macroeconomic Features of the French Revolution". + +filename: assignat.xlsx +title: French Revolution — assignat issues, budgets and seigniorage (Sargent-Velde) +description: > + Working spreadsheet behind the monetary side of the French Revolution + analysis: assignat circulation and denominations, revolutionary budgets, + seigniorage, exchange rates and price indices, roughly 1789-1796. + +# The authors' own research workbook, republished as they assembled it. We +# apply no processing to the file itself -- the lecture slices ranges out of it +# at read time. +class: verbatim + +source: + name: Sargent and Velde, "Macroeconomic Features of the French Revolution" — supporting spreadsheets + url: https://www.journals.uchicago.edu/doi/10.1086/261992 + doi: 10.1086/261992 + citation: > + Sargent, Thomas J. and François R. Velde (1995). Macroeconomic Features of + the French Revolution. Journal of Political Economy, 103(3), 474-518. + note: > + The consuming lecture names the provenance directly: "This lecture uses + data from three spreadsheets assembled by [Sargent and Velde 1995]" -- + this file, dette.xlsx and fig_3.xlsx. The spreadsheets are the authors' + working files and are not published alongside the article. + +license: + name: null + url: null + redistribution: permitted + verified: 2026-08-06 + # No formal licence stated; the authors' own research data, served publicly + # from lecture-python-intro for years. Republished here with attribution + # under the record-and-track policy (#35). + +retrieved: null # unrecorded -- inherited bytes +maintainer: QuantEcon + +# --------------------------------------------------------------------------- +# Integrity (PLAN Phase 7) +# --------------------------------------------------------------------------- +# Migration check (repoint gate): verified 2026-08-06 by git blob hash against +# lecture-python-intro's lectures/datasets/assignat.xlsx -- identical. + +integrity: + sha256: 72213ff18a994f63520e9fc79d0c2117c451697cb29419da07c4d0c554503bc9 + upstream: + status: unverifiable + date: 2026-08-06 + against: null + note: > + Author-assembled research workbook with prose-only provenance. Not + distributed by the journal, no internal source list, no builder. Values + can be spot-checked against the figures in the published article but not + re-derived. + +# --------------------------------------------------------------------------- +# Shape +# --------------------------------------------------------------------------- +# READ POSITIONALLY. Every consuming read uses header=None with explicit +# usecols/skiprows/nrows, so the contract is a set of CELL RANGES, not named +# columns. Inserting or deleting a row or column anywhere above or left of a +# range silently shifts what the lecture plots, with no error. Treat the +# ranges below as the file's public interface. + +schema: + format: xlsx + sheet_count: 18 + positional_reads: true + sheets: + - name: Budgets + read_as: {usecols: 'J:K', skiprows: 22, nrows: 52, header: null} + shape: [52, 2] + dtypes: [float64, float64] + known_nulls_total: 2 + description: revolutionary government receipts and expenditures + - name: seignor + read_as: {usecols: 'F', skiprows: 6, nrows: 75, header: null} + shape: [75, 1] + dtypes: [float64] + known_nulls_total: 0 + description: monthly seigniorage from assignat issues + - name: Data + read_as: {usecols: 'P:Q', skiprows: 4, nrows: 80, header: null} + shape: [80, 2] + dtypes: [float64, float64] + known_nulls_total: 17 + description: assignat circulation and price level + - name: Data + read_as: {usecols: 'L', skiprows: 4, nrows: 80, header: null} + shape: [80, 1] + dtypes: [float64] + known_nulls_total: 0 + description: monthly index aligned with the P:Q block above + # 14 further sheets ship with the workbook and are not read by any lecture: + # Intro, Fig6, Data2, Denomina, Dom-nat, Exchge, Exchge2, Exchge3, + # Inflatio, Metal, Mint, Ramel, Ramel2, Revoluti, Post-95. + # Kept as the authors assembled them -- they are the working provenance for + # the ranges above. + date_range: {start: 1789, end: 1796} + row_count_floor: null # per-range, declared above + known_nulls: {} + +consumers: [] # empty until the Track A repoint lands + +builder: null +builder_status: not-applicable # verbatim republication -- nothing to build diff --git a/lectures/caron.npy.yml b/lectures/caron.npy.yml new file mode 100644 index 0000000..6534684 --- /dev/null +++ b/lectures/caron.npy.yml @@ -0,0 +1,104 @@ +# Manifest for caron.npy — assignat specie value, French Revolution. +# +# One of two files in this migration with NO recorded provenance anywhere. The +# consuming lecture's "Data Sources" section names three spreadsheets and does +# not mention this file or its twin, nom_balances.npy. Recorded honestly rather +# than given a plausible-sounding source. + +filename: caron.npy +title: French Revolution — monthly specie value of the assignat, 1791-1796 +description: > + 63 monthly observations of the assignat's value against specie, falling from + 96.7 at the start of 1791 to 0.431 in early 1796. The consuming lecture + differences its logarithm to obtain the monthly inflation rate used in its + real-balances regressions. + +# Constructed: a two-column numeric array is not a third-party file +# republished as distributed -- somebody produced it. What that process was is +# not recorded, which is precisely what builder_status captures. +class: constructed + +source: + name: unrecorded + url: null + citation: > + Consumed by analysis reproducing Sargent, Thomas J. and François R. Velde + (1995). Macroeconomic Features of the French Revolution. Journal of + Political Economy, 103(3), 474-518. + note: > + The filename is the only provenance signal. It most likely points to Pierre + Caron's published depreciation tables for the assignat, the standard source + for this series -- but that is an INFERENCE FROM THE NAME, stated nowhere in + the lecture or the file, and it should not be promoted to a source claim + without evidence. Recorded as a lead for whoever recovers this, not as fact. + +license: + name: null + url: null + redistribution: permitted + verified: 2026-08-06 + # Source unestablished, so no licence can be established either. These bytes + # have been served publicly from lecture-python-intro for years; migrated + # under the record-and-track policy (#35) and logged there, since a licence + # question does not hold up data we already serve. This one needs its source + # settled before its licence can be. + +retrieved: null # unrecorded -- inherited bytes +maintainer: QuantEcon + +# --------------------------------------------------------------------------- +# Integrity (PLAN Phase 7) +# --------------------------------------------------------------------------- +# Migration check (repoint gate): verified 2026-08-06 by git blob hash against +# lecture-python-intro's lectures/datasets/caron.npy -- identical. +# +# This file is one of the two LOCAL-PATH reads in lecture-python-intro +# (`np.load('datasets/caron.npy')`), which is what breaks downloaded and Colab +# notebooks. Repointing it at a public URL is the whole point of the migration +# -- the same failure mode pilot P1 fixed for msy_fishery. + +integrity: + sha256: 7e0bcc9a42fd05bb4a062041df029ca986ba70ddbc2611a7cbaf4d8d5d820885 + upstream: + status: unverifiable + date: 2026-08-06 + against: null + note: > + No source, no builder, no internal metadata -- a bare .npy carries no + provenance at all. Actively checked 2026-08-06 whether the series is a + dump of a column in the committed Sargent-Velde workbooks: it is not. + Neither terminal value (96.696, 0.431) appears anywhere in assignat.xlsx, + dette.xlsx or fig_3.xlsx, and no contiguous column matches. So it was not + derived from the data we hold, and cannot be re-derived from it. + +# --------------------------------------------------------------------------- +# Shape +# --------------------------------------------------------------------------- + +schema: + format: npy + dtype: float64 + shape: [63, 2] + columns: + - {index: 0, name: date, dtype: float64, description: "decimal year, monthly steps of 1/12"} + - {index: 1, name: specie_value, dtype: float64, description: "assignat value against specie, index units"} + row_count_floor: 63 # frozen historical extract: exact by design + date_range: {start: 1791.0, end: 1796.25} + known_nulls: {} # no NaNs present + note: > + Row alignment is load-bearing across files: the lecture pairs all 63 rows + of this array with rows 14:77 of nom_balances.npy. Changing the length of + either silently misaligns the other. + +consumers: [] # empty until the Track A repoint lands + +# --------------------------------------------------------------------------- +# Builder +# --------------------------------------------------------------------------- +# AGENTS.md: a constructed dataset without its builder is a bug -- inherited +# here rather than introduced. PLAN Phase 9 tracks recovery. Unlike the other +# unrecovered files in this migration, this one lacks even a named source, so +# recovery starts with identifying where the series came from. + +builder: null +builder_status: unrecovered diff --git a/lectures/chapter_3.xlsx.yml b/lectures/chapter_3.xlsx.yml new file mode 100644 index 0000000..f7d0a0c --- /dev/null +++ b/lectures/chapter_3.xlsx.yml @@ -0,0 +1,103 @@ +# Manifest for chapter_3.xlsx — QuantEcon's transcription of the appendix +# tables to chapter 3 of Sargent's "Rational Expectations and Inflation". + +filename: chapter_3.xlsx +title: The Ends of Four Big Inflations — appendix tables, transcribed +description: > + Twenty-seven tables underlying the four post-World-War-I hyperinflation + figures for Austria, Hungary, Poland and Germany: budget receipts and + expenditures, note circulation, retail and wholesale price indices, and + exchange rates against the US dollar, 1919-1925. + +# Constructed, and unambiguously so: the numbers were transcribed by hand from +# printed appendix tables into this workbook. That transcription is our +# processing, and it is the step a builder would have to reproduce. +class: constructed + +source: + name: Sargent, "Rational Expectations and Inflation", chapter 3 appendix tables + url: https://press.princeton.edu/books/paperback/9780691158709/rational-expectations-and-inflation + citation: > + Sargent, Thomas J. (2013). Rational Expectations and Inflation, third + edition. Princeton, New Jersey: Princeton University Press. Chapter 3, + "The Ends of Four Big Inflations", and its appendix tables. + note: > + The consuming lecture is explicit about the provenance: "Data underlying + our graphs appear in tables in an appendix to chapter 3 ... We have + transcribed all of these data into a spreadsheet." The transcription is + QuantEcon's own work; the underlying figures are the book's. + +license: + name: null + url: null + redistribution: permitted + verified: 2026-08-06 + # The transcription is QuantEcon's own work. The underlying figures are + # historical macroeconomic statistics reproduced in a published book -- data, + # not expression. Served publicly from lecture-python-intro for years and + # republished here with attribution under the record-and-track policy (#35). + +retrieved: null # unrecorded -- inherited bytes +maintainer: QuantEcon + +# --------------------------------------------------------------------------- +# Integrity (PLAN Phase 7) +# --------------------------------------------------------------------------- +# Migration check (repoint gate): verified 2026-08-06 by git blob hash against +# lecture-python-intro's lectures/datasets/chapter_3.xlsx -- identical. + +integrity: + sha256: ee17d0031f17c663c4b96765dad85dc4d873b2c6cd00636a728b093da1d30923 + upstream: + status: unverifiable + date: 2026-08-06 + against: null + note: > + A hand transcription from a printed source. Verification means checking + cells against the book's appendix, which is possible for spot-checks but + cannot be automated or re-derived, and no record exists of who + transcribed it or when. Recorded honestly rather than being labelled + verbatim, which would imply an upstream file that does not exist. + +# --------------------------------------------------------------------------- +# Shape +# --------------------------------------------------------------------------- +# 27 sheets named Table3.1 .. Table3.27. The lecture loops over a subset, +# reading each with header=1 and truncating to a per-table row count held in +# the lecture rather than here. Table shapes vary by design -- these are +# distinct historical tables, not one series split up. + +schema: + format: xlsx + sheets_pattern: 'Table3\.\d+' + sheet_count: 27 + read_as: {header: 1} + # Column lists are complete for the sheets shown, not truncated -- the count + # must equal the second element of `shape`, or the sample is not a contract. + sheets_sample: + - name: Table3.1 + shape: [6, 6] + columns: [Budget Beginning, Budget Ending, Receipts, Expenditures, Deficit, + "Percentage of expenditures covered by new issues of paper money"] + - {name: Table3.2, shape: [78, 3], columns: [Year, Month, Total Note Circulation]} + - {name: Table3.3, shape: [44, 3], columns: [Year, Month, "Retail price index, 52 commodities"]} + - {name: Table3.4, shape: [74, 3], columns: [Year, Month, Exchange Rate]} + date_range: {start: 1919, end: 1925} + row_count_floor: null # varies per sheet; the invariant is sheet_count + known_nulls: {} + note: > + Entries are strings needing cleanup (the lecture applies a `process_entry` + map to strip whitespace and handle footnote markers), so dtypes are not + stable across sheets and are deliberately not asserted here. + +consumers: [] # empty until the Track A repoint lands + +# --------------------------------------------------------------------------- +# Builder +# --------------------------------------------------------------------------- +# A constructed dataset should ship its builder. This one cannot: the +# construction was manual transcription from print. Recorded as `unrecovered` +# per AGENTS.md rather than misclassified as verbatim (PLAN Phase 9). + +builder: null +builder_status: unrecovered diff --git a/lectures/dette.xlsx.yml b/lectures/dette.xlsx.yml new file mode 100644 index 0000000..c9421f5 --- /dev/null +++ b/lectures/dette.xlsx.yml @@ -0,0 +1,114 @@ +# Manifest for dette.xlsx — one of three research spreadsheets assembled by +# Sargent and Velde for "Macroeconomic Features of the French Revolution". + +filename: dette.xlsx +title: French Revolution — public debt, military spending and revenues (Sargent-Velde) +description: > + Working spreadsheet behind the fiscal side of the French Revolution + analysis: French and British military expenditure through the eighteenth + century wars, the structure and servicing of the French royal debt, tax + revenues, annuities and interest rates, roughly 1685-1800. + +class: verbatim + +source: + name: Sargent and Velde, "Macroeconomic Features of the French Revolution" — supporting spreadsheets + url: https://www.journals.uchicago.edu/doi/10.1086/261992 + doi: 10.1086/261992 + citation: > + Sargent, Thomas J. and François R. Velde (1995). Macroeconomic Features of + the French Revolution. Journal of Political Economy, 103(3), 474-518. + note: > + One of the three spreadsheets the consuming lecture attributes to the + authors of the 1995 article. Working files, not journal-distributed. + +license: + name: null + url: null + redistribution: permitted + verified: 2026-08-06 + # No formal licence stated; the authors' own research data, served publicly + # from lecture-python-intro for years. Republished with attribution under the + # record-and-track policy (#35). + +retrieved: null # unrecorded -- inherited bytes +maintainer: QuantEcon + +# --------------------------------------------------------------------------- +# Integrity (PLAN Phase 7) +# --------------------------------------------------------------------------- +# Migration check (repoint gate): verified 2026-08-06 by git blob hash against +# lecture-python-intro's lectures/datasets/dette.xlsx -- identical. + +integrity: + sha256: 94f766090a000d25509baea2cae8a8adc6f89b22dda068ec041cec2d88a827ae + upstream: + status: unverifiable + date: 2026-08-06 + against: null + note: > + Author-assembled research workbook with prose-only provenance. Not + distributed by the journal, no internal source list, no builder. Values + can be spot-checked against the figures in the published article but not + re-derived. + +# --------------------------------------------------------------------------- +# Shape +# --------------------------------------------------------------------------- +# READ POSITIONALLY -- see the warning in assignat.xlsx.yml, which applies +# identically here. Every read is header=None with explicit usecols/skiprows/ +# nrows, so a row or column inserted above or left of a range silently changes +# what the lecture plots. These ranges are the file's public interface. + +schema: + format: xlsx + sheet_count: 27 + positional_reads: true + sheets: + - name: Militspe + read_as: {usecols: 'M:X', skiprows: 7, nrows: 102, header: null} + shape: [102, 12] + dtypes: [float64] + known_nulls_total: 0 + description: British military spending, 1689-1790; column index 4 is plotted + - name: Militspe + read_as: {usecols: 'D', skiprows: 3, nrows: 105, header: null} + shape: [105, 1] + dtypes: [float64] + known_nulls_total: 32 + description: French military spending 1685-1789, in 1726 livres + - name: Debt + read_as: {usecols: 'R:S', skiprows: 5, nrows: 99, header: null} + shape: [99, 2] + dtypes: [float64, float64] + known_nulls_total: 78 + description: French debt service and revenue series + - name: Debt + read_as: {usecols: 'P', skiprows: 89, nrows: 15, header: null} + shape: [15, 1] + dtypes: [float64] + known_nulls_total: 0 + description: terminal-period continuation of the block above + - name: Debt + read_as: {usecols: 'K', skiprows: 41, nrows: 120, header: null} + shape: [120, 1] + dtypes: [float64] + known_nulls_total: 8 + description: debt stock series + # 25 further sheets ship with the workbook and are not read by any lecture: + # Amortiss, Britain, Cambon, Divers, Experim, Indes, Interest, Irr-matr, + # Macro, Mortalit, P-Etat, Path2, Policy, Popul, Rate-mon, Rentier1, + # Return, Revenues, Studies, Studies2, Units, Viager, White, Viagere, + # Yearly. Kept as the authors assembled them. + date_range: {start: 1685, end: 1800} + row_count_floor: null # per-range, declared above + known_nulls: {} + note: > + High null counts in the Debt ranges are structural -- the blocks are + sparse layouts in a working spreadsheet, and the lecture indexes specific + columns out of them rather than consuming the block whole. + +consumers: [] # empty until the Track A repoint lands + +builder: null +builder_status: not-applicable # verbatim republication -- nothing to build diff --git a/lectures/employ.csv.yml b/lectures/employ.csv.yml index 797d14d..26f1668 100644 --- a/lectures/employ.csv.yml +++ b/lectures/employ.csv.yml @@ -48,7 +48,7 @@ integrity: schema: format: csv columns: - - {name: "", dtype: int64, description: pandas row index (0-based), not used by the lecture} + - {name: "", dtype: int64, description: "pandas row index (0-based), not used by the lecture"} - {name: GEO, dtype: string, description: geography (country or EU aggregate)} - {name: AGE, dtype: string, description: age band (e.g. "From 15 to 24 years")} - {name: UNIT, dtype: string, description: unit of measure (e.g. "Thousand persons")} diff --git a/lectures/epl_match_goals.csv.yml b/lectures/epl_match_goals.csv.yml index f613e55..72c5860 100644 --- a/lectures/epl_match_goals.csv.yml +++ b/lectures/epl_match_goals.csv.yml @@ -50,7 +50,7 @@ schema: format: csv columns: - {name: season, dtype: object, description: "season label, e.g. 2023-24"} - - {name: date, dtype: object, description: match date, ISO 8601} + - {name: date, dtype: object, description: "match date, ISO 8601"} - {name: home_team, dtype: object, description: home team name} - {name: away_team, dtype: object, description: away team name} - {name: home_goals, dtype: int64, description: full-time goals scored by the home team} diff --git a/lectures/fig_3.xlsx.yml b/lectures/fig_3.xlsx.yml new file mode 100644 index 0000000..87702f7 --- /dev/null +++ b/lectures/fig_3.xlsx.yml @@ -0,0 +1,89 @@ +# Manifest for fig_3.xlsx — one of three research spreadsheets assembled by +# Sargent and Velde for "Macroeconomic Features of the French Revolution". +# +# Note for anyone reading git history: a source-format twin, fig_3.ods, was +# dropped in #10 after being verified a pure format duplicate (identical single +# sheet, identical 34x6 frame, DataFrame.equals true). It remains recoverable +# from history. The full-filename sidecar convention exists because of exactly +# this pair -- see manifest-schema.yml. + +filename: fig_3.xlsx +title: French Revolution — figure 3 series (Sargent-Velde) +description: > + Small working sheet holding the series behind one figure of the French + Revolution analysis. Four numeric columns over thirty rows, read as a block. + +class: verbatim + +source: + name: Sargent and Velde, "Macroeconomic Features of the French Revolution" — supporting spreadsheets + url: https://www.journals.uchicago.edu/doi/10.1086/261992 + doi: 10.1086/261992 + citation: > + Sargent, Thomas J. and François R. Velde (1995). Macroeconomic Features of + the French Revolution. Journal of Political Economy, 103(3), 474-518. + note: > + One of the three spreadsheets the consuming lecture attributes to the + authors of the 1995 article. Working files, not journal-distributed. + +license: + name: null + url: null + redistribution: permitted + verified: 2026-08-06 + # No formal licence stated; the authors' own research data, served publicly + # from lecture-python-intro for years. Republished with attribution under the + # record-and-track policy (#35). + +retrieved: null # unrecorded -- inherited bytes +maintainer: QuantEcon + +# --------------------------------------------------------------------------- +# Integrity (PLAN Phase 7) +# --------------------------------------------------------------------------- +# Migration check (repoint gate): verified 2026-08-06 by git blob hash against +# lecture-python-intro's lectures/datasets/fig_3.xlsx -- identical. + +integrity: + sha256: bf66aa332439c8a5146d96a71dc4b4b0201bb2ed381480c79e8ed7ce055e3ecf + upstream: + status: unverifiable + date: 2026-08-06 + against: null + note: > + Author-assembled research sheet with prose-only provenance and no + internal labelling -- the columns are unheaded, so even their meaning + comes from how the lecture plots them rather than from the file. Values + can be spot-checked against the published figure but not re-derived. + +# --------------------------------------------------------------------------- +# Shape +# --------------------------------------------------------------------------- +# READ POSITIONALLY -- see the warning in assignat.xlsx.yml. The single sheet +# is 35x6; the lecture takes a 30x4 block out of the middle of it, so the five +# rows and two columns outside that block are load-bearing padding: deleting +# them shifts the read. + +schema: + format: xlsx + sheet_count: 1 + positional_reads: true + sheets: + - name: Sheet1 + full_shape: [35, 6] + read_as: {usecols: 'C:F', skiprows: 5, nrows: 30, header: null} + shape: [30, 4] + dtypes: [float64, float64, float64, float64] + known_nulls_per_column: [8, 8, 4, 9] + description: > + Four unheaded series read as a block. Nulls are ragged series ends + within the block, not damage -- the lecture plots each column over its + own valid span. + date_range: {start: 1789, end: 1796} + row_count_floor: 30 # exact by design -- a frozen 30-row block + known_nulls: {} # declared per column above + +consumers: [] # empty until the Track A repoint lands + +builder: null +builder_status: not-applicable # verbatim republication -- nothing to build diff --git a/lectures/japan_earthquakes.csv.yml b/lectures/japan_earthquakes.csv.yml index 4fbe7d4..a6a86f0 100644 --- a/lectures/japan_earthquakes.csv.yml +++ b/lectures/japan_earthquakes.csv.yml @@ -59,10 +59,10 @@ integrity: schema: format: csv columns: - - {name: time, dtype: object, description: event origin time, ISO 8601 UTC} - - {name: magnitude, dtype: float64, description: reported magnitude (mixed scales, as ComCat supplies)} - - {name: latitude, dtype: float64, description: epicentre latitude, degrees north} - - {name: longitude, dtype: float64, description: epicentre longitude, degrees east} + - {name: time, dtype: object, description: "event origin time, ISO 8601 UTC"} + - {name: magnitude, dtype: float64, description: "reported magnitude (mixed scales, as ComCat supplies)"} + - {name: latitude, dtype: float64, description: "epicentre latitude, degrees north"} + - {name: longitude, dtype: float64, description: "epicentre longitude, degrees east"} - {name: depth_km, dtype: float64, description: hypocentre depth in kilometres} row_count_floor: 3557 date_range: {start: 2000, end: 2024} diff --git a/lectures/japan_population_by_age.csv.yml b/lectures/japan_population_by_age.csv.yml index 789514e..10f369b 100644 --- a/lectures/japan_population_by_age.csv.yml +++ b/lectures/japan_population_by_age.csv.yml @@ -71,8 +71,8 @@ schema: format: csv columns: - {name: age, dtype: int64, description: single year of age; 100 means "100 and over"} - - {name: total_population, dtype: int64, description: total population at this age, in thousands} - - {name: japanese_population, dtype: int64, description: Japanese-national population at this age, in thousands} + - {name: total_population, dtype: int64, description: "total population at this age, in thousands"} + - {name: japanese_population, dtype: int64, description: "Japanese-national population at this age, in thousands"} row_count_floor: 101 date_range: {start: 2024, end: 2024} known_nulls: {} diff --git a/lectures/longprices.xls.yml b/lectures/longprices.xls.yml new file mode 100644 index 0000000..9c81a1e --- /dev/null +++ b/lectures/longprices.xls.yml @@ -0,0 +1,94 @@ +# Manifest for longprices.xls — four centuries of price levels, assembled by +# Sargent and Velde for "The Big Problem of Small Change". + +filename: longprices.xls +title: Price levels in four hard-currency countries, 1600-2000 +description: > + Annual price-level index for the United Kingdom, United States, France and + Castile (Spain). The consuming lecture plots 1600-1914, the period in which + all four were on commodity-money standards, to contrast with twentieth + century paper-money inflation. + +# The authors' own assembly of historical price series, republished here as +# they distributed it. We apply no processing, so this is not `constructed` -- +# but it also cannot be re-fetched from a publisher, which is what makes the +# integrity status below `unverifiable` rather than a clean re-download. +class: verbatim + +source: + name: Sargent and Velde, "The Big Problem of Small Change" — supporting data + url: https://press.princeton.edu/books/hardcover/9780691116358/the-big-problem-of-small-change + citation: > + Sargent, Thomas J. and François R. Velde (2002). The Big Problem of Small + Change. Princeton, New Jersey: Princeton University Press. + note: > + The consuming lecture states these data "originally appeared on page 35" of + the book. The spreadsheet is the authors' working assembly and is not + published at a citable location of its own; it reached QuantEcon directly. + +license: + name: null + url: null + redistribution: permitted + verified: 2026-08-06 + # No formal licence is stated. These bytes have been served publicly from + # lecture-python-intro for years and are republished here with attribution + # under the record-and-track policy (#35): a migration is not held up for a + # licence question on data we already serve. Logged there for review before + # data.quantecon.org is promoted as a public open-data host. + +retrieved: null # unrecorded -- inherited bytes +maintainer: QuantEcon + +# --------------------------------------------------------------------------- +# Integrity (PLAN Phase 7) +# --------------------------------------------------------------------------- +# Migration check (repoint gate, not a manifest field): verified 2026-08-06 by +# git blob hash against lecture-python-intro's lectures/datasets/longprices.xls +# -- identical. + +integrity: + sha256: 1160f9d3ac42cc225731a5446912acbb948e07753981e0cdc1b1da5dc2b344f7 + upstream: + status: unverifiable + date: 2026-08-06 + against: null + note: > + Author-assembled with prose-only provenance: no publisher distribution + point, no per-series source list inside the workbook, and no builder. + The figures can be spot-checked against page 35 of the cited book but + not re-derived. Trust rests on the authors' provenance. + +# --------------------------------------------------------------------------- +# Shape +# --------------------------------------------------------------------------- + +schema: + format: xls + sheets: + - name: all + read_as: {header: 2, index_col: 0, then: "iloc[1:]"} + shape: [401, 4] + columns: + - {name: UK, dtype: float64, description: United Kingdom price index} + - {name: US, dtype: float64, description: United States price index} + - {name: France, dtype: float64, description: France price index} + - {name: Castile, dtype: float64, description: Castile (Spain) price index} + known_nulls: {UK: 0, US: 152, France: 126, Castile: 0} + date_range: {start: 1600, end: 2000} + note: > + Nulls are series that do not span the full 1600-2000 index -- the US + series begins in the eighteenth century and France has gaps. Expected, + not damage. + # Nine further per-country sheets ship with the workbook and are not read: + # England 1264-1800, UK 1661-1913, UK 1947-2000, UK 1750-1998, + # US 1750-1932, US 1860-1939, US 1913-2000, Castile, France 1726-2000. + # They are the sources the `all` sheet is compiled from -- useful provenance, + # so they are kept rather than stripped. + row_count_floor: 401 # frozen historical extract: exact by design + known_nulls: {} # declared per sheet above + +consumers: [] # empty until the Track A repoint lands + +builder: null +builder_status: not-applicable # verbatim republication -- nothing to build diff --git a/lectures/mpd2020.xlsx.yml b/lectures/mpd2020.xlsx.yml new file mode 100644 index 0000000..14f4ec5 --- /dev/null +++ b/lectures/mpd2020.xlsx.yml @@ -0,0 +1,151 @@ +# Manifest for mpd2020.xlsx — Maddison Project Database 2020. +# +# Landed in the Feb 2025 migration (data#5-#7) and byte-identical to the copy +# lecture-python-intro consumes today; the repoint is Track A, PR set 1. + +filename: mpd2020.xlsx +title: Maddison Project Database 2020 — GDP per capita and population, 1 CE to 2018 +description: > + Historical estimates of real GDP per capita (2011 US$) and mid-year + population for 169 countries, some series reaching back to the first + century. The reference dataset for long-run comparative growth. + +# NOT verbatim, despite being an upstream release. Three header labels on the +# `Regional data` sheet were edited locally -- see integrity.upstream below. +# A file we have modified is not "republished as distributed", so it is +# classified `constructed` rather than misreporting it as untouched. +class: constructed + +source: + name: Maddison Project Database, version 2020 + url: https://www.rug.nl/ggdc/historicaldevelopment/maddison/releases/maddison-project-database-2020 + file_url: https://www.rug.nl/ggdc/historicaldevelopment/maddison/data/mpd2020.xlsx + version: "2020" + citation: > + Bolt, Jutta and Jan Luiten van Zanden (2020). Maddison style estimates of + the evolution of the world economy. A new 2020 update. Maddison Project + Database, version 2020. + # The workbook's own `Notes` sheet states a citation policy beyond the + # licence: the ORIGINAL papers (listed on the `Sources` sheet) must be cited + # when the data is shown in any graphical form, or when a subset of fewer + # than twelve countries is used. The consuming lecture does both, so this is + # a live obligation rather than a formality. + citation_policy: > + Original source papers must be cited when data is shown graphically or + when fewer than 12 countries are used; see the `Sources` sheet. + # Provenance chain of THESE bytes: they did not come here from Maddison + # directly. The file was downloaded, locally edited (see integrity.upstream) + # and committed to lecture-python-intro on 2023-03-23 in + # QuantEcon/lecture-python-intro#120, where the blob has never changed. It + # was copied into this repo by #5 in Feb 2025 and is byte-identical to + # intro's copy. So the immediate source is the lecture series; Maddison is + # the ultimate source. + +license: + name: CC BY 4.0 + url: https://creativecommons.org/licenses/by/4.0/ + redistribution: permitted + verified: 2026-08-06 + # Established against the release page itself, which states the database + # "is licensed under a Creative Commons Attribution 4.0 International + # License". Attribution is the sole condition; the citation above and the + # citation_policy note carry it. + +retrieved: null # unrecorded -- inherited bytes, see AGENTS.md +maintainer: QuantEcon + +# --------------------------------------------------------------------------- +# Integrity (PLAN Phase 7) +# --------------------------------------------------------------------------- +# Migration check (a repoint gate, not a manifest field): verified 2026-08-06 +# by git blob hash against lecture-python-intro's copy at +# lectures/datasets/mpd2020.xlsx -- identical, so the repoint cannot change +# lecture output. + +integrity: + sha256: f67af0fd599fcebd9b92a0ad65472998f4548ea006e6b7c0018d4ade76e3bceb + upstream: + # `diverged`, not `verified`: the check was performed in full and the file + # does NOT match upstream. Recording `verified` here would be a claim a + # reader would act on -- someone refreshing this file from the source would + # find it silently changes lecture output. + status: diverged + date: 2026-08-06 + against: https://www.rug.nl/ggdc/historicaldevelopment/maddison/data/mpd2020.xlsx + note: > + Checked in full against upstream and found to differ in three header + labels; every data value matches. See `delta`. + delta_kind: local-edit + delta: > + Re-fetched from the Maddison release and compared cell by cell across all + six sheets. Every data value matches, including all 21,682 data rows of + `Full data`. THREE header labels on `Regional data` differ: row0 col1 + "GDP pc 2011 prices" -> "gdppc_2011", row0 col9 "Population" -> "pop", + and row0 col18 "gdppc_2011" added where upstream is empty. Byte sizes + differ (1,765,204 vs 1,764,793), consistent with a re-save plus these + edits. The lecture reads that sheet with header=(0,1,2), so the renames + are LOAD-BEARING: replacing this file with a pristine upstream copy would + silently break long_run_growth, which is why this cannot be resolved by + swapping the file alone -- file and lecture must move together. + delta_evidence: > + `local-edit` is established, not assumed. The alternative -- that Maddison + renamed its own headers after we took the file, which would make our copy + a faithful older vintage and the class `verbatim` -- is ruled out: the + Internet Archive holds the upstream file with one unchanging content + digest (4OWZWHTE5HXBBF4XLCCQTGDXY4HXOGOK) across every snapshot from + 2021-01-10 to 2026-01-02, and the copy fetched on 2026-08-06 matches that + digest exactly. Upstream was already byte-stable two years before this + file was committed to lecture-python-intro on 2023-03-23. Corollary: the + data held here corresponds to the CURRENT Maddison release, so there is no + vintage ambiguity and these three labels are the only delta. + register: QuantEcon/data-lectures#39 + +# --------------------------------------------------------------------------- +# Shape +# --------------------------------------------------------------------------- +# Multi-sheet workbook: the contract is the sheets the lectures actually read, +# not a single flat table. Sheets not listed here are shipped as part of the +# upstream release and carry provenance (`Notes`, `Sources`) but are not read. + +schema: + format: xlsx + sheets: + - name: Full data + read_as: {header: 0} + shape: [21682, 5] + columns: + - {name: countrycode, dtype: str, description: ISO3 country code} + - {name: country, dtype: str, description: country name} + - {name: year, dtype: int64, description: calendar year} + - {name: gdppc, dtype: float64, description: "real GDP per capita, 2011 US$"} + - {name: pop, dtype: float64, description: "mid-year population, thousands"} + known_nulls: {gdppc: 1976, pop: 4483} + date_range: {start: 1, end: 2018} + - name: Regional data + read_as: {header: [0, 1, 2], index_col: 0} + shape: [23, 18] + date_range: {start: 1820, end: 2018} + note: column labels are locally edited -- see integrity.upstream + # Not read by any lecture, but shipped with the release and kept: + # Notes, Sources (provenance), GDP pc, Population (wide reshapes of Full data) + row_count_floor: 21682 # frozen release: exact by design + known_nulls: {} # declared per sheet above + +# --------------------------------------------------------------------------- +# Consumers +# --------------------------------------------------------------------------- +# Empty until the Track A repoint lands. lecture-python-intro and lecture-wasm +# both read this file today, but from intro's own copy, not from here. + +consumers: [] + +# --------------------------------------------------------------------------- +# Builder +# --------------------------------------------------------------------------- +# The transformation is exactly three header renames, enumerated in +# integrity.upstream. No script was committed for them, so the status is +# `unrecovered` -- but unusually for that state, recovery here is cheap and +# fully specified rather than lost. + +builder: null +builder_status: unrecovered diff --git a/lectures/nom_balances.npy.yml b/lectures/nom_balances.npy.yml new file mode 100644 index 0000000..8d2ad73 --- /dev/null +++ b/lectures/nom_balances.npy.yml @@ -0,0 +1,97 @@ +# Manifest for nom_balances.npy — nominal assignat balances, French Revolution. +# +# Twin of caron.npy, and the second of the two files in this migration with NO +# recorded provenance. The consuming lecture's "Data Sources" section names +# three spreadsheets and does not mention either .npy file. + +filename: nom_balances.npy +title: French Revolution — monthly nominal assignat balances, 1789-1796 +description: > + 81 monthly observations of assignats in circulation, rising from 90 in late + 1789 to a peak of 37,540.9 and standing at 33,555.59 in mid-1796. The + consuming lecture multiplies a 63-row slice of this series by the assignat's + specie value to obtain real balances. + +class: constructed + +source: + name: unrecorded + url: null + citation: > + Consumed by analysis reproducing Sargent, Thomas J. and François R. Velde + (1995). Macroeconomic Features of the French Revolution. Journal of + Political Economy, 103(3), 474-518. + note: > + No source recorded anywhere. Unlike its twin, the filename offers no lead + beyond the quantity it holds. The natural candidate is the assignat + circulation series in the authors' own workbooks -- see integrity.upstream + for why that was tested and rejected. + +license: + name: null + url: null + redistribution: permitted + verified: 2026-08-06 + # Source unestablished, so no licence can be established either. Served + # publicly from lecture-python-intro for years; migrated under the + # record-and-track policy (#35) and logged there. Needs its source settled + # before its licence can be. + +retrieved: null # unrecorded -- inherited bytes +maintainer: QuantEcon + +# --------------------------------------------------------------------------- +# Integrity (PLAN Phase 7) +# --------------------------------------------------------------------------- +# Migration check (repoint gate): verified 2026-08-06 by git blob hash against +# lecture-python-intro's lectures/datasets/nom_balances.npy -- identical. +# +# The second of the two LOCAL-PATH reads in lecture-python-intro +# (`np.load('datasets/nom_balances.npy')`) that break downloaded and Colab +# notebooks. Repointing it is the point of the migration. + +integrity: + sha256: 8790a75e023f552dd9b444ffe4b490a0abbccf4ffc745bc34c76facf847b972a + upstream: + status: unverifiable + date: 2026-08-06 + against: null + note: > + No source, no builder, no internal metadata. Actively checked 2026-08-06 + against the committed Sargent-Velde workbooks: the terminal value + 33,555.59 does occur in several assignat.xlsx sheets (Fig6, Data, Data2, + Ramel, Ramel2, seignor), but no column in any of them contains this + 81-value series contiguously, and the series maximum 37,540.9 appears + nowhere. So the array overlaps the workbooks without being extractable + from them -- a partial lead, not a recovery. + +# --------------------------------------------------------------------------- +# Shape +# --------------------------------------------------------------------------- + +schema: + format: npy + dtype: float64 + shape: [81, 2] + columns: + - {index: 0, name: date, dtype: float64, description: "decimal year, monthly steps of 1/12"} + - {index: 1, name: nominal_balances, dtype: float64, description: "assignats in circulation, millions of livres"} + row_count_floor: 81 # frozen historical extract: exact by design + date_range: {start: 1789.833, end: 1796.5} + known_nulls: {} # no NaNs present + note: > + Row alignment is load-bearing across files: the lecture takes rows 14:77 + of this array to pair with all 63 rows of caron.npy. The leading 14 rows + (late 1789 through 1790) exist to extend the circulation series before the + price series begins, and must not be trimmed. + +consumers: [] # empty until the Track A repoint lands + +# --------------------------------------------------------------------------- +# Builder +# --------------------------------------------------------------------------- +# Inherited constructed-without-builder, tracked for recovery in PLAN Phase 9. +# Recovery starts with identifying the source, as for caron.npy. + +builder: null +builder_status: unrecovered diff --git a/lectures/realwage.csv.yml b/lectures/realwage.csv.yml index 6f358f7..3b1379a 100644 --- a/lectures/realwage.csv.yml +++ b/lectures/realwage.csv.yml @@ -51,7 +51,7 @@ integrity: schema: format: csv columns: - - {name: "", dtype: int64, description: pandas row index (0-based), not used by the lecture} + - {name: "", dtype: int64, description: "pandas row index (0-based), not used by the lecture"} - {name: Time, dtype: string, description: observation year as ISO date (YYYY-01-01)} - {name: Country, dtype: string, description: OECD country name} - {name: Series, dtype: string, description: price basis (e.g. "In 2015 constant prices at 2015 USD PPPs")} diff --git a/manifest-schema.yml b/manifest-schema.yml index 06cd8e7..ba522c9 100644 --- a/manifest-schema.yml +++ b/manifest-schema.yml @@ -86,7 +86,8 @@ maintainer: QuantEcon # who fixes this when it breaks integrity: sha256: null # of the committed file upstream: - status: unverified # verified | spot-checked | unverifiable | unverified | failing + status: unverified # verified | spot-checked | diverged | + # unverifiable | unverified | failing date: null # ISO date of the last check against: null # what a verified check compared to (URL, # DOI, or repo path) — makes "verified" @@ -97,6 +98,37 @@ integrity: # with a note is a known status the # catalog can show; silence is not. + # --- `diverged` only ------------------------------------------------ + # `verified` means "checked, and it matches". A file that was checked and + # found to DIFFER has no honest home in the rest of this vocabulary: + # `failing` implies something is broken, and `verified` is a lie a reader + # would act on. `diverged` is the third outcome — checked, differs, and the + # difference is known and accepted (PLAN "Repoint rules" 3, register #39). + # + # It is not a defect state. A migration deliberately moves the copy the + # lectures already consume, so a known delta against today's upstream is an + # expected result, recorded rather than resolved mid-migration. + delta_kind: null # local-edit | upstream-moved + # local-edit our copy was modified; + # upstream is unchanged + # upstream-moved a newer vintage exists; + # adopting it means a NEW + # filename (see AGENTS.md + # "Corrections vs vintages") + # The two look alike and need opposite + # responses, so the register needs them + # distinguished machine-readably. + delta: null # what differs, precisely enough that + # resolving it later needs no + # re-investigation + delta_evidence: null # how delta_kind was ESTABLISHED rather + # than assumed — ruling out the other + # kind is the whole work, and an + # unevidenced claim here misclassifies + # the file's class as well as its status + register: null # the issue tracking the resolution, + # e.g. QuantEcon/data-lectures#39 + # --------------------------------------------------------------------------- # Shape — what a consumer can rely on; also the basis for PR validation tests # --------------------------------------------------------------------------- diff --git a/migration.yml b/migration.yml index a8e45f5..885704a 100644 --- a/migration.yml +++ b/migration.yml @@ -172,6 +172,97 @@ datasets: date: 2026-08-04 cutover: null + # --------------------------------------------------------------------------- + # Track A, the intro static batch -- landed in the Feb 2025 migration + # (data#5-#7) but never repointed, which is the gap #4 recorded. Manifests + # written 2026-08-06; each flips to `repointed` when its intro + wasm PR set + # merges. Every one of these has TWO consuming repos, so all consumers move + # together (PLAN, "Repoint rules"). + # --------------------------------------------------------------------------- + + mpd2020.xlsx: + pilot: A1 + status: landed + prior_pattern: own-repo + landed: + pr: QuantEcon/data-lectures#5 + date: 2025-02-14 + repoints: [] + cutover: null + + longprices.xls: + pilot: A2 + status: landed + prior_pattern: own-repo + landed: + pr: QuantEcon/data-lectures#5 + date: 2025-02-14 + repoints: [] + cutover: null + + chapter_3.xlsx: + pilot: A2 + status: landed + prior_pattern: own-repo + landed: + pr: QuantEcon/data-lectures#5 + date: 2025-02-14 + repoints: [] + cutover: null + + assignat.xlsx: + pilot: A3 + status: landed + prior_pattern: own-repo + landed: + pr: QuantEcon/data-lectures#5 + date: 2025-02-14 + repoints: [] + cutover: null + + dette.xlsx: + pilot: A3 + status: landed + prior_pattern: own-repo + landed: + pr: QuantEcon/data-lectures#5 + date: 2025-02-14 + repoints: [] + cutover: null + + fig_3.xlsx: + pilot: A3 + status: landed + prior_pattern: own-repo + landed: + pr: QuantEcon/data-lectures#5 + date: 2025-02-14 + repoints: [] + cutover: null + + caron.npy: + pilot: A3 + status: landed + # Local-path read in intro (np.load('datasets/caron.npy')) -- one of the + # two Colab-breaking reads this batch retires, the same failure mode P1 + # fixed for msy_fishery. + prior_pattern: local-path + landed: + pr: QuantEcon/data-lectures#5 + date: 2025-02-14 + repoints: [] + cutover: null + + nom_balances.npy: + pilot: A3 + status: landed + prior_pattern: local-path + landed: + pr: QuantEcon/data-lectures#5 + date: 2025-02-14 + repoints: [] + cutover: null + # Planned waves that have not landed anything here yet. `datasets` names the # files as the audit sees them today, so the dashboard can join the two views. # `title` is the reader-facing milestone name (the dashboard is read by people diff --git a/scripts/build_catalog.py b/scripts/build_catalog.py index 468b5a2..b7e6469 100644 --- a/scripts/build_catalog.py +++ b/scripts/build_catalog.py @@ -97,8 +97,11 @@ def fmt_integrity(integrity) -> str: return "?" up = integrity.get("upstream") or {} status = up.get("status", "?") - mark = {"verified": "✅", "spot-checked": "◑", "unverifiable": "⚠️", - "unverified": "…", "failing": "❌"}.get(status, "") + # `diverged` is checked-and-differs: not a clean pass, but not a defect + # either — a known, accepted delta tracked in the register (#39). It needs + # its own mark so it cannot be misread as either ✅ or ❌. + mark = {"verified": "✅", "spot-checked": "◑", "diverged": "⇄", + "unverifiable": "⚠️", "unverified": "…", "failing": "❌"}.get(status, "") return f"{mark} {status}".strip() @@ -124,15 +127,24 @@ def build(manifests) -> str: lines.append("# Dataset catalog — `QuantEcon/data-lectures`") lines.append("") lines.append( - "The migrated-dataset registry, **auto-generated** from the sidecar " - "manifests (`lectures/*.yml`). Do not edit by hand — run " + "The dataset registry, **auto-generated** from the sidecar manifests " + "(`lectures/*.yml`). Do not edit by hand — run " "`python scripts/build_catalog.py`. A dataset appears here once it has a " - "manifest; files not yet migrated are tracked in " - "[PLAN.md](PLAN.md) Phase 9." + "manifest, which may be before its consuming lectures are repointed — " + "an empty **Used by** column means the file is here and documented but " + "no lecture reads it from this repo yet. Files still to migrate are " + "tracked in [PLAN.md](PLAN.md)." ) lines.append("") + # `consumers` is the honest test of "in use": a manifest can land ahead of + # its repoint, so counting manifests would overstate what lectures read. + in_use = sum(1 for m in manifests if m.get("consumers")) + awaiting = len(manifests) - in_use + headline = f"**{len(manifests)} datasets** · {in_use} read by lectures today" + if awaiting: + headline += f", {awaiting} awaiting repoint" lines.append( - f"**{len(manifests)} datasets migrated** · {human_size(total)} total · " + f"{headline} · {human_size(total)} total · " f"{permitted} permitted / {restricted} restricted redistribution" ) lines.append("") @@ -162,7 +174,8 @@ def build(manifests) -> str: lines.append("") lines.append( "**Legend** — *Integrity* is the `integrity.upstream.status` (is this " - "what the source says?): ✅ verified · ◑ spot-checked · ⚠️ unverifiable · " + "what the source says?): ✅ verified · ◑ spot-checked · ⇄ diverged " + "(checked, differs, delta known and tracked) · ⚠️ unverifiable · " "… unverified · ❌ failing. *Redist.* ⚠️ restricted files are cached as " "inherited exposures and tracked for licence review " "([workspace-lectures#20](https://github.com/QuantEcon/workspace-lectures/issues/20)). "