From 94f7d737f0977eca5c6abe27c318f6d5f63a69c5 Mon Sep 17 00:00:00 2001 From: Matt McKay Date: Thu, 13 Aug 2026 17:01:11 +1000 Subject: [PATCH 1/2] =?UTF-8?q?Land=20wave=20B2'=20=E2=80=94=20the=20hanse?= =?UTF-8?q?n=5Fsingleton=20pair=20and=20their=20builders=20(Track=20B)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Migrates the last two Track B datasets out of QuantEcon/lecture-python.myst, where they sat beside their builders at lectures/_static/lecture_specific/hansen_singleton_198{2,3}/ and were read by URL. Bytes are byte-identical to the copies the lectures read today: 3e9d4f37… hansen_singleton_1982_data.csv 11,662 B dc5c1f8d… hansen_singleton_1983_data.csv 26,084 B both against lecture-python.myst @ d2a5b21. A repoint to these files therefore cannot change a figure. This is the first wave to move BUILDERS as well as data. Both were named make_data.py beside their own output, where the parent directory supplied the meaning; flattened into builders/ they collide, so each takes its dataset's stem per the naming rule. The write target moves to lectures/ and a validate() stage is added, per the four-stage contract — the arithmetic is untouched. D3 settled as `committed`, not `committed-frozen`, on evidence: each builder was run twice on 2026-08-13, once unmodified as it stood in lecture-python.myst and once after these edits, and all four runs reproduced the committed bytes exactly. FRED and Ken French are live public sources, which is not what committed-frozen means. Each validate() was also negative-controlled — truncation, percent units, net returns, a NaN, a renamed column, a missing month, and (1983) a duplicated T-bill leg are all rejected. Provenance worth reading before the numbers: these are NOT Hansen and Singleton's data. CRSP is a paid product, so the market leg substitutes the Ken French value-weighted research factor and the consumption leg uses modern BEA chain-type indices. The paper is the specification being replicated; the bytes are ours. Recorded at the head of both manifests. Licence recorded as found and split across the two providers: the three FRED inputs are US federal statistics in the public domain, while the Ken French library states no licence and no terms at all — only a copyright notice — and derives from CRSP with an Ibbotson T-bill leg covering this whole sample. Registered for review on #35. hansen_singleton_1982_data.csv is a bitwise strict subset of the 1983 file on an identical index. Recorded in both manifests, including the rule that a correction to the shared columns must be mirrored; consolidating the two files is a rename-class decision, deferred under D6. Acceptance test — strict audit, dry-run both directions: status: landed → exit 0 status: repointed → exit 1, "marked repointed but consumers still read via ['own-repo']", both files Also corrects three counts this change invalidates, one of which wave B1' had already left stale: builders/README.md said 18 constructed datasets when there were 19 (now 21), and the fetch-vs-sources split in AGENTS.md and sources/README.md moves from six-of-seven to eight-of-nine. PLAN.md's Track B row said coupling "none", which B1' refuted three ways and B2' adds a fourth to. Work plan: QuantEcon/workspace-lectures#39 Co-Authored-By: Claude Opus 5 (1M context) --- AGENTS.md | 2 +- CATALOG.md | 4 +- PLAN.md | 2 +- builders/README.md | 13 +- builders/hansen_singleton_1982_data.py | 203 +++++++++++++++++ builders/hansen_singleton_1983_data.py | 239 +++++++++++++++++++ lectures/hansen_singleton_1982_data.csv | 240 ++++++++++++++++++++ lectures/hansen_singleton_1982_data.csv.yml | 188 +++++++++++++++ lectures/hansen_singleton_1983_data.csv | 240 ++++++++++++++++++++ lectures/hansen_singleton_1983_data.csv.yml | 197 ++++++++++++++++ migration.yml | 43 ++++ scripts/audit_annotations.yml | 8 - sources/README.md | 3 +- 13 files changed, 1368 insertions(+), 14 deletions(-) create mode 100644 builders/hansen_singleton_1982_data.py create mode 100644 builders/hansen_singleton_1983_data.py create mode 100644 lectures/hansen_singleton_1982_data.csv create mode 100644 lectures/hansen_singleton_1982_data.csv.yml create mode 100644 lectures/hansen_singleton_1983_data.csv create mode 100644 lectures/hansen_singleton_1983_data.csv.yml diff --git a/AGENTS.md b/AGENTS.md index 6b9a645..ae6e61a 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -116,7 +116,7 @@ Where one builder produces a **set** of files, name it for the set and let each `scripts/` is repo tooling — the audit dashboard and the catalog generator — and produces no dataset. Keep the two apart. -**Where a builder reads its input from.** The normal case is the third-party upstream, fetched at run time: six of the seven builders here do that, and it is the fetch stage of the contract below. A builder reads from `sources/` **only when the input cannot be re-fetched** — the upstream is gone, unlocatable, or was inherited with no recoverable source. `sources/` is that exception layer, not a general input tree, and it is emphatically not "the big-file directory": the defining property is un-refetchability, not size. What it must never be is a network read from another QuantEcon repo — that is how a retired repo becomes load-bearing again. +**Where a builder reads its input from.** The normal case is the third-party upstream, fetched at run time: eight of the nine `committed` builders here do that, and it is the fetch stage of the contract below. A builder reads from `sources/` **only when the input cannot be re-fetched** — the upstream is gone, unlocatable, or was inherited with no recoverable source. `sources/` is that exception layer, not a general input tree, and it is emphatically not "the big-file directory": the defining property is un-refetchability, not size. What it must never be is a network read from another QuantEcon repo — that is how a retired repo becomes load-bearing again. Builders follow four stages — **fetch → pre-process → validate → write** — and only write on validation pass (expected columns/dtypes, row-count floor, recency of date range, no all-NaN columns, values unchanged in the overlap window with the previous vintage). Lectures always read the last-good snapshot: an upstream outage may fail a refresh, it must never break a lecture build. diff --git a/CATALOG.md b/CATALOG.md index 9409264..d2bfd7d 100644 --- a/CATALOG.md +++ b/CATALOG.md @@ -6,7 +6,7 @@ The dataset registry, **auto-generated** from the sidecar manifests (`lectures/*.yml`). Do not edit by hand — run `python scripts/build_catalog.py`. A dataset appears here once it has a manifest, which may be before its consuming lectures are repointed — an empty **Used by** column means the file is here and documented but no lecture reads it from this repo yet. Files still to migrate are tracked in [PLAN.md](PLAN.md). -**31 datasets** · 31 read by lectures today · 113.0 MB total · 26 permitted / 5 restricted redistribution +**33 datasets** · 31 read by lectures today, 2 awaiting repoint · 113.0 MB total · 28 permitted / 5 restricted redistribution | Dataset | Class | Source | Licence | Redist. | Integrity | Builder | Size | Used by | | --- | --- | --- | --- | --- | --- | --- | --- | --- | @@ -27,6 +27,8 @@ The dataset registry, **auto-generated** from the sidecar manifests (`lectures/* | [**forbes-billionaires.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/forbes-billionaires.csv)
Forbes Billionaires — individual net worth | constructed | [Forbes Billionaires](https://www.forbes.com/billionaires/) | | ⚠️ restricted | ⚠️ unverifiable | committed-frozen | 775.7 KB | [lecture-python-intro · heavy_tails.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/heavy_tails.md)
[lecture-wasm · heavy_tails.md](https://github.com/QuantEcon/lecture-wasm/blob/main/lectures/heavy_tails.md)
[lecture-intro.zh-cn · heavy_tails.md](https://github.com/QuantEcon/lecture-intro.zh-cn/blob/main/lectures/heavy_tails.md)
[test-actions-lecture-intro · heavy_tails.md](https://github.com/QuantEcon/test-actions-lecture-intro/blob/main/lectures/heavy_tails.md) | | [**forbes-global2000.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/forbes-global2000.csv)
Forbes Global 2000 — firm size measures | constructed | [Forbes Global 2000](https://www.forbes.com/lists/global2000/) | | ⚠️ restricted | ⚠️ unverifiable | committed-frozen | 115.6 KB | [lecture-python-intro · heavy_tails.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/heavy_tails.md)
[lecture-wasm · heavy_tails.md](https://github.com/QuantEcon/lecture-wasm/blob/main/lectures/heavy_tails.md)
[lecture-intro.zh-cn · heavy_tails.md](https://github.com/QuantEcon/lecture-intro.zh-cn/blob/main/lectures/heavy_tails.md)
[test-actions-lecture-intro · heavy_tails.md](https://github.com/QuantEcon/test-actions-lecture-intro/blob/main/lectures/heavy_tails.md) | | [**fp.dta**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/fp.dta)
Treisman (2016) Russia's Billionaires — country-year panel of billionaire counts and covariates | verbatim | [Replication package for Treisman (2016), "Russia's Billionaires" (AER Papers & Proceedings)](https://www.aeaweb.org/articles?id=10.1257/aer.p20161068) | | ✅ permitted | ✅ verified | n/a (verbatim) | 1000.1 KB | [lecture-python.myst · mle.md](https://github.com/QuantEcon/lecture-python.myst/blob/main/lectures/mle.md)
[lecture-python.zh-cn · mle.md](https://github.com/QuantEcon/lecture-python.zh-cn/blob/main/lectures/mle.md)
[lecture-stats · mle.md](https://github.com/QuantEcon/lecture-stats/blob/main/lectures/mle.md) | +| [**hansen_singleton_1982_data.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/hansen_singleton_1982_data.csv)
Hansen-Singleton (1982) replication — monthly US gross real market return and consumption growth, 1959-1978 | constructed | [FRED (BEA and BLS monthly series) and the Ken French data library (F-F_Research_Data_Factors)](https://mba.tuck.dartmouth.edu/pages/faculty/ken.french/data_library.html) | | ✅ permitted | ✅ verified | ✅ committed | 11.4 KB | — | +| [**hansen_singleton_1983_data.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/hansen_singleton_1983_data.csv)
Hansen-Singleton (1983) replication — monthly US returns, consumption and inflation, 1959-1978 | constructed | [FRED (BEA and BLS monthly series) and the Ken French data library (F-F_Research_Data_Factors)](https://mba.tuck.dartmouth.edu/pages/faculty/ken.french/data_library.html) | | ✅ permitted | ✅ verified | ✅ committed | 25.5 KB | — | | [**japan_deaths_by_age.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/japan_deaths_by_age.csv)
Japan — deaths by single year of age, 2023 | constructed | [United Nations, Department of Economic and Social Affairs, Population Division — World Population Prospects 2024](https://population.un.org/wpp/downloads) | CC BY 3.0 IGO | ✅ permitted | ✅ verified | ✅ committed | 1.7 KB | [lecture-python-intro · observed_distributions.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/observed_distributions.md)
[lecture-python-intro · fitting_distributions.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/fitting_distributions.md) | | [**japan_earthquakes.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/japan_earthquakes.csv)
Japan region — earthquakes of magnitude 5 and above, 2000-2024 | constructed | [Advanced National Seismic System (ANSS) Comprehensive Earthquake Catalog (ComCat), US Geological Survey](https://earthquake.usgs.gov/earthquakes/search/) | US Government work — public domain | ✅ permitted | ✅ verified | ✅ committed | 172.8 KB | [lecture-python-intro · fitting_distributions.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/fitting_distributions.md) | | [**japan_population_by_age.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/japan_population_by_age.csv)
Japan — population by single year of age, 2024 | constructed | [Population Estimates, Statistics Bureau of Japan, Ministry of Internal Affairs and Communications](https://www.stat.go.jp/english/data/jinsui/index.html) | Japan Statistics Bureau terms of use | ✅ permitted | ✅ verified | ✅ committed | 1.3 KB | [lecture-python-intro · prob_dist.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/prob_dist.md) | diff --git a/PLAN.md b/PLAN.md index 50e3b38..f53bf48 100644 --- a/PLAN.md +++ b/PLAN.md @@ -214,7 +214,7 @@ The remaining work decomposes by **consuming series** rather than by hosting pat | Track | Datasets | Coupling | Blocked on | | --- | --- | --- | --- | | **A — `intro` + `wasm`** | 17, **all done**. The last two CSVs landed as wave A4 ([#74](https://github.com/QuantEcon/data-lectures/pull/74), flipped in [#75](https://github.com/QuantEcon/data-lectures/pull/75)); `graph.txt` was never a migration — see below | — | — | -| **B — `python.myst`** | 7: `maketable1/2/4.dta`, `fp.dta`, `hansen_singleton_1982/1983_data.csv`, `NEWQDATA.csv` | none | nothing | +| **B — `python.myst`** | 7, cut into two waves. **B1′ done**: the `ols` trio, `fp.dta` and `NEWQDATA.csv` landed in [#79](https://github.com/QuantEcon/data-lectures/pull/79), flipped in [#80](https://github.com/QuantEcon/data-lectures/pull/80). **B2′**: `hansen_singleton_1982/1983_data.csv` | **three consumers, not one** — `lecture-python.zh-cn` reads by URL *and* holds byte-identical copies of all 7 plus both builders (and is outside `SCAN_REPOS`, so the audit cannot see it); `lecture-python.notebooks` lags a publish tag; `lecture-stats` carried a published-site prose link to `fp.dta` behind a daily linkcheck. B2′ adds a fourth kind: the two builders migrate too, and each lecture names them twice outside its data cell | nothing | | **C — `advanced.myst`** | 6: `dataBHS.mat`, `acs_data_summary.csv`, `bbh` ×2, `fred_data.csv`, `hansen_jagannathan_1991_data.json` | none | nothing (builder recovery is in-wave work, not a gate) | | **D — `programming`** | 1: `test_pwt.csv` | none | nothing — a single-PR track | | **E — dynamic / live-API** | the UNRATE twin, then the 15 incidental API lectures | wasm is the forcing customer | [#14](https://github.com/QuantEcon/data-lectures/issues/14) schema decisions, [#26](https://github.com/QuantEcon/data-lectures/issues/26) fetch layer | diff --git a/builders/README.md b/builders/README.md index 9b7707e..a64b962 100644 --- a/builders/README.md +++ b/builders/README.md @@ -40,6 +40,8 @@ re-fetched** — see `AGENTS.md`. | `japan_population_by_age.py` | `japan_population_by_age.csv` | committed | | `us_adult_heights.py` | `us_adult_heights.csv` | committed | | `NEWQDATA.py` | `NEWQDATA.csv` | committed — the **only** builder here that reads a committed input (`sources/NEWQDATA.MAT`) instead of fetching. Its upstream is published nowhere; see `sources/README.md`. Reproduces its output byte for byte | +| `hansen_singleton_1982_data.py` | `hansen_singleton_1982_data.csv` | committed — fetches FRED and the Ken French factors live. Reproduces its output byte for byte (2026-08-13) | +| `hansen_singleton_1983_data.py` | `hansen_singleton_1983_data.csv` | committed — the same construction plus a T-bill leg, so its output is a strict superset of the 1982 file's. Reproduces its output byte for byte (2026-08-13) | | `business_cycle.py` | `business_cycle_data.csv`, `business_cycle_info.md`, `business_cycle_metadata.md` | run by hand, no validate stage yet (PLAN Phase 5); its three outputs are the repo's only unmanifested files | | `webscrape_forbes.ipynb` | `forbes-global2000.csv`, `forbes-billionaires.csv` | **committed-frozen** — an undocumented Forbes API, a spoofed user-agent and hardcoded GDPR consent cookies. Defects recorded in the two manifests rather than fixed | | `generating_mini.md` | `SCF_plus_mini.csv`, `SCF_plus_mini_no_weights.csv` | **committed-frozen** — its `to_csv` calls are commented out upstream and stay that way. As written it still fetches the `high_dim_data` URL; that URL is historical, and the input is now committed at `sources/SCF_plus.dta`. See `sources/README.md` | @@ -52,14 +54,21 @@ asserts is that the path exists. `usa-gini-nwealth-tincome-lincome.ipynb` takes the opposite choice deliberately: its upstream name was `data.ipynb`, which is meaningless in a flat directory, so it is renamed to its output stem. +The two `hansen_singleton_*` builders are the case that leaves no choice at all. +Both were called `make_data.py`, sitting beside their own output in separate +`_static/lecture_specific//` directories where the parent directory +supplied the meaning. Flattened into one tree they collide outright, so each +takes its dataset's stem — the rule's default, arrived at by necessity rather +than by preference. + **A frozen builder is committed verbatim and not edited.** That is what makes it provenance rather than code, and it is why the pandas-3 defect above is recorded here instead of patched — fixing it would mean this file is no longer the thing that produced those bytes. The fix belongs in `lecture-python-intro`, which still serves that notebook to readers. -**This listing is the coverage report.** The repo has 18 `constructed` datasets -and 10 builders; the difference is the Phase 9 recovery backlog, carried as +**This listing is the coverage report.** The repo has 21 `constructed` datasets +and 12 builders; the difference is the Phase 9 recovery backlog, carried as `builder_status: unrecovered` in each manifest rather than hidden by reclassifying the file as `verbatim`. diff --git a/builders/hansen_singleton_1982_data.py b/builders/hansen_singleton_1982_data.py new file mode 100644 index 0000000..33bde73 --- /dev/null +++ b/builders/hansen_singleton_1982_data.py @@ -0,0 +1,203 @@ +#!/usr/bin/env python3 +""" +Builder for lectures/hansen_singleton_1982_data.csv. + +Constructs the monthly gross real market return and gross consumption-growth +series the `hansen_singleton_1982` lecture estimates on, from FRED and the Ken +French data library, over Hansen-Singleton's 1959:2-1978:12 sample. + +The lecture's ND+VWR specification wants CRSP's value-weighted NYSE return and +BEA per-capita nondurables consumption. CRSP is not open access, so the market +leg uses the Ken French research factors (`Mkt-RF + RF`) as the closest public +proxy -- which is why the lecture matches the paper qualitatively rather than +exactly. Consumption is FRED's real nondurables index divided by the 16+ +civilian noninstitutional population; the nominal return is deflated by the +month-over-month gross inflation of the nondurables price deflator. + +`hansen_singleton_1983_data.csv` is built the same way and its columns are a +strict superset of this file's, bitwise -- see that builder and the two +manifests. + +Migrated from QuantEcon/lecture-python.myst, where it lived beside its output +at lectures/_static/lecture_specific/hansen_singleton_1982/make_data.py. Both +hansen builders were named make_data.py there and collide in this flat tree, +so each takes its dataset's stem per the naming rule in AGENTS.md. The write +target moved to lectures/, and a validate() stage was added; the arithmetic is +unchanged, and the run below reproduces the migrated bytes exactly. + +Stages: fetch -> pre-process -> validate -> write. + +Requires only the standard library plus pandas. +""" +import io +import os +import urllib.request +import zipfile + +import pandas as pd + +CURRENT_FILE_DIR = os.path.dirname(os.path.abspath(__file__)) +REPO_ROOT = os.path.dirname(CURRENT_FILE_DIR) +PUBLISHED_DIR = os.path.join(REPO_ROOT, 'lectures') + +FRED_CODES = { + "population_16plus": "CNP16OV", + "cons_nd_real_index": "DNDGRA3M086SBEA", + "cons_nd_price_index": "DNDGRG3M086SBEA", +} +START = "1959-02-01" +END = "1978-12-01" + +OUT_FILE = 'hansen_singleton_1982_data.csv' + +COLUMNS = ["gross_real_return", "gross_cons_growth"] + +# 1959-02 to 1978-12 inclusive, monthly, no gaps. Exact by design: the sample +# is the paper's and does not grow. +N_MONTHS = 239 +FIRST_MONTH = pd.Timestamp("1959-02-28") +LAST_MONTH = pd.Timestamp("1978-12-31") + + +def read_fred(codes, start, end): + """Download FRED series as a date-indexed DataFrame (columns = codes).""" + base = "https://fred.stlouisfed.org/graph/fredgraph.csv" + columns = [] + for code in codes: + url = f"{base}?id={code}&cosd={start:%Y-%m-%d}&coed={end:%Y-%m-%d}" + columns.append( + pd.read_csv(url, index_col=0, parse_dates=True, na_values=".")) + fred = pd.concat(columns, axis=1).astype("float64") + fred.index.name = "DATE" + return fred + + +def read_famafrench_factors(start, end): + """Download the monthly Fama-French research factors (percent).""" + url = ("https://mba.tuck.dartmouth.edu/pages/faculty/ken.french/ftp/" + "F-F_Research_Data_Factors_CSV.zip") + with urllib.request.urlopen(url) as response: + payload = response.read() + with zipfile.ZipFile(io.BytesIO(payload)) as archive: + text = archive.read(archive.namelist()[0]).decode("utf-8") + + # Preamble, then a monthly table (rows keyed by YYYYMM), then an annual + # table (rows keyed by YYYY). Keep the contiguous monthly block. + records = [] + for line in text.splitlines(): + cells = [cell.strip() for cell in line.split(",")] + key = cells[0] + if len(key) == 6 and key.isdigit(): + records.append([key] + [float(x) for x in cells[1:5]]) + elif records: + break + factors = pd.DataFrame( + records, columns=["date", "Mkt-RF", "SMB", "HML", "RF"]) + factors.index = pd.PeriodIndex( + pd.to_datetime(factors["date"], format="%Y%m"), freq="M") + factors = factors.drop(columns="date") + window = ((factors.index >= pd.Period(start, "M")) + & (factors.index <= pd.Period(end, "M"))) + return factors.loc[window] + + +def to_month_end(index): + return pd.PeriodIndex(pd.DatetimeIndex(index), freq="M").to_timestamp("M") + + +def sample_window(start=START, end=END): + """Fetch and sample bounds. One extra prior month builds the first growth + rate, which is why the fetch window starts before the sample.""" + start_period = pd.Timestamp(start).to_period("M") + end_period = pd.Timestamp(end).to_period("M") + return { + "fetch_start": (start_period - 1).to_timestamp(how="start"), + "fetch_end": end_period.to_timestamp("M"), + "sample_start": start_period.to_timestamp("M"), + "sample_end": end_period.to_timestamp("M"), + } + + +def fetch(start=START, end=END): + window = sample_window(start, end) + fred = read_fred(list(FRED_CODES.values()), + window["fetch_start"], window["fetch_end"]) + ff = read_famafrench_factors(window["fetch_start"], window["fetch_end"]) + return fred, ff + + +def pre_process(raw, start=START, end=END): + fred, ff = raw + window = sample_window(start, end) + + fred = fred.rename(columns={v: k for k, v in FRED_CODES.items()}) + fred.index = to_month_end(fred.index) + fred["cons_real_level"] = fred["cons_nd_real_index"] + fred["cons_price_index"] = fred["cons_nd_price_index"] + fred["consumption_per_capita"] = ( + fred["cons_real_level"] / fred["population_16plus"]) + fred["gross_cons_growth"] = ( + fred["consumption_per_capita"] + / fred["consumption_per_capita"].shift(1)) + fred["gross_inflation_cons"] = ( + fred["cons_price_index"] / fred["cons_price_index"].shift(1)) + + ff = ff.copy() + ff.columns = [str(col).strip() for col in ff.columns] + if ("Mkt-RF" not in ff.columns) or ("RF" not in ff.columns): + raise KeyError( + "Fama-French data missing required columns: 'Mkt-RF' and 'RF'.") + # Mkt-RF and RF are reported in percent per month. + ff["gross_nom_return"] = 1.0 + (ff["Mkt-RF"] + ff["RF"]) / 100.0 + ff.index = ff.index.to_timestamp(how="end") + ff.index = to_month_end(ff.index) + market = ff[["gross_nom_return"]] + + out = fred.join(market, how="inner") + out["gross_real_return"] = ( + out["gross_nom_return"] / out["gross_inflation_cons"]) + out = out.loc[window["sample_start"]:window["sample_end"]].dropna() + + frame = out[COLUMNS].copy() + frame.index.name = "date" + return frame + + +def validate(frame): + """Refuse to write anything that is not the shape we expect.""" + assert list(frame.columns) == COLUMNS + assert frame.index.name == "date" + assert (frame.dtypes == "float64").all() + assert not frame.isnull().values.any() + + # The sample is the paper's, frozen. A short fetch -- an upstream outage + # part-way through, or a truncated Ken French block -- fails here rather + # than silently publishing a shorter series. + assert len(frame) == N_MONTHS, f'expected {N_MONTHS} months, got {len(frame)}' + assert frame.index[0] == FIRST_MONTH + assert frame.index[-1] == LAST_MONTH + assert frame.index.is_monotonic_increasing + # Month-end stamps on an unbroken monthly grid: 28-31 days apart, always. + gaps = frame.index.to_series().diff().dropna().dt.days + assert gaps.between(28, 31).all(), 'index is not an unbroken monthly grid' + + # Both series are GROSS -- ratios near 1, not percentages and not net + # returns. A units change upstream would pass every structural check above + # and quietly rescale every figure in the lecture. Bands are wide relative + # to the observed 1959-1978 spread (returns 0.87-1.16, growth 0.97-1.03). + assert frame["gross_real_return"].between(0.5, 1.5).all(), \ + 'gross_real_return is not a gross monthly ratio' + assert frame["gross_cons_growth"].between(0.9, 1.1).all(), \ + 'gross_cons_growth is not a gross monthly ratio' + + +def run(): + frame = pre_process(fetch()) + validate(frame) + frame.to_csv(os.path.join(PUBLISHED_DIR, OUT_FILE)) + print(f'wrote {OUT_FILE}: {frame.shape[0]} rows x {frame.shape[1]} cols ' + f'({frame.index.min().date()} .. {frame.index.max().date()})') + + +if __name__ == '__main__': + run() diff --git a/builders/hansen_singleton_1983_data.py b/builders/hansen_singleton_1983_data.py new file mode 100644 index 0000000..9a58095 --- /dev/null +++ b/builders/hansen_singleton_1983_data.py @@ -0,0 +1,239 @@ +#!/usr/bin/env python3 +""" +Builder for lectures/hansen_singleton_1983_data.csv. + +Constructs the monthly series the `hansen_singleton_1983` lecture estimates on +-- gross real market return, gross consumption growth, gross consumption +inflation, per-capita real nondurables consumption, and gross real T-bill +return -- from FRED and the Ken French data library, over the same +1959:2-1978:12 sample as the companion 1982 lecture. + +The construction is the companion builder's, plus the T-bill leg: the market +return is Ken French `Mkt-RF + RF` standing in for CRSP's value-weighted NYSE +return, the T-bill return is `RF` alone, consumption is FRED's real nondurables +index per head of the 16+ civilian noninstitutional population, and both +nominal returns are deflated by month-over-month gross inflation of the +nondurables price deflator. This file's `gross_real_return` and +`gross_cons_growth` columns are bitwise identical to +`hansen_singleton_1982_data.csv` on the same index -- that file is a strict +subset of this one, kept separate because the two lectures are separate. + +Migrated from QuantEcon/lecture-python.myst, where it lived beside its output +at lectures/_static/lecture_specific/hansen_singleton_1983/make_data.py. Both +hansen builders were named make_data.py there and collide in this flat tree, +so each takes its dataset's stem per the naming rule in AGENTS.md. The write +target moved to lectures/, and a validate() stage was added; the arithmetic is +unchanged, and the run below reproduces the migrated bytes exactly. + +Stages: fetch -> pre-process -> validate -> write. + +Requires only the standard library plus pandas. +""" +import io +import os +import urllib.request +import zipfile + +import pandas as pd + +CURRENT_FILE_DIR = os.path.dirname(os.path.abspath(__file__)) +REPO_ROOT = os.path.dirname(CURRENT_FILE_DIR) +PUBLISHED_DIR = os.path.join(REPO_ROOT, 'lectures') + +FRED_CODES = { + "population_16plus": "CNP16OV", + "cons_nd_real_index": "DNDGRA3M086SBEA", + "cons_nd_price_index": "DNDGRG3M086SBEA", +} +START = "1959-02-01" +END = "1978-12-01" + +OUT_FILE = 'hansen_singleton_1983_data.csv' + +COLUMNS = [ + "gross_real_return", + "gross_cons_growth", + "gross_inflation_cons", + "consumption_per_capita", + "gross_real_tbill", +] + +# 1959-02 to 1978-12 inclusive, monthly, no gaps. Exact by design: the sample +# is the paper's and does not grow. +N_MONTHS = 239 +FIRST_MONTH = pd.Timestamp("1959-02-28") +LAST_MONTH = pd.Timestamp("1978-12-31") + + +def read_fred(codes, start, end): + """Download FRED series as a date-indexed DataFrame (columns = codes).""" + base = "https://fred.stlouisfed.org/graph/fredgraph.csv" + columns = [] + for code in codes: + url = f"{base}?id={code}&cosd={start:%Y-%m-%d}&coed={end:%Y-%m-%d}" + columns.append( + pd.read_csv(url, index_col=0, parse_dates=True, na_values=".")) + fred = pd.concat(columns, axis=1).astype("float64") + fred.index.name = "DATE" + return fred + + +def read_famafrench_factors(start, end): + """Download the monthly Fama-French research factors (percent).""" + url = ("https://mba.tuck.dartmouth.edu/pages/faculty/ken.french/ftp/" + "F-F_Research_Data_Factors_CSV.zip") + with urllib.request.urlopen(url) as response: + payload = response.read() + with zipfile.ZipFile(io.BytesIO(payload)) as archive: + text = archive.read(archive.namelist()[0]).decode("utf-8") + + # Preamble, then a monthly table (rows keyed by YYYYMM), then an annual + # table (rows keyed by YYYY). Keep the contiguous monthly block. + records = [] + for line in text.splitlines(): + cells = [cell.strip() for cell in line.split(",")] + key = cells[0] + if len(key) == 6 and key.isdigit(): + records.append([key] + [float(x) for x in cells[1:5]]) + elif records: + break + factors = pd.DataFrame( + records, columns=["date", "Mkt-RF", "SMB", "HML", "RF"]) + factors.index = pd.PeriodIndex( + pd.to_datetime(factors["date"], format="%Y%m"), freq="M") + factors = factors.drop(columns="date") + window = ((factors.index >= pd.Period(start, "M")) + & (factors.index <= pd.Period(end, "M"))) + return factors.loc[window] + + +def to_month_end(index): + return pd.PeriodIndex(pd.DatetimeIndex(index), freq="M").to_timestamp("M") + + +def sample_window(start=START, end=END): + """Fetch and sample bounds. One extra prior month builds the first growth + rate, which is why the fetch window starts before the sample.""" + start_period = pd.Timestamp(start).to_period("M") + end_period = pd.Timestamp(end).to_period("M") + return { + "fetch_start": (start_period - 1).to_timestamp(how="start"), + "fetch_end": end_period.to_timestamp("M"), + "sample_start": start_period.to_timestamp("M"), + "sample_end": end_period.to_timestamp("M"), + } + + +def fetch(start=START, end=END): + window = sample_window(start, end) + fred = read_fred(list(FRED_CODES.values()), + window["fetch_start"], window["fetch_end"]) + ff = read_famafrench_factors(window["fetch_start"], window["fetch_end"]) + return fred, ff + + +def pre_process(raw, start=START, end=END): + fred, ff = raw + window = sample_window(start, end) + + fred = fred.rename(columns={v: k for k, v in FRED_CODES.items()}) + fred.index = to_month_end(fred.index) + fred["cons_real_level"] = fred["cons_nd_real_index"] + fred["cons_price_index"] = fred["cons_nd_price_index"] + fred["consumption_per_capita"] = ( + fred["cons_real_level"] / fred["population_16plus"]) + fred["gross_cons_growth"] = ( + fred["consumption_per_capita"] + / fred["consumption_per_capita"].shift(1)) + fred["gross_inflation_cons"] = ( + fred["cons_price_index"] / fred["cons_price_index"].shift(1)) + + ff = ff.copy() + ff.columns = [str(col).strip() for col in ff.columns] + if ("Mkt-RF" not in ff.columns) or ("RF" not in ff.columns): + raise KeyError( + "Fama-French data missing required columns: 'Mkt-RF' and 'RF'.") + # Mkt-RF and RF are reported in percent per month. + ff["gross_nom_return"] = 1.0 + (ff["Mkt-RF"] + ff["RF"]) / 100.0 + ff["gross_nom_tbill"] = 1.0 + ff["RF"] / 100.0 + ff.index = ff.index.to_timestamp(how="end") + ff.index = to_month_end(ff.index) + market = ff[["gross_nom_return", "gross_nom_tbill"]] + + out = fred.join(market, how="inner") + out["gross_real_return"] = ( + out["gross_nom_return"] / out["gross_inflation_cons"]) + out["gross_real_tbill"] = ( + out["gross_nom_tbill"] / out["gross_inflation_cons"]) + out = out.loc[window["sample_start"]:window["sample_end"]].dropna() + + frame = out[COLUMNS].copy() + frame.index.name = "date" + return frame + + +def validate(frame): + """Refuse to write anything that is not the shape we expect.""" + assert list(frame.columns) == COLUMNS + assert frame.index.name == "date" + assert (frame.dtypes == "float64").all() + assert not frame.isnull().values.any() + + # The sample is the paper's, frozen. A short fetch -- an upstream outage + # part-way through, or a truncated Ken French block -- fails here rather + # than silently publishing a shorter series. + assert len(frame) == N_MONTHS, f'expected {N_MONTHS} months, got {len(frame)}' + assert frame.index[0] == FIRST_MONTH + assert frame.index[-1] == LAST_MONTH + assert frame.index.is_monotonic_increasing + # Month-end stamps on an unbroken monthly grid: 28-31 days apart, always. + gaps = frame.index.to_series().diff().dropna().dt.days + assert gaps.between(28, 31).all(), 'index is not an unbroken monthly grid' + + # Four of the five series are GROSS -- ratios near 1, not percentages and + # not net returns. A units change upstream would pass every structural + # check above and quietly rescale every figure in the lecture. Bands are + # wide relative to the observed 1959-1978 spread (returns 0.87-1.16, growth + # 0.97-1.03, inflation 1.00-1.02, T-bill 0.98-1.01). + assert frame["gross_real_return"].between(0.5, 1.5).all(), \ + 'gross_real_return is not a gross monthly ratio' + assert frame["gross_cons_growth"].between(0.9, 1.1).all(), \ + 'gross_cons_growth is not a gross monthly ratio' + assert frame["gross_inflation_cons"].between(0.95, 1.05).all(), \ + 'gross_inflation_cons is not a gross monthly ratio' + assert frame["gross_real_tbill"].between(0.95, 1.05).all(), \ + 'gross_real_tbill is not a gross monthly ratio' + + # The one LEVEL series, and the one whose units are a ratio of two FRED + # index bases rather than a rate. It has no natural scale, so it is pinned + # by its observed order of magnitude (2.1e-4 to 2.7e-4): a rebased FRED + # index or a population series in thousands rather than units would shift + # it by orders of magnitude and fail here. + assert frame["consumption_per_capita"].between(1e-4, 1e-3).all(), \ + 'consumption_per_capita is off its expected order of magnitude' + + # The two real returns are built from different Ken French columns and + # share a deflator. Asserted on the MEAN, not month by month: the market + # leg beats the T-bill leg in only 54% of months, so a per-month or + # majority test would sit a few revisions away from failing for no real + # reason. A positive average excess return over twenty years is the robust + # form of the same economics. + excess = frame["gross_real_return"] - frame["gross_real_tbill"] + assert excess.mean() > 0, 'average excess return over the T-bill is not positive' + # And they must be distinct series. If the T-bill leg silently picks up the + # market column -- the natural failure of a mis-joined Ken French frame -- + # every band above still passes and the equity premium quietly vanishes. + assert not frame["gross_real_return"].equals(frame["gross_real_tbill"]), \ + 'the market and T-bill legs are identical' + + +def run(): + frame = pre_process(fetch()) + validate(frame) + frame.to_csv(os.path.join(PUBLISHED_DIR, OUT_FILE)) + print(f'wrote {OUT_FILE}: {frame.shape[0]} rows x {frame.shape[1]} cols ' + f'({frame.index.min().date()} .. {frame.index.max().date()})') + + +if __name__ == '__main__': + run() diff --git a/lectures/hansen_singleton_1982_data.csv b/lectures/hansen_singleton_1982_data.csv new file mode 100644 index 0000000..9b28375 --- /dev/null +++ b/lectures/hansen_singleton_1982_data.csv @@ -0,0 +1,240 @@ +date,gross_real_return,gross_cons_growth +1959-02-28,1.0109579062397553,1.0006229224411516 +1959-03-31,1.00571436382525,1.0180902221035215 +1959-04-30,1.037975704918033,0.9817704517570749 +1959-05-31,1.0206154266958425,1.0184807553981456 +1959-06-30,0.997762130888052,0.9958074219092394 +1959-07-31,1.0332284787350057,0.993247983171137 +1959-08-31,0.9862179047619046,1.001525294171654 +1959-09-30,0.9525118058948054,1.00821529850278 +1959-10-31,1.0133797259977257,0.9992750451488477 +1959-11-30,1.02031276849642,0.9946087169562763 +1959-12-31,1.0277770065075922,1.004162246542396 +1960-01-31,0.9350211841390549,0.9964357385736891 +1960-02-29,1.0144448970724023,0.9893925342902301 +1960-03-31,0.9853802764976959,1.0163173699295078 +1960-04-30,0.9786453316092882,1.017764785481483 +1960-05-31,1.0343458146930415,0.9709179722539227 +1960-06-30,1.0234206480133703,1.0085616910776072 +1960-07-31,0.9760739799763161,1.0025345810761046 +1960-08-31,1.029428143294484,0.99079410331375 +1960-09-30,0.9410440365002685,0.996384087876349 +1960-10-31,0.9934469213868494,1.008487858402606 +1960-11-30,1.044505804453463,0.998696477646169 +1960-12-31,1.0473899061533538,0.9853033898833323 +1961-01-31,1.0641837217984962,1.0048685493734337 +1961-02-28,1.0366024043074953,1.0060859901137909 +1961-03-31,1.0330478230674713,1.011079709815748 +1961-04-30,1.0092366983309182,0.9981161184493256 +1961-05-31,1.0267367533304685,1.0027101407890602 +1961-06-30,0.9728199730965832,1.002476272282421 +1961-07-31,1.026840189873418,0.9909863637951827 +1961-08-31,1.0265493942318,1.0038014943010605 +1961-09-30,0.9809888298728473,1.0049599207987134 +1961-10-31,1.0289144491593705,1.0005889440900915 +1961-11-30,1.045943817810721,1.0078399370066953 +1961-12-31,1.0013294402322954,1.0046414973238167 +1962-01-31,0.9621478982122725,0.9998486478698215 +1962-02-28,1.0156832745349584,0.9968025400551506 +1962-03-31,0.9947746099593928,1.0099083657192807 +1962-04-30,0.9340541975374449,0.9959690580913071 +1962-05-31,0.9170709260741926,1.0038630150586507 +1962-06-30,0.9186180622393326,0.9888965939087042 +1962-07-31,1.0686570555108608,1.004257045558508 +1962-08-31,1.0208077663671373,1.0032008132019383 +1962-09-30,0.9390012372442236,1.008872742785775 +1962-10-31,1.0046497376093295,0.9902798033248339 +1962-11-30,1.1110766885802141,1.0035193294325306 +1962-12-31,1.0147671803592304,1.0014352218681717 +1963-01-31,1.0481125807476441,1.001428214563085 +1963-02-28,0.9775236125072746,0.9915834699129015 +1963-03-31,1.036389952762592,1.0078801105544286 +1963-04-30,1.0489919005102042,0.9956865775751438 +1963-05-31,1.0186465686534685,0.9989658846886131 +1963-06-30,0.9784570076658737,1.0050600698980607 +1963-07-31,0.9946981519507188,1.000088934489468 +1963-08-31,1.0506999422268908,1.008744726271068 +1963-09-30,0.9914783159227603,0.9882926288372472 +1963-10-31,1.0264046658591814,0.9941407674824702 +1963-11-30,0.9918539775115595,1.003306442060827 +1963-12-31,1.018524179855361,1.0094163852654314 +1964-01-31,1.0219723597618302,0.9969309553486984 +1964-02-29,1.0177810463659147,1.0103641242285242 +1964-03-31,1.017465629080274,1.0048580805656955 +1964-04-30,1.0036379124895571,0.9958384002706671 +1964-05-31,1.0183483290622877,1.019303124872573 +1964-06-30,1.0136325521920666,0.9989575150840528 +1964-07-31,1.0186986244268446,1.0077285243985779 +1964-08-31,0.9891215350922932,1.0072981181249718 +1964-09-30,1.0265952692867542,0.9906379404291648 +1964-10-31,1.008218629539197,1.0047084123320786 +1964-11-30,1.0011356879829885,0.9826758836425177 +1964-12-31,1.0019410439645797,1.0251178687188183 +1965-01-31,1.0370722222222222,0.993420687639879 +1965-02-28,1.0080778922304467,1.0098922458637234 +1965-03-31,0.9878478595404079,0.9842224236080818 +1965-04-30,1.0314834672435103,1.0148702669060463 +1965-05-31,0.9920809487139997,1.003708609359037 +1965-06-30,0.9392863331299572,0.9867187240607261 +1965-07-31,1.0173448223272838,1.0169657327375092 +1965-08-31,1.0331209600978442,0.9985898079603674 +1965-09-30,1.033437932618683,1.0100855351490021 +1965-10-31,1.027159983695929,1.031222998180067 +1965-11-30,1.0019237940158763,0.9926644342395888 +1965-12-31,1.0088823100303954,1.0031400596884208 +1966-01-31,1.0072241849197536,1.0030154325138518 +1966-02-28,0.9845428829190056,1.0025556170670897 +1966-03-31,0.9742078627020555,1.0005930499735936 +1966-04-30,1.021336986573844,1.0087165432532905 +1966-05-31,0.9483431557989048,0.9912941713789765 +1966-06-30,0.9881211970570689,1.00766318282403 +1966-07-31,0.9865624323115901,1.0086434828390984 +1966-08-31,0.9182677010360139,0.9885676485039826 +1966-09-30,0.9899812192723696,1.0035629859147115 +1966-10-31,1.041768339389178,0.9947317349453374 +1966-11-30,1.0175501840852192,1.0026958464253002 +1966-12-31,1.004757447748013,0.9975934450405057 +1967-01-31,1.0866530295590693,1.0048229877833104 +1967-02-28,1.0108036808009422,1.0052226050463773 +1967-03-31,1.0445689700898777,1.00083279682869 +1967-04-30,1.0425608343160377,1.0027883252312702 +1967-05-31,0.9585927786499215,0.9969769901187401 +1967-06-30,1.0229355226506378,1.0019075616143613 +1967-07-31,1.0447114907285733,0.9878937593061994 +1967-08-31,0.9903404945454546,1.0053931198406931 +1967-09-30,1.0319484760522495,1.0073852460033181 +1967-10-31,0.9718775125628141,0.9829686324961886 +1967-11-30,1.0035662061329613,1.0089850965808387 +1967-12-31,1.033054040792765,1.013531234047572 +1968-01-31,0.9586732832463992,0.996618192741401 +1968-02-29,0.9626763822688275,1.0034102821507809 +1968-03-31,1.0037470630202143,1.0144003772849155 +1968-04-30,1.0909715308396148,0.9927375972057049 +1968-05-31,1.0231285188158084,1.0023733703506579 +1968-06-30,1.0090607132761706,1.0117684709448878 +1968-07-31,0.9751435403551628,1.007210916081518 +1968-08-31,1.0130833933498575,0.9997056149271085 +1968-09-30,1.0421195260751936,0.9885652798055251 +1968-10-31,1.003737031191979,1.0001906700265046 +1968-11-30,1.055511409395973,1.0121418411010206 +1968-12-31,0.964018356683775,0.9850571280382645 +1969-01-31,0.9889133014001473,1.0082139433036967 +1969-02-28,0.9453468955677268,1.0078666002342456 +1969-03-31,1.0268519363857187,0.9975673833773808 +1969-04-30,1.014136540126692,0.9971018638456864 +1969-05-31,1.0000262257127293,1.0075298934862833 +1969-06-30,0.9260949497679867,0.9851917552258751 +1969-07-31,0.9311470263724436,1.0048978054068127 +1969-08-31,1.0496402792696027,1.0091713231994486 +1969-09-30,0.9718027792624265,0.9878632302339346 +1969-10-31,1.0556596297614809,1.0120173869077764 +1969-11-30,0.9618071687759981,0.9932846657059041 +1969-12-31,0.9741939652518143,0.9985286940605688 +1970-01-31,0.9212316015419659,1.0112022620493186 +1970-02-28,1.0529872993719471,0.9918189353702483 +1970-03-31,0.9953170527417878,1.0027697854897362 +1970-04-30,0.8890496685298322,0.9902986375379047 +1970-05-31,0.9343163135593221,1.0069967730099987 +1970-06-30,0.9469810955590116,1.0000796481595717 +1970-07-31,1.0708379418729819,0.9970257158016894 +1970-08-31,1.0493963217896323,1.0008126665645092 +1970-09-30,1.0423570419218087,1.0046092184891302 +1970-10-31,0.9778801100260139,0.9975297864731273 +1970-11-30,1.0499492585648544,1.0011273292953267 +1970-12-31,1.058937765120925,1.0124788876843573 +1971-01-31,1.0539815499254845,0.9951243762978286 +1971-02-28,1.0165775612661676,0.9985130749361691 +1971-03-31,1.0412760627916844,0.9873714034196678 +1971-04-30,1.0299738476488232,1.0083586571050156 +1971-05-31,0.9593924993686338,0.9932845622433929 +1971-06-30,0.9981365955842305,1.0051700066447553 +1971-07-31,0.9584173951268526,0.991440910682584 +1971-08-31,1.0379071267816953,0.998618699830775 +1971-09-30,0.9932372145263945,1.002297245019121 +1971-10-31,0.9581448303078138,0.9943196841276655 +1971-11-30,0.9981049217744947,1.0085871519718006 +1971-12-31,1.0857983972240584,1.0000851203496202 +1972-01-31,1.0239623281468675,0.9795919590714202 +1972-02-29,1.0249978642445072,1.0057831401155264 +1972-03-31,1.0092303401137805,1.0176466007666984 +1972-04-30,1.0062942303754965,1.002537882028125 +1972-05-31,1.014321677092277,1.0090974304940825 +1972-06-30,0.978640011448197,0.9979709716441202 +1972-07-31,0.9909672557003257,1.0044556152379327 +1972-08-31,1.0329503309913495,1.0019206535386223 +1972-09-30,0.9847214674460794,1.0030088650530669 +1972-10-31,1.0067647697566862,1.015012111236451 +1972-11-30,1.0463335538184007,0.9957679937499486 +1972-12-31,1.0065508830110277,0.9976564431715872 +1973-01-31,0.9646849504165014,1.0054534426599542 +1973-02-28,0.9472085626670861,1.0029914982298926 +1973-03-31,0.9779890500193873,0.989248586225157 +1973-04-30,0.9384658711583471,0.9942346637985785 +1973-05-31,0.9691159728575784,0.9965352430069263 +1973-06-30,0.9814309245604084,1.0019997690857858 +1973-07-31,1.0551640866873064,1.017181918623719 +1973-08-31,0.9462215797625194,0.9667306001463677 +1973-09-30,1.0534230820252044,1.0144124866136586 +1973-10-31,0.9898111517100264,0.9934181638693607 +1973-11-30,0.8654669068376685,1.0037280242950093 +1973-12-31,1.0000506584098514,0.9908907468835305 +1974-01-31,0.9875830190659436,0.9956525119112044 +1974-02-28,0.98031451668151,0.9910821959505793 +1974-03-31,0.9595848668280871,0.9896974308812007 +1974-04-30,0.9478784373956594,1.0007275502562654 +1974-05-31,0.9495931537598203,0.9973613030939844 +1974-06-30,0.97195321299639,0.9975897053862942 +1974-07-31,0.9204789669340637,1.004966234283919 +1974-08-31,0.8991692737968946,0.9973952658120752 +1974-09-30,0.8792332560723923,0.989318272633984 +1974-10-31,1.1588045494699646,0.9969716375104358 +1974-11-30,0.9518714294651237,0.9898766288310575 +1974-12-31,0.9642454181784351,0.9845061814135657 +1975-01-31,1.1373505850029328,1.0081651656309403 +1975-02-28,1.0579404276954363,1.001536828811441 +1975-03-31,1.0313355872105572,1.0052761564076116 +1975-04-30,1.0471811753817677,0.9935362730578393 +1975-05-31,1.0539289649441383,1.0331915679240886 +1975-06-30,1.0453038992788168,0.993308307847106 +1975-07-31,0.9271832920866783,0.9969921925239672 +1975-08-31,0.9711535316543092,1.0035698292820647 +1975-09-30,0.9614013160261409,0.9952147892289859 +1975-10-31,1.0520194918535728,0.9890868041021199 +1975-11-30,1.0265333709131903,1.0073759465907024 +1975-12-31,0.9857875950190781,1.0021354245060932 +1976-01-31,1.1243044882773283,1.0142100453652187 +1976-02-29,1.0109178776914407,0.994119885230983 +1976-03-31,1.0298266412940058,1.0037780407827317 +1976-04-30,0.9905959747528877,1.011408985199085 +1976-05-31,0.9858094425180048,0.9915899111107992 +1976-06-30,1.0405823588441898,1.0134757714559979 +1976-07-31,0.9903584962803963,0.995576046841543 +1976-08-31,0.994202673029629,1.0020517454497475 +1976-09-30,1.0206372806250363,0.9993560142074526 +1976-10-31,0.9753222238342569,1.0053898361696996 +1976-11-30,1.0035523611351946,0.9967222024565706 +1976-12-31,1.0565306057814117,1.007819656591786 +1977-01-31,0.9587727991746904,0.9968859272175131 +1977-02-28,0.9716811046347121,0.9941048911407744 +1977-03-31,0.9870324205420342,0.993808284687326 +1977-04-30,0.9989612347270488,1.004466953570016 +1977-05-31,0.9835700231320196,0.9990643544993814 +1977-06-30,1.0456919101746605,0.9926203877235478 +1977-07-31,0.9846248341074982,1.008972476016858 +1977-08-31,0.9840791441491121,0.9936099082340789 +1977-09-30,1.0002761537614275,1.0019378571638178 +1977-10-31,0.9572044980800877,1.013567427204734 +1977-11-30,1.0379116837832683,1.0100645126409113 +1977-12-31,1.0035811916648578,0.9924702747962896 +1978-01-31,0.9407902963503254,0.9909629361948763 +1978-02-28,0.9886968052410225,1.0131171043457206 +1978-03-31,1.0258074850860062,1.0061359586059442 +1978-04-30,1.074129084499576,0.9917395346788931 +1978-05-31,1.013486002469071,1.0049967361563237 +1978-06-30,0.9808226302483777,0.9998135892775268 +1978-07-31,1.0521110326933056,0.9959300072415749 +1978-08-31,1.0400536933581626,1.0033159988277387 +1978-09-30,0.9873057466529351,1.0104543799417738 +1978-10-31,0.8813554851242205,0.9911387220819502 +1978-11-30,1.0281085688148,1.009087222697169 +1978-12-31,1.0113599312297732,1.0089199884598332 diff --git a/lectures/hansen_singleton_1982_data.csv.yml b/lectures/hansen_singleton_1982_data.csv.yml new file mode 100644 index 0000000..8c5b1bc --- /dev/null +++ b/lectures/hansen_singleton_1982_data.csv.yml @@ -0,0 +1,188 @@ +# Manifest for hansen_singleton_1982_data.csv — migrated out of +# QuantEcon/lecture-python.myst, where it sat at +# lectures/_static/lecture_specific/hansen_singleton_1982/ and was read by URL +# (PLAN Phase 8, wave B2'). +# +# READ THE `source` NOTE FIRST. These are NOT Hansen and Singleton's data. The +# paper is the specification being replicated; the bytes are a modern +# reconstruction from open sources, and the return leg is a deliberate +# substitute for the CRSP series the authors used. A reader who takes this file +# for the paper's dataset will misread every number in it. +# +# Its two columns are BITWISE IDENTICAL to the same-named columns of +# hansen_singleton_1983_data.csv on an identical index — that file is a strict +# superset of this one (verified 2026-08-13, max abs diff 0.0). They are kept +# as two files because they serve two lectures, which is the status quo this +# migration preserves; consolidating them is a rename-class decision and is +# deferred with the rest (D6). + +filename: hansen_singleton_1982_data.csv +title: Hansen-Singleton (1982) replication — monthly US gross real market return and consumption growth, 1959-1978 +description: > + The two monthly series the `hansen_singleton_1982` lecture estimates its + Euler equation on, over Hansen and Singleton's 1959:2-1978:12 sample: the + gross real return on the aggregate stock market, and gross growth of real + per-capita nondurable consumption. Both are gross ratios near 1, not + percentages. Reconstructed from FRED and the Ken French data library rather + than taken from the paper — see the source note. + +# Constructed: every value is computed from upstream series by +# builders/hansen_singleton_1982_data.py. Nothing here is republished as +# distributed. +class: constructed + +source: + name: FRED (BEA and BLS monthly series) and the Ken French data library (F-F_Research_Data_Factors) + # The Ken French library is the single downloadable FILE the builder + # retrieves; the three FRED inputs are separate series, enumerated below. + url: https://mba.tuck.dartmouth.edu/pages/faculty/ken.french/data_library.html + doi: null # neither provider issues one for these series + version: > + Ken French research factors, zip header "This file was created using the + 202606 CRSP database. The 1-month TBill rate data until 202405 are from + Ibbotson Associates" — read 2026-08-13. FRED series are unversioned and + revisable; the 1959-1978 window used here has been stable across every + check so far (see integrity.upstream). + series: > + FRED CNP16OV (civilian noninstitutional population 16+, thousands of + persons, BLS Employment Situation); + FRED DNDGRA3M086SBEA (real PCE, nondurable goods, chain-type quantity + index, 2017=100, SA, BEA Personal Income and Outlays); + FRED DNDGRG3M086SBEA (PCE, nondurable goods, chain-type price index, + 2017=100, SA, BEA); + Ken French F-F_Research_Data_Factors, monthly `Mkt-RF` and `RF`. + citation: > + Constructed by QuantEcon from U.S. Bureau of Economic Analysis and U.S. + Bureau of Labor Statistics series retrieved via FRED (Federal Reserve Bank + of St. Louis), and from the Fama-French research factors published by + Kenneth R. French, Tuck School of Business, Dartmouth College. Replicates + the ND+VWR specification of Hansen, Lars Peter, and Kenneth J. Singleton. + 1982. "Generalized Instrumental Variables Estimation of Nonlinear Rational + Expectations Models." Econometrica 50 (5): 1269-1286. + doi:10.2307/1911873. + note: > + NOT the authors' data, and not a faithful copy of it. Hansen and Singleton + used CRSP equally- and value-weighted NYSE returns paired with NIPA + nondurables consumption. CRSP is a paid subscription product, so the market + leg here substitutes the Ken French value-weighted research factor + (`Mkt-RF + RF`), which is close to but not the same series — which is why + the lecture states plainly that it matches the paper qualitatively and not + exactly. The consumption leg uses FRED's modern BEA chain-type indices, not + the 1970s NIPA vintage the authors had. The paper is the specification + being replicated; the bytes are ours. + +license: + name: null + url: https://mba.tuck.dartmouth.edu/pages/faculty/ken.french/data_library.html + # Recorded as found, per the record-and-track policy (AGENTS.md, "Licensing + # and attribution"). Split answer across the two providers, and only one half + # has a licence to name. + redistribution: permitted + verified: 2026-08-13 + note: > + The three FRED inputs are U.S. federal statistics — BEA chain-type indices + and the BLS civilian noninstitutional population — which are U.S. + Government works in the public domain; FRED is the delivery channel, not the + rights holder, and these are not among the third-party series FRED re-serves + under restriction. The Ken French data library states NO licence and NO + redistribution terms: the page carries only "Copyright (c) Eugene F. Fama + and Kenneth R. French" (checked 2026-08-13), and the factors are themselves + computed from the proprietary CRSP database with the pre-2024 T-bill leg + from Ibbotson Associates. `permitted` therefore rests on what is actually + published here rather than on a grant: not the library file, but two of its + columns summed, converted to a gross ratio and deflated by a FRED price + index over a fixed 1959-1978 window — from which the underlying factor + values cannot be recovered without the deflator. An inherited exposure, + served publicly from lecture-python.myst since 2026-06-26. No permission has + been sought from either provider. Registered for licence review on + QuantEcon/data-lectures#35. + +retrieved: null # the builder fetches live and no retrieval date + # was recorded upstream; not reconstructed from + # git history (AGENTS.md). It costs nothing + # here — a re-run on 2026-08-13 reproduced these + # bytes exactly, so the vintage is pinned by + # content rather than by a date. For context and + # NOT as a retrieval date: the bytes and their + # builder landed in lecture-python.myst together + # in a3e8443 (#926), 2026-06-26. +maintainer: QuantEcon + +# --------------------------------------------------------------------------- +# Integrity (PLAN Phase 7) +# --------------------------------------------------------------------------- +# Migration check (repoint gate, not a manifest field): the bytes landing here +# are byte-identical to the copy the lecture reads today — sha256 3e9d4f37… +# against lecture-python.myst @ d2a5b21, +# lectures/_static/lecture_specific/hansen_singleton_1982/. The +# lecture-python.zh-cn copy carries the identical git blob. Verified +# 2026-08-13; a repoint to this file therefore cannot change a figure. + +integrity: + sha256: 3e9d4f37c31dbcab26624418f3eeb630d57cc362bd162fa5501cfd992225609b + upstream: + # `verified` in the sense AGENTS.md defines for a constructed dataset — + # "re-run the builder and compare". Run twice on the date below: once + # unmodified as it stood in lecture-python.myst, and once after the + # migration edits (rename, new write target, added validate()), both + # reproducing this file byte for byte. The second run is what licenses the + # `committed` status below rather than `committed-frozen`. + status: verified + date: 2026-08-13 + against: builders/hansen_singleton_1982_data.py + note: > + Both live upstreams answered and neither had revised the 1959-1978 + window. The builder's own docstring in lecture-python.myst warned that + "re-running this script may produce small differences from the committed + data"; that is a real risk for a FRED-sourced series and it has not + materialised — the migrated builder records what was measured instead of + repeating the hedge. A future divergence is what validate() and this + sha256 exist to catch. + +# --------------------------------------------------------------------------- +# Shape +# --------------------------------------------------------------------------- +# Measured from the committed bytes 2026-08-13, and asserted by the builder's +# validate() stage on every run. + +schema: + format: csv + columns: + - {name: date, dtype: datetime64, description: "month-end stamp, 1959-02-28 to 1978-12-31 on an unbroken monthly grid — the CSV index column"} + - {name: gross_real_return, dtype: float64, description: "gross real monthly return on the aggregate stock market: (1 + (Mkt-RF + RF)/100) divided by the gross monthly inflation of the PCE nondurables price deflator. A ratio near 1 (observed 0.865-1.159), not a percentage and not a net return"} + - {name: gross_cons_growth, dtype: float64, description: "gross monthly growth of real per-capita nondurable consumption — the ratio of consecutive months of (DNDGRA3M086SBEA / CNP16OV). Observed 0.967-1.033"} + row_count_floor: 239 # exact by design, not a floor with headroom: + # the sample is the paper's 1959:2-1978:12 and + # does not grow. validate() asserts equality. + date_range: {start: 1959-02-28, end: 1978-12-31} + known_nulls: {} # genuinely none — a complete monthly panel + +# The gross monthly inflation of the nondurables deflator is used to deflate the +# nominal return but is NOT a column here. hansen_singleton_1983_data.csv +# carries it, along with the per-capita consumption level and the T-bill leg. + +# --------------------------------------------------------------------------- +# Consumers — how a correction knows what to rebuild +# --------------------------------------------------------------------------- +# Empty at landing and filled at the flip, per the B1' convention: no lecture +# reads this file from here yet. Two will — +# lecture-python.myst/lectures/hansen_singleton_1982.md and its +# lecture-python.zh-cn twin. zh-cn is outside the audit's SCAN_REPOS, so it is +# recorded here or nowhere. The generated lecture-python.notebooks mirror is a +# third reference-holder; it self-heals after a publish tag and is not a repo +# anyone repoints, so it will deliberately not be listed (wave A4 precedent). +# +# Both reads build the URL by string concatenation across three source lines and +# must be COLLAPSED at the repoint rather than patched: editing the stem line +# alone leaves the filename appended to a path that no longer exists, since the +# flat published tree has no _static/lecture_specific// segment. +# Note for any future sweep: a line-based grep for the whole old URL returns a +# confident ZERO against the wrapped form. Grep the filename separately. +# +# Each lecture ALSO carries two references to the BUILDER — a prose link, and +# the path named again in a docstring inside the data cell. Neither is a code +# read; both break silently when the builder moves here. +consumers: [] + +builder: builders/hansen_singleton_1982_data.py +builder_status: committed diff --git a/lectures/hansen_singleton_1983_data.csv b/lectures/hansen_singleton_1983_data.csv new file mode 100644 index 0000000..491212f --- /dev/null +++ b/lectures/hansen_singleton_1983_data.csv @@ -0,0 +1,240 @@ +date,gross_real_return,gross_cons_growth,gross_inflation_cons,consumption_per_capita,gross_real_tbill +1959-02-28,1.0109579062397553,1.0006229224411516,1.0004373018476003,0.0002135783527442639,1.001462058791389 +1959-03-31,1.00571436382525,1.0180902221035215,0.9992896951152881,0.0002174420325819119,1.0029123735578762 +1959-04-30,1.037975704918033,0.9817704517570749,1.0006014544261577,0.00021347816255892023,1.001397704918033 +1959-05-31,1.0206154266958425,1.0184807553981456,0.9989071038251366,0.00021742340026401722,1.003296498905908 +1959-06-30,0.997762130888052,0.9958074219092394,1.0022428884026258,0.00021651183567965163,1.0001567600021832 +1959-07-31,1.0332284787350057,0.993247983171137,1.0010370612957806,0.00021504994412149461,1.0014614231188659 +1959-08-31,0.9862179047619046,1.001525294171654,1.001908396946565,0.00021537795854787765,0.999991619047619 +1959-09-30,0.9525118058948054,1.00821529850278,1.002612244897959,0.0002171473527682678,1.000486484285947 +1959-10-31,1.0133797259977257,0.9992750451488477,1.0023883189491396,0.00021698993074146356,1.0006102236421726 +1959-11-30,1.02031276849642,0.9946087169562763,0.9983213299398929,0.00021582007660719832,1.0042858646127142 +1959-12-31,1.0277770065075922,1.004162246542396,1.000216966804079,0.00021671837297483626,1.0031823427331887 +1960-01-31,0.9350211841390549,0.9964357385736891,0.9983731019522776,0.00021594593203766918,1.004934926670288 +1960-02-29,1.0144448970724023,0.9893925342902301,1.000054318305269,0.0002136552929684153,1.0028455271305194 +1960-03-31,0.9853802764976959,1.0163173699295078,1.0018467220683287,0.00021714158542117832,1.0016502304147465 +1960-04-30,0.9786453316092882,1.017764785481483,1.0062889671997832,0.00022099905910529468,0.9956384623673292 +1960-05-31,1.0343458146930415,0.9709179722539227,0.9995689887398309,0.00021457195833653753,1.0031323613431788 +1960-06-30,1.0234206480133703,1.0085616910776072,0.9997844014445102,0.00021640905715773214,1.0026161625963665 +1960-07-31,0.9760739799763161,1.0025345810761046,1.0015634265998168,0.00021695756345870175,0.9997369846054472 +1960-08-31,1.029428143294484,0.99079410331375,1.0022069114005814,0.0002149602745442004,0.9994942048445137 +1960-09-30,0.9410440365002685,0.996384087876349,1.0005907943498575,0.0002141829970813727,1.0010086097691897 +1960-10-31,0.9934469213868494,1.008487858402606,1.0016639828234033,0.00021600095203284516,1.0005351267349014 +1960-11-30,1.044505804453463,0.998696477646169,1.0035367879534858,0.00021571938996342157,0.9977710952101246 +1960-12-31,1.0473899061533538,0.9853033898833323,1.0014417685694452,0.00021254904619452377,1.0001580036258932 +1961-01-31,1.0641837217984962,1.0048685493734337,0.9997333902100883,0.00021358385172019805,1.002167187583338 +1961-02-28,1.0366024043074953,1.0060859901137909,1.0004800256013655,0.00021488372093023255,1.0009195329992535 +1961-03-31,1.0330478230674713,1.011079709815748,0.9979208870881758,0.0002172645702022677,1.004087611517709 +1961-04-30,1.0092366983309182,0.9981161184493256,0.9954057374859767,0.00021685526948684843,1.006323313476091 +1961-05-31,1.0267367533304685,1.0027101407890602,0.9990876402082327,0.0002174429777980074,1.0027148366996133 +1961-06-30,0.9728199730965832,1.002476272282421,0.9983347657928664,0.00021798142581693568,1.0036713478611783 +1961-07-31,1.026840189873418,0.9909863637951827,1.003174603174603,0.00021601662054521444,0.9986297468354433 +1961-08-31,1.0265493942318,1.0038014943010605,1.000536365586784,0.0002168378064971514,1.0008631714377614 +1961-09-30,0.9809888298728473,1.0049599207987134,0.9991958829205532,0.00021791330484354402,1.0025061323032352 +1961-10-31,1.0289144491593705,1.0005889440900915,0.9988196791673374,0.00021804164359658394,1.0030839608959552 +1961-11-30,1.045943817810721,1.0078399370066953,1.000053714347102,0.00021975107634721747,1.001446207970781 +1961-12-31,1.0013294402322954,1.0046414973238167,0.9988720592974543,0.00022077105037998892,1.0030313598967573 +1962-01-31,0.9621478982122725,0.9998486478698215,1.0016131634134537,0.00022073763621123217,1.0007855693348364 +1962-02-28,1.0156832745349584,0.9968025400551506,1.0043485263327427,0.000220031836461126,0.9976616420782554 +1962-03-31,0.9947746099593928,1.0099083657192807,1.0004276245456492,0.00022221199236666778,1.0015717033554177 +1962-04-30,0.9340541975374449,0.9959690580913071,1.0024043599059627,0.00022131626873402282,0.9997961302702416 +1962-05-31,0.9170709260741926,1.0038630150586507,0.9986141463674645,0.00022217121681286672,1.0037911075527088 +1962-06-30,0.9186180622393326,0.9888965939087042,0.9982385908726983,0.00021970435957079613,1.0037680461982674 +1962-07-31,1.0686570555108608,1.004257045558508,0.9969521976259222,0.00022063965103889184,1.0057653740949317 +1962-08-31,1.0208077663671373,1.0032008132019383,1.0027353177795657,0.00022134587734680818,0.9995658697047495 +1962-09-30,0.9390012372442236,1.008872742785775,1.0116067608044501,0.00022330982238319812,0.9906023158673928 +1962-10-31,1.0046497376093295,0.9902798033248339,0.9974620631311795,0.00022113920699013702,1.0051510098065202 +1962-11-30,1.1110766885802141,1.0035193294325306,0.9998409753511794,0.00022191746870998388,1.002159368041565 +1962-12-31,1.0147671803592304,1.0014352218681717,0.9976672675220021,0.00022223596951400574,1.0046435646721221 +1963-01-31,1.0481125807476441,1.001428214563085,1.0036135614836856,0.00022255337016210698,0.9988904479508632 +1963-02-28,0.9775236125072746,0.9915834699129015,1.0007942391189242,0.00022068024302615243,1.0015045658959842 +1963-03-31,1.036389952762592,1.0078801105544286,0.9968255647849321,0.00022241922773837667,1.0054918687967729 +1963-04-30,1.0489919005102042,0.9956865775751438,0.9986731065230082,0.00022145983965373076,1.0038319781037415 +1963-05-31,1.0186465686534685,0.9989658846886131,1.0013286564625852,0.00022123082464268757,1.0010699219786634 +1963-06-30,0.9784570076658737,1.0050600698980607,1.0039276046918952,0.0002223502680789852,0.998378762886598 +1963-07-31,0.9946981519507188,1.000088934489468,1.0041237113402062,0.00022237004268655985,0.9985821355236139 +1963-08-31,1.0506999422268908,1.008744726271068,1.002474595903754,0.00022431460784073955,1.0000253413865545 +1963-09-30,0.9914783159227603,0.9882926288372472,0.995483193277311,0.0002216884734695207,1.0072495515458477 +1963-10-31,1.0264046658591814,0.9941407674824702,1.00184657592065,0.00022038954915700652,1.0010514824372003 +1963-11-30,0.9918539775115595,1.003306442060827,1.002264468903049,0.00022111825443210596,1.0004345470786045 +1963-12-31,1.018524179855361,1.0094163852654314,1.0026271542664986,0.00022320038910505835,1.0002721308038989 +1964-01-31,1.0219723597618302,0.9969309553486984,1.0033539461272403,0.00022251537714470702,0.9996472370207876 +1964-02-29,1.0177810463659147,1.0103641242285242,1.0003133813851457,0.00022482155415619166,1.002285902255639 +1964-03-31,1.017465629080274,1.0048580805656955,0.9997389306599832,0.0002259137553791873,1.0033619470413122 +1964-04-30,1.0036379124895571,0.9958384002706671,1.0002611375150154,0.00022497359275594868,1.0026381735588972 +1964-05-31,1.0183483290622877,1.019303124872573,0.9983813700918963,0.00022931628610994815,1.0042254693792165 +1964-06-30,1.0136325521920666,0.9989575150840528,1.0020396422781237,0.0002290772273406975,1.0009584029227554 +1964-07-31,1.0186986244268446,1.0077285243985779,1.001670146137787,0.00023084765628135864,1.0013276365152146 +1964-08-31,0.9891215350922932,1.0072981181249718,0.9992705293872447,0.0002325324097457729,1.0035320471373448 +1964-09-30,1.0265952692867542,0.9906379404291648,1.0030242986755658,0.0002303554274735831,0.9997763776252858 +1964-10-31,1.008218629539197,1.0047084123320786,1.0006758161779994,0.00023144003580906095,1.0022226816977506 +1964-11-30,1.0011356879829885,0.9826758836425177,1.001662424022027,0.0002274305416989249,1.0012355220164928 +1964-12-31,1.0019410439645797,1.0251178687188183,1.0015559358954411,0.00023314311218796823,1.001541665372068 +1965-01-31,1.0370722222222222,0.993420687639879,1.0010874631039304,0.00023160919082827286,1.001710676598386 +1965-02-28,1.0080778922304467,1.0098922458637234,0.9993275398303331,0.00023390032588824417,1.0036749314146693 +1965-03-31,0.9878478595404079,0.9842224236080818,1.0023810756250324,0.00023020994562844782,1.0012160289181513 +1965-04-30,1.0314834672435103,1.0148702669060463,1.002633617350891,0.00023363322896436924,1.0004651576019776 +1965-05-31,0.9920809487139997,1.003708609359037,1.0032447466007417,0.00023449968334388855,0.9998557215462807 +1965-06-30,0.9392863331299572,0.9867187240607261,1.0097027568150316,0.00023138522834172603,0.9938568486882245 +1965-07-31,1.0173448223272838,1.0169657327375092,1.0001525320317266,0.0002353108482851793,1.0029470184535612 +1965-08-31,1.0331209600978442,0.9985898079603674,0.9975598596919324,0.00023497901480008836,1.0057541813178412 +1965-09-30,1.033437932618683,1.0100855351490021,0.9983182999541353,0.00023734890391313254,1.0047897549770293 +1965-10-31,1.027159983695929,1.031222998180067,1.001888718734048,0.0002447596483080532,1.0012089978091405 +1965-11-30,1.0019237940158763,0.9926644342395888,1.0012737555408366,0.00024296419781239434,1.0022234123753309 +1965-12-31,1.0088823100303954,1.0031400596884208,1.0044779157337673,0.00024372711989567456,0.998827335359676 +1966-01-31,1.0072241849197536,1.0030154325138518,1.0037487335359676,0.0002444620625775154,1.0000510749974765 +1966-02-28,0.9845428829190056,1.0025556170670897,1.0069647723831634,0.00024508681399689446,0.9965591920609463 +1966-03-31,0.9742078627020555,1.0005930499735936,1.0046110665597434,0.00024523216272546343,0.9991926561564558 +1966-04-30,1.021336986573844,1.0087165432532905,1.0032927559369387,0.00024736973947895794,1.000106892093486 +1966-05-31,0.9483431557989048,0.9912941713789765,0.999005469915465,0.0002452161809210269,1.0050996017919362 +1966-06-30,0.9881211970570689,1.00766318282403,1.0012941762070682,0.00024709531734683514,1.0025025850069598 +1966-07-31,0.9865624323115901,1.0086434828390984,1.0006462517399086,0.00024923108148194406,1.0028519052113865 +1966-08-31,0.9182677010360139,0.9885676485039826,1.007004818917979,0.0002463817841547099,0.9971153872718304 +1966-09-30,0.9899812192723696,1.0035629859147115,1.0034533793783917,0.00024725963898129463,1.0005447394296951 +1966-10-31,1.041768339389178,0.9947317349453374,1.0012782694198623,0.000245957009665821,1.0032176175979575 +1966-11-30,1.0175501840852192,1.0026958464253002,1.0002455072179122,0.0002466200719911061,1.0037535712532522 +1966-12-31,1.004757447748013,0.9975934450405057,1.000539983309607,0.00024602656723374505,1.0034581493474632 +1967-01-31,1.0866530295590693,1.0048229877833104,0.9992149936218231,0.0002472131503618832,1.0050890012766376 +1967-02-28,1.0108036808009422,1.0052226050463773,1.0004910144358246,0.000248504247008494,1.0031074597565763 +1967-03-31,1.0445689700898777,1.00083279682869,0.999263839811543,0.0002487112005573187,1.0046395756593487 +1967-04-30,1.0425608343160377,1.0027883252312702,0.9995579784882864,0.0002494046882731321,1.0036436320754718 +1967-05-31,0.9585927786499215,0.9969769901187401,1.001572327044025,0.0002486507354360499,1.0017249607535323 +1967-06-30,1.0229355226506378,1.0019075616143613,1.0038755886970172,0.0002491250520343504,0.9988289498118555 +1967-07-31,1.0447114907285733,0.9878937593061994,1.0041049699457558,0.00024610908419156696,0.9990987297415681 +1967-08-31,0.9903404945454546,1.0053931198406931,1.0037961746240327,0.00024743637997649533,0.9993064581818183 +1967-09-30,1.0319484760522495,1.0073852460033181,1.0021818181818183,0.00024926375851279224,1.0010159651669086 +1967-10-31,0.9718775125628141,0.9829686324961886,1.0012578616352201,0.00024501845583617957,1.0026388190954774 +1967-11-30,1.0035662061329613,1.0089850965808387,1.0037205257054502,0.0002472199703259556,0.9998799210513649 +1967-12-31,1.033054040792765,1.013531234047572,1.000722091176046,0.00025056516160566994,1.002576048681932 +1968-01-31,0.9586732832463992,0.996618192741401,1.0052434096594187,0.00024971779852339985,0.9987630760396231 +1968-02-29,0.9626763822688275,1.0034102821507809,1.0039718619897593,0.0002505694066744365,0.999928422306959 +1968-03-31,1.0037470630202143,1.0144003772849155,1.002144899904671,0.0002541777006666058,1.0016515576694414 +1968-04-30,1.0909715308396148,0.9927375972057049,1.0032342449464924,0.0002523317598230371,1.0010623192528325 +1968-05-31,1.0231285188158084,1.0023733703506579,1.0040771820035082,0.00025293063654033043,1.0004211010907031 +1968-06-30,1.0090607132761706,1.0117684709448878,1.0022191793757969,0.0002559072433875273,1.0020762131348346 +1968-07-31,0.9751435403551628,1.007210916081518,1.0028267219447848,0.00025775256904424736,1.0019677158695854 +1968-08-31,1.0130833933498575,0.9997056149271085,1.004556985812271,0.0002576766905354213,0.9996446335874293 +1968-09-30,1.0421195260751936,0.9885652798055251,1.0025721367441425,0.00025473022967851047,1.0017234303573095 +1968-10-31,1.003737031191979,1.0001906700265046,1.004944491090587,0.0002547787990981548,0.9994581878945412 +1968-11-30,1.055511409395973,1.0121418411010206,1.002831414779057,0.0002578722827927135,1.0013647211293681 +1968-12-31,0.964018356683775,0.9850571280382645,1.0010182828049061,0.0002540189302884615,1.0032783788782542 +1969-01-31,0.9889133014001473,1.0082139433036967,1.0039302723447543,0.00025610542737991663,1.0013643653279294 +1969-02-28,0.9453468955677268,1.0078666002342456,1.0006908621960207,0.00025812010639493505,1.003906438992958 +1969-03-31,1.0268519363857187,0.9975673833773808,1.0042343627744281,0.0002574921991334865,1.0003640955130848 +1969-04-30,1.014136540126692,0.9971018638456864,1.0056831202163252,0.00025674595168172405,0.9996190447978855 +1969-05-31,1.0000262257127293,1.0075298934862833,1.0038736726974433,0.00025867922135092185,1.0009227528599964 +1969-06-30,0.9260949497679867,0.9851917552258751,1.007672053749773,0.00025484863612317736,0.9974475289453532 +1969-07-31,0.9311470263724436,1.0048978054068127,1.0044600621705635,0.00025609683515110033,1.0008362082884825 +1969-08-31,1.0496402792696027,1.0091713231994486,1.0021528525296017,0.000258445581996627,1.0028410311493017 +1969-09-30,0.9718027792624265,0.9878632302339346,1.0048335123523093,0.00025530888747087716,1.0013599144842331 +1969-10-31,1.0556596297614809,1.0120173869077764,1.000890789239266,0.00025837703315260865,1.0051046635813456 +1969-11-30,0.9618071687759981,0.9932846657059041,1.0056069775720895,0.0002566419450010722,0.9995952916187276 +1969-12-31,0.9741939652518143,0.9985286940605688,1.006062483405611,0.00025626434618308494,1.0003354827358697 +1970-01-31,0.9212316015419659,1.0112022620493186,1.0040906091928745,0.00025913508654292513,1.0019016120553705 +1970-02-28,1.0529872993719471,0.9918189353702483,1.004380585246189,0.00025701508565208116,1.001811479413817 +1970-03-31,0.9953170527417878,1.0027697854897362,0.999781926029309,0.0002577269623069636,1.0059193648300835 +1970-04-30,0.8890496685298322,0.9902986375379047,1.0068053919644027,0.000255226659629369,0.998206811386975 +1970-05-31,0.9343163135593221,1.0069967730099987,1.0021231422505308,0.00025701242263289584,1.0031701271186442 +1970-06-30,0.9469810955590116,1.0000796481595717,1.0008647526807333,0.0002570328931993456,1.0049309832382929 +1970-07-31,1.0708379418729819,0.9970257158016894,1.0033264212890962,0.00025626840432665677,1.0018673670613565 +1970-08-31,1.0493963217896323,1.0008126665645092,1.0008611410118406,0.0002564766650903932,1.004435039793504 +1970-09-30,1.0423570419218087,1.0046092184891302,1.004646160464616,0.00025765882207715826,1.0007503532736695 +1970-10-31,0.9778801100260139,0.9975297864731273,1.0041108208795444,0.0002570223497695452,1.000487176425434 +1970-11-30,1.0499492585648544,1.0011273292953267,1.0008102690946308,0.00025731209859399407,1.00378666268962 +1970-12-31,1.058937765120925,1.0124788876843573,1.0025140617010397,0.0002605230673721748,1.0016817103753135 +1971-01-31,1.0539815499254845,0.9951243762978286,0.9982148170187444,0.00025925285492993264,1.0055951713859912 +1971-02-28,1.0165775612661676,0.9985130749361691,1.0008090270385352,0.0002588673653620676,1.0024889593260722 +1971-03-31,1.0412760627916844,0.9873714034196678,1.0028080326752893,0.0002555982338370966,1.0001914297836232 +1971-04-30,1.0299738476488232,1.0083586571050156,1.0042002545608826,0.00025773469183038846,0.9986056022645653 +1971-05-31,0.9593924993686338,0.9932845622433929,1.0037601926570618,0.0002560038905496832,0.9991430297163061 +1971-06-30,0.9981365955842305,1.0051700066447553,1.0046721104470073,0.00025732743236490826,0.9990324102392224 +1971-07-31,0.9584173951268526,0.991440910682584,1.000712220872261,0.0002551249438874757,1.003285439169388 +1971-08-31,1.0379071267816953,0.998618699830775,1.0045214770158253,0.0002547725397593104,1.0001777194298573 +1971-09-30,0.9932372145263945,1.002297245019121,1.001875468867217,0.00025535781470728124,1.0018211156870085 +1971-10-31,0.9581448303078138,0.9943196841276655,1.0014143683181496,0.0002539073016592748,1.0022824035226188 +1971-11-30,0.9981049217744947,1.0085871519718006,1.000996967557014,0.0002560876422453728,1.0027003402913226 +1971-12-31,1.0857983972240584,1.0000851203496202,1.004606382537245,0.00025610944051501414,0.9990977734633182 +1972-01-31,1.0239623281468675,0.9795919590714202,1.003552544613351,0.00025088274857078805,0.9993497653741662 +1972-02-29,1.0249978642445072,1.0057831401155264,1.0060508767596936,0.0002523336386583413,0.9964704799312631 +1972-03-31,1.0092303401137805,1.0176466007666984,0.9996726811505258,0.0002567864696397534,1.0030283100724429 +1972-04-30,1.0062942303754965,1.002537882028125,0.9995088609667253,0.0002574381634061178,1.0033928053724253 +1972-05-31,1.014321677092277,1.0090974304940825,1.0015560378362884,0.00025978018920422925,1.0014417187947176 +1972-06-30,0.978640011448197,0.9979709716441202,0.9999591152540986,0.00025925308783403804,1.0029410049881429 +1972-07-31,0.9909672557003257,1.0044556152379327,1.0041704145882737,0.0002604082198426725,0.9989340309446255 +1972-08-31,1.0329503309913495,1.0019206535386223,1.0025651465798047,0.00026090837381159964,1.0003339966697802 +1972-09-30,0.9847214674460794,1.0030088650530669,1.0073914632660521,0.0002616934118996139,0.9960378230195526 +1972-10-31,1.0067647697566862,1.015012111236451,1.0024188671638783,0.0002656219825088973,1.0015773175145788 +1972-11-30,1.0463335538184007,0.9957679937499486,1.0032173738186205,0.0002644978686187686,1.0004810783724194 +1972-12-31,1.0065508830110277,0.9976564431715872,1.0033273201042292,0.00026387800283266644,1.0003714439827394 +1973-01-31,0.9646849504165014,1.0054534426599542,1.0072718555218156,0.00026531704639033764,0.9971488774295914 +1973-02-28,0.9472085626670861,1.0029914982298926,1.0089646965489885,0.0002661107418649747,0.9952776379933952 +1973-03-31,0.9779890500193873,0.989248586225157,1.0139172825916025,0.0002632496751692539,0.9908106087630864 +1973-04-30,0.9384658711583471,0.9942346637985785,1.0105854982551377,0.00026173195228698816,0.9946709127882439 +1973-05-31,0.9691159728575784,0.9965352430069263,1.0064842880712122,0.000260825114674991,0.9986246302226288 +1973-06-30,0.9814309245604084,1.0019997690857858,1.0081198536139067,0.0002613467046761146,0.997004469653999 +1973-07-31,1.0551640866873064,1.017181918623719,1.0015503875968992,0.0002658371424884367,1.004842105263158 +1973-08-31,0.9462215797625194,0.9667306001463677,1.0238616627652344,0.00025699290029904186,0.9835313076185559 +1973-09-30,1.0534230820252044,1.0144124866136586,1.0007375175160411,0.0002606968070344071,1.0060580145920848 +1973-10-31,0.9898111517100264,0.9934181638693607,1.0084752008254108,0.00025898094337072574,0.9980413987138265 +1973-11-30,0.8654669068376685,1.0037280242950093,1.014250219234142,0.00025994643061955623,0.9914713163772607 +1973-12-31,1.0000506584098514,0.9908907468835305,1.0122487210894155,0.0002575785127863199,0.9942220513915582 +1974-01-31,0.9875830190659436,0.9956525119112044,1.0173321944622393,0.0002564586932700517,0.9891557600139933 +1974-02-28,0.98031451668151,0.9910821959505793,1.0213048801819136,0.0002541716448966989,0.9848185586079332 +1974-03-31,0.9595848668280871,0.9896974308812007,1.0185654586558883,0.00025155302395711175,0.9872708635996771 +1974-04-30,0.9478784373956594,1.0007275502562654,1.007196663976325,0.00025173604142415606,1.0003011686143573 +1974-05-31,0.9495931537598203,0.9973613030939844,1.0114858096828048,0.0002510717863105175,0.9960594507163134 +1974-06-30,0.97195321299639,0.9975897053862942,1.0058097312999272,0.0002504666293363198,1.0001891696750904 +1974-07-31,0.9204789669340637,1.004966234283919,1.006432556613062,0.00025171050529790745,1.0005638166047086 +1974-08-31,0.8991692737968946,0.9973952658120752,1.0143807474075524,0.00025105486633929816,0.9917380653872119 +1974-09-30,0.8792332560723923,0.989318272633984,1.0124730767994343,0.0002483731667031502,0.995680796951897 +1974-10-31,1.1588045494699646,0.9969716375104358,1.006381965391332,0.0002476210027216921,0.9987261641847552 +1974-11-30,0.9518714294651237,0.9898766288310575,1.0086446239273092,0.00024511424340191467,0.9967831842352208 +1974-12-31,0.9642454181784351,0.9845061814135657,1.0086643728495466,0.0002413164877816943,0.99834992402394 +1975-01-31,1.1373505850029328,1.0081651656309403,1.004527552950662,0.0002432868768739086,1.0012667119439385 +1975-02-28,1.0579404276954363,1.001536828811441,1.0018522520297597,0.0002436607671557339,1.0024432225063937 +1975-03-31,1.0313355872105572,1.0052761564076116,0.9993837241549316,0.00024494635947364617,1.0047191841642773 +1975-04-30,1.0471811753817677,0.9935362730578393,0.99944500971233,0.00024336309309053218,1.0049577417862101 +1975-05-31,1.0539289649441383,1.0331915679240886,1.0023445935523678,0.00025144069572506287,1.002050598627312 +1975-06-30,1.0453038992788168,0.993308307847106,1.0071712166446092,0.00024975813199456127,0.9969506508984233 +1975-07-31,0.9271832920866783,0.9969921925239672,1.0125290306808459,0.00024900690761794805,0.99236660831774 +1975-08-31,0.9711535316543092,1.0035698292820647,1.005402305788616,0.000249895819768199,0.9994009305676462 +1975-09-30,0.9614013160261409,0.9952147892289859,1.0013508240026414,0.0002487000155998128,1.0039438485520717 +1975-10-31,1.0520194918535728,0.9890868041021199,1.0064452305294083,0.0002459859036097662,0.9991601822893397 +1975-11-30,1.0265333709131903,1.0073759465907024,1.0039615166949634,0.0002478002824968575,1.0001379368658398 +1975-12-31,0.9857875950190781,1.0021354245060932,1.003055835756245,0.0002483294412927181,1.001738850601911 +1976-01-31,1.1243044882773283,1.0142100453652187,1.00168594161318,0.00025185821391900706,1.0030089854130986 +1976-02-29,1.0109178776914407,0.994119885230983,0.9956298352329771,0.00025037725871564367,1.007804270715938 +1976-03-31,1.0298266412940058,1.0037780407827317,0.997449433536983,0.00025132319421013996,1.006567316841104 +1976-04-30,0.9905959747528877,1.011408985199085,0.9986917221693626,0.00025419053681307024,1.0055154936286768 +1976-05-31,0.9858094425180048,0.9915899111107992,1.0045551982851018,0.00025205277180367864,0.9991486796479062 +1976-06-30,1.0405823588441898,1.0134757714559979,1.0041492546160458,0.0002554493773513558,1.0001501224875298 +1976-07-31,0.9903584962803963,0.995576046841543,1.003777928632567,0.00025431928127159643,1.0009186009585695 +1976-08-31,0.994202673029629,1.0020517454497475,1.0043223852509628,0.0002548410796997285,0.9998781414685561 +1976-09-30,1.0206372806250363,0.9993560142074526,1.004274505211383,0.0002546769656650444,1.0001249606436942 +1976-10-31,0.9753222238342569,1.0053898361696996,1.0046936038714944,0.0002560496327861752,0.9994091692539825 +1976-11-30,1.0035523611351946,0.9967222024565706,1.0040333110872528,0.00025521035392883266,0.9999668227270102 +1976-12-31,1.0565306057814117,1.007819656591786,1.0037570082654184,0.0002572060112552243,1.000242082229644 +1977-01-31,0.9587727991746904,0.9968859272175131,1.0047218703213177,0.0002564050530160824,0.9988834021091242 +1977-02-28,0.9716811046347121,0.9941048911407744,1.0127808344795965,0.00025489351731649705,0.9908362854394206 +1977-03-31,0.9870324205420342,0.993808284687326,1.0033105087431387,0.00025331528922222716,1.0004878761386389 +1977-04-30,0.9989612347270488,1.004466953570016,1.0063453566090412,0.00025444683685775803,0.99747069274745 +1977-05-31,0.9835700231320196,0.9990643544993814,1.0055206815379443,0.0002542087648197054,0.998189314679078 +1977-06-30,1.0456919101746605,0.9926203877235478,1.0052674117220812,0.00025233280269806017,0.9987392292764071 +1977-07-31,0.9846248341074982,1.008972476016858,1.0027169392847242,0.00025459685271853505,1.0014790422472903 +1977-08-31,0.9840791441491121,0.9936099082340789,1.0027648750276488,0.0002529699554663489,1.0016306165214512 +1977-09-30,1.0002761537614275,1.0019378571638178,1.001323480754384,0.00025346017510678004,1.0029725850864633 +1977-10-31,0.9572044980800877,1.013567427204734,1.0039651944046701,0.0002568989775818404,1.0009311135490948 +1977-11-30,1.0379116837832683,1.0100645126409113,1.0068294020844761,0.00025948454058915003,0.9981830068920428 +1977-12-31,1.0035811916648578,0.9924702747962896,1.004004467569261,0.0002575306933039027,1.0008919606034294 +1978-01-31,0.9407902963503254,0.9909629361948763,1.0043683525070546,0.0002552033719967376,1.000529335170327 +1978-02-28,0.9886968052410225,1.0131171043457206,1.0020260961179996,0.0002585509012565985,1.0025686994500163 +1978-03-31,1.0258074850860062,1.0061359586059442,1.00779143750674,0.00026013735888423854,0.9975278242957653 +1978-04-30,1.074129084499576,0.9917395346788931,1.0092827907225597,0.00025798850325245095,0.9961529209075488 +1978-05-31,1.013486002469071,1.0049967361563237,1.0090913910093298,0.0002592776037345683,0.9960445693572538 +1978-06-30,0.9808226302483777,0.9998135892775268,1.007827480234299,0.000259229271609135,0.9975913732440252 +1978-07-31,1.0521110326933056,0.9959300072415749,1.004456722875238,0.000258174210350914,1.0011382044628956 +1978-08-31,1.0400536933581626,1.0033159988277387,1.0032174364296835,0.00025903031572979,1.002374922408442 +1978-09-30,0.9873057466529351,1.0104543799417738,1.004552038071591,0.00026173831706686687,1.0016404943357364 +1978-10-31,0.8813554851242205,0.9911387220819502,1.0073120494335737,0.00025941898109753473,0.9994916675186585 +1978-11-30,1.0281085688148,1.009087222697169,1.00582762498722,0.0002617763791506407,1.0011655824354544 +1978-12-31,1.0113599312297732,1.0089199884598332,1.0050823338076846,0.0002641114214317213,1.00270392394822 diff --git a/lectures/hansen_singleton_1983_data.csv.yml b/lectures/hansen_singleton_1983_data.csv.yml new file mode 100644 index 0000000..2337f09 --- /dev/null +++ b/lectures/hansen_singleton_1983_data.csv.yml @@ -0,0 +1,197 @@ +# Manifest for hansen_singleton_1983_data.csv — migrated out of +# QuantEcon/lecture-python.myst, where it sat at +# lectures/_static/lecture_specific/hansen_singleton_1983/ and was read by URL +# (PLAN Phase 8, wave B2'). +# +# READ THE `source` NOTE FIRST. These are NOT Hansen and Singleton's data. The +# paper is the specification being replicated; the bytes are a modern +# reconstruction from open sources, and the return leg is a deliberate +# substitute for the CRSP series the authors used. A reader who takes this file +# for the paper's dataset will misread every number in it. +# +# This file is a STRICT SUPERSET of hansen_singleton_1982_data.csv: identical +# index, and its `gross_real_return` and `gross_cons_growth` columns are bitwise +# identical to that file's two (verified 2026-08-13, max abs diff 0.0). They are +# kept as two files because they serve two lectures, which is the status quo +# this migration preserves; consolidating them is a rename-class decision and is +# deferred with the rest (D6). + +filename: hansen_singleton_1983_data.csv +title: Hansen-Singleton (1983) replication — monthly US returns, consumption and inflation, 1959-1978 +description: > + The five monthly series the `hansen_singleton_1983` lecture estimates on, + over the 1959:2-1978:12 sample: gross real return on the aggregate stock + market, gross growth of real per-capita nondurable consumption, gross + consumption-deflator inflation, the per-capita real consumption level itself, + and the gross real return on one-month Treasury bills. Four of the five are + gross ratios near 1, not percentages. Reconstructed from FRED and the Ken + French data library rather than taken from the paper — see the source note. + +# Constructed: every value is computed from upstream series by +# builders/hansen_singleton_1983_data.py. Nothing here is republished as +# distributed. +class: constructed + +source: + name: FRED (BEA and BLS monthly series) and the Ken French data library (F-F_Research_Data_Factors) + # The Ken French library is the single downloadable FILE the builder + # retrieves; the three FRED inputs are separate series, enumerated below. + url: https://mba.tuck.dartmouth.edu/pages/faculty/ken.french/data_library.html + doi: null # neither provider issues one for these series + version: > + Ken French research factors, zip header "This file was created using the + 202606 CRSP database. The 1-month TBill rate data until 202405 are from + Ibbotson Associates" — read 2026-08-13. That Ibbotson clause covers the + whole of this file's sample, since it ends in 1978. FRED series are + unversioned and revisable; the 1959-1978 window used here has been stable + across every check so far (see integrity.upstream). + series: > + FRED CNP16OV (civilian noninstitutional population 16+, thousands of + persons, BLS Employment Situation); + FRED DNDGRA3M086SBEA (real PCE, nondurable goods, chain-type quantity + index, 2017=100, SA, BEA Personal Income and Outlays); + FRED DNDGRG3M086SBEA (PCE, nondurable goods, chain-type price index, + 2017=100, SA, BEA); + Ken French F-F_Research_Data_Factors, monthly `Mkt-RF` and `RF`. + citation: > + Constructed by QuantEcon from U.S. Bureau of Economic Analysis and U.S. + Bureau of Labor Statistics series retrieved via FRED (Federal Reserve Bank + of St. Louis), and from the Fama-French research factors published by + Kenneth R. French, Tuck School of Business, Dartmouth College. Replicates + the data construction of Hansen, Lars Peter, and Kenneth J. Singleton. + 1983. "Stochastic Consumption, Risk Aversion, and the Temporal Behavior of + Asset Returns." Journal of Political Economy 91 (2): 249-265. + doi:10.1086/261141. + note: > + NOT the authors' data, and not a faithful copy of it. Hansen and Singleton + used CRSP value-weighted NYSE returns paired with NIPA nondurables + consumption. CRSP is a paid subscription product, so the market leg here + substitutes the Ken French value-weighted research factor (`Mkt-RF + RF`) + and the T-bill leg its `RF` column — close to but not the same series, which + is why the lecture states plainly that it matches the paper qualitatively + and not exactly. The consumption leg uses FRED's modern BEA chain-type + indices, not the 1970s NIPA vintage the authors had. The paper is the + specification being replicated; the bytes are ours. + +license: + name: null + url: https://mba.tuck.dartmouth.edu/pages/faculty/ken.french/data_library.html + # Recorded as found, per the record-and-track policy (AGENTS.md, "Licensing + # and attribution"). Split answer across the two providers, and only one half + # has a licence to name. + redistribution: permitted + verified: 2026-08-13 + note: > + The three FRED inputs are U.S. federal statistics — BEA chain-type indices + and the BLS civilian noninstitutional population — which are U.S. + Government works in the public domain; FRED is the delivery channel, not the + rights holder, and these are not among the third-party series FRED re-serves + under restriction. The Ken French data library states NO licence and NO + redistribution terms: the page carries only "Copyright (c) Eugene F. Fama + and Kenneth R. French" (checked 2026-08-13), and the factors are themselves + computed from the proprietary CRSP database with the pre-2024 T-bill leg + from Ibbotson Associates — which is the whole of this file's sample. + `permitted` therefore rests on what is actually published here rather than + on a grant: not the library file, but two of its columns converted to gross + ratios and deflated by a FRED price index over a fixed 1959-1978 window — + from which the underlying factor values cannot be recovered without the + deflator. Note this file does publish `gross_inflation_cons`, so its two + real return columns ARE invertible back to the nominal `Mkt-RF + RF` and + `RF` for those 239 months; that is the strongest form of the exposure here + and the reason it is written down rather than waved through. An inherited + exposure, served publicly from lecture-python.myst since 2026-06-26. No + permission has been sought from either provider. Registered for licence + review on QuantEcon/data-lectures#35. + +retrieved: null # the builder fetches live and no retrieval date + # was recorded upstream; not reconstructed from + # git history (AGENTS.md). It costs nothing + # here — a re-run on 2026-08-13 reproduced these + # bytes exactly, so the vintage is pinned by + # content rather than by a date. For context and + # NOT as a retrieval date: the bytes and their + # builder landed in lecture-python.myst together + # in a3e8443 (#926), 2026-06-26. +maintainer: QuantEcon + +# --------------------------------------------------------------------------- +# Integrity (PLAN Phase 7) +# --------------------------------------------------------------------------- +# Migration check (repoint gate, not a manifest field): the bytes landing here +# are byte-identical to the copy the lecture reads today — sha256 dc5c1f8d… +# against lecture-python.myst @ d2a5b21, +# lectures/_static/lecture_specific/hansen_singleton_1983/. The +# lecture-python.zh-cn copy carries the identical git blob. Verified +# 2026-08-13; a repoint to this file therefore cannot change a figure. + +integrity: + sha256: dc5c1f8dac4b50abb3613dd297f6c22b4ebde80b9a7f02971e8d986b0c6eff18 + upstream: + # `verified` in the sense AGENTS.md defines for a constructed dataset — + # "re-run the builder and compare". Run twice on the date below: once + # unmodified as it stood in lecture-python.myst, and once after the + # migration edits (rename, new write target, added validate()), both + # reproducing this file byte for byte. The second run is what licenses the + # `committed` status below rather than `committed-frozen`. + status: verified + date: 2026-08-13 + against: builders/hansen_singleton_1983_data.py + note: > + Both live upstreams answered and neither had revised the 1959-1978 + window. The builder's own docstring in lecture-python.myst warned that + "re-running this script may produce small differences from the committed + data"; that is a real risk for a FRED-sourced series and it has not + materialised — the migrated builder records what was measured instead of + repeating the hedge. A future divergence is what validate() and this + sha256 exist to catch. + +# --------------------------------------------------------------------------- +# Shape +# --------------------------------------------------------------------------- +# Measured from the committed bytes 2026-08-13, and asserted by the builder's +# validate() stage on every run. + +schema: + format: csv + columns: + - {name: date, dtype: datetime64, description: "month-end stamp, 1959-02-28 to 1978-12-31 on an unbroken monthly grid — the CSV index column"} + - {name: gross_real_return, dtype: float64, description: "gross real monthly return on the aggregate stock market: (1 + (Mkt-RF + RF)/100) divided by gross_inflation_cons. A ratio near 1 (observed 0.865-1.159), not a percentage and not a net return"} + - {name: gross_cons_growth, dtype: float64, description: "gross monthly growth of real per-capita nondurable consumption — the ratio of consecutive months of consumption_per_capita. Observed 0.967-1.033"} + - {name: gross_inflation_cons, dtype: float64, description: "gross month-over-month inflation of the PCE nondurables price deflator (DNDGRG3M086SBEA); the deflator applied to both nominal return legs. Observed 0.995-1.024"} + - {name: consumption_per_capita, dtype: float64, description: "real per-capita nondurable consumption, DNDGRA3M086SBEA / CNP16OV. A LEVEL, and the one column with no interpretable unit: it divides a chain-type index (2017=100) by a population in thousands, so it lands near 2.4e-4 and is meaningful only through its growth rate. Observed 2.14e-4 to 2.66e-4"} + - {name: gross_real_tbill, dtype: float64, description: "gross real monthly return on one-month Treasury bills: (1 + RF/100) divided by gross_inflation_cons. Observed 0.984-1.008. Exceeded by gross_real_return in only 54% of months, though the mean excess is positive"} + row_count_floor: 239 # exact by design, not a floor with headroom: + # the sample is the paper's 1959:2-1978:12 and + # does not grow. validate() asserts equality. + date_range: {start: 1959-02-28, end: 1978-12-31} + known_nulls: {} # genuinely none — a complete monthly panel + +# --------------------------------------------------------------------------- +# Consumers — how a correction knows what to rebuild +# --------------------------------------------------------------------------- +# Empty at landing and filled at the flip, per the B1' convention: no lecture +# reads this file from here yet. Two will — +# lecture-python.myst/lectures/hansen_singleton_1983.md and its +# lecture-python.zh-cn twin. zh-cn is outside the audit's SCAN_REPOS, so it is +# recorded here or nowhere. The generated lecture-python.notebooks mirror is a +# third reference-holder; it self-heals after a publish tag and is not a repo +# anyone repoints, so it will deliberately not be listed (wave A4 precedent). +# +# Both reads build the URL by string concatenation across three source lines and +# must be COLLAPSED at the repoint rather than patched: editing the stem line +# alone leaves the filename appended to a path that no longer exists, since the +# flat published tree has no _static/lecture_specific// segment. +# Note for any future sweep: a line-based grep for the whole old URL returns a +# confident ZERO against the wrapped form. Grep the filename separately. +# +# Each lecture ALSO carries two references to the BUILDER — a prose link, and +# the path named again in a docstring inside the data cell. Neither is a code +# read; both break silently when the builder moves here. +# +# A correction to this file must be mirrored into hansen_singleton_1982_data.csv +# whenever it touches gross_real_return or gross_cons_growth: those two columns +# are that file, bitwise. +consumers: [] + +builder: builders/hansen_singleton_1983_data.py +builder_status: committed diff --git a/migration.yml b/migration.yml index 11b5c37..43087fe 100644 --- a/migration.yml +++ b/migration.yml @@ -653,6 +653,49 @@ datasets: date: 2026-08-13 cutover: null + # -------------------------------------------------------------------------- + # Wave B2' — the lecture-python.myst block, part two (Track B). Two files, + # two lectures, one URL form. + # + # The first wave in this migration to move BUILDERS as well as data. Both + # were named make_data.py beside their output and collide in the flat + # builders/ tree, so each takes its dataset's stem. That rename is what makes + # this wave more coupled than B1' was: each lecture carries FIVE references, + # not one — a prose link to the builder, three lines of a concatenated data + # URL, and the builder's path named again in a docstring inside the data + # cell. Only the URL is a code read; the other two break silently. + # + # Both carry `prior_pattern: own-repo`. Consumers are recorded at the flip, + # and include lecture-python.zh-cn, which is OUTSIDE the audit's SCAN_REPOS + # and so is recorded in the manifests or nowhere. lecture-stats holds no + # reference to either file (swept 2026-08-13, exit 1) — unlike B1', it is not + # a consumer here. + # + # 1982 is a bitwise strict subset of 1983 on an identical index. Recorded in + # both manifests; consolidating them is a rename-class decision, deferred + # with the rest under D6. + # -------------------------------------------------------------------------- + + hansen_singleton_1982_data.csv: + pilot: B2 + status: landed + prior_pattern: own-repo + landed: + pr: QuantEcon/data-lectures#82 + date: 2026-08-13 + repoints: [] + cutover: null + + hansen_singleton_1983_data.csv: + pilot: B2 + status: landed + prior_pattern: own-repo + landed: + pr: QuantEcon/data-lectures#82 + date: 2026-08-13 + repoints: [] + cutover: null + # Planned waves that have not landed anything here yet. `datasets` names the # files as the audit sees them today, so the dashboard can join the two views. # `title` is the reader-facing milestone name (the dashboard is read by people diff --git a/scripts/audit_annotations.yml b/scripts/audit_annotations.yml index 8bbdb7a..19ada0c 100644 --- a/scripts/audit_annotations.yml +++ b/scripts/audit_annotations.yml @@ -71,14 +71,6 @@ datasets: description: Hansen–Jagannathan (1991) asset-returns bundle provenance: constructed-lost note: lecture documents 3 sources (FRED yields deflated by CPIAUCSL, …); no build script - hansen_singleton_1982_data.csv: - description: FRED-derived consumption/returns snapshot - provenance: constructed-committed - note: make_data.py + README beside the data (exact series and sample documented) - hansen_singleton_1983_data.csv: - description: FRED-derived snapshot - provenance: constructed-committed - note: make_data.py + README beside the data longprices.xls: description: Long-run price levels (Sargent–Velde) provenance: author-assembled diff --git a/sources/README.md b/sources/README.md index 2a2ba0c..d9c464d 100644 --- a/sources/README.md +++ b/sources/README.md @@ -11,7 +11,8 @@ The defining property is **un-refetchability**, not size. The normal case for a builder is to fetch from its third-party upstream at run time, and that is what most of them do: `jse.amstat.org`, `earthquake.usgs.gov`, -`wwwn.cdc.gov`, `stat.go.jp`, `openfootball`. **Six of the seven `committed` +`wwwn.cdc.gov`, `stat.go.jp`, `openfootball`, `fred.stlouisfed.org`, +`mba.tuck.dartmouth.edu`. **Eight of the nine `committed` builders in this repo fetch at run time and have no committed input**; the exception is `NEWQDATA.py`, whose upstream is not published anywhere. So `sources/` is not "where builder inputs live" as a general rule — it is the From 883d8c163078a1c02f127976a43c4742d87e6cd7 Mon Sep 17 00:00:00 2001 From: Matt McKay Date: Thu, 13 Aug 2026 17:15:47 +1000 Subject: [PATCH 2/2] builders/README.md: make the coverage numbers independently checkable MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The coverage sentence said "21 constructed datasets and 12 builders" over a table listing 13, and claimed their difference was the Phase 9 backlog. Both halves misled. The 13th is business_cycle.py, which no manifest references — its own table row already says its three outputs are the repo's only unmanifested files, but the reader had to connect that themselves. The arithmetic was worse: 21 - 12 = 9, while the backlog is 7. Two builders each produce two datasets (generating_mini.md writes both SCF files, webscrape_forbes.ipynb both Forbes files), so 14 constructed datasets ship a builder but only 12 distinct files carry them. That slip predates this wave — at 19 and 10 the same sentence was off by the same two — and is fixed here rather than left reading as freshly verified. Every number now stands on its own, so the next refresh cannot reintroduce it. Raised by Copilot on #82. Co-Authored-By: Claude Opus 5 (1M context) --- builders/README.md | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/builders/README.md b/builders/README.md index a64b962..6656647 100644 --- a/builders/README.md +++ b/builders/README.md @@ -67,10 +67,15 @@ here instead of patched — fixing it would mean this file is no longer the thin that produced those bytes. The fix belongs in `lecture-python-intro`, which still serves that notebook to readers. -**This listing is the coverage report.** The repo has 21 `constructed` datasets -and 12 builders; the difference is the Phase 9 recovery backlog, carried as -`builder_status: unrecovered` in each manifest rather than hidden by -reclassifying the file as `verbatim`. +**This listing is the coverage report.** The repo has 21 `constructed` datasets. +Fourteen ship a builder (9 `committed`, 5 `committed-frozen`), carried by **12** +distinct builder files — fewer than the datasets because `generating_mini.md` +and `webscrape_forbes.ipynb` each produce two. The remaining **7** have none: +they carry `builder_status: unrecovered` in their manifests, which is the Phase +9 recovery backlog, kept visible rather than hidden by reclassifying the file as +`verbatim`. The table above lists a **13th** builder, `business_cycle.py`, which +no manifest references — its three outputs are the repo's only unmanifested +files. Repo tooling — the audit dashboard and the catalog generator — lives in `scripts/` and is not a builder.