Flip the six high_dim_data datasets to repointed - #69
Merged
Conversation
All 28 consuming reads are on main across the four repos, so the P3 wave is complete and the tracker should say so. This is the last push of the fold and the same-day close of the window PLAN describes as red by design. Verified locally against the real repos before pushing, both directions: migration.yml at `landed` -> exit 1, 6 warnings migration.yml at `repointed` -> exit 0, 0 warnings The six warnings are "marked landed but some consumer already reads data-lectures" -- the pre-existing assertion at build_audit.py:444. A non-zero strict audit also skips the Pages deploy, and Pages serves lectures/ as well as the dashboard, so the served data tree would have gone stale alongside. That window opened when the last C2 PR merged and closes here. consumed-file-check: 24 manifests, 25 files hash-checked, 0 errors. Records all FOUR repoint PRs, not the usual two: lecture-python-intro #832 lecture-wasm #60 lecture-intro.zh-cn #292 test-actions-lecture-intro #53 and populates each manifest's `consumers` with all four as well. This departs from the 18 existing records, which name only the two repos the audit scans. The departure is deliberate: zh-cn and the canary are real consumers that no CI can see, so this file is the only place they are written down at all, and the habit of recording only the scanned repos is what produced the undercount three times (ws#28, rule 6's enumeration table, and the "in 2020" prose count). Safe to record: build_audit derives its own consumer list from the scan (:403) and asserts on that, never on the manifest's; check_consumed_files uses `consumers` only to require existence and a hash, both already true. Confirmed by the green runs above. The other 18 manifests undercount the same way and want a separate backfill. Part of QuantEcon/workspace-lectures#23. Data: #62 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Updates the migration tracker and dataset manifests to reflect that the P3 “high_dim_data fold” repoint wave is complete, moving the six affected datasets from landed to repointed and recording the known consumers/repoint PRs.
Changes:
- Flip six P3 datasets in
migration.ymlfromstatus: landedtostatus: repointed, recording repoint PR references for four consuming repos. - Populate each affected dataset manifest’s
consumerslist with the four known consuming repos and lecture file paths.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| migration.yml | Marks the six P3 fold datasets as repointed and records repoint PR provenance across four repos. |
| lectures/SCF_plus_mini.csv.yml | Replaces empty consumers: [] with four recorded consumers for the repointed SCF mini dataset. |
| lectures/SCF_plus_mini_no_weights.csv.yml | Replaces empty consumers: [] with four recorded consumers for the repointed no-weights extract. |
| lectures/forbes-global2000.csv.yml | Replaces empty consumers: [] with four recorded consumers for the repointed Forbes Global 2000 dataset. |
| lectures/forbes-billionaires.csv.yml | Replaces empty consumers: [] with four recorded consumers for the repointed Forbes billionaires dataset. |
| lectures/cities_us.csv.yml | Replaces empty consumers: [] with four recorded consumers for the repointed US cities dataset. |
| lectures/cities_brazil.csv.yml | Replaces empty consumers: [] with four recorded consumers for the repointed Brazil cities dataset. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
mmcky
added a commit
that referenced
this pull request
Aug 11, 2026
…ete (#70) * Retire the dashboard's high_dim_data references now the fold is complete Every consumer was repointed and migration.yml flipped in #69, so the dashboard was the last place still describing a repo the lecture family no longer reads. It named it once on the overview page. Three changes, none of them a plain deletion: - audit_annotations.yml loses the six entries for the folded datasets. Annotations are curated judgment for refs that are NOT yet migrated; all six now carry full manifests, which is the authoritative record. The strict audit fails a ref with neither, so this is only safe *because* the manifests landed first. - render_audit.py's `external` pattern label is generalised rather than removed. It reads "QuantEcon/high_dim_data via raw and media (LFS) hosts", which describes one retired repo, but the entry is still live: migration.yml records `prior_pattern: external` for all six and render_audit.py:610 looks the label up to render it. Deleting it would have degraded those rows to a raw enum string. - The branch-pin finding is restated as history. It described SCF_plus_mini_no_weights.csv as "now reads main" on a repo we have since stopped reading entirely; it now says the file was folded in and every consumer repointed, so the pin is doubly retired. Verified locally: scan --strict exit 0 with zero warnings, render exit 0, and grep for high_dim_data across the three generated pages returns 0, down from 1. Close-out of QuantEcon/workspace-lectures#23 step 3. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Describe `external` by what it excludes, not by whose repo it was Copilot caught that the generalisation in 2dcfa6f was still wrong, just one level up. classify_url reaches `external` by TWO routes: a non-QuantEcon org (build_audit.py:180-181) and a QuantEcon repo that is none of this repo, the consuming repo, a legacy name or a scanned sibling (:190-191). "another QuantEcon repo" describes only the second, which is the branch high_dim_data happened to take. The pattern is defined by what it excludes, so the label now says that: a GitHub repo outside the audited lecture set, any org. Added a comment naming both branches so the next edit does not have to re-derive the definition from classify_url -- this string has now been too narrow twice, first naming one repo and then one org. Note the description is currently dormant: PATTERN_META's label (index 1) renders 30 times in migration.html for `prior_pattern`, but the description (index 2) only surfaces when a live ref classifies external, and none does today. A wrong one would have sat unnoticed until the next external read appeared, which is the argument for fixing it rather than deferring. scan --strict exit 0 / 0 warnings; render exit 0; high_dim_data still 0 across the generated pages. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This was referenced Aug 11, 2026
mmcky
added a commit
that referenced
this pull request
Aug 11, 2026
`scripts/build_catalog.py` has asserted in its docstring since it was written that "CI asserts the catalog is current with `git diff --exit-code CATALOG.md`". Nothing did. No workflow ran the generator at all — only `audit-dashboard.yml` and `consumed-file-check.yml` exist, and neither touched it. So #69 could populate `consumers` in the six P3 manifests without regenerating, and `main` has since carried a catalog that contradicts both the manifests and the live dashboard: the header still reads "18 read by lectures today, 6 awaiting repoint" and the six fold rows still show "Used by: —". Regenerating produces exactly 7 insertions / 7 deletions. Two changes: - **Regenerate.** The header becomes "24 read by lectures today" and the six rows gain their four consumers each, matching the manifests, `migration.yml` and the dashboard. - **Add the gate the docstring promised**, in `consumed-file-check.yml` — the job that already parses these manifests and already installs pyyaml. It regenerates and runs `git diff --exit-code -- CATALOG.md` with a legible error, so a manifest edit can no longer drift the catalog silently. Exercised both directions before landing: clean tree exits 0, a one-word manifest edit exits 1 with the error. The docstring now names the workflow that enforces it rather than asserting a gate in the abstract. A docstring is not a gate. Closes #71
mmcky
added a commit
that referenced
this pull request
Aug 11, 2026
Seven statements in PLAN.md and one comment in migration.yml still described the six high_dim_data files as unrepointed. They were true when written on 2026-08-10 and false by the following afternoon, when PR set C landed and #69 flipped the records. PLAN.md's own line 9 is the diagnosis: "every figure below that restates one is a copy that can drift." Figures re-read from audit.json (2026-08-11), not carried forward: - the headline: 18 of 41 migrated / 23 to go / 5 URL forms -> **24 of 41 / 17 to go / 4 URL forms** - rule 2: "8 of the 23 remaining datasets have two consuming repos" -> **2 of 17**, both intro + wasm, both in step 4 - Track A: "17, of which 9 remain" -> **3 remain**; usa-gini's SCF dependency is discharged - Phase 6: 21 non-.yml files / 18 manifests -> **27 / 24** - Phase 9: 23 datasets to repoint -> **17** - the CATALOG-vs-`migrated` paragraph explained a six-file gap that has since closed; it now explains when the gap appears rather than asserting one Two boxes were unchecked for work that had landed — the 28 consuming repoints (Phase 3) and P3 itself (Phase 8). Both are ticked, with the four repoint PRs named, and P3 gains the record of its consumer half: the C0 -> C1 -> C2 ordering, the flip as its acceptance test (dry-run `landed` -> exit 1 with 6 warnings, `repointed` -> exit 0), and the independent validation against QuantEcon/workspace-lectures#36. Two things worth more than the arithmetic: **Rule 6's tables are now labelled as the pre-fold record.** The "Current host" column said `media` for four `data.ipynb:37` rows that have read raw.githubusercontent since 2026-08-11 — a table titled "current" that describes a state no repo is in is worse than a stale number, because it reads as a description of today. It is the worked example now, not open work. **Rule 2 gains the distinction the audit cannot make.** "Two consuming repos" is `SCAN_REPOS`' count, not the consumer set: both remaining pairs have five reference-holders each, and three of the five are invisible to every audit run. That gap is exactly what rule 1 exists for, and rule 2 previously read as though the dashboard's number were the whole answer. P3's "three things it proved" list is now five, adding the two findings that outlived the step: the `# i18n` markers were never exercised, so the hand-diff is still what protects a localisation; and the translation sync is `.md`-only, so no hand-localised `_static` asset can be carried by it (QuantEcon/action-translation#271). Also drops migration.yml's verbatim quote of a render_audit.py label that #70 deleted — a quoted string that outlives its source is an unfindable false record. Strict audit green.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
All 28 consuming reads are on
mainacross the four repos, so the P3 wave is complete and the tracker should say so. This is the last push of the fold and the same-day close of the windowPLAN.mddescribes as red by design.This PR is the fold's acceptance test
The strict audit reads each consuming repo's
main, so it is the only thing that checks all the repoints together against reality rather than file by file. Run locally against the live repos before pushing, in both directions:migration.ymlstatelanded(before this PR)repointed(this PR)The six warnings are
marked landed but some consumer already reads data-lectures— the pre-existing assertion atbuild_audit.py:444. So the red window was real rather than theoretical: it opened the moment the last C2 PR merged, and closes here. A non-zero strict audit also skips the Pages deploy, and Pages serveslectures/as well as the dashboard, so the served data tree would have gone stale alongside the dashboard.Post-flip scan: 41 static files, 26 orphans, 22 live-API lectures, zero warnings.
check_consumed_files.py: 24 manifests, 25 files hash-checked, 0 errors.It records four repoint PRs, not two
lecture-python-introlecture-wasmlecture-intro.zh-cntest-actions-lecture-introEach manifest's
consumersblock gets all four too. This departs from the 18 existing records, which name only the two repos the audit scans, so it is worth being explicit about why.lecture-intro.zh-cnandtest-actions-lecture-introare real consumers that no CI anywhere can see — the first is excluded fromSCAN_REPOSby decision, the second is not a Python-family repo and was never in scope for it. This file is therefore the only place they are written down at all. Recording only the scanned repos is the habit that produced the same undercount three separate times in this programme: QuantEcon/workspace-lectures#28 (two published sites left serving 404s), repoint rule 6's enumeration table (corrected in #68), and the "in 2020" prose count (QuantEcon/workspace-lectures#35).Safe to record — confirmed by the green runs above rather than assumed.
build_auditderives its own consumer list from the scan (:403) and asserts on that, never on the manifest's;check_consumed_filesusesconsumersonly to require existence and a recorded hash, both already true for all six.The other 18 manifests undercount the same way —
lecture-intro.zh-cnreads thefrench_revset, for instance — and want a separate backfill rather than being quietly fixed here.Not in this PR
The close-out still has the stale
scripts/audit_annotations.ymlentries and the twohigh_dim_datastrings inscripts/render_audit.py. Nothing fails the build with them present, so they are kept out of the time-sensitive flip; the dashboard will keep describing a retired repo until they go. Archivinghigh_dim_dataitself comes after intro republishes. Both tracked on QuantEcon/workspace-lectures#23.🤖 Generated with Claude Code