Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion PLAN.md
Original file line number Diff line number Diff line change
Expand Up @@ -296,7 +296,7 @@ Only one file genuinely forces LFS, and it is not a dataset:
Full automation:

- [x] Audit dashboard workflow ([#20](https://github.com/QuantEcon/data-lectures/issues/20), added 2026-07-17): `.github/workflows/audit-dashboard.yml` rebuilds the full-universe data audit + migration tracker from the 8 lecture repos' `main` (push to main / weekly / dispatch) and deploys it with the published tree to Pages. Strict mode fails the build on an unannotated data reference or a `migration.yml` status the scan contradicts
- [ ] PR validation: manifest schema check + per-dataset invariant tests (expected columns/dtypes, row-count floor, date-range recency, no all-NaN columns, overlap-window agreement with the previous vintage) on every PR touching data. The schema decisions these tests force — column patterns for wide files, `known_nulls` exact-vs-ceiling, a canonical dtype vocabulary — are researched in [#14](https://github.com/QuantEcon/data-lectures/issues/14)
- [ ] PR validation: manifest schema check + per-dataset invariant tests (expected columns/dtypes, row-count floor, date-range recency, no all-NaN columns, overlap-window agreement with the previous vintage) on every PR touching data. The schema decisions these tests force — column patterns for wide files, `known_nulls` exact-vs-ceiling, a canonical dtype vocabulary — have their own issues under the [#14](https://github.com/QuantEcon/data-lectures/issues/14) tracker: [#120](https://github.com/QuantEcon/data-lectures/issues/120) column patterns, [#121](https://github.com/QuantEcon/data-lectures/issues/121) `known_nulls`, [#122](https://github.com/QuantEcon/data-lectures/issues/122) dtype vocabulary; the shared manifest-driven `validate()` and this workflow are [#119](https://github.com/QuantEcon/data-lectures/issues/119)
- [x] Retrofit `builders/business_cycle.py` to the four-stage builder contract — **done 2026-09-01**, with the two provenance dumps moved out of the published tree to `provenance/` ([#13](https://github.com/QuantEcon/data-lectures/issues/13)). Its `validate()` is the first to face a *revised* upstream: it bounds the overlap window (5 pp) and prints the revision summary rather than asserting equality, which is the review surface the refresh-as-PR workflow below will use. It previously had fetch/transform/write but no validate stage. Builder architecture and a copy-able template: [#14](https://github.com/QuantEcon/data-lectures/issues/14)
- [x] Scheduled refresh workflow for dynamic datasets — **landed 2026-09-01** as `.github/workflows/refresh-snapshots.yml`, manifest-driven rather than cron-per-class: a weekly run asks `scripts/snapshots.py due` which `dynamic-snapshot` datasets have their cadence elapsed (or are `diverged`, or were never refreshed), runs each builder in place, stamps the manifest (`retrieved`, `sha256`, `integrity.upstream: verified`, `date_range.end`), regenerates the catalog, and opens a PR on `refresh/<stem>` whose body is the builder's overlap summary. Nothing auto-merges; the first consumer is `business_cycle_data.csv`, not UNRATE — the pilot's order inverted once the World Bank file turned out to be the one already here
- [x] Weekly sources-alive canary: fetch + validate, no commit, opens an issue on failure — **landed 2026-09-01** as the `canary` job of the same workflow: every dynamic snapshot's builder runs with `--out-dir`, and a failure opens or updates one `upstream-break` issue classified by exit code (2 = the data broke the contract, a human; anything else = the fetch, a retry). Covers the live APIs only as their snapshot twins land here — the 23 live-API lectures without a twin are still guarded by nothing but their own CI
Expand Down
Loading