fix(security)!: harden artifact serialization and loading - #2339
Open
XianBW (XianBW) wants to merge 28 commits into
Open
XianBW (XianBW) wants to merge 28 commits into
XianBW (XianBW) wants to merge 28 commits into
Conversation
XianBW (XianBW)
changed the title
Collaborator
Author
|
Local verification update:
The branch also contains minimal CI compatibility fixes for current Node/commitlint, Plotly, MLflow, and Windows/Python 3.8 OSQP behavior. |
Collaborator
Author
|
Final verification: all 76 GitHub checks pass on commit |
Allow the exact MaskedArray and SparseDtype module paths emitted by the older dependency versions in CI. Preserve restricted loading and cover both old and new paths. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
PIT normalization uses report dates and calendar-day offsets, so do not request an unused Baostock calendar. Cover annual and quarterly normalization without online calendar access. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Expose default-off artifact trust at online, delayed-training and meta-workflow entry points. Keep data loading restricted, preserve legacy default recorder calls and restored workflow defaults, and document migration and trust boundaries. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Exercise real training, cache and recorder restoration, saved-task replay, signals and portfolio backtests. Fix the cache trust, grouping, relative-path, LightGBM early-stopping and HIST CPU blockers exposed by full execution. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Use trusted consistently across public APIs and CLI options. Migrate pre-release saved consent only at serialization boundaries, reject invalid or conflicting settings, and preserve restricted data reads. Add a canonical upgrade guide with README, installation, changelog and runtime-error entry points. BREAKING CHANGE: executable artifact restoration requires explicit trusted=True. HIST mappings use JSON and high-frequency cache paths must remain within artifact_root. See docs/start/artifact_migration.rst for upgrading existing workflows. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
XianBW (XianBW)
changed the title
Target upgrades from pre-PR Qlib rather than artifacts generated by intermediate PR revisions. Retain restricted defaults for saved objects without consent, unify current APIs, and remove state/keyword migration branches and documentation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Download the same existing v2 archives used by successful jobs without probing nonexistent package-version assets. Allow five download attempts 60 seconds apart; preserve the full matrix, datasets, noninteractive setup and failure propagation. Cover the exact archives and retry policy in offline CI tests. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Show explicit consent for restored online components while preserving the original manager, schema-aware Arrow conversions, timezone-preserving ZoneInfo migration, and restricted round trips into a separate recorder. Explain shared-component scope, staging artifacts and trusted-source prerequisites. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Keep immutable cache references intact at open rolling tails. Add explicit per-load record-template consent and per-artifact collector loading options without widening other reads. Restore the original template class after parent-load failures, and support audited business-frequency and NumPy record representations through exact pickle globals. Document the scoped APIs and cover fresh trained-model reads, cache generation, numerical round trips and refusal paths. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
trustedopt-ins for workflows and record templates, plus per-artifact collector consent. Automatic prediction/label reads remain restricted.HighFreqProvider's configured and generated artifact paths toartifact_root.Upgrade notes
Breaking change: Reloading saved models, datasets and tasks containing executable objects requires explicit
trusted=True(--trusted=Truein example CLIs), after verifying their source and storage. Previously saved workflow components without consent default to restricted loading.HIST's
stock_indexmust use JSON, and high-frequency artifact paths must remain within the configured root. The migration guide explains component-level authorization and data-format compatibility, with examples for restored managers, Arrow columns and timezones.