fix(security)!: harden artifact serialization and loading

.github/ci/README.md

@@ -76,8 +76,17 @@ selection, add caches, or reduce test data sizes.
 
 ## Network and workflow retries
 
-Dataset commands have at most three attempts, with a 15-minute timeout per
-attempt. `--delete_old False` makes retries noninteractive in these fresh CI
+Dataset commands name the existing `v2` release archives explicitly: source
+jobs use `qlib_data_simple_cn_1d_latest.zip`, and PyPI jobs use
+`qlib_data_cn_1d_latest.zip`. These are the same datasets selected by successful
+version-probing downloads, not smaller substitutes. Naming the archive avoids
+an unnecessary request for a nonexistent package-version asset: the downloader
+can otherwise mistake an HTTP 504 response for confirmation that the file exists.
+
+Download steps have at most five attempts, with a 15-minute timeout per
+attempt and a 60-second pause between failures, allowing transient GitHub
+download outages more time to recover than three closely spaced attempts.
+`--delete_old False` makes retries noninteractive in these fresh CI
 directories, including when an earlier attempt already extracted one dataset.
 Bash command blocks use `set -euo pipefail`, so an earlier failure cannot be
 hidden by a later successful command. Exhausted retries still fail the job.

.github/workflows/test_qlib_from_pip.yml

@@ -64,13 +64,13 @@ jobs:
       uses: nick-fields/retry@ad984534de44a9489a53aefd81eb77f87c70dc60 # v4
       with:
         timeout_minutes: 15
-        max_attempts: 3
-        retry_wait_seconds: 10
+        max_attempts: 5
+        retry_wait_seconds: 60
         shell: bash
         command: |
           set -euo pipefail
           cd "$RUNNER_TEMP"
-          python -m qlib.cli.data qlib_data --target_dir ~/.qlib/qlib_data/cn_data --region cn --delete_old False
+          python -m qlib.cli.data download_data --file_name v2/qlib_data_cn_1d_latest.zip --target_dir ~/.qlib/qlib_data/cn_data --delete_old False
 
     - name: Test workflow by config
       uses: nick-fields/retry@ad984534de44a9489a53aefd81eb77f87c70dc60 # v4

.github/workflows/test_qlib_from_source.yml

@@ -108,12 +108,12 @@ jobs:
       uses: nick-fields/retry@ad984534de44a9489a53aefd81eb77f87c70dc60 # v4
       with:
         timeout_minutes: 15
-        max_attempts: 3
-        retry_wait_seconds: 10
+        max_attempts: 5
+        retry_wait_seconds: 60
         shell: bash
         command: |
           set -euo pipefail
-          python scripts/get_data.py qlib_data --name qlib_data_simple --target_dir ~/.qlib/qlib_data/cn_data --interval 1d --region cn --delete_old False
+          python scripts/get_data.py download_data --file_name v2/qlib_data_simple_cn_1d_latest.zip --target_dir ~/.qlib/qlib_data/cn_data --delete_old False
           python scripts/get_data.py download_data --file_name rl_data.zip --target_dir tests/.data/rl --delete_old False
 
     - name: Check Qlib ipynb with nbconvert

.github/workflows/test_qlib_from_source_slow.yml

@@ -78,10 +78,10 @@ jobs:
       uses: nick-fields/retry@ad984534de44a9489a53aefd81eb77f87c70dc60 # v4
       with:
         timeout_minutes: 15
-        max_attempts: 3
-        retry_wait_seconds: 10
+        max_attempts: 5
+        retry_wait_seconds: 60
         shell: bash
-        command: python scripts/get_data.py qlib_data --name qlib_data_simple --target_dir ~/.qlib/qlib_data/cn_data --interval 1d --region cn --delete_old False
+        command: python scripts/get_data.py download_data --file_name v2/qlib_data_simple_cn_1d_latest.zip --target_dir ~/.qlib/qlib_data/cn_data --delete_old False
 
     - name: Unit tests with Pytest
       uses: nick-fields/retry@ad984534de44a9489a53aefd81eb77f87c70dc60 # v4

CHANGELOG.md

@@ -0,0 +1,13 @@
+# Changelog
+
+## Unreleased
+
+- **BREAKING:** New source builds restrict recorder artifact loading by default.
+  Reloading executable artifacts requires verified source/storage and explicit
+  `trusted=True` (CLI: `--trusted=True`). Supported data-only reads and fresh
+  in-memory training need no opt-in. See the
+  [artifact loading migration guide](https://qlib.readthedocs.io/en/latest/start/artifact_migration.html)
+  for workflow, HIST and high-frequency cache upgrades.
+- Merging into `main` affects source installs before a PyPI release. These changes
+  remain unreleased until included in a tagged release; its versioned upgrade notes
+  should link to the same guide.

CHANGES.rst

@@ -2,6 +2,18 @@ Changelog
 =========
 Here you can see the full list of changes between each QLib release.
 
+Unreleased
+----------
+- **BREAKING:** New source builds restrict recorder artifact loading by default.
+  Reloading executable artifacts requires verified source/storage and explicit
+  ``trusted=True`` (CLI: ``--trusted=True``); supported data-only reads and fresh
+  in-memory training need no opt-in. See the
+  `artifact loading migration guide <https://qlib.readthedocs.io/en/latest/start/artifact_migration.html>`_
+  for workflow, HIST and high-frequency cache upgrades.
+- Merging into ``main`` affects source installs before a PyPI release. These
+  changes remain unreleased until included in a tagged release; its versioned
+  upgrade notes should link to the same guide.
+
 Version 0.1.0
 -------------
 This is the initial release of QLib library.

README.md

@@ -7,6 +7,20 @@
 [![License](https://img.shields.io/pypi/l/pyqlib)](LICENSE)
 [![Join the chat at https://gitter.im/Microsoft/qlib](https://badges.gitter.im/Microsoft/qlib.svg)](https://gitter.im/Microsoft/qlib?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
 
+## Unreleased breaking change: artifact loading
+
+New source builds introduce a **backward-incompatible safety default**: recorder
+loads are restricted. If you reload saved models/datasets, resume online or delayed
+training, or use DDG-DA, verify the artifact writer and storage permissions before
+opting in with `trusted=True` (CLI: `--trusted=True`). Fresh in-memory training and
+supported data-only reads need no opt-in. HIST mappings and high-frequency cache
+paths also have migration requirements.
+
+Follow the [artifact loading migration guide](https://qlib.readthedocs.io/en/latest/start/artifact_migration.html)
+before upgrading. Merging into `main` affects source installs before a new PyPI
+release; this change remains **unreleased until the first tagged release containing
+it**. That release's versioned upgrade notes should link to the same guide.
+
 ## :newspaper: **What's NEW!**     :sparkling_heart: 
 
 Recent released features

docs/advanced/serial.rst

@@ -23,6 +23,13 @@ Example
 ``Qlib``'s serializable class includes  ``DataHandler``, ``DataSet``, ``Processor`` and ``Model``, etc., which are subclass of  ``qlib.utils.serial.Serializable``.
 Specifically, ``qlib.data.dataset.DatasetH`` is one of them. Users can serialize ``DatasetH`` as follows.
 
+.. warning::
+
+    The direct local ``pickle.load`` below can execute code and must only read a
+    file from an independently trusted source and store. Recorder/workflow
+    ``trusted`` settings do not authorize or protect this API. For the separate
+    recorder loading policy and upgrade steps, see :ref:`artifact_loading_migration`.
+
 .. code-block:: Python
 
     ##=============dump dataset=============

docs/advanced/task_management.rst

@@ -79,6 +79,17 @@ Meanwhile, ``Qlib`` provides a module called ``Trainer``.
 If you do not want to use ``Task Manager`` to manage tasks, then use TrainerR to train a list of tasks generated by ``TaskGen`` is enough.
 `Here <../reference/api.html#Trainer>`_ are the details about different ``Trainer``.
 
+Delayed training reloads saved task configurations from recorders. For tasks
+containing executable objects such as reweighters, first verify their writer and
+artifact store, then configure
+``DelayTrainerR(trusted=True)`` or
+``DelayTrainerRM(trusted=True)``. This consent is forwarded to delayed
+training completion, including the ``DelayTrainerRM`` worker. Direct callers can
+instead pass ``trusted=True`` to ``end_task_train`` or ``end_train``.
+Do not pass this constructor option to ordinary ``TrainerR`` or ``TrainerRM``.
+The option concerns recorder artifacts, not the independent trust requirements of
+MongoDB task contents. See :ref:`artifact_loading_migration`.
+
 Task Collecting
 ===============
 Before collecting model training results, you need to use the ``qlib.init`` to specify the path of mlruns.
@@ -87,6 +98,14 @@ To collect the results of ``task`` after training, ``Qlib`` provides `Collector
 
 `Collector <../reference/api.html#Collector>`_ can collect objects from everywhere and process them such as merging, grouping, averaging and so on. It has 2 step action including ``collect`` (collect anything in a dict) and ``process_collect`` (process collected dict).
 
+``RecorderCollector`` keeps artifact reads restricted by default. To collect
+verified executable objects alongside predictions, use per-artifact loading
+options, for example ``artifact_load_kwargs={"model": {"trusted": True}}`` with
+``artifacts_path={"model": "params.pkl", "pred": "pred.pkl"}``. Only the named
+``model`` entry receives consent; prediction reads remain restricted. Use
+``collect(only_exist=False)`` when loading failures must not produce a partial
+collection. See :ref:`artifact_loading_migration` for a complete example.
+
 `Group <../reference/api.html#Group>`_ also has 2 steps including ``group`` (can group a set of object based on `group_func` and change them to a dict) and ``reduce`` (can make a dict become an ensemble based on some rule).
 For example: {(A,B,C1): object, (A,B,C2): object} ---``group``---> {(A,B): {C1: object, C2: object}} ---``reduce``---> {(A,B): object}
 

docs/component/highfreq.rst

@@ -35,6 +35,13 @@ An example of a nested decision execution framework for high-frequency can be fo
 
 Besides, the above examples, here are some other related works about high-frequency trading in Qlib.
 
+.. note::
+
+    New source builds require ``HighFreqProvider`` artifact paths, including derived
+    cache files, to remain inside ``artifact_root`` (the current directory by
+    default). Cached pickle contents still require independent trust. See
+    :ref:`artifact_loading_migration` before reusing existing provider configurations.
+
 - `Prediction with high-frequency data <https://github.com/microsoft/qlib/tree/main/examples/highfreq#benchmarks-performance-predicting-the-price-trend-in-high-frequency-data>`_
 - `Examples <https://github.com/microsoft/qlib/blob/main/examples/orderbook_data/>`_ to extract features from high-frequency data without fixed frequency.
 - `A paper <https://github.com/microsoft/qlib/tree/high-freq-execution#high-frequency-execution>`_ for high-frequency trading.

docs/component/meta.rst

@@ -66,3 +66,16 @@ which adapts to the market dynamics.
 4. Apply guide information to the forecasting models to improve their performances.
 
 The `above example <https://github.com/microsoft/qlib/tree/main/examples/benchmarks_dynamic/DDG-DA>`_ can be found in ``examples/benchmarks_dynamic/DDG-DA/workflow.py``.
+
+DDG-DA uses restricted loading by default for recorder artifacts and local
+handler/internal-data pickle caches. To reload executable meta-models, tasks and
+caches from a trusted writer and access-controlled storage, configure
+``DDGDA(..., trusted=True)`` at the workflow entry point.
+Lower-level callers can configure ``MetaDatasetDS`` or ``InternalData.setup`` with
+the same option for recorder task reads. Prediction, label and numerical-report reads
+remain restricted. DDG-DA's opt-in also authorizes its handler/internal-data
+cache reads, so verify ``working_dir``, the configuration directory and any
+explicit ``h_path`` as well as the MLflow store. There is no automatic unsafe retry
+and the global restricted loader is unchanged.
+See :ref:`artifact_loading_migration`
+and the example README for CLI commands.

docs/component/model.rst

@@ -110,6 +110,21 @@ Also, the above example has been given in ``examples/train_backtest_analyze.ipyn
 Technically, the meaning of the model prediction depends on the label setting designed by user.
 By default, the meaning of the score is normally the rating of the instruments by the forecasting model. The higher the score, the more profit the instruments.
 
+Reloading a saved model
+=======================
+
+Training and prediction with in-memory objects, as above, do not require a recorder
+trust grant. Reloading a saved model or dataset may execute Python code, however.
+For artifacts from a verified writer in an access-controlled store, explicitly use
+``recorder.load_object("params.pkl", trusted=True)`` (or the corresponding ``R``
+facade). Keep prediction and label data reads restricted by omitting ``trusted``.
+Do not automatically retry a refused load with unrestricted deserialization.
+
+Workflows that resume training can configure ``DelayTrainerR`` or
+``DelayTrainerRM`` with ``trusted=True`` at construction, rather than
+changing internal loads. See :ref:`artifact_loading_migration` for migration details,
+version limitations and the separate trust requirements of local pickle files.
+
 
 Custom Model
 ============

docs/component/online.rst

@@ -26,6 +26,31 @@ The `examples <https://github.com/microsoft/qlib/tree/main/examples/online_srv>`
 Known limitations currently
 - Currently, the daily updating prediction for the next trading day is supported. But generating orders for the next trading day is not supported due to the `limitations of public data <https://github.com/microsoft/qlib/issues/215#issuecomment-766293563>_`
 
+Recorder artifact trust
+=======================
+
+Online updates may reload executable model, dataset and task objects from recorders.
+The default is restricted loading. After verifying the artifact writer and the
+store's write permissions, opt in with ``trusted=True`` on each
+``RollingStrategy`` or on a directly constructed ``OnlineToolR``/updater.
+``RollingStrategy`` forwards this setting through its online tool and updater;
+prediction, label and numerical-report reads remain restricted.
+
+``OnlineManager`` does not grant trust globally. Configure newly added strategies
+as well as initial strategies, and configure ``DelayTrainerR`` or ``DelayTrainerRM``
+separately if used. A supplied trainer keeps its caller-selected trust policy.
+Local serialized manager files must also be independently trusted; restoring one
+retains the settings saved in it, rather than applying a new manager-wide grant.
+Legacy components without a saved flag default to restricted loading. Explicitly
+reconfigure or recreate each strategy, its ``strategy.tool``, and any delayed
+trainer as needed; an example constructor flag does not override a subsequently
+loaded manager.
+
+See :ref:`artifact_loading_migration` for supported data, refusal handling and custom
+loader migration, and the
+`example commands <https://github.com/microsoft/qlib/blob/main/examples/README.md#recorder-artifact-trust>`_
+for the default-off ``--trusted=True`` CLI option.
+
 
 Online Manager
 ==============

docs/component/recorder.rst

@@ -91,9 +91,23 @@ Here are some important APIs that are not included in the ``QlibRecorder``:
 
 For other interfaces such as `save_objects`, `load_object`, please refer to `Recorder API <../reference/api.html#recorder>`_.
 
+.. _artifact_trust_migration:
+
+Migration: explicit artifact trust
+==================================
+
+The migration guide has moved to :ref:`artifact_loading_migration`. See that guide
+for the unreleased restricted-loading default, explicit ``trusted=True`` opt-in,
+workflow and custom-loader changes, and HIST/high-frequency cache migrations.
+
 Record Template
 ===============
 
+``RecordTemp.load(name, trusted=True)`` can explicitly authorize an individual
+artifact from a verified source and store, including parent-path lookup. The
+default remains restricted, and this does not authorize other loads or the
+template's ``generate`` method. See :ref:`artifact_loading_migration` for examples.
+
 The ``RecordTemp`` class is a class that enables generate experiment results such as IC and backtest in a certain format. We have provided three different `Record Template` class:
 
 - ``SignalRecord``: This class generates the `prediction` results of the model.
@@ -152,3 +166,5 @@ For more information about the APIs, please refer to `Record Template API <../re
 Known Limitations
 =================
 - The Python objects are saved based on pickle, which may results in issues when the environment dumping objects and loading objects are different.
+- Restricted loading is intentionally not compatible with arbitrary Python objects.
+  See :ref:`artifact_loading_migration` before changing trust settings.

docs/component/report.rst

@@ -23,6 +23,14 @@ Introduction
 All of the accumulated profit metrics(e.g. return, max drawdown) in Qlib are calculated by summation.
 This avoids the metrics or the plots being skewed exponentially over time.
 
+.. note::
+
+    Supported numerical report artifacts load in the restricted default mode.
+    Saved backtest artifacts containing ``Position`` instances or indicator objects
+    instead require explicit ``trusted=True`` after verifying their writer and
+    store. See :ref:`artifact_loading_migration`; a report filename alone does not
+    establish that its contents are data-only.
+
 Graphical Reports
 =================
 

docs/index.rst

@@ -26,6 +26,7 @@ Document Structure
    :caption: FIRST STEPS:
 
    Installation <start/installation.rst>
+   Artifact Loading Migration <start/artifact_migration.rst>
    Initialization <start/initialization.rst>
    Data Retrieval <start/getdata.rst>
    Custom Model Integration <start/integration.rst>