Skip to content

Python: Fix sub-workflow checkpoint restore to preserve sub-workflow state#7097

Draft
TaoChenOSU wants to merge 2 commits into
mainfrom
feature/python-subworkflow-checkpoint
Draft

Python: Fix sub-workflow checkpoint restore to preserve sub-workflow state#7097
TaoChenOSU wants to merge 2 commits into
mainfrom
feature/python-subworkflow-checkpoint

Conversation

@TaoChenOSU

Copy link
Copy Markdown
Contributor

Motivation & Context

When a workflow that contains sub-workflows (via WorkflowExecutor) is checkpointed and later resumed, each sub-workflow's own mid-progress state was lost. On restore, the parent only replayed the sub-workflow's pending request-info events; any executor state or in-flight progress inside the sub-workflow that had advanced before the checkpoint was not restored. A resumed parent therefore re-ran sub-workflows from an effectively empty state instead of continuing where they left off, producing incorrect results for nested/hierarchical workflows that checkpoint mid-run.

This change makes sub-workflow checkpoint/restore preserve the full nested state, so resuming a parent workflow faithfully continues each sub-workflow.

Description & Review Guide

  • What are the major changes?

    • Runner gains two internal methods:
      • capture_checkpoint_object() — captures a sub-workflow's state as a WorkflowCheckpoint object. It is quiescent-only: it raises WorkflowCheckpointException if the runner still has in-flight messages, so only settled state is embedded.
      • restore_from_checkpoint_object(checkpoint) — validates the graph signature, clears and re-imports shared/executor state, applies the checkpoint, and marks the runner as resumed.
    • WorkflowExecutor.on_checkpoint_save() now embeds the nested checkpoint under a sub_workflow_checkpoint key, and on_checkpoint_restore() decodes it and restores the sub-workflow via restore_from_checkpoint_object().
    • A backward-compatibility fallback is retained: when sub_workflow_checkpoint is absent (checkpoints written by older versions), restore falls back to the previous behavior of replaying pending request-info events.
    • New regression tests in test_runner.py (capture/restore roundtrip, quiescent-only guard, graph-signature-mismatch rejection) and test_sub_workflow.py (mid-progress sub-workflow resume preserves state).
  • What is the impact of these changes?

    • Resuming a parent workflow now restores each sub-workflow's mid-progress state instead of only its pending request-info events.
    • No public API changes; the new Runner methods are internal. Existing checkpoints remain loadable via the fallback path.
  • What do you want reviewers to focus on?

    • The quiescent-only invariant in capture_checkpoint_object() and the graph-signature validation in restore_from_checkpoint_object().
    • The backward-compat fallback in WorkflowExecutor.on_checkpoint_restore() for checkpoints lacking sub_workflow_checkpoint.

Related Issue

Part of the multi-PR workflow engine refactor series (follows #6695 and #6776). No standalone tracking issue.

Contribution Checklist

  • The code builds clean without any errors or warnings
  • All unit tests pass, and I have added new tests where possible
  • The PR follows the Contribution Guidelines
  • This PR is linked to an issue and there is no other open PR for this issue (see Related Issue above).
  • This is not a breaking change. If it is a breaking change, add the breaking change label (or add "[BREAKING]" to the title prefix, before or after any language prefix) — a workflow keeps the label and title prefix in sync automatically.

Add Runner.capture_checkpoint_object/restore_from_checkpoint_object (quiescent-only nested checkpoint) and embed a sub_workflow_checkpoint in WorkflowExecutor.on_checkpoint_save/on_checkpoint_restore so a resumed parent restores each sub-workflow's mid-progress state instead of only replaying pending request-info events. Keeps a backward-compat fallback when sub_workflow_checkpoint is absent.
Copilot AI review requested due to automatic review settings July 13, 2026 22:31
@giles17 giles17 added the python Usage: [Issues, PRs], Target: Python label Jul 13, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes a correctness gap in Python workflow checkpointing for nested workflows: when a parent workflow is resumed, the sub-workflow now resumes with its own internal runner/executor state restored (not just the parent WorkflowExecutor’s pending-request bookkeeping).

Changes:

  • Add Runner.capture_checkpoint_object() / Runner.restore_from_checkpoint_object() to support in-memory checkpoint capture/restore for embedded (sub-workflow) state.
  • Update WorkflowExecutor checkpoint save/restore to embed and restore the child workflow’s checkpoint, with a backward-compat fallback when the embedded checkpoint is absent.
  • Add regression tests covering checkpoint object roundtrip, quiescent-only capture, graph-signature mismatch rejection, and mid-progress sub-workflow resume behavior.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File Description
python/packages/core/agent_framework/_workflows/_runner.py Adds in-memory checkpoint capture/restore APIs used for nested workflow checkpoint embedding.
python/packages/core/agent_framework/_workflows/_workflow_executor.py Embeds the sub-workflow checkpoint on save and restores it on resume, keeping a fallback for older checkpoints.
python/packages/core/tests/workflow/test_runner.py Adds unit tests for the new runner checkpoint object APIs and invariants.
python/packages/core/tests/workflow/test_sub_workflow.py Adds regression coverage ensuring sub-workflow mid-progress state survives parent checkpoint restore.

Comment on lines +529 to +533
sub_workflow_checkpoint = state.get("sub_workflow_checkpoint")
if sub_workflow_checkpoint is not None:
sub_workflow_checkpoint = decode_checkpoint_value(sub_workflow_checkpoint)
await self.workflow._runner.restore_from_checkpoint_object(sub_workflow_checkpoint) # pyright: ignore[reportPrivateUsage]
return

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated Code Review

Reviewers: 5 | Confidence: 86% | Result: All clear

Reviewed: Correctness, Security Reliability, Test Coverage, Failure Modes, Design Approach


Automated review by TaoChenOSU's agents

@github-actions

Copy link
Copy Markdown
Contributor

Python Test Coverage

Python Test Coverage Report •
FileStmtsMissCoverMissing
packages/core/agent_framework/_workflows
   _runner.py196298%417–418
   _workflow_executor.py1993283%95, 405, 465, 494, 496, 504–505, 510, 512, 517, 519, 538, 543, 620–626, 630–632, 640, 645, 656, 666, 670, 676, 680, 690, 694
TOTAL44173526588% 

Python Unit Test Overview

Tests Skipped Failures Errors Time
8872 33 💤 0 ❌ 0 🔥 2m 8s ⏱️

Add RunnerContext.create_checkpoint_object alongside create_checkpoint (create_checkpoint now delegates to it and persists), so Runner.capture_checkpoint_object builds the snapshot via the context instead of a one-off get_messages peek primitive. In-flight messages are captured non-destructively (per-source lists copied). The checkpoint-less capturing contexts (azurefunctions, durabletask) raise NotImplementedError to match create_checkpoint.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

python Usage: [Issues, PRs], Target: Python

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants