Skip to content

Clarify Pipeline __getitem__ and apply step handling#268

Open
taimoorsohail wants to merge 1 commit into
ACCESS-Community-Hub:developfrom
taimoorsohail:ts/fix-getitem-inconsistency-vs-apply
Open

Clarify Pipeline __getitem__ and apply step handling#268
taimoorsohail wants to merge 1 commit into
ACCESS-Community-Hub:developfrom
taimoorsohail:ts/fix-getitem-inconsistency-vs-apply

Conversation

@taimoorsohail

Copy link
Copy Markdown

Summary

This updates Pipeline.__getitem__ and Pipeline.apply() so they handle pipeline step types consistently and fail with clearer errors when a step cannot be applied to an already-retrieved sample.

What changed

  • Allows Operation and PipelineBranchPoint steps in the post-source __getitem__ path.
  • Runs PipelineStep instances via run() and Operation instances via apply() in __getitem__.
  • Adds explicit errors when PipelineIndex is used where an already-retrieved sample is expected.
  • Improves error messages for unsupported pipeline step types.

Validation

  • PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 pytest -q -o addopts='' packages/pipeline/tests/test_controller
  • 34 passed, 1 warning

raise TypeError(
f"Pipeline.__getitem__ encountered a {type(step)} in the step list "
"after the data-source position, which cannot be applied to a sample. "
"Steps after the data source must be PipelineStep, Operation, "

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I need to dive into this to understand the example fully. A cache object is an index which can be included after the initial data source index, so that circumstance needs to be accounted for. The last one (e.g. the cache or temporal window) is supposed to use the parent pipeline to retrieve the individual samples, since the data source index might not be the same as the cache/subsequent index.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, I see, reading ahead I've just understood the situation more clearly. The use case is attempting to use the pipeline apply method on a single already-retrieved sample, and the index steps are interfering with its execution in an unexpected way.

"""
for step in self.steps:
if isinstance(step, PipelineIndex):
raise TypeError(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thinking about this, particularly in the context of a caching step, maybe the index class in question should have a flag to indicate if skipping is safe. Skipping caching is still valid mathematically, so you could either skip it silents or raise a warning rather than an exception. However, other things like a temporal window aren't valid, because they need to reach back upstream in the pipeline and retrieve additional data, and perhaps it's appropriate to raise the exception in those circumstances. What do you think about that idea?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants