Clarify Pipeline __getitem__ and apply step handling#268
Conversation
| raise TypeError( | ||
| f"Pipeline.__getitem__ encountered a {type(step)} in the step list " | ||
| "after the data-source position, which cannot be applied to a sample. " | ||
| "Steps after the data source must be PipelineStep, Operation, " |
There was a problem hiding this comment.
I need to dive into this to understand the example fully. A cache object is an index which can be included after the initial data source index, so that circumstance needs to be accounted for. The last one (e.g. the cache or temporal window) is supposed to use the parent pipeline to retrieve the individual samples, since the data source index might not be the same as the cache/subsequent index.
There was a problem hiding this comment.
Oh, I see, reading ahead I've just understood the situation more clearly. The use case is attempting to use the pipeline apply method on a single already-retrieved sample, and the index steps are interfering with its execution in an unexpected way.
| """ | ||
| for step in self.steps: | ||
| if isinstance(step, PipelineIndex): | ||
| raise TypeError( |
There was a problem hiding this comment.
Thinking about this, particularly in the context of a caching step, maybe the index class in question should have a flag to indicate if skipping is safe. Skipping caching is still valid mathematically, so you could either skip it silents or raise a warning rather than an exception. However, other things like a temporal window aren't valid, because they need to reach back upstream in the pipeline and retrieve additional data, and perhaps it's appropriate to raise the exception in those circumstances. What do you think about that idea?
Summary
This updates
Pipeline.__getitem__andPipeline.apply()so they handle pipeline step types consistently and fail with clearer errors when a step cannot be applied to an already-retrieved sample.What changed
OperationandPipelineBranchPointsteps in the post-source__getitem__path.PipelineStepinstances viarun()andOperationinstances viaapply()in__getitem__.PipelineIndexis used where an already-retrieved sample is expected.Validation
PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 pytest -q -o addopts='' packages/pipeline/tests/test_controller34 passed, 1 warning