[SPARK-58575][PYTHON] Inline redundant error_class parameter in worker.py verify helpers - #57773
Open
Yicong-Huang wants to merge 1 commit into
Open
Conversation
HyukjinKwon
approved these changes
Aug 4, 2026
…r.py verify helpers
Yicong-Huang
force-pushed
the
inline-redundant-verify-error-class
branch
from
August 5, 2026 00:18
7e7e7bb to
6e5b20c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This is a follow-up to SPARK-58529, which consolidated the row-count verification helpers in
python/pyspark/worker.pybut left two of them over-parameterized with anerror_classargument that is always passed the same constant. This PR inlines the constant and drops the parameter:verify_output_row_limit: both call sites (SQL_SCALAR_ARROW_ITER_UDFandSQL_SCALAR_PANDAS_ITER_UDF) passerror_class="OUTPUT_EXCEEDS_INPUT_ROWS". The constant is inlined into theraiseand the parameter is removed from the signature and both call sites.verify_iterator_exhausted: both call sites passerror_class="INPUT_NOT_FULLY_CONSUMED". Same treatment.Both error classes are semantically bound to these two helpers, which together enforce the iterator-UDF contract that the number of output rows equals the number of input rows. No other code path can reuse them: for example, UDTFs have no such contract (one input row may produce many output rows) and use a different set of verifiers, so parameterizing the error class buys no reuse.
Why are the changes needed?
Addresses a non-blocking review comment on SPARK-58529. Removing the always-constant parameter makes the helpers read more clearly: the helper name states the contract and the inlined error class is the fixed violation signal, with no indirection to trace through.
Does this PR introduce any user-facing change?
No. The raised error class and message are unchanged, so there is no behavior change, no
error-conditions.jsonchange, and no change to existing test assertions.How was this patch tested?
Existing test
ScalarPandasUDFTests.test_vectorized_udf_invalid_lengthalready exercises theINPUT_NOT_FULLY_CONSUMEDpath throughverify_iterator_exhausted. This PR additionally adds a case to that test covering an iterator UDF that emits more output rows than input rows, which exercises theOUTPUT_EXCEEDS_INPUT_ROWSfail-fast path throughverify_output_row_limit(previously untested).Was this patch authored or co-authored using generative AI tooling?
No.