Skip to content

[SPARK-58575][PYTHON] Inline redundant error_class parameter in worker.py verify helpers - #57773

Open
Yicong-Huang wants to merge 1 commit into
apache:masterfrom
Yicong-Huang:inline-redundant-verify-error-class
Open

[SPARK-58575][PYTHON] Inline redundant error_class parameter in worker.py verify helpers#57773
Yicong-Huang wants to merge 1 commit into
apache:masterfrom
Yicong-Huang:inline-redundant-verify-error-class

Conversation

@Yicong-Huang

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

This is a follow-up to SPARK-58529, which consolidated the row-count verification helpers in python/pyspark/worker.py but left two of them over-parameterized with an error_class argument that is always passed the same constant. This PR inlines the constant and drops the parameter:

  • verify_output_row_limit: both call sites (SQL_SCALAR_ARROW_ITER_UDF and SQL_SCALAR_PANDAS_ITER_UDF) pass error_class="OUTPUT_EXCEEDS_INPUT_ROWS". The constant is inlined into the raise and the parameter is removed from the signature and both call sites.
  • verify_iterator_exhausted: both call sites pass error_class="INPUT_NOT_FULLY_CONSUMED". Same treatment.

Both error classes are semantically bound to these two helpers, which together enforce the iterator-UDF contract that the number of output rows equals the number of input rows. No other code path can reuse them: for example, UDTFs have no such contract (one input row may produce many output rows) and use a different set of verifiers, so parameterizing the error class buys no reuse.

Why are the changes needed?

Addresses a non-blocking review comment on SPARK-58529. Removing the always-constant parameter makes the helpers read more clearly: the helper name states the contract and the inlined error class is the fixed violation signal, with no indirection to trace through.

Does this PR introduce any user-facing change?

No. The raised error class and message are unchanged, so there is no behavior change, no error-conditions.json change, and no change to existing test assertions.

How was this patch tested?

Existing test ScalarPandasUDFTests.test_vectorized_udf_invalid_length already exercises the INPUT_NOT_FULLY_CONSUMED path through verify_iterator_exhausted. This PR additionally adds a case to that test covering an iterator UDF that emits more output rows than input rows, which exercises the OUTPUT_EXCEEDS_INPUT_ROWS fail-fast path through verify_output_row_limit (previously untested).

Was this patch authored or co-authored using generative AI tooling?

No.

@Yicong-Huang
Yicong-Huang force-pushed the inline-redundant-verify-error-class branch from 7e7e7bb to 6e5b20c Compare August 5, 2026 00:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants