GH-50801: [Python] Expose the record_batch_reader_source Acero node (RecordBatchReaderSourceNodeOptions)

[owenowenisme]

Rationale for this change

Closes #50801
As title, this exposes RecordBatchReaderSourceNodeOptions to Python. With it, a hash join can build its hash table once and stream the probe side through as a generator-backed RecordBatchReader, releasing each input chunk as it is consumed, instead of materializing the full probe table up front (table_source) or pinning all fragments for the plan's lifetime (dataset scan).

Are these changes tested?

Yes

Are there any user-facing changes?

Yeah, users can pass RecordBatchReaderSourceNodeOptions into Declaration, but its not a breaking change

• GitHub Issue: [Python] Expose the record_batch_reader_source Acero node (RecordBatchReaderSourceNodeOptions) #50801

[github-actions]

⚠️ GitHub issue #50801 has been automatically assigned in GitHub to PR creator.

[Copilot]

Pull request overview

Exposes Acero’s record_batch_reader_source node options (RecordBatchReaderSourceNodeOptions) to Python so execution plans can stream data from a pyarrow.RecordBatchReader (including generator-backed readers) instead of requiring fully-materialized tables or dataset scans.

Changes:

• Added a Cython binding for arrow::acero::RecordBatchReaderSourceNodeOptions and exposed it as pyarrow.acero.RecordBatchReaderSourceNodeOptions.
• Added Python tests covering basic usage, generator-backed laziness, generator error propagation, and a hash-join probe-side streaming example.
• Updated Python API docs to include the newly exposed options class.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
python/pyarrow/_acero.pyx	Adds the RecordBatchReaderSourceNodeOptions Cython wrapper that constructs the underlying Acero options object.
python/pyarrow/includes/libarrow_acero.pxd	Declares arrow::acero::RecordBatchReaderSourceNodeOptions for Cython to construct.
python/pyarrow/acero.py	Re-exports the new options class from the _acero extension for public Python use.
python/pyarrow/tests/test_acero.py	Adds coverage for streaming source behavior (including generator-backed readers and join integration).
docs/source/python/api/acero.rst	Documents the new options class in the public Acero Python API listing.

[Reranko05]

Could you edit the title to GH-<Issue Number>: [<Component>] <Title>, this will make it easier for maintainers.

[owenowenisme]

@Reranko05 I updated the title, ty

[owenowenisme]

@wgtmac @rok
Hi, would you mind taking a look? Thanks!

[rok]

This looks ok to me, but I am not familiar with this part of the codebase. Perhaps @wgtmac or @zanmato1984 can review.

[owenowenisme]

cc @wgtmac too, PTAL thank you!!!

[zanmato1984]

+1. Thanks for working on this!

[owenowenisme]

@zanmato1984 Thanks for reviewing!

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

[owenowenisme]

@zanmato1984 Hi Rossi, can I please get a merge? Thank you!

[zanmato1984]

Let's wait for CI then I'll merge. Thanks!

[zanmato1984]

Merging.