[SPARK-58282][PYTHON][DOCS] Refresh PySpark README and clarify project scope - #57452
[SPARK-58282][PYTHON][DOCS] Refresh PySpark README and clarify project scope#57452nchammas wants to merge 9 commits into
Conversation
| This packaging is currently experimental and may change in future versions (although we will do our best to keep compatibility). | ||
| Using PySpark requires the Spark JARs, and if you are building this from source please see the builder instructions at | ||
| ["Building Spark"](https://spark.apache.org/docs/latest/building-spark.html). | ||
| PySpark is not intended to be a complete distribution of Spark. It's meant for local development or for interacting with an existing cluster (be it Spark standalone, YARN, or Kubernetes). Using PySpark to set up a new standalone Spark cluster is not supported. To set up a standalone cluster you need to [use the full distribution of Spark](https://spark.apache.org/downloads.html). |
There was a problem hiding this comment.
This wording change from "does not contain the tools required" to "is not supported" is deliberate. I would like us to have the freedom to add more sbin scripts (like for Connect) to the pyspark package while making it clear that we don't support using PySpark to create a cluster.
I will propose packaging additional sbin scripts with PySpark in a separate PR.
| See also [Dependencies](https://spark.apache.org/docs/latest/api/python/getting_started/install.html#dependencies) for production, and [pyproject.toml](https://github.com/apache/spark/blob/master/pyproject.toml) for development. | ||
| At its core PySpark depends on Py4J, but some additional sub-packages have their own extra requirements for some features. See [Dependencies](https://spark.apache.org/docs/latest/api/python/getting_started/install.html#dependencies) for the full list of production dependencies and [pyproject.toml](https://github.com/apache/spark/blob/master/pyproject.toml) for development dependencies. | ||
|
|
||
| PySpark also requires the Spark JARs, which are included in this distribution. If you are building this distribution from source please see [the builder instructions](https://spark.apache.org/docs/latest/building-spark.html). |
There was a problem hiding this comment.
The new line "PySpark also requires the Spark JARs, which are included in this distribution" is inaccurate for two of the three PyPI packages that reuse this file verbatim as their long_description. Confirmed against master: python/packaging/ {classic,connect,client}/setup.py all read README.md into long_description.
The classic pyspark package bundles the assembly jars (statement true), but pyspark-connect obtains them only transitively via its pyspark==VERSION dependency, and pyspark-client bundles no jars and does not even depend on pyspark, so on those two PyPI pages "included in this distribution" is false.
A one-word hedge ("included in the PySpark distribution", or noting that the client/connect packages obtain them via the pyspark dependency) fixes it.
There was a problem hiding this comment.
Oh, I didn't realize that packages other than pyspark use the same README. Let me take another look.
There was a problem hiding this comment.
OK this is a good catch, and it looks like the old text is also inaccurate due to the shared use of this README. python-client, for example, doesn't require Spark JARs and doesn't even depend on Py4J.
I'll update this accordingly.
|
I left just one comment, but otherwise looks good |
not just standalone clusters
|
Yeah I think we need at least @HyukjinKwon and @zhengruifeng to go over this change to confirm the newly added explanation is correct and the deleted information is outdated. |
|
|
||
| Spark is a unified analytics engine for large-scale data processing. It provides | ||
| high-level APIs in Scala, Java, Python, and R, and an optimized engine that | ||
| high-level APIs in Scala, Java, and Python, and an optimized engine that |
There was a problem hiding this comment.
we deprecated R but I think the support is still there (?)
There was a problem hiding this comment.
Updated to match the main README.
Line 4 in 8920dea
| [py]: https://github.com/apache/spark/blob/master/pyproject.toml | ||
|
|
||
| ## Python Requirements | ||
| ## Python vs. "Full" Distribution of Spark |
There was a problem hiding this comment.
why we need such a comparsion Python vs. "Full" Distribution of Spark ?
There was a problem hiding this comment.
The information in this section is not new. I rephrased it from the original README.
The comparison is useful because PySpark does not have the same capabilities or the same support as "full" Spark. Users cannot setup a standalone cluster using PySpark (except for local development); and even if they could, we do not support that use case.
It's important to explain this difference to users, and this README is where we have explained that difference in the past.
5 participants
What changes were proposed in this pull request?
Refresh the PySpark README.
Key changes:
Why are the changes needed?
The README hasn't seen a significant update since it was introduced in #15659 about 10 years ago, and contains some obsolete language.
Clarifying the intended use of PySpark vs. "Full" Spark will also help focus related discussions coming out of the effort to potentially introduce a new CLI in #57313.
Does this PR introduce any user-facing change?
Documentation only.
How was this patch tested?
No testing.
Was this patch authored or co-authored using generative AI tooling?
No.