[SPARK-58240] Add Unified Spark CLI

[nchammas]

What changes were proposed in this pull request?

This PR adds a new command line tool named spark that unifies Spark's command line interface.

It's installed on the user's PATH when a full distribution of Spark is installed, similar to spark-submit, spark-sql, and other Spark executables. It's a pure Python script that sticks to the standard library and acts simply (at least for now) as a dispatcher to various scripts that already exist under bin/ and sbin/.

Here are the top-level commands:

$ spark
usage: spark [-h] COMMAND ...

Unified CLI for Apache Spark.

options:
  -h, --help  show this help message and exit

standalone cluster commands:
  cluster     Manage the standalone cluster (master + workers).
  master      Manage the standalone cluster master.
  worker      Manage the standalone cluster workers.

shells:
  scala       Start the Scala shell.
  python      Start the PySpark shell.
  sql         Start the SQL shell.

commands:
  connect     Manage the Connect server.
  history     Manage the History server.
  thrift      Manage the Thrift server.
  pipelines   Run Spark Declarative Pipelines.
  submit      Submit an application.

Each command has its own --help and may also have its own sub-commands. For example:

$ spark connect
usage: spark connect [-h] COMMAND ...

options:
  -h, --help  show this help message and exit

commands:
  start       Start the server.
  stop        Stop the server.
  status      Show server status.

Why are the changes needed?

Today, Spark's CLI is spread across 20+ scripts under bin/ and sbin/. This fragmented interface makes discovery more difficult for users and maintenance more difficult for contributors.

A unified CLI gives users a single entry point with a clear help menu that makes it easy to discover and use what's available. This follows a pattern that other projects have successfully adopted already for their CLIs, like kubectl, docker, and ray.

Does this PR introduce any user-facing change?

Yes, it introduces a major new user interface.

How was this patch tested?

Local manual tests + new automated tests.

Was this patch authored or co-authored using generative AI tooling?

I authored this patch with assistance from GitHub Copilot.

TODO

• Secure committer buy-in for this idea + design.
• Add new tests to CI build.
• Figure out how this new CLI should be packaged.
• Determine what interface stability guarantees the new CLI should have.
• Add user-facing documentation.

[nchammas]

@gaogaotiantian - You expressed some interest in this idea. As a committer, do you have any guidance on how I should move this forward? It's a major new user interface, so I assume we first want to agree as a project that we want this. Then, we can sort out the implementation details.

[gaogaotiantian]

I did express interest in this idea and I think it should be bigger than the original thought.

I believe this requires an SPIP because it's a pretty large new feature.

If we do this, we need to eventually replace the old scripts, instead of having two sets of code that do the same thing. That will raise a problem - making python a hard requirement for using spark. Admittedly, python is so popular and almost every developer has some version of python on their computer, but making it a requirement for using spark is another story. If we can move past this blocker, then we should be able to make a much better CLI for spark usage. I don't like shell scripts in general.

If we can't do that, then a fallback option would be having a CLI just for spark developers. Users can still use the shell scripts. That should be an easier push.

In any case, I believe features like this need an SPIP and a vote from the community. Unless we are creating a CLI without replacing the old one - then I'm just against the idea because it does not make sense.

[nchammas]

I believe this requires an SPIP because it's a pretty large new feature.

That's what I expected, but wanted to confirm with a committer. I will write one up.

Is there a PMC member you think might be interested in being the shepherd?

If we do this, we need to eventually replace the old scripts, instead of having two sets of code that do the same thing.

I agree, and that is something I mentioned in my initial email to the dev list about this idea.

If we can't do that, then a fallback option would be having a CLI just for spark developers. Users can still use the shell scripts.

My motivation for proposing a unified CLI is specifically to help users and make various features of Spark more discoverable. I am not as attracted to the idea of a new CLI just for Spark developers.

Unless we are creating a CLI without replacing the old one - then I'm just against the idea because it does not make sense.

I think having the new CLI be a wrapper around the existing scripts will be a transitional phase and will allow us to merge the various scripts into the CLI bit by bit, rather than having to do everything all at once. I expect at least some committers will express hesitation about rewriting old and stable scripts. This will be part of the SPIP discussion.

[gaogaotiantian]

Yeah I would hate if we stuck at the "transition phase". That means we have more code and no progress. So if we are going to make an SPIP, we need to be crystal clear that we are going to replace all the shell/cmd scripts and make python a hard requirement. I think this would be a challenge. I'm not sure if @HyukjinKwon is interested in this.

[nchammas]

Yeah I would hate if we stuck at the "transition phase". That means we have more code and no progress.

A new unified CLI is still progress! It offers a single entry point and easy discoverability for users. I consider that alone to be a big win, and it's not that much more code, but I agree that ideally we would replace the old shell scripts.

make python a hard requirement

There are alternatives to this that allow us to offer a unified CLI without making Python a runtime dependency. The CLI could be implemented in another language (like Scala) that produces executable artifacts, or we could use one of the many tools for building standalone executables from Python.

We can discuss this in more detail if committers object to making Python a runtime requirement.

[gaogaotiantian]

None of the standalone libraries for Python is good enough to ship scripts. I would be against that.

I don't really think a single entry would be significantly better than current situation - it would be better probably. What I really want is a util tool that pyspark users can just use when they pip install pyspark (I think Python is the majority language spark users are using). So not like ./bin/spark (this can't be used when you pip install), but something installed like spark-cli ..., which could be easily implemented in Python. I think you should be clear about whether you want this to be user-interfacing or dev-focusing, because I believe most of the spark users do not have the source code package.

Of course, if pyspark is not installed with pip install, we can always have a fall back with python -m pyspark.cli or something like that. I want the cli to be part of pyspark. We should of course also support users that do not use Python. We probably can think of a strategy to organize code.

I would not be interested in solutions where we ship an executable (or a shell script) just to be a unified entry. That's just not what python users are used to. However, I'm not against the idea. I won't oppose it - just indifferent.

I understand if you want this to be more about spark itself than pyspark - then you'll need to find other committers that are interested in the idea.

[nchammas]

I think there is a misunderstanding of what I am proposing. Let me try to clear that up before I address some of your other concerns.

So not like ./bin/spark (this can't be used when you pip install)

I think this is the source of some confusion. My point was to show the commands that would be supported, not the executable's path. I am working with a prototype and currently have the executable under bin/.

My goal, however, is to place spark on the user's path so they can call it just like they call the current executables like spark-sql, spark-submit, etc.

I will edit the PR description to clear up this point of confusion.

I want the cli to be part of pyspark.

Could you elaborate on why you want the CLI to be part of PySpark? To me, the CLI is independent of any language runtime. Users use it to manage various servers, submit applications, and launch shells. There is nothing about it that seems like it should be tied specifically to PySpark.

[gaogaotiantian]

I thought about it - it should not be part of pyspark. What I originally thought was, it should be usable when users just pip install pyspark. There are many ways for users to install spark and we should make the CLI available in all case.

However, I took another look at the existing python packaging, the scripts are already available. For example, if you pip install pyspark, then pyspark will just bring the python shell. So we have no problem with that part.

I was thinking about how to manage the code, but it's okay for the CLI itself to be standalone code.

[nchammas]

There are many ways for users to install spark and we should make the CLI available in all case.

Agreed. When Spark is installed via a Python installer, we need to configure scripts or console_scripts accordingly. Homebrew has its own mechanism for installing executables on the PATH. Other package managers do their own thing.

I think as long as we provide the executable and a clear specification of runtime dependencies, each package manager or distributor will figure out how to do the right thing.