[SPARK-58439][BUILD][INFRA] Add dedicated validation script for distributions

[nchammas]

What changes were proposed in this pull request?

Add a new Python script, dev/validate-distribution, that runs at the end of dev/make-distribution.sh and checks several basic things about the distribution.

The checks that run depend on the flags that are passed to make-distribution.sh. If a Python package is built, Python-specific tests are run; if an R package is built, R-specific tests are run; etc. I used the same command line flags in the validation script as in the build script.

Why are the changes needed?

This extends the work done in #56453 and checks additional, basic things that would likely fail a release if done incorrectly somehow. Since the script is in Python, it's also a bit easier to maintain.

This will also make it easier to add more checks in the future, should they become relevant. I would add smoke tests for the new CLI, for example, if #57313 is accepted.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Here's what the output looks like when the distribution is fine:

$ ./dev/validate-distribution dist/
Testing distribution at: .../spark/dist

Checking distribution structure... OK
Checking bin/spark-submit --version... OK
Checking bin/spark-shell --version... OK
Checking bin/pyspark --version... OK
Checking bin/spark-sql --version... OK
Checking sbin/start-thriftserver.sh --help... OK
Checking sbin/start-connect-server.sh --help... OK
All distribution validation checks passed.

And here's what it looks like when the distribution has some problems:

$ ./dev/validate-distribution dist/ --pip /tmp/fake-pip
Testing distribution at: .../spark/dist

Checking distribution structure... FAILED
  MISSING: LICENSE
  MISSING: RELEASE
Checking bin/spark-submit --version... OK
Checking bin/spark-shell --version... OK
Checking bin/pyspark --version... FAILED
Checking bin/spark-sql --version... OK
Checking sbin/start-thriftserver.sh --help... OK
Checking sbin/start-connect-server.sh --help... OK
Checking pyspark-fake.tar.gz for LICENSE and NOTICE... FAILED
  pyspark-fake.tar.gz: missing LICENSE at the package root
  pyspark-fake.tar.gz: missing NOTICE at the package root

5 check(s) failed:
  MISSING: LICENSE
  MISSING: RELEASE
  bin/pyspark --version: [Errno 13] Permission denied: '.../spark/dist/bin/pyspark'
  pyspark-fake.tar.gz: missing LICENSE at the package root
  pyspark-fake.tar.gz: missing NOTICE at the package root

Was this patch authored or co-authored using generative AI tooling?

I wrote this with assistance from GitHub Copilot.

[nchammas]

@huaxingao - Helpful or overkill?

[uros-b]

subprocess.run for each smoke test has no timeout= argument. If a JVM invocation hangs (e.g. missing JAVA_HOME pointing at a stale path, or a classloader deadlock during --version), the validation script and therefore make-distribution.sh will hang indefinitely rather than failing with a clear message. Adding timeout=60 (or a similar reasonable ceiling) and catching subprocess.TimeoutExpired with a clean error would make failures actionable without false-passing on hangs.

[nchammas]

Fixed.

[HyukjinKwon]

0 blocking, 0 non-blocking, 0 nits.
Useful, well-scoped build-validation tooling; robust and correctly wired.

Verification

Confirmed make-distribution.sh assembles the command as a bash array conditional on MAKE_PIP/MAKE_R/MAKE_TGZ/MAKE_SPARK_CONNECT (validates only what was built; array form avoids quoting bugs). The script accumulates errors and sys.exit(1)s so a validation failure fails the build, and the smoke-test subprocess calls use a timeout with TimeoutExpired handling (the one reviewer concern, already fixed).