feat(sparkeks): add JAR/--class entrypoint support (chipmunk EMR->EKS migration, UoW-F3)

[prasadlohakpure]

What

Adds JAR/--class entrypoint support to the Heimdall sparkeks plugin so it can run a custom-class
(Java/Scala) Spark application, not just the built-in Python SQL wrapper. Additive and opt-in — no
existing sparkeks command sets a JAR wrapper, so current behavior is unchanged.

Entrypoint strategy (internal/pkg/object/command/sparkeks/entrypoint.go) — the entrypoint-specific
part of the spec (Type / MainClass / Arguments) is dispatched behind an entrypointStrategy
interface, selected by the file extension of the command's wrapper_uri:

wrapper_uri	Entrypoint	Spec.Type	Spec.MainClass
.py	Python SQL wrapper (pre-existing)	Python	not set
.jar	JVM JAR by main class	Scala / Java (from application_type, default Scala)	entry_point
anything else	falls back to SQL wrapper	Python	not set

New optional job-context fields: parameters.entry_point (fully-qualified main class) and
parameters.application_type (Scala or Java, case-insensitive).

Argument contract. Both entrypoints submit the same positional vector, matching the deployed
com.pattern.chipmunk.Writer:

[appName, queryURI, user]              # return_result = false
[appName, queryURI, user, resultURI]   # return_result = true

queryURI is an s3a:// URI — the application reads the SQL from it rather than receiving it inline.
Only Type and MainClass differ between the JAR and SQL-wrapper paths.

s3:// vs s3a:// in properties. Cluster/job properties are no longer blanket-rewritten
s3:// → s3a://. Only keys read through Hadoop's FileSystem abstraction need s3a://, so rewriting
is restricted to an explicit hadoopPathProperties set (driver/executor podTemplateFile). App-specific
keys such as spark.chipmunk.output.path are read by application code via the AWS SDK, which requires
the literal s3:// scheme. Plugin-managed URIs (query, result, event log, wrapper) are still always
rewritten. ⚠️ Reviewers: this narrows an earlier commit in this branch — worth confirming no existing
cluster config depends on a non-podTemplateFile property being rewritten.

Also in this branch: submission-failure/unknown-state reasons are now written to job stderr, and
s3aQueryURI/s3aResultURI are precomputed once on the execution context.

jobContext.arguments is retained for config compatibility but is ignored — both strategies use a
fixed 3/4-element vector, so there is no positional slot to append to. Job-specific inputs travel as
SparkConf properties instead.

Why

Foundation step (UoW-F3) of migrating Chipmunk's collection-build Spark jobs off EMR-on-EKS onto
Spark-on-EKS (sparkeks). The chipmunk collection writer is a Scala JAR
(com.pattern.chipmunk.Writer); sparkeks could previously only launch the Python SQL wrapper, so it
could not run that JAR. This unlocks the spark-chipmunk-eks command + operator wiring (step 4).

Migration roadmap (where this PR fits)

Foundation phase. Steps 1–3 are mutually independent — none is consumed until step 4 wires them together
— so they are reviewable in parallel. Step 4 gates on all three.

Step	Unit	Repo(s)	Status
1	Add Snowflake jars to sparkeks image	data-platform-tf	raised — patterninc/data-platform-tf#1836
2	Mount snowflake-key secret on sparkeks driver	heimdall-config	raised — patterninc/heimdall-config#444
3	JAR/--class support in sparkeks plugin	heimdall	▶️ this PR
4	spark-chipmunk-eks command + ChipmunkCollection(use_eks=) operator (depends on 1–3)	heimdall-config + data-airflow	designed, not implemented
∥	Shadow-parity validation harness (off critical path; required before cutover validation)	TBD	pending

After the foundation lands: per-pipeline cutovers (chipmunk_daily, product_catalog_chipmunk_daily,
amazon_merchant_asins_daily, chipmunk_org_collection, sensors_universal, core_dims_hourly), each
adding the sparkeks path alongside EMR → validating → cutting over, with EMR removal as a separate later
PR per pipeline. Then EMR-on-EKS decommission, gated on FinOps sign-off.

Test plan

• go build ./internal/... ./cmd/... — clean.
• go vet ./internal/pkg/object/command/sparkeks/ — clean.
• gofmt -l internal/pkg/object/command/sparkeks/ — clean.
• go test ./internal/pkg/object/command/sparkeks/ — ok (existing package tests).
• No new unit tests in this PR (the ones added earlier in the branch were removed per request).
  The JAR argument contract was verified against the deployed chipmunk JAR by running it on the
  chipmunk_eks_migration_test collection, not by unit test.

Notes / rollback

• Rollback: revert the branch — additive and opt-in; no existing command sets a .jar wrapper_uri.
  The one non-additive piece is the properties-rewriting narrowing described above.
• Pre-existing, unrelated: go build ./... across the whole repo reports "function main is undeclared"
  for every plugins/* package — these are -buildmode=plugin packages with no func main() and fail
  the same way on an untouched origin/main checkout. Not introduced by this PR.

---

🤖 Generated with Claude Code

[Copilot]

Pull request overview

Adds opt-in JAR/--class entrypoint support to the sparkeks plugin so it can launch JVM Spark applications (e.g., Chipmunk Writer) in addition to the existing SQL-wrapper path, with a defensive fallback to ensure SparkApplication.Spec.Type is never submitted empty.

Changes:

• Introduces entry_point and arguments job-context fields to enable JAR-mode SparkApplication submission.
• Updates SparkApplication spec generation to support a JAR-mode argument vector and to default Spec.Type when not set by templates.
• Adds unit tests covering JAR mode, JAR+return_result, SQL-wrapper non-regression, and the default-Type fallback.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
internal/pkg/object/command/sparkeks/sparkeks.go	Adds JAR-mode (--class) SparkApplication configuration and a defensive default for Spec.Type.
internal/pkg/object/command/sparkeks/sparkeks_test.go	Adds tests validating the new JAR-mode behavior and guarding existing SQL-wrapper behavior.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.