FEAT: Support for custom entrypoint in sparkeks - #137
Open
prasadlohakpure wants to merge 2 commits into
Open
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR extends the Spark EKS plugin to support JVM JAR (--class) entrypoints in addition to the existing Python SQL wrapper flow, selecting behavior based on the command context wrapper_uri file extension.
Changes:
- Add an entrypoint strategy layer to configure SparkApplication
Type/MainClass/Argumentsfor.jarvs.pywrappers. - Introduce job parameters for JAR execution (
entry_point,application_type) and ensure a non-empty defaultSpec.Typeis set before submission. - Improve error reporting when SparkApplication enters failed-submission/unknown states, and document the new entrypoint behavior.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| plugins/sparkeks/README.md | Documents entrypoint selection and JAR configuration. |
| internal/pkg/object/command/sparkeks/sparkeks.go | Wires entrypoint selection into SparkApplication spec creation, adds new job parameters, sets default app type, improves error output. |
| internal/pkg/object/command/sparkeks/entrypoint.go | Implements extension-based entrypoint strategy selection and JAR application type resolution. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+99
to
+103
| var mainClass, applicationType string | ||
| if jobContext.Parameters != nil { | ||
| mainClass = jobContext.Parameters.EntryPoint | ||
| applicationType = jobContext.Parameters.ApplicationType | ||
| } |
Comment on lines
+21
to
+29
| // resolveJarApplicationType maps an optional configured application type (e.g. "Java", "Scala", | ||
| // case-insensitive) to its SparkApplication type, falling back to the default (Scala) when | ||
| // unset or unrecognized. | ||
| func resolveJarApplicationType(applicationType string) v1beta2.SparkApplicationType { | ||
| if t, ok := jarApplicationTypes[strings.ToLower(strings.TrimSpace(applicationType))]; ok { | ||
| return t | ||
| } | ||
| return defaultJarApplicationType | ||
| } |
Comment on lines
+84
to
+98
| ```json | ||
| { | ||
| "query": "SELECT key, value FROM my_table", | ||
| "wrapper_uri": "s3://mybucket/application-assembly.jar", | ||
| "parameters": { | ||
| "entry_point": "com.org.customapplication.main", | ||
| "application_type": "Scala", | ||
| "properties": { | ||
| "spark.application.custom.path": "s3://mybucket/output/v1" //custom param to be used in jar | ||
| } | ||
| }, | ||
| "return_result": false | ||
| } | ||
| ``` | ||
|
|
Enable the sparkeks plugin to run custom-class (Scala/Java) Spark applications (e.g. com.pattern.chipmunk.Writer) alongside the existing Python SQL wrapper. - entrypoint.go: entrypoint strategy pattern selected by wrapper_uri extension (.jar -> JAR/--class, .py/default -> SQL wrapper). JAR mode sets Spec.Type (Scala/Java via parameters.application_type, default Scala) and MainClass from parameters.entry_point. - Managed positional args [appName, queryURI, user, (resultURI)] shared by both entrypoints; job context `arguments`, when set, is passed to the app verbatim so callers can match an app's exact CLI contract. - Precompute s3a query/result URIs once on the execution context. - Write submission-failure / unknown-state reason to job stderr. - Default Spec.Type=Python guards templateless jobs. - plugins/sparkeks/README.md: document JAR entrypoint support, extension dispatch, entry_point/application_type, and the argument contract.
prasadlohakpure
force-pushed
the
feat/sparkeks_entrypoint_support
branch
from
July 30, 2026 10:20
427246c to
de0231c
Compare
Nitin Bhakar (nitinbhakar)
previously approved these changes
Jul 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds JAR/
--classentrypoint support to the Heimdallsparkeksplugin so it can run a custom-class(Java/Scala) Spark application, not just the built-in Python SQL wrapper. Additive and opt-in — no
existing
sparkekscommand sets a JAR wrapper, so current behavior is unchanged.Entrypoint strategy (
internal/pkg/object/command/sparkeks/entrypoint.go) — the entrypoint-specificpart of the spec (
Type/MainClass/Arguments) is dispatched behind anentrypointStrategyinterface, selected by the file extension of the command's
wrapper_uri:wrapper_uriSpec.TypeSpec.MainClass.pyPython.jarScala/Java(fromapplication_type, defaultScala)entry_pointPythonNew optional job-context fields:
parameters.entry_point(fully-qualified main class) andparameters.application_type(ScalaorJava, case-insensitive).Also in this branch: submission-failure/unknown-state reasons are now written to job stderr, and
s3aQueryURI/s3aResultURIare precomputed once on the execution context.Test plan
go build ./internal/... ./cmd/...— clean.go vet ./internal/pkg/object/command/sparkeks/— clean.gofmt -l internal/pkg/object/command/sparkeks/— clean.go test ./internal/pkg/object/command/sparkeks/— ok (existing package tests).The JAR argument contract was verified against the deployed chipmunk JAR by running it on the
chipmunk_eks_migration_testcollection, not by unit test.Notes / rollback
.jarwrapper_uri.The one non-additive piece is the properties-rewriting narrowing described above.
go build ./...across the whole repo reports "function main is undeclared"for every
plugins/*package — these are-buildmode=pluginpackages with nofunc main()and failthe same way on an untouched
origin/maincheckout. Not introduced by this PR.Testing details:
Job ID: 0304b4ff-b674-4b92-8b8a-2b2fd059ed6b
Job ID: 5369ba25-d7f2-437c-a28a-69b0a7d2b7d5