Spark: Add support for 4.2.0

[manuzhang]

This PR adds support for Apache Spark 4.2.0.

Release publishing: Spark 4.2 artifacts are excluded from dev/stage-binaries.sh and will not be published by default.

The first two commits rename the Spark 4.1 module to 4.2 and then restore 4.1 so Git retains file history. The remaining commits contain the Spark 4.2 compatibility changes and review follow-ups.

Changes

• Adds and registers the Spark 4.2 modules in Gradle, settings, packaging, benchmarks, CI, and CVE scanning.
• Integrates Spark 4.2's RelationCatalog and View APIs while preserving Iceberg property-only updates for view properties, including cached and session catalogs.
• Updates view validation and metadata handling for Spark-owned view expansion, catalog-qualified cycle detection and output, compatibility with existing Iceberg views, metric-view dependencies, and de-duplicated table/view listings.
• Converts geospatial values between Spark's BinaryView representation and Iceberg's pure WKB Parquet representation, including StructInternalRow and nested values.
• Updates Spark 4.2 tests, benchmarks, APIs, and error expectations, and resolves the released Spark 4.2.0 artifacts from Maven Central.

Geospatial compatibility details

• Validate WKB during Spark conversion: Spark 4.2 parses and validates WKB when constructing Geometry and Geography values. It accepts the seven OGC base geometry types, including Z/M variants, and enforces longitude and latitude bounds for geography. Existing Iceberg data with extended geometry types or out-of-bounds geography coordinates can now fail with wkbParseError instead of passing through as it did in Spark 4.1. POINT EMPTY remains accepted. Iceberg uses Spark's public conversion API and cannot bypass this validation.
• Keep nested geospatial conversion in the Spark 4.2 module: Spark 4.2 requires nested geometry and geography values to use its BinaryView physical representation, so arrays, maps, and structs must convert Iceberg WKB values explicitly. Older Spark versions use different physical types and can be handled separately.

View compatibility details

• Adapt to Spark 4.2's view and relation APIs: Spark 4.2 requires catalogs that provide both tables and views to implement RelationCatalog. It also represents query text, current catalog and namespace, schema mode, SQL configs, dependencies, and properties through its new View model. SparkCatalog and SparkSessionCatalog translate that model to and from Iceberg view metadata and implement create, replace, create-or-replace, load, drop, and rename through the new signatures.
• Use Spark-owned view expansion: ResolveRelations now calls RelationCatalog.loadRelation before extension resolution rules, and BaseCatalog.loadRelation falls back from tables to views. Spark therefore expands Iceberg views into logical View nodes and applies GetViewColumnByNameAndOrdinal plus schema-mode-driven casting. This replaces Iceberg 4.1's positional expansion. The unreachable ResolveViews relation expansion and its identifier rewrites have been removed; Spark function resolution now owns references such as system.bucket.
• Route Iceberg commands around session-catalog fallbacks: Spark's ResolveSessionCatalog exits early or selects V1 handling for several view commands. RewriteViewCommands recognizes only Iceberg-backed V2 view catalogs before that happens, while preserving local and global temporary-view behavior, and routes create, drop, rename, describe, show-create, show-properties, and show-views commands to Iceberg-aware logical and physical nodes. Resolved commands carry the already-loaded view into inspection planning, avoiding a second catalog lookup.
• Restore validation and cycle detection: Rewriting CreateView means Spark's native CheckViewReferences no longer sees the original node. CheckViews explicitly reuses Spark's checks for temporary objects, generated aliases, and column counts, and traverses Spark logical View nodes for fully qualified recursive-view detection. Iceberg retains its complete nested-expression traversal because Spark's helper only matches a SubqueryExpression at the root of an expression.
• Preserve Spark-only metadata across Iceberg commits: Iceberg view metadata does not have dedicated fields for every Spark value. SparkView round-trips query column names as JSON (with legacy comma-separated read compatibility), SQL configs, schema-binding mode, metric-view dependencies, engine versions, and non-default view types through reserved Iceberg properties. Views without Spark properties explicitly use COMPENSATION, including views created by older Iceberg versions or other engines, and type-promotion coverage verifies that behavior. Metric-view dependencies remain persisted because Iceberg has no native dependency field and Spark must receive the structured list again when loading the view.
• Keep updates compatible with Iceberg metadata: Iceberg-specific execution nodes keep ALTER VIEW SET/UNSET TBLPROPERTIES as property-only metadata commits because Spark 4.2 removed ViewCatalog.alterView, and its replacement path would discard non-Spark SQL representations. These commands translate identifiers through the owning catalog and invalidate cached plans. Rename preserves an existing cache entry. CREATE OR REPLACE VIEW removes omitted properties on a best-effort follow-up commit because Iceberg's ViewBuilder cannot yet express atomic property removals. The owning Spark catalog remains persisted in portable default-catalog metadata; the load path only falls back to the adapter's catalog name for older metadata where it is absent.
• Match Spark SQL inspection behavior: Iceberg-specific describe, show-create, and show-properties execution preserves property redaction, emits collation and schema mode in their DDL clauses, keeps metric-view type information, fully quotes catalog-qualified identifiers, and uses one reserved-property list so collation and internal metadata do not leak through another inspection command. View listing retains temporary views and avoids duplicate table/view entries without loading every view in the namespace.
• Generated aliases are now validated: Aligning with Spark's persistent-view checks means statements such as CREATE VIEW v AS SELECT id + 1 FROM t are rejected unless the expression has an explicit alias. spark.sql.legacy.allowAutoGeneratedAliasForView=true provides the Spark compatibility escape hatch.

Verification

• Focused TestViews coverage for recursive cycles, rename behavior, property replacement, and property ALTER across the existing catalog configurations.
• TestViewCatalogCache across named and session catalogs, covering replace, property removal, ALTER, rename, and cache invalidation.
• TestSparkCatalogIdentifierMapping, TestSparkCatalogOperations, TestSparkSessionCatalog, TestSparkParquetReader, TestSparkParquetWriter, TestAlterTable, and TestTimestampWithoutZone.
• Spark 4.2 core and extensions Spotless and Checkstyle checks for main and test sources.
• git diff --check.

---

AI Disclosure

• Model: GPT-5 Codex
• Platform/Tool: Codex
• Human Oversight: partially reviewed
• Prompt Summary: Add Spark 4.2 support and address compatibility issues in views, catalogs, relation listings, geospatial values, tests, and CI coverage.

[manuzhang]

This failure from testing Spark 4.2.0-preview2 is caused by apache/spark#53788, after which an AnalysisException would be thrown on Iceberg metadata tables like default.table.partitions.

TestAddFilesProcedure > addPartitionsWithNullValueShouldAddFilesToNullPartition() > catalogName = spark_catalog, implementation = org.apache.iceberg.spark.SparkSessionCatalog, config = {type=hive, default-namespace=default, parquet-enabled=true, cache-enabled=false}, formatVersion = 2 FAILED
    org.apache.spark.sql.AnalysisException: [REQUIRES_SINGLE_PART_NAMESPACE] spark_catalog requires a single-part namespace, but got `default`.`table`. SQLSTATE: 42K05

[manuzhang]

Failed tests after upgrading to Spark 4.2.0-preview3-rc1

1. testJoinsHourToDays() in TestStoragePartitionedJoins.java:

Assertion failed: "SPJ should not change query output: number of results should match"
The actual and expected query result sizes differ, indicating that either the join logic or test data setup causes a mismatch.

2. readFromViewReferencingTempFunction() in TestViews.java:

Assertion failed: Expected a specific routine not found error, but got an AnalysisException with different message details.

[manuzhang]

apache/spark#54884 has been opened to fix the first failure.

[manuzhang]

I will update HourToDaysReducer following interface changes from apache/spark#54884 in next preview release. All other test failures have been fixed.

[szehon-ho]

leave a note here to implement the new method in the Reducer once apache/spark#54884 is in (next Spark 4.2 preview)

Sorry just saw, it is the same comment

[manuzhang]

The failed tests in 4.2.0-preview3 have been fixed in 4.2.0-preview4.

[nssalian]

I took a look at the variant module.
The variant + DSv2 side looks like a clean carry-forward from 4.1. SparkV2Filters, SparkScanBuilder, PruneColumnsWithoutReordering, and the reader/writer/shredding code are all identical to 4.1, and the Spark APIs they depend on (VariantVal, getVariant, VariantGet incl. its try_variant_get/variant_get representation, VariantUtil.Type) are unchanged from 4.1.0 to 4.2.0, so variant_get filter pushdown and reads should stay coherent.

Also, I checked the spark-tests (4.2, 2.13, core) last workflow run and all six variant cases are succeeding: TestSparkVariants, TestSparkVariantRead, TestStructInternalRowVariant, TestVariantShredding, TestFilterPushDown#testVariantExtractFiltering, and TestSelect#variantTypeInFilter. testVariantExtractFiltering passing confirms the try_variant_get(...) residual-filter assertions hold on 4.2, so no concern.

Only nit: the Verification section lists the geospatial suites but not the variant ones. Mind adding a one-liner for the record?

[manuzhang]

@nssalian Thanks for review. One-liner on variant tests has been added.

[szehon-ho]

Reviewed the view changes against the Spark 4.2.0 API. Most comments are inline; a few notes that don't map to a single line.

Moving BaseCatalog to RelationCatalog was mandatory — Spark 4.2 rejects a plugin that implements both TableCatalog and ViewCatalog without it (Catalogs.scala:118-122). But that change also rewires how Iceberg views are read: ResolveRelations sits ahead of extendedResolutionRules in the Resolution batch, loadRelation falls back to loadView, and RelationResolution.createRelation turns the result into a V1 scan. So Spark now owns view expansion. Worth stating explicitly in the PR description, because it is user-visible: 4.1 expanded views strictly positionally, while 4.2 uses GetViewColumnByNameAndOrdinal plus schema-mode-driven casting.

I also looked at whether the IcebergAlterV2View*Exec nodes could be dropped in favour of Spark's, and I don't think they can. Details in the reply on CheckViews.scala, but the short version is that Spark expresses a property change as a full replaceView, which against Iceberg's model makes UNSET a silent no-op and drops non-Spark dialect representations. Keeping these execs looks correct for the life of this module.

Two smaller items without a good anchor:

• collation appears in SHOW TBLPROPERTIES but is filtered out of SHOW CREATE VIEW. Spark filters all three inspection commands with one list. Worth making consistent, and a test pinning the expected output of the three commands would keep the forked display code from re-drifting.
• The DESCRIBE / SHOW paths call loadView during analysis, which adds a catalog round-trip per statement.

For the release notes: CREATE VIEW v AS SELECT id + 1 FROM t now fails, from adopting Spark's verifyAutoGeneratedAliasesNotExists. I read that as the intended consequence of aligning with Spark rather than a regression, but it is a breaking change and should be called out, along with the spark.sql.legacy.allowAutoGeneratedAliasForView escape hatch.

Two questions. What is the intent behind normalizeViewCurrentCatalog nulling out default-catalog, and does it round-trip for views shared with other engines? And should spark.view-dependencies be persisted at all in 4.2 — only CreateV2MetricViewExec populates viewDependencies, plain CREATE VIEW leaves it null, and CatalogV2Util.viewInfoBuilderFrom drops it on the ALTER path, so committing to an encoding now means supporting it indefinitely.

[szehon-ho]

Filed and fixed the Spark-side bug I mentioned in the dependency question in my review: SPARK-58604 / apache/spark#57802.

[GGraziadei]

I’ve reviewed the PR. Overall, it is well structured, but there are a few points I’d like to clarify and discuss before moving forward.

Thanks for the effort you’ve put into this. This is a fairly large PR touching several areas, and I appreciate that putting all the pieces together and keeping everything consistent is not a trivial task.

I’ve left a few comments and questions where I think we could clarify the behavior or make the intent more explicit. Happy to discuss them and work through the open points together.

Thanks again for putting this together!

[nssalian]

Discussed with @manuzhang offline, at some point we should freeze this PR to include all the current changes and allow reviewers to focus on this baseline. Once the baseline 4.2 work is in, we can forward port the active changes going into 4.0 and 4.1 Spark into the 4.2 branch.

[GGraziadei]

Agreed @nssalian. I think this PR is simply too large, the right approach here is to split the sub-problems into separate issues and land them as smaller PR (once this merged). I follow this approach for my open comments.

[szehon-ho]

Thanks @manuzhang, this is a lot of careful work. A few structural checks first, since they're hard to see in a 643-file diff and I think they're worth recording:

• spark/v4.1 at this head is byte-identical to the merge base, so the rename-and-restore commits net out correctly and every real change is confined to spark/v4.2 plus nine build/CI files.
• That registration set matches what the 4.1 PR (3179684) touched, file for file, plus cve-scan.yml - good catch, the 4.1 PR missed that one.
• CI exercises the new module for real: spark-tests (4.2, 2.13, core), (4.2, 2.13, extensions) and spark-runtime-4.2_2.13 all pass.

On the geospatial changes, I went through these against Spark's branch-4.2 sources and agree that nearly all of it is forced: GeometryVal/GeographyVal are deleted in 4.2, SpecializedGetters replaced getGeometry/getGeography with getBinaryView, and STUtils.stAsBinary is gone. The geography SRID change isn't compile-forced but is required for correctness, and your own TestSparkSchemaUtil change is the evidence - EPSG:4269 throws on 4.1 and round-trips on 4.2 while Iceberg's SparkSchemaUtil is unchanged between the modules, so non-CRS84 geography columns only became reachable now. I also convinced myself that not using stGeomAsBinary in the writer is the right call, since it re-serializes through toWkb(ByteOrder.LITTLE_ENDIAN); geospatialWriterStoresPureWkb pins that well with the big-endian fixture.

The view/extensions layer is where I'd most like your take, since a few of the new paths can fail in ways 4.1 couldn't. Two of these come out of 4.2 API removals and I think the approach is right - they just have a rough edge each:

• ALTER VIEW reaching the Iceberg catalog directly is forced, since 4.1's ViewCatalog.alterView(ident, ViewChange...) is gone in 4.2 and replaceView would discard non-Spark SQL representations. That's worth stating in the description. The part I'd change is that the identifier is rebuilt with Spark3Util rather than buildIdentifier, so a catalog subclass that overrides the mapping breaks on ALTER VIEW only.
• The two-phase commit in removeDroppedViewProperties follows from the same thing: replaceView is full-state, so absent properties must be dropped, and Iceberg's ViewBuilder can't express that in one commit. The window is real but the fix belongs in core - here I'd just make the second commit best-effort rather than failing a replace that already succeeded.

The other two aren't forced, and both look cheap to resolve:

• The per-view loadView in SparkSessionCatalog#listRelationSummaries exists only to distinguish METRIC_VIEW in the tableType column, and SparkCatalog doesn't do it at all - so the two catalogs already disagree.
• listTables no longer agrees with listTableSummaries about views. The split itself is forced by RelationCatalog (summaries are tables-only now, where 4.1 put views in listTableSummaries); filtering listTables to match is the missing half.

Separately, RenameV2ViewExec looks like it was simply missed by the Iceberg* renaming - Spark 4.2 has its own class by that name in the same package, so the two shadow each other.

I also went through the change looking for anything not actually forced by 4.2, on the theory that this PR is easier to review and to revert if it stays close to "make Iceberg work on 4.2". The renames, the View-object ViewCatalog signatures, RelationCatalog, the BinaryView migration, PlanUtils#collectPushedSparkExpressions, the reportDriverMetrics disambiguation, property-only ALTER VIEW commits and the reserved-property round-tripping all check out as unavoidable. Four things don't, and I've suggested each inline as a follow-up rather than a change here:

• nested array/map geospatial handling in StructInternalRow (new capability - 4.1 throws)
• ALTER VIEW ... WITH SCHEMA support (new capability, no 4.1 counterpart)
• the metric-view tableType distinction in listings, which is what forces the per-view load above
• the two new cached-catalog configurations in TestViews, roughly +66% on that class's runtime
• the private cycle-detection copy in CheckViews, now that 4.2 exposes ViewHelper.checkCyclicViewReference

Plus the GenericsHelpers map-value fix, which is a real bug but a pre-existing one in every Spark module. None of these are blocking - if you'd rather keep them, the descriptions just want a line each.

[nssalian]

Give the large surface and 4.2 being relatively new, this PR would be great to have in main at some point. But, based on the discussion in the mailing list thread: https://lists.apache.org/thread/cvn448s85v2g835dfwxpz2z1j2hczok9 I've removed it from the 1.12 milestone so this gets more eyes and time before going in.

[manuzhang]

@szehon-ho Thanks for the thorough review. I've addressed all the comments, either fixing as recommended, or updating comments, or opening follow-up issues. Please take another look at your convenience. We are not in a rush to get this in now, thanks for @nssalian's heads-up.

[szehon-ho]

Thank you @manuzhang for persistence, sorry about the previous review if its not so clear. I'll try to make it more clear in the future. It's just a huge pr so hard to review as other folks mention.

I'm a bit swamped but try to take a look in next day or two.

By the way, @nssalian would one option maybe, we merge some version and then fix different parts in smaller prs? For the release, we will have the source code, but this won't make any release artifact, would that work?

[huan233usc]

Thank you @manuzhang for persistence, sorry about the previous review if its not so clear. I'll try to make it more clear in the future. It's just a huge pr so hard to review as other folks mention.

I'm a bit swamped but try to take a look in next day or two.

By the way, @nssalian would one option maybe, we merge some version and then fix different parts in smaller prs? For the release, we will have the source code, but this won't make any release artifact, would that work?

+1 to merging a version first and then fixing the individual parts in smaller PRs — right now it does feel like we're reviewing a moving target.

IIUC about the changes -- the bulk of the real logic is the view migration to Spark 4.2's RelationCatalog (Spark now owns view parsing/resolution, so ResolveViews changes a lot, plus the new Iceberg*V2View exec nodes and catalog wiring). The geo side looks mostly like a mechanical GeometryVal/GeographyVal → BinaryView migration.

We could do additional follow-ups items and check like cross-version view testing to guard against regressions.

[manuzhang]

@szehon-ho No worry. The review comments are valuable. It's just that how we can collaborate more efficiently on such a huge PR involving both AI and human.

By the way, @nssalian would one option maybe, we merge some version and then fix different parts in smaller prs? For the release, we will have the source code, but this won't make any release artifact, would that work?

+1. I can remove the codes to publish the 4.2 artifacts.

[nssalian]

By the way, @nssalian would one option maybe, we merge some version and then fix different parts in smaller prs? For the release, we will have the source code, but this won't make any release artifact, would that work?

This makes more sense. Let's get the baseline there and then we can add features on top of it.
@manuzhang is there a good baseline PR you can make if haven't already that we can aim for the 1.12 release?

[manuzhang]

@nssalian I think this PR is already in good shape after many rounds of reviews. The only remaining work is to remove the release codes.

[manuzhang]

This excludes Spark 4.2 support from source-release tarball.