feat: support Iceberg 1.11, audit existing Iceberg diffs for old configs

[mbutrovich]

Draft while I test.

Which issue does this PR close?

Closes #4381.

Rationale for this change

Iceberg 1.11.0 is the first Iceberg release to support Spark 4.1, and it publishes a real iceberg-spark-runtime-4.1 artifact. Until now we tested Iceberg only against Spark 3.4/3.5, so the Spark 4.1 path went uncovered. This adds an Iceberg 1.11 / Spark 4.1 target so we can catch compatibility issues in that configuration.

Iceberg 1.11 also removed the legacy upstream Comet reader classes (the ones the older diffs had to delete), so the diff needed to enable Comet in Iceberg's own tests shrinks dramatically to just enabling Comet in the test session builders.

What changes are included in this PR?

• spark/pom.xml: the spark-4.1 profile now depends on the real iceberg-spark-runtime-4.1_2.13:1.11.0 instead of reusing the 4.0 runtime, so CometIcebergNativeSuite runs against genuine Spark 4.1 Iceberg.
• dev/diffs/iceberg/1.11.0.diff: new diff generated against the apache-iceberg-1.11.0 tag. Adds the comet version, wires Comet into build.gradle, and injects the Comet plugin / shuffle manager config into the spark/v4.1 test bases. Because 1.11 removed the legacy Comet integration entirely, upstream no longer provides a Comet dependency, so the diff adds it as testImplementation in the iceberg-spark and iceberg-spark-extensions modules (plus integrationImplementation in iceberg-spark-runtime); without these the test JVMs fail to load CometPlugin. No class deletions or reader edits are needed. Includes the new TestVariantShredding base (variant shredding is new in 1.11 and builds its own session).
• spark/src/main/scala/org/apache/comet/iceberg/IcebergReflection.scala: Iceberg 1.11 renamed two SparkScan accessors the native scan reflects on (expectedSchema() -> projection(), filterExpressions() -> filters()). Without this the reflection returned None and every Iceberg read silently fell back to Spark. Both use a new-name-then-old-name lookup so one build still supports Iceberg 1.8-1.11.
• spark/src/test/scala/org/apache/comet/CometIcebergNativeSuite.scala: tests that exercise the native scan now assert it actually engages (CometIcebergNativeScanExec present) instead of only checking results, so a silent fallback like the one above can't pass green again. Intentional-fallback tests assert the fallback with its reason.
• CI: iceberg_1_11 runs on every PR (our only Spark 4.1 Iceberg coverage) and iceberg_1_10 is demoted to main-only. A new run-iceberg-tests PR label enables the main-only Iceberg jobs (1.8/1.9/1.10) on a PR; the label is added to the preflight gate so applying it actually re-triggers the pipeline.
• Dropped dead/removed configs from all Iceberg test-enable blocks (1.8.1/1.9.1/1.10.0/1.11.0): spark.comet.schemaEvolution.enabled and spark.comet.exec.broadcastExchange.enabled (both gone from CometConf), and spark.comet.use.lazyMaterialization (no-op for the Iceberg native scan).
• Docs updated for the new version, pairing, and label.

How are these changes tested?

Existing Iceberg Spark test suites (iceberg-spark, iceberg-spark-extensions, iceberg-spark-runtime) run with Comet enabled via the new iceberg_1_11 CI job against Spark 4.1. CometIcebergNativeSuite exercises the native Iceberg scan against the real 4.1 runtime.

CI coverage across Spark and Iceberg versions (rows are the Spark version; columns are the Iceberg runtime paired with it):

Spark \ Iceberg	1.5.2	1.8.1	1.9.1	1.10.0	1.11.0
3.4	Comet suites — every PR	Iceberg tests — main + run-iceberg-tests	—	—	—
3.5	—	Comet suites — every PR	Iceberg tests — main + run-iceberg-tests	Iceberg tests — main + run-iceberg-tests	—
4.0	—	—	—	Comet suites — every PR	—
4.1	—	—	—	—	Comet suites + Iceberg tests — every PR
4.2	—	—	—	Comet suites — every PR	—

Comet suites = CometIcebergNativeSuite (and the other Iceberg-touching Comet unit suites) in the pr_build_linux "scans" bucket; the Iceberg runtime is pulled by the active spark-* profile in spark/pom.xml. Runs on every PR across all five Spark profiles.

Iceberg tests = Apache Iceberg's own iceberg-spark / -extensions / -runtime suites run with Comet enabled, via the iceberg_1_8 / iceberg_1_9 / iceberg_1_10 / iceberg_1_11 jobs. 1.11 (Spark 4.1) runs on every PR; the older three run on push-to-main or on a PR labeled run-iceberg-tests.

The projection() / filters() rename is only in Iceberg 1.11's spark/v4.1, so 4.1 x 1.11 is the only cell that exercises the new accessor names; every other Comet-suite cell exercises the old-name fallback, so backward compatibility is covered on every PR.

[mbutrovich]

9 failed tests from Iceberg 1.11 is not bad at all. I will take a look tomorrow.

[mbutrovich]

Reproduced the same 9 failures locally

TestRewriteTablePathsAction. formatVersion = 2
TestRewriteTablePathsAction. formatVersion = 2
TestSparkReaderDeletes. fileFormat = PARQUET, formatVersion = 2, vectorized = false, planningMode = DISTRIBUTED
TestSparkReaderDeletes. fileFormat = PARQUET, formatVersion = 2, vectorized = true, planningMode = LOCAL
TestFilterPushDown. catalogName = testhadoop, implementation = org.apache.iceberg.spark.SparkCatalog, config = {type=hadoop, cache-enabled=false}, planningMode = testhadoop
TestFilterPushDown. catalogName = testhadoop, implementation = org.apache.iceberg.spark.SparkCatalog, config = {type=hadoop, cache-enabled=false}, planningMode = testhadoop
TestSelect. catalogName = testhive, implementation = org.apache.iceberg.spark.SparkCatalog, config = {type=hive, default-namespace=default}, binaryTableName = testhive.default.binary_table
TestSelect. catalogName = testhadoop, implementation = org.apache.iceberg.spark.SparkCatalog, config = {type=hadoop, cache-enabled=false}, binaryTableName = testhadoop.default.binary_table
TestSelect. catalogName = spark_catalog, implementation = org.apache.iceberg.spark.SparkSessionCatalog, config = {type=hive, default-namespace=default, parquet-enabled=true, cache-enabled=false}, binaryTableName = default.binary_table