feat: add native Delta Lake scan contrib module (page/row-group pruning) #5365
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,263 @@ | ||
| # Licensed to the Apache Software Foundation (ASF) under one | ||
| # or more contributor license agreements. See the NOTICE file | ||
| # distributed with this work for additional information | ||
| # regarding copyright ownership. The ASF licenses this file | ||
| # to you under the Apache License, Version 2.0 (the | ||
| # "License"); you may not use this file except in compliance | ||
| # with the License. You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, | ||
| # software distributed under the License is distributed on an | ||
| # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
| # KIND, either express or implied. See the License for the | ||
| # specific language governing permissions and limitations | ||
| # under the License. | ||
|
|
||
| name: Delta Contrib Tests | ||
|
|
||
| # Reusable: invoked by ci.yml. Triggering, path filters, and concurrency | ||
| # live in the umbrella workflow. | ||
| on: | ||
| workflow_call: | ||
|
|
||
| permissions: | ||
| contents: read | ||
|
|
||
| env: | ||
| RUST_VERSION: stable | ||
| RUST_BACKTRACE: 1 | ||
| # Force GNU ld on Linux: rust-lld cannot resolve -ljvm against the Zulu JDK | ||
| # layout installed by setup-java (same rationale as pr_build_linux.yml). | ||
| RUSTFLAGS: "-Clink-arg=-fuse-ld=bfd" | ||
| # The container's default locale is POSIX, which makes the JVM's file-path encoder | ||
| # reject non-ASCII partition directory names the suites create. | ||
| LANG: "C.UTF-8" | ||
| LC_ALL: "C.UTF-8" | ||
|
|
||
| jobs: | ||
|
|
||
| contrib-delta: | ||
| name: Delta contrib (Spark ${{ matrix.profile.spark }}) | ||
| runs-on: ubuntu-24.04 | ||
| container: | ||
|
Member
There was a problem hiding this comment. This container has no Docker socket, and Could you either mount
Contributor
Author
There was a problem hiding this comment.
Added a plain-runner job, |
||
| image: amd64/rust | ||
| strategy: | ||
| matrix: | ||
| profile: | ||
| - spark: "3.5" | ||
| java_version: "17" | ||
| - spark: "4.0" | ||
| java_version: "17" | ||
| - spark: "4.1" | ||
| java_version: "17" | ||
| # spark-4.2 is intentionally absent: the contrib profile is dormant | ||
| # until a Delta release supports Spark 4.2. | ||
| fail-fast: false | ||
| steps: | ||
| - uses: actions/checkout@v7 | ||
|
|
||
| - name: Setup Rust & Java toolchain | ||
| uses: ./.github/actions/setup-builder | ||
| with: | ||
| rust-version: ${{ env.RUST_VERSION }} | ||
| jdk-version: ${{ matrix.profile.java_version }} | ||
|
|
||
| - name: Restore Maven dependencies | ||
| id: maven-cache | ||
| uses: actions/cache/restore@v6 | ||
| with: | ||
| path: | | ||
| ~/.m2/repository | ||
| /root/.m2/repository | ||
| key: ${{ runner.os }}-java-maven-${{ hashFiles('**/pom.xml') }}-delta-${{ matrix.profile.spark }} | ||
| restore-keys: | | ||
| ${{ runner.os }}-java-maven- | ||
|
|
||
| - name: Restore Cargo cache | ||
| uses: actions/cache/restore@v6 | ||
| with: | ||
| path: | | ||
| ~/.cargo/registry | ||
| ~/.cargo/git | ||
| native/target | ||
| key: ${{ runner.os }}-cargo-ci-${{ hashFiles('native/**/Cargo.lock', 'native/**/Cargo.toml') }}-${{ hashFiles('native/**/*.rs') }} | ||
| restore-keys: | | ||
| ${{ runner.os }}-cargo-ci-${{ hashFiles('native/**/Cargo.lock', 'native/**/Cargo.toml') }}- | ||
|
|
||
| - name: Build native library (CI profile) | ||
| run: | | ||
| cd native | ||
| # `delta` is in the default feature set, so no feature flag is needed. | ||
| cargo build --profile ci | ||
| env: | ||
| # Must match the flags spark_sql_test_reusable.yml builds with: | ||
| # cargo folds RUSTFLAGS into its fingerprints, so any divergence | ||
| # would make the shared cargo cache restore without ever hitting. | ||
| RUSTFLAGS: "-Ctarget-cpu=x86-64-v3 -Clink-arg=-fuse-ld=bfd" | ||
|
|
||
| # No cache save: this workflow never runs on a push to main, so the restore above | ||
| # falls back to the cache build_linux saves there. | ||
|
|
||
| - name: Stage native library at release path | ||
| run: | | ||
| # Maven's -Prelease profile (activated below) expects libcomet.so | ||
| # under native/target/release/; --profile ci builds it under | ||
| # native/target/ci/ instead (same as the other native-building | ||
| # workflows), so copy it into place. | ||
| mkdir -p native/target/release | ||
| cp native/target/ci/libcomet.so native/target/release/libcomet.so | ||
|
|
||
| - name: Install Comet core jars | ||
| run: | | ||
| # The delta profile is what produces the spark test-jar the contrib suites depend on. | ||
| ./mvnw -B -q -Prelease -Pspark-${{ matrix.profile.spark }},delta install -pl common,spark -DskipTests -Dspotless.check.skip=true | ||
|
|
||
| - name: Run Delta contrib test suites | ||
| run: | | ||
| SPARK_HOME=$(pwd) COMET_CONF_DIR=$(pwd)/conf ./mvnw -B -Prelease -Pspark-${{ matrix.profile.spark }},delta test -pl contrib/delta-spark | ||
|
|
||
| # Saved only on main: an entry written from a pull request or a | ||
| # `gh-readonly-queue/*` branch cannot be restored by any later run, and | ||
| # it evicts main's from the shared budget. See "Large caches are written | ||
| # on main only" in README.md. | ||
| - name: Save Maven dependencies | ||
| if: ${{ github.ref == 'refs/heads/main' && steps.maven-cache.outputs.cache-hit != 'true' }} | ||
| uses: actions/cache/save@v6 | ||
| with: | ||
| path: | | ||
| ~/.m2/repository | ||
| /root/.m2/repository | ||
| key: ${{ runner.os }}-java-maven-${{ hashFiles('**/pom.xml') }}-delta-${{ matrix.profile.spark }} | ||
|
|
||
| # The container jobs above have no Docker socket, so CometDeltaS3Suite cancels itself | ||
| # there. This job runs it on the plain runner, where Testcontainers can start MinIO. | ||
| contrib-delta-s3: | ||
| name: Delta contrib S3 suite (Spark 3.5, MinIO) | ||
| runs-on: ubuntu-24.04 | ||
| steps: | ||
| - uses: actions/checkout@v7 | ||
|
|
||
| # setup-builder runs apt-get as root inside the container jobs; the hosted runner needs | ||
| # sudo for that, so the same steps are spelled out here. | ||
| - name: Install build dependencies | ||
| run: | | ||
| sudo apt-get update | ||
| sudo apt-get install -y protobuf-compiler clang | ||
|
|
||
| - name: Install JDK 17 | ||
| uses: actions/setup-java@v4 | ||
| with: | ||
| distribution: 'zulu' | ||
| java-version: "17" | ||
|
|
||
| - name: Setup Rust toolchain | ||
| run: | | ||
| rustup toolchain install ${{ env.RUST_VERSION }} | ||
| rustup default ${{ env.RUST_VERSION }} | ||
| rustup component add rustfmt clippy | ||
|
|
||
| - name: Restore Maven dependencies | ||
| id: maven-cache | ||
| uses: actions/cache/restore@v6 | ||
| with: | ||
| path: | | ||
| ~/.m2/repository | ||
| key: ${{ runner.os }}-java-maven-${{ hashFiles('**/pom.xml') }}-delta-3.5 | ||
| restore-keys: | | ||
| ${{ runner.os }}-java-maven- | ||
|
|
||
| - name: Restore Cargo cache | ||
| uses: actions/cache/restore@v6 | ||
| with: | ||
| path: | | ||
| ~/.cargo/registry | ||
| ~/.cargo/git | ||
| native/target | ||
| key: ${{ runner.os }}-cargo-ci-${{ hashFiles('native/**/Cargo.lock', 'native/**/Cargo.toml') }}-${{ hashFiles('native/**/*.rs') }} | ||
| restore-keys: | | ||
| ${{ runner.os }}-cargo-ci-${{ hashFiles('native/**/Cargo.lock', 'native/**/Cargo.toml') }}- | ||
|
|
||
| - name: Build native library (CI profile) | ||
| run: | | ||
| cd native | ||
| cargo build --profile ci | ||
| env: | ||
| RUSTFLAGS: "-Ctarget-cpu=x86-64-v3 -Clink-arg=-fuse-ld=bfd" | ||
|
|
||
| - name: Stage native library at release path | ||
| run: | | ||
| mkdir -p native/target/release | ||
| cp native/target/ci/libcomet.so native/target/release/libcomet.so | ||
|
|
||
| - name: Install Comet core jars | ||
| run: | | ||
| ./mvnw -B -q -Prelease -Pspark-3.5,delta install -pl common,spark -DskipTests -Dspotless.check.skip=true | ||
|
|
||
| - name: Run the MinIO-backed S3 suite | ||
| run: | | ||
| docker info > /dev/null | ||
| SPARK_HOME=$(pwd) COMET_CONF_DIR=$(pwd)/conf ./mvnw -B -Prelease -Pspark-3.5,delta test -pl contrib/delta-spark -DwildcardSuites=org.apache.comet.contrib.delta.CometDeltaS3Suite | ||
|
|
||
| # Saved only on main, for the same reason as the container job above. | ||
| - name: Save Maven dependencies | ||
| if: ${{ github.ref == 'refs/heads/main' && steps.maven-cache.outputs.cache-hit != 'true' }} | ||
| uses: actions/cache/save@v6 | ||
| with: | ||
| path: | | ||
| ~/.m2/repository | ||
| key: ${{ runner.os }}-java-maven-${{ hashFiles('**/pom.xml') }}-delta-3.5 | ||
|
|
||
| # `delta` is in the default feature set, so no regular job builds without it; | ||
| # this keeps the feature-off build and its "built without the delta feature" | ||
| # error arm (planner.rs cfg(not(feature = "delta"))) from becoming dead code. | ||
| feature-off-build: | ||
|
github-advanced-security[bot] marked this conversation as resolved.
Fixed
|
||
| name: Feature-off native build | ||
| runs-on: ubuntu-24.04 | ||
| container: | ||
| image: amd64/rust | ||
| steps: | ||
| - uses: actions/checkout@v7 | ||
|
|
||
| - name: Setup Rust & Java toolchain | ||
| uses: ./.github/actions/setup-builder | ||
| with: | ||
| rust-version: ${{ env.RUST_VERSION }} | ||
| jdk-version: "17" | ||
|
|
||
| - name: Restore Cargo cache | ||
| uses: actions/cache/restore@v6 | ||
| with: | ||
| path: | | ||
| ~/.cargo/registry | ||
| ~/.cargo/git | ||
| native/target | ||
| key: ${{ runner.os }}-cargo-ci-${{ hashFiles('native/**/Cargo.lock', 'native/**/Cargo.toml') }}-${{ hashFiles('native/**/*.rs') }} | ||
| restore-keys: | | ||
| ${{ runner.os }}-cargo-ci-${{ hashFiles('native/**/Cargo.lock', 'native/**/Cargo.toml') }}- | ||
|
|
||
| - name: Test the delta-off error path | ||
| run: | | ||
| cd native | ||
| # The test binary links JNI; libjvm.so must be resolvable at load | ||
| # time (same as .github/actions/rust-test). | ||
| export LD_LIBRARY_PATH=${JAVA_HOME}/lib/server:${LD_LIBRARY_PATH} | ||
| cargo test -p datafusion-comet --no-default-features --features hdfs-opendal delta_scan | ||
|
|
||
| # The contrib's dev tooling (benchmark and regression-harness scripts) is | ||
| # Python; keep it import-clean on one supported interpreter. | ||
| dev-scripts-python: | ||
|
github-advanced-security[bot] marked this conversation as resolved.
Fixed
|
||
| name: Delta dev scripts (Python) | ||
| runs-on: ubuntu-24.04 | ||
| steps: | ||
| - uses: actions/checkout@v7 | ||
|
|
||
| - name: Setup Python | ||
| uses: actions/setup-python@v6 | ||
| with: | ||
| python-version: "3.12" | ||
|
|
||
| - name: Byte-compile contrib dev scripts | ||
| run: | | ||
| python -m compileall -q contrib/delta-spark/dev | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,66 @@ | ||
| <!-- | ||
| Licensed to the Apache Software Foundation (ASF) under one | ||
| or more contributor license agreements. See the NOTICE file | ||
| distributed with this work for additional information | ||
| regarding copyright ownership. The ASF licenses this file | ||
| to you under the Apache License, Version 2.0 (the | ||
| "License"); you may not use this file except in compliance | ||
| with the License. You may obtain a copy of the License at | ||
|
|
||
| http://www.apache.org/licenses/LICENSE-2.0 | ||
|
|
||
| Unless required by applicable law or agreed to in writing, | ||
| software distributed under the License is distributed on an | ||
| "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
| KIND, either express or implied. See the License for the | ||
| specific language governing permissions and limitations | ||
| under the License. | ||
| --> | ||
|
|
||
| # Comet Delta Lake Contrib (experimental) | ||
|
|
||
| Native Delta Lake reads for Comet. Delta tables are scanned through Comet's | ||
| existing native Parquet reader, so they get row-group pruning, page-index | ||
| pruning, and filter pushdown, with deletion vectors applied inside the scan. | ||
|
|
||
| Support is experimental and explicitly opt-in. Two things are required: | ||
|
|
||
| 1. This module's jar (`comet-contrib-delta-spark`) on the classpath, alongside | ||
| `delta-spark`. It is never bundled into `comet-spark`; without it, Comet | ||
| has no Delta surface at all. | ||
| 2. `spark.comet.scan.delta.enabled=true`. The default is `false`, so the jar | ||
| alone does nothing. | ||
|
|
||
| Unsupported tables and features fall back to Spark's reader. See the | ||
| [user guide](https://datafusion.apache.org/comet/user-guide/latest/delta.html) | ||
| for configuration details. | ||
|
|
||
| ## Supported versions | ||
|
|
||
| | Spark | Delta | Status | | ||
| | ----- | -------------- | --------------------------------------------- | | ||
| | 3.5 | 3.3.x | supported | | ||
| | 4.0 | 4.0.x | supported | | ||
| | 4.1 | 4.3.x | supported | | ||
| | 3.4 | delta-core 2.4 | not supported (older Delta, would need shims) | | ||
| | 4.2 | none released | inert until Delta ships a Spark 4.2 release | | ||
|
|
||
| ## Building and testing | ||
|
|
||
| The module builds under the `delta` Maven profile. It resolves `comet-spark` | ||
| from the local Maven repository, so install `common` and `spark` from the same | ||
| checkout immediately before, as CI does, with the `delta` profile active so that | ||
| install produces the spark test-jar the contrib suites depend on; a stale sibling | ||
| install is the trap the contributor guide warns about: | ||
|
|
||
| ```shell | ||
| ./mvnw -Pspark-3.5,delta install -pl common,spark -DskipTests | ||
| ./mvnw -Pspark-3.5,delta install -pl contrib/delta-spark | ||
| ``` | ||
|
|
||
| Run the test suites the same way (`test` instead of `install` on the second | ||
| line). CI runs them | ||
| on Spark 3.5, 4.0, and 4.1 via `.github/workflows/delta_contrib_test.yml`. | ||
|
|
||
| `dev/` contains a benchmark script (`bench_delta_comet.py`) and a harness for | ||
| running Delta's own test suites against Comet (`run-delta-regression.sh`). |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.