Core: Add support for HashiCorp Vault KMS client

[ebyhr]

Restores #14451 with additional changes:

• Extracted the module so non-HashiCorp users can avoid the dependency
• Added hashicorp to encryption.kms-type
• Added test with hashicorp/vault docker image
• Replaced environment variables with properties
• Made Vault KMS client serializable

Mailing list: https://lists.apache.org/thread/1269w5pzoy723sr1c6xxq8jg02zcf3on

Fixes #14437

[ebyhr]

@pvary could you take a look at this PR when you have a moment?

cc: @mrendi29

[pvary]

I think we should talk on the dev list about adding a new project and supporting a new KMS client.

[mrendi29]

tysm @ebyhr for picking this up.

I will test this KMS implementation against our staging env on Monday and get back to you if I spot anything.

[caushie-akamai]

I tried reproducing this today with the following steps:

1. Build jar ./gradlew build :iceberg-hashicorp:build -DsparkVersions=3.5 -x test -x integrationTest -x revapi --parallel
2. Copy iceberg jar + hashicorp jar into spark
3. Spawn spark-sql shell with vault configuration

But i am getting a java.lang.NoClassDefFoundError: org/apache/hc/core5/http/HttpEntity
Should we also shade http jars in here or would it be better to change the imports to the shaded version? WDYT

[mrendi29]

What are your thoughts about using an env var instead of explicitly specifying secretID in the config?

I believe this would be beneficial for folks who use spark-operator in k8s and would not like sensitive credentials to live in the spark config.

Internally we also grab the secretID from an environment variable to avoid this.

[ebyhr]

Environment variables don't align well with Trino's catalog-based architecture, because in theory the value can differ per catalog (even if that's not practical). I've added support for both properties and environment variables.

[mrendi29]

sounds good to me, this way we are supporting both query engines.

[mrendi29]

WDYT about also allowing clients to configure SSL config in the http client? There may be folks who do a mTLS vault deployment or who need to trust their vault deployment's CA.

I believe the bettercloud vault had a similar solution: https://github.com/BetterCloud/vault-java-driver/blob/master/src/main/java/com/bettercloud/vault/SslConfig.java#L44

I am trying to find if such a thing is also possible with the apache http client

[ebyhr]

Adding TLS support with the Apache HTTP client is straightforward. However, I'd like to keep the initial PR focused. Can we add TLS support in a follow-up?

[ebyhr]

@caushie-akamai I've added the following lines to iceberg-hashicorp project in build.gradle:

    apply plugin: 'com.gradleup.shadow'

    build.dependsOn shadowJar

Could you please confirm whether the NoClassDefFoundError issue has been resolved?

[mrendi29]

TYSM @ebyhr , i'll try to test this against staging tomorrow morning and get back to you

[mrendi29]

@ebyhr I tested this again today, I still see the error java.lang.NoClassDefFoundError: org/apache/hc/core5/http/HttpEntity . I'll see if i can spot the root cause of this tonight

[github-actions]

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.

[mrendi29]

I was on PTO the last 3 weeks, saw you pushed some changes. will test these this week

[mrendi29]

@ebyhr I took a stab at this today, and i still see the java.lang.NoClassDefFoundError: org/apache/hc/core5/http/HttpEntity. I was able to fix this by adding the hashicorp dependency on spark gradle build:

ecaushi@bos-lhvg11:~/iceberg$ git diff spark/v4.1/build.gradle
diff --git a/spark/v4.1/build.gradle b/spark/v4.1/build.gradle
index e6455fa34..a22114c88 100644
--- a/spark/v4.1/build.gradle
+++ b/spark/v4.1/build.gradle
@@ -236,6 +236,7 @@ project(":iceberg-spark:iceberg-spark-runtime-${sparkMajorVersion}_${scalaVersio
     }
     implementation project(':iceberg-gcp')
     implementation project(':iceberg-bigquery')
+    implementation project(':iceberg-hashicorp')
     implementation project(':iceberg-hive-metastore')
     implementation(project(':iceberg-nessie')) {
       exclude group: 'com.google.code.findbugs', module: 'jsr305'

IMO this should be fine as we are not introducing any 3rd party dependency wdyt?

After that I tried testing the integration with vault but I was facing constantly certificate issues.

Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

I then added the CA in the truststore and keystore but that also didn't help.

spark.driver.extraJavaOptions -Divy.cache.dir=/tmp -Divy.home=/tmp -Djavax.net.ssl.trustStore=/opt/spark/work-dir/vault-truststore.jks -Djavax.net.ssl.tru
stStorePassword=changeit -Djavax.net.ssl.keyStore=/opt/spark/work-dir/vault-truststore.jks -Djavax.net.ssl.keyStorePassword=changeit
spark.executor.extraJavaOptions -Djavax.net.ssl.trustStore=/opt/spark/work-dir/vault-truststore.jks -Djavax.net.ssl.trustStorePassword=changeit -Djavax.ne
t.ssl.keyStore=/opt/spark/work-dir/vault-truststore.jks -Djavax.net.ssl.keyStorePassword=changeit

I'll try to do a deeper dive on this tomorrow as well. What are your thoughts moving forward?

[github-actions]

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.