Spec: Document standard encryption key metadata by xanderbailey · Pull Request #16527 · apache/iceberg

xanderbailey · 2026-05-22T11:59:29Z

Before this PR

The key_metadata field in manifest entries and the encrypted-key-metadata field in table metadata are described as "implementation-specific" in the spec. While implementing table encryption in iceberg-rust, we found that the actual binary formats used by the Java implementation are required for cross-implementation interop but aren't documented anywhere - you have to reverse-engineer them from the Java source.

See mailing list thread: [DISCUSS] Specifying the encryption key metadata formats for cross-implementation interop

After this PR

The spec now documents:

The version-prefixed Avro binary format for per-file key_metadata (DEK, AAD prefix, file length)
The two-tier key hierarchy (KEKs wrapped by KMS vs manifest list keys encrypted by KEKs)

Another implementation can now correctly read/write encrypted tables without referencing the Java source.

xanderbailey · 2026-05-22T12:01:12Z

cc: @ggershinsky in case you have any thoughts here

…ementation interop Specify the binary wire format for the key_metadata field in manifest entries and the key hierarchy in table metadata encryption-keys, which were previously described only as "implementation-specific" but are required for implementations to interoperate.

szlta

Looks good to me, just a minor nits.

szlta · 2026-05-22T12:46:46Z

+
+The standard encryption scheme uses a two-tier key hierarchy tracked in the table metadata `encryption-keys` list:
+
+1. **Key Encryption Keys (KEKs):** Entries where `encrypted-by-id` equals the table's encryption key ID (configured via `encryption.key-id`). The `encrypted-key-metadata` contains the KEK wrapped by the KMS and is opaque to Iceberg — its format is determined by the KMS provider.


Perhaps mention here that a KEY_TIMESTAMP property is expected to be present for KEKs - AFAIK without it decryption flow will error out.

Yep, good shout. 5f38182

szlta · 2026-05-22T12:49:38Z

+| Field name | Avro type | Required | Description |
+|---|---|---|---|
+| **`encryption_key`** | `bytes` | _required_ | The data encryption key (DEK) for this file. Must be 16, 24, or 32 bytes (corresponding to AES-128, AES-192, or AES-256). |
+| **`aad_prefix`** | `bytes` | _optional_ | Random AAD prefix used for [AES GCM Stream](gcm-stream-spec.md) block authentication. |


ggershinsky · 2026-05-22T13:02:51Z

I think this makes a lot of sense, thank you. I'll have a look at the details.

- Clarify that aad_prefix is for integrity protection, not authentication - Document that KEY_TIMESTAMP property is required on KEK entries

ggershinsky · 2026-05-23T06:22:19Z

    | _optional_ | _optional_ | _optional_ | **`125  lower_bounds`**           | `map<126: int, 127: binary>`                                                | Map from column id to lower bound in the column serialized as binary [1]. Each value must be less than or equal to all non-null, non-NaN values in the column for the file [2] |
    | _optional_ | _optional_ | _optional_ | **`128  upper_bounds`**           | `map<129: int, 130: binary>`                                                | Map from column id to upper bound in the column serialized as binary [1]. Each value must be greater than or equal to all non-null, non-Nan values in the column for the file [2] |
-    | _optional_ | _optional_ | _optional_ | **`131  key_metadata`**           | `binary`                                                                    | Implementation-specific key metadata for encryption |
+    | _optional_ | _optional_ | _optional_ | **`131  key_metadata`**           | `binary`                                                                    | Per-file encryption key metadata. See [Standard Key Metadata](#standard-key-metadata) for the interoperable format used by the standard encryption scheme. |


there is also a key_metadata field in the Manifest File struct (field id 519)

ggershinsky · 2026-05-23T06:31:54Z

+| Field name | Avro type | Required | Description |
+|---|---|---|---|
+| **`encryption_key`** | `bytes` | _required_ | The data encryption key (DEK) for this file. Must be 16, 24, or 32 bytes (corresponding to AES-128, AES-192, or AES-256). |
+| **`aad_prefix`** | `bytes` | _optional_ | Random AAD prefix used for [AES GCM Stream](gcm-stream-spec.md) integrity protection. |


AAD prefix is used not only in AES GCM Stream files, but also in encrypted Parquet files (https://parquet.apache.org/docs/file-format/data-pages/encryption/ or https://github.com/apache/parquet-format/blob/master/Encryption.md)

ggershinsky · 2026-05-23T06:32:47Z

+| **`aad_prefix`** | `bytes` | _optional_ | Random AAD prefix used for [AES GCM Stream](gcm-stream-spec.md) integrity protection. |
+| **`file_length`** | `long` | _optional_ | The plaintext file length before encryption. Used to detect truncation attacks (see [AES GCM Stream file length](gcm-stream-spec.md#file-length)). |
+
+The AAD prefix is combined with a 4-byte little-endian block index to form the AAD for each AES GCM Stream cipher block, as described in the [AES GCM Stream AAD section](gcm-stream-spec.md#additional-authenticated-data).


in Parquet encryption, this works differently, https://parquet.apache.org/docs/file-format/data-pages/encryption/

ggershinsky · 2026-05-23T06:35:45Z

+|---|---|---|---|
+| **`encryption_key`** | `bytes` | _required_ | The data encryption key (DEK) for this file. Must be 16, 24, or 32 bytes (corresponding to AES-128, AES-192, or AES-256). |
+| **`aad_prefix`** | `bytes` | _optional_ | Random AAD prefix used for [AES GCM Stream](gcm-stream-spec.md) integrity protection. |
+| **`file_length`** | `long` | _optional_ | The plaintext file length before encryption. Used to detect truncation attacks (see [AES GCM Stream file length](gcm-stream-spec.md#file-length)). |


This keeps file length after encryption, https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/encryption/AesGcmInputFile.java#L45

Only for AES GCM Stream files. Not set/used for encrypted Parquet data files.

Clarify that file_length stores the encrypted (ciphertext) length, not plaintext. Broaden aad_prefix and file_length descriptions to cover both AES GCM Stream and Parquet modular encryption formats.

xanderbailey · 2026-05-26T11:08:33Z

Have hopefully addressed comments! Thanks folks. What's the process for getting spec changes merged? Do people typically vote or is just a regular PR approval?

ggershinsky · 2026-05-26T12:40:35Z

+| Field name | Avro type | Required | Description |
+|---|---|---|---|
+| **`encryption_key`** | `bytes` | _required_ | The data encryption key (DEK) for this file. Must be 16, 24, or 32 bytes (corresponding to AES-128, AES-192, or AES-256). |
+| **`aad_prefix`** | `bytes` | _optional_ | Random AAD prefix used for encryption integrity protection. For [AES GCM Stream](gcm-stream-spec.md) files, the prefix is combined with a block index to form the per-block AAD. For [Parquet modular encryption](https://parquet.apache.org/docs/file-format/data-pages/encryption/), the prefix is passed as the `aad_file_unique` component. |


For Parquet, it also passed as an AAD Prefix parameter; it then combined with AAD Suffixes.
aad_file_unique is something else (a Parquet-internal parameter)

ggershinsky · 2026-05-26T12:43:07Z

+
+The usage of the `encryption_key` and `aad_prefix` fields depends on the file format:
+
+* **AES GCM Stream files** (manifest lists, manifests, and non-Parquet data files): The `encryption_key` is used directly as the AES-GCM key. The `aad_prefix` is combined with a 4-byte little-endian block index to form the AAD for each cipher block, as described in the [AES GCM Stream AAD section](gcm-stream-spec.md#additional-authenticated-data). The `file_length` field stores the encrypted file length for truncation detection.


Iceberg table encryption is not supported yet for ORC data format, so "manifest lists, manifests, and non-Parquet data files" should be something like "manifest lists, manifests, and Avro data files"

Actually delete vectors (Puffin files) are also AES GCM encrypted for encrypted tables. I think the list is long enough to worth showing it as a proper list in the spec rather than in just brackets?

Made it a list here bcaf862

ggershinsky · 2026-05-27T07:31:57Z

+| Field name | Avro type | Required | Description |
+|---|---|---|---|
+| **`encryption_key`** | `bytes` | _required_ | The data encryption key (DEK) for this file. Must be 16, 24, or 32 bytes (corresponding to AES-128, AES-192, or AES-256). |
+| **`aad_prefix`** | `bytes` | _optional_ | Random AAD prefix used for encryption integrity protection. For [AES GCM Stream](gcm-stream-spec.md) files, the prefix is combined with a block index to form the per-block AAD. For [Parquet modular encryption](https://parquet.apache.org/docs/file-format/data-pages/encryption/), the prefix is passed as the AAD prefix parameter, which is combined with a per-file random `aad_file_unique` to form the full file AAD. |


I'd suggest changing "the prefix is passed as the AAD prefix parameter, which is combined with a per-file random aad_file_unique to form the full file AAD." to something like "the prefix is passed as the AAD prefix parameter, which is combined with a module AAD suffix to form the full AAD for each Parquet module."
(per the PME spec, aad_file_unique is a part of the suffix)

rdblue · 2026-05-27T19:59:17Z

I don't think that this should go into the table spec. The intent when we started work on encryption was to keep the table parts generic and write an encryption-specific spec.

I think an encryption spec is the right path forward.

xanderbailey · 2026-05-27T21:40:08Z

Thanks for taking a look @rdblue! Apologies if I haven't understood correctly, is your suggestion to add a new file in addition to the gcm stream encryption file? And reference it from the main spec?

xanderbailey · 2026-06-02T18:51:34Z

Hi @rdblue, any guidance here on how to move forward? Happy to change things up here I'm just not sure on the next steps.

github-actions · 2026-07-03T00:37:53Z

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.

xanderbailey · 2026-07-03T04:58:39Z

Please don't close, not stale.

xanderbailey · 2026-07-03T04:59:36Z

Gentle nudge here @rdblue for some guidance on what the next steps here should be. Thanks in advance!

mbutrovich · 2026-07-06T16:33:43Z

I will try to raise this on the July 15 community call if we don't have some movement here by then.

xanderbailey

Wonder if we need to bring some of the https://github.com/apache/iceberg/blob/main/docs/docs/encryption.md into this spec also. Interested to hear people's thoughts.

github-actions · 2026-08-24T00:12:04Z

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.

xanderbailey · 2026-08-24T06:58:40Z

@rdblue whats the best way for me to get a review here?

singhpk234

Thanks @xanderbailey, i am supportive of documenting this !

i added some questions inline it more intended to folks interested / working group of encryption not just specific to the pr

singhpk234 · 2026-08-24T16:36:02Z

+where:
+
+* `VersionByte` is a single byte indicating the key metadata schema version. Currently, the only valid version is `0x01`.
+* `Payload` is an Avro binary-encoded record (not a container file — only the raw binary encoding of the fields) using the schema for the given version.


[doubt] how are other language implementations dealing with it ? they are using avro library, or this encoding are standard enough to be replicated ?

https://github.com/apache/iceberg-rust/blob/4908b0b1d160043c9b8a4e1da160e72e85610812/crates/iceberg/src/encryption/key_metadata.rs#L143-L168

This is the rust implementation and it works correctly.

singhpk234 · 2026-08-24T16:37:38Z

+* **AES GCM Stream files**:
+  - Manifest lists
+  - Manifests
+  - Avro data files


what about ORC ?

is this abstracted carefully with the FileFormat API ?

ORC has it's own encryption spec I believe and doesn't stream encrypt https://orc.apache.org/specification/ORCv1/

singhpk234 · 2026-08-24T16:39:32Z

+
+The standard encryption scheme uses a two-tier key hierarchy tracked in the table metadata [`encryption-keys`](spec.md#encryption-keys) list:
+
+1. **Key Encryption Keys (KEKs):** Entries where `encrypted-by-id` equals the table's encryption key ID (configured via `encryption.key-id`). The `encrypted-key-metadata` contains the KEK wrapped by the KMS and is opaque to Iceberg — its format is determined by the KMS provider. KEK entries must include a `KEY_TIMESTAMP` property recording the creation time in milliseconds since epoch; this timestamp is used as the AAD when encrypting manifest list key metadata.


[doubt] why is KEK_TIMESTAMP hard coded expiry 720 days ? i know its the maximal time, i wonder if this is something we should make configurable, systems would prefer monthly rotations ?

It's constant today in java

iceberg/core/src/main/java/org/apache/iceberg/encryption/StandardEncryptionManager.java

Lines 40 to 42 in 5a1e297

// Maximal lifespan of key encryption keys is 2 years according to NIST SP 800-57 (PART 1 REV. 5,

// section 5.3.6.7.b)

private static final long KEY_ENCRYPTION_KEY_LIFESPAN_MS = TimeUnit.DAYS.toMillis(730);

I think making it configurable is fine. But it does not affect this spec patch.

cool its not for this spec but i wonder if this is something we should document, how do you feel clients having different KEK lifespan_ms for now ?

can be workable, I think. But if this made say a table property, then all clients will use the same lifespan.

xanderbailey · 2026-08-24T18:03:05Z

Thanks @xanderbailey, i am supportive of documenting this !

i added some questions inline it more intended to folks interested / working group of encryption not just specific to the pr

I think that would be valuable as different client libraries start to implement this. Might be worth a mailing list thread?

singhpk234

This mostly looks good to me !

if @ggershinsky is fine lets move ahead to VOTE, there seems to general concencus of adding this

singhpk234 · 2026-08-25T18:31:11Z

+### Encryption Key Hierarchy
+
+The standard encryption scheme uses a two-tier key hierarchy tracked in the table metadata [`encryption-keys`](spec.md#encryption-keys) list:
+
+1. **Key Encryption Keys (KEKs):** Entries where `encrypted-by-id` equals the table's encryption key ID (configured via `encryption.key-id`). The `encrypted-key-metadata` contains the KEK wrapped by the KMS and is opaque to Iceberg — its format is determined by the KMS provider. KEK entries must include a `KEY_TIMESTAMP` property recording the creation time in milliseconds since epoch; this timestamp is used as the AAD when encrypting manifest list key metadata.
+
+2. **Manifest List Keys:** Entries where `encrypted-by-id` references a KEK. The `encrypted-key-metadata` contains the Standard Key Metadata (defined above) encrypted with AES GCM using the referenced unwrapped KEK. The ciphertext format is:
+
+    ```
+    Nonce Ciphertext Tag
+    ```
+
+where `Nonce` is 12 bytes, `Ciphertext` is the encrypted Standard Key Metadata payload, and `Tag` is the 16-byte GCM authentication tag. The AAD for this encryption is the KEK's `KEY_TIMESTAMP` property value encoded as UTF-8 bytes.
+
+The snapshot field `key-id` references the encryption key entry used to encrypt that snapshot's manifest list key metadata.


[optional] I understand the text ! i wonder if we should have some sort of a DAG diagram of these key hierrarchy ?

KEKs can be shared by snapshots and is encrypted by TMK, MLK is encrypted by KEK .... .

Yes, I think the intention is that KEK is used to encrypt multiple manifest list keys (this doesn't have to be the case by the spec I think and implementations would work with multiple KEK - MLKs). I don't think this needs to be enforced by the spec. WDYT?

I agree i was saying it just from visualization pov, its just an optional comment !

singhpk234 · 2026-08-25T18:34:58Z

+
+* **AES GCM Stream files**:
+  - Manifest lists
+  - Manifests


[orthogonal] is there a plan to utilize the PME when in v4 manifests are written in parquet ? if yes how

I'm not aware of a conversation about this but would be happy to start a thread in the mailing list.

xanderbailey · 2026-09-03T07:17:20Z

VOTE thread here https://lists.apache.org/thread/hpdmzppsl87l0olkpzm3q47occ9cfydo

gaborkaszab

I believe we should reduce the coupling between the table spec and the encryption spec. I feel that the current version talks too much about the exact table metadata elements where each encryption technique is meant to be used, and I think rather that we could be more generic with the encryption spec.

gaborkaszab · 2026-09-03T06:29:05Z

+
+Encrypted key material is tracked in two places:
+
+* The `key_metadata` fields in [manifest entries](spec.md#manifests), [manifest list entries](spec.md#manifest-lists), and [statistics files](spec.md#table-metadata-fields) store the per-file key material.


key_metadata in table stats files is going to be deprecated here: #17533
Listing where the field is found now might not age well if the list changes by time.

gaborkaszab · 2026-09-03T06:42:41Z

+Encrypted key material is tracked in two places:
+
+* The `key_metadata` fields in [manifest entries](spec.md#manifests), [manifest list entries](spec.md#manifest-lists), and [statistics files](spec.md#table-metadata-fields) store the per-file key material.
+* The table metadata [`encryption-keys`](spec.md#encryption-keys) list tracks the key hierarchy used to protect that per-file material.


If I'm not mistaken encrypted-keys in table metadata is not to protect the key_metadata fields listed above.

gaborkaszab · 2026-09-03T06:45:01Z

+
+## Standard Key Metadata
+
+The `key_metadata` field in manifest entries stores per-file encryption key material as a binary blob. To enable cross-implementation interoperability, the standard encryption scheme defines the following binary format for this field:


I think the spec's purpose is cross-implementation interoperability, no need to spell it out: "To enable cross-implementation interoperability". I think we can remove this part of the sentence and just say that the this is the scheme for key_metadata

gaborkaszab · 2026-09-03T08:26:29Z

+
+## Standard Key Metadata
+
+The `key_metadata` field in manifest entries stores per-file encryption key material as a binary blob. To enable cross-implementation interoperability, the standard encryption scheme defines the following binary format for this field:


I'd rather remove reference to the table format such as mentioning that manifest entries have a key-metadata field. I think that in the encryption spec we should be more general and not assuming that which table metadata structs have what fields.

gaborkaszab · 2026-09-03T08:40:19Z

+
+1. **Key Encryption Keys (KEKs):** Entries where `encrypted-by-id` equals the table's encryption key ID (configured via `encryption.key-id`). The `encrypted-key-metadata` contains the KEK wrapped by the KMS and is opaque to Iceberg — its format is determined by the KMS provider. KEK entries must include a `KEY_TIMESTAMP` property recording the creation time in milliseconds since epoch; this timestamp is used as the AAD when encrypting manifest list key metadata.
+
+2. **Manifest List Keys:** Entries where `encrypted-by-id` references a KEK. The `encrypted-key-metadata` contains the Standard Key Metadata (defined above) encrypted with AES GCM using the referenced unwrapped KEK. The ciphertext format is:


Saying "Manifest List" is too narrow IMO. In V4, root manifest files, and as a future enhancement, stat files could also store their encryption metadata within this list.

gaborkaszab · 2026-09-03T08:45:11Z

+
+The usage of the `encryption_key` and `aad_prefix` fields depends on the file format:
+
+* **AES GCM Stream files**:


For a regular reader "AES GCM Stream files" might not mean much in an Iceberg spec. Taking a look at the list below, isn't this AVRO files and Puffin files?

gaborkaszab · 2026-09-03T08:49:40Z

+
+### Encryption Key Hierarchy
+
+The standard encryption scheme uses a two-tier key hierarchy tracked in the table metadata [`encryption-keys`](spec.md#encryption-keys) list:


This gives the impression that all encryption related information should go into this structure. I think that what is true instead is that there are 2 different way: 1) raw key-metadata stored per file as described in the above sections, or 2) key-id driven encrypted key-metadata stored together (where sharing between encrypted files is possible).
I think we should articulate this more.

Also, for 2) it's essential to have a KMS. Might make sense to dedicate a sort paragraph to describe what it is and what assumptions we might have.

gaborkaszab · 2026-09-03T08:54:07Z

@@ -731,7 +664,7 @@ The `data_file` struct consists of the following fields:
    | _optional_ | _optional_ |            | ~~**`111  distinct_counts`**~~    | `map<123: int, 124: long>`                                                  | **Deprecated. Do not write.** |
    | _optional_ | _optional_ | _optional_ | **`125  lower_bounds`**           | `map<126: int, 127: binary>`                                                | Map from column id to lower bound in the column serialized as binary [1]. Each value must be less than or equal to all non-null, non-NaN values in the column for the file [2] |
    | _optional_ | _optional_ | _optional_ | **`128  upper_bounds`**           | `map<129: int, 130: binary>`                                                | Map from column id to upper bound in the column serialized as binary [1]. Each value must be greater than or equal to all non-null, non-Nan values in the column for the file [2] |
-    | _optional_ | _optional_ | _optional_ | **`131  key_metadata`**           | `binary`                                                                    | Implementation-specific key metadata for encryption |
+    | _optional_ | _optional_ | _optional_ | **`131  key_metadata`**           | `binary`                                                                    | Per-file encryption key metadata. See [Standard Key Metadata](encryption-spec.md#standard-key-metadata) for the interoperable format used by the standard encryption scheme. |


I think this should be enough:
"Per-file encryption key metadata. See Standard Key Metadata."
Same goes for other occurrences.

github-actions Bot added the Specification Issues that may introduce spec changes. label May 22, 2026

xanderbailey force-pushed the xb/spec-encryption-key-metadata branch from 57ab53b to d67101b Compare May 22, 2026 12:07

xanderbailey force-pushed the xb/spec-encryption-key-metadata branch from d67101b to ae07984 Compare May 22, 2026 12:09

szlta reviewed May 22, 2026

View reviewed changes

Spec: Address review feedback on encryption key metadata

5f38182

- Clarify that aad_prefix is for integrity protection, not authentication - Document that KEY_TIMESTAMP property is required on KEK entries

ggershinsky reviewed May 23, 2026

View reviewed changes

comments

7d60b4c

xanderbailey force-pushed the xb/spec-encryption-key-metadata branch from b58d594 to 7d60b4c Compare May 26, 2026 08:48

Spec: Distinguish AES GCM Stream vs Parquet encryption in key metadata

4f12e28

Clarify that file_length stores the encrypted (ciphertext) length, not plaintext. Broaden aad_prefix and file_length descriptions to cover both AES GCM Stream and Parquet modular encryption formats.

xanderbailey force-pushed the xb/spec-encryption-key-metadata branch from 4690c89 to 4f12e28 Compare May 26, 2026 08:54

ggershinsky reviewed May 26, 2026

View reviewed changes

comments

bcaf862

xanderbailey force-pushed the xb/spec-encryption-key-metadata branch from 4fce858 to bcaf862 Compare May 26, 2026 14:06

xanderbailey mentioned this pull request May 26, 2026

Support client-side encryption apache/iceberg-rust#2034

Open

ggershinsky reviewed May 27, 2026

View reviewed changes

github-actions Bot added the stale label Jul 3, 2026

github-actions Bot removed the stale label Jul 4, 2026

github-actions Bot added the docs label Jul 24, 2026

move to new spec

21b7a21

xanderbailey force-pushed the xb/spec-encryption-key-metadata branch from 2bd9086 to 21b7a21 Compare July 24, 2026 13:10

xanderbailey commented Jul 24, 2026

View reviewed changes

github-actions Bot added the stale label Aug 24, 2026

singhpk234 reviewed Aug 24, 2026

View reviewed changes

github-actions Bot removed the stale label Aug 25, 2026

singhpk234 added the not-stale label Aug 25, 2026

singhpk234 reviewed Aug 25, 2026

View reviewed changes

add v3

b3756cd

xanderbailey force-pushed the xb/spec-encryption-key-metadata branch from 126b741 to b3756cd Compare August 25, 2026 21:28

gaborkaszab reviewed Sep 3, 2026

View reviewed changes


		The standard encryption scheme uses a two-tier key hierarchy tracked in the table metadata `encryption-keys` list:

		1. Key Encryption Keys (KEKs): Entries where `encrypted-by-id` equals the table's encryption key ID (configured via `encryption.key-id`). The `encrypted-key-metadata` contains the KEK wrapped by the KMS and is opaque to Iceberg — its format is determined by the KMS provider.

	\| `aad_prefix` \| `bytes` \| _optional_ \| Random AAD prefix used for [AES GCM Stream](gcm-stream-spec.md) block authentication. \|
	\| `aad_prefix` \| `bytes` \| _optional_ \| Random AAD prefix used for [AES GCM Stream](gcm-stream-spec.md) integrity protection. \|


		The usage of the `encryption_key` and `aad_prefix` fields depends on the file format:

		* AES GCM Stream files (manifest lists, manifests, and non-Parquet data files): The `encryption_key` is used directly as the AES-GCM key. The `aad_prefix` is combined with a 4-byte little-endian block index to form the AAD for each cipher block, as described in the [AES GCM Stream AAD section](gcm-stream-spec.md#additional-authenticated-data). The `file_length` field stores the encrypted file length for truncation detection.

	// Maximal lifespan of key encryption keys is 2 years according to NIST SP 800-57 (PART 1 REV. 5,
	// section 5.3.6.7.b)
	private static final long KEY_ENCRYPTION_KEY_LIFESPAN_MS = TimeUnit.DAYS.toMillis(730);


		Encrypted key material is tracked in two places:

		* The `key_metadata` fields in [manifest entries](spec.md#manifests), [manifest list entries](spec.md#manifest-lists), and [statistics files](spec.md#table-metadata-fields) store the per-file key material.


		## Standard Key Metadata

		The `key_metadata` field in manifest entries stores per-file encryption key material as a binary blob. To enable cross-implementation interoperability, the standard encryption scheme defines the following binary format for this field:


		1. Key Encryption Keys (KEKs): Entries where `encrypted-by-id` equals the table's encryption key ID (configured via `encryption.key-id`). The `encrypted-key-metadata` contains the KEK wrapped by the KMS and is opaque to Iceberg — its format is determined by the KMS provider. KEK entries must include a `KEY_TIMESTAMP` property recording the creation time in milliseconds since epoch; this timestamp is used as the AAD when encrypting manifest list key metadata.

		2. Manifest List Keys: Entries where `encrypted-by-id` references a KEK. The `encrypted-key-metadata` contains the Standard Key Metadata (defined above) encrypted with AES GCM using the referenced unwrapped KEK. The ciphertext format is:


		The usage of the `encryption_key` and `aad_prefix` fields depends on the file format:

		* AES GCM Stream files:


		### Encryption Key Hierarchy

		The standard encryption scheme uses a two-tier key hierarchy tracked in the table metadata [`encryption-keys`](spec.md#encryption-keys) list:

Uh oh!

Conversation

xanderbailey commented May 22, 2026

Before this PR

After this PR

Uh oh!

xanderbailey commented May 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

szlta left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ggershinsky commented May 22, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xanderbailey commented May 26, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rdblue commented May 27, 2026

Uh oh!

xanderbailey commented May 27, 2026

Uh oh!

xanderbailey commented Jun 2, 2026

Uh oh!

github-actions Bot commented Jul 3, 2026

Uh oh!

xanderbailey commented Jul 3, 2026

Uh oh!

xanderbailey commented Jul 3, 2026

Uh oh!

mbutrovich commented Jul 6, 2026

Uh oh!

xanderbailey left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Aug 24, 2026

Uh oh!

xanderbailey commented Aug 24, 2026

Uh oh!

singhpk234 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xanderbailey commented May 22, 2026 •

edited

Loading

singhpk234 Aug 25, 2026 •

edited

Loading