You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@iceberg.apache.org by bl...@apache.org on 2025/05/05 17:08:20 UTC
(iceberg) branch main updated: Spec additions for encryption (#12162)
This is an automated email from the ASF dual-hosted git repository.
blue pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/iceberg.git
The following commit(s) were added to refs/heads/main by this push:
new 7f3f450bbd Spec additions for encryption (#12162)
7f3f450bbd is described below
commit 7f3f450bbddf55bb383ff1409d6d0ca4557c9ffc
Author: ggershinsky <gg...@users.noreply.github.com>
AuthorDate: Mon May 5 20:08:14 2025 +0300
Spec additions for encryption (#12162)
---
format/spec.md | 25 +++++++++++++++++++++++++
1 file changed, 25 insertions(+)
diff --git a/format/spec.md b/format/spec.md
index 918582f016..6dc0a7d65d 100644
--- a/format/spec.md
+++ b/format/spec.md
@@ -53,6 +53,7 @@ Version 3 of the Iceberg spec extends data types and existing metadata structure
* Multi-argument transforms for partitioning and sorting
* Row Lineage tracking
* Binary deletion vectors
+* Table encryption keys
## Goals
@@ -742,6 +743,7 @@ A snapshot consists of the following fields:
| _optional_ | _required_ | _required_ | **`summary`** | A string map that summarizes the snapshot changes, including `operation` as a _required_ field (see below) |
| _optional_ | _optional_ | _optional_ | **`schema-id`** | ID of the table's current schema when the snapshot was created |
| | | _required_ | **`first-row-id`** | The first `_row_id` assigned to the first row in the first data file in the first manifest, see [Row Lineage](#row-lineage) |
+| | | _optional_ | **`key-id`** | ID of the encryption key that encrypts the manifest list key metadata |
The snapshot summary's `operation` field is used by some operations, like snapshot expiration, to skip processing certain snapshots. Possible `operation` values are:
@@ -941,6 +943,7 @@ Table metadata consists of the following fields:
| _optional_ | _optional_ | _optional_ | **`statistics`** | A list (optional) of [table statistics](#table-statistics). |
| _optional_ | _optional_ | _optional_ | **`partition-statistics`** | A list (optional) of [partition statistics](#partition-statistics). |
| | | _required_ | **`next-row-id`** | A `long` higher than all assigned row IDs; the next snapshot's `first-row-id`. See [Row Lineage](#row-lineage). |
+| | | _optional_ | **`encryption-keys`** | A list (optional) of [encryption keys](#encryption-keys) used for table encryption. |
For serialization details, see Appendix C.
@@ -1029,6 +1032,22 @@ The unified partition type looks like `Struct<field#1, field#2, field#3>`.
and then the table has evolved into `spec#1` which has just one field `{field#2}`.
The unified partition type looks like `Struct<field#1, field#2>`.
+#### Encryption Keys
+
+Keys used for table encryption can be tracked in table metadata as a list named `encryption-keys`. The schema of each key is a struct with the following fields:
+
+| v1 | v2 | v3 | Field name | Type. | Description |
+|----|----|------------|------------------------------|-----------------------|-------------|
+| | | _required_ | **`key-id`** | `string` | ID of the encryption key |
+| | | _required_ | **`encrypted-key-metadata`** | `string` | Encrypted key and metadata, base64 encoded [1] |
+| | | _optional_ | **`encrypted-by-id`** | `string` | Optional ID of the key used to encrypt or wrap `key-metadata` |
+| | | _optional_ | **`properties`** | `map<string, string>` | A string to string map of additional metadata used by the table's encryption scheme |
+
+Notes:
+
+1. The format of encrypted key metadata is determined by the table's encryption scheme and can be a wrapped format specific to the table's KMS provider.
+
+
### Commit Conflict Resolution and Retry
When two commits happen at the same time and are based on the same version, only one commit will succeed. In most cases, the failed commit can be applied to the new current version of table metadata and retried. Updates verify the conditions under which they can be applied to a new version and retry if those conditions are met.
@@ -1528,6 +1547,7 @@ Table metadata is serialized as a JSON object according to the following table.
|**`sort-orders`**|`JSON sort orders (list of sort field object)`|`See above`|
|**`default-sort-order-id`**|`JSON int`|`0`|
|**`refs`**|`JSON map with string key and object value:`<br />`{`<br /> `"<name>": {`<br /> `"snapshot-id": <id>,`<br /> `"type": <type>,`<br /> `"max-ref-age-ms": <long>,`<br /> `...`<br /> `}`<br /> `...`<br />`}`|`{`<br /> `"test": {`<br /> `"snapshot-id": 123456789000,`<br /> `"type": "tag",`<br /> `"max-ref-age-ms": 10000000`<br /> `}`<br />`}`|
+|**`encryption-keys`**|`JSON list of encryption key objects`|`[ {"key-id": "5f819b", "key-metadata": "aWNlYmVyZwo="} ]`|
### Name Mapping Serialization
@@ -1654,6 +1674,11 @@ Row-level delete changes:
* These position delete files must be merged into the DV for a data file when one is created
* Position delete files that contain deletes for more than one data file need to be kept in table metadata until all deletes are replaced by DVs
+Encryption changes:
+
+* Encryption keys are tracked by table metadata `encryption-keys`
+* The encryption key used for a snapshot is specified by `key-id`
+
### Version 2
Writing v1 metadata: