You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "ggershinsky (via GitHub)" <gi...@apache.org> on 2025/11/19 07:18:51 UTC
[PR] Docs: encryption [iceberg]
ggershinsky opened a new pull request, #14621:
URL: https://github.com/apache/iceberg/pull/14621
(no comment)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Docs: encryption [iceberg]
Posted by "palladium-coder (via GitHub)" <gi...@apache.org>.
palladium-coder commented on code in PR #14621:
URL: https://github.com/apache/iceberg/pull/14621#discussion_r2576173348
##########
docs/docs/encryption.md:
##########
@@ -0,0 +1,152 @@
+---
+title: "Encryption"
+---
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements. See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License. You may obtain a copy of the License at
+ -
+ - http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+# Encryption
+
+Iceberg table encryption protects confidentiality and integrity of table data in an untrusted storage. The `data`, `delete`, `manifest` and `manifest list` files are encrypted and tamper-proofed before being sent to the storage backend.
+
+The `metadata.json` file does not contain data or stats, and is therefore not encrypted.
+
+Currently, encryption is supported in the Hive and REST catalogs for tables with Parquet and Avro data formats.
+
+Two parameters are required to activate encryption of a table,
+1. Catalog property `encryption.kms-impl`, that specifies the class path for a client of a KMS ("key management service").
+2. Table property `encryption.key-id`, that specifies the ID of a master key used to encrypt and decrypt the table. Master keys are stored and managed in the KMS.
+
+The `encryption.key-id` must be set during the table creation, and never modified or removed during the table lifetime.
Review Comment:
Is there a document/spec which tracks how catalogs should behave .
I would prefer to avoid accidents like these and any custom catalogue to know these behaviours exist .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Docs: encryption [iceberg]
Posted by "ggershinsky (via GitHub)" <gi...@apache.org>.
ggershinsky commented on code in PR #14621:
URL: https://github.com/apache/iceberg/pull/14621#discussion_r2787630310
##########
docs/docs/encryption.md:
##########
@@ -0,0 +1,153 @@
+---
+title: "Encryption"
+---
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements. See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License. You may obtain a copy of the License at
+ -
+ - http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+# Encryption
+
+Iceberg table encryption protects confidentiality and integrity of table data in an untrusted storage. The `data`, `delete`, `manifest` and `manifest list` files are encrypted and tamper-proofed before being sent to the storage backend.
+
+The `metadata.json` file does not contain data or stats, and is therefore not encrypted.
Review Comment:
Thanks again @singhpk234, can you please clarify a few points
1. Is the following true? - partition summary is not a part of the snapshot summary in the Iceberg spec https://iceberg.apache.org/spec/#optional-snapshot-summary-fields ; but in the implementation, it is added sometimes to the snapshots and can contain data stats.
2. If yes, when the partition summaries are enabled, and when do they have stats? Is any of this under the writing user control?
3. Can you give an example of such a usecase (that writes stats to the metadata.json)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Docs: encryption [iceberg]
Posted by "singhpk234 (via GitHub)" <gi...@apache.org>.
singhpk234 commented on code in PR #14621:
URL: https://github.com/apache/iceberg/pull/14621#discussion_r2783581749
##########
docs/docs/encryption.md:
##########
@@ -0,0 +1,153 @@
+---
+title: "Encryption"
+---
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements. See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License. You may obtain a copy of the License at
+ -
+ - http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+# Encryption
+
+Iceberg table encryption protects confidentiality and integrity of table data in an untrusted storage. The `data`, `delete`, `manifest` and `manifest list` files are encrypted and tamper-proofed before being sent to the storage backend.
+
+The `metadata.json` file does not contain data or stats, and is therefore not encrypted.
Review Comment:
I believe this is not true, please check this PR on stats contained in the metadata.json https://github.com/apache/iceberg/pull/14502#issue-3587751964, thoughts on exploit
cc @ggershinsky
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Docs: encryption [iceberg]
Posted by "palladium-coder (via GitHub)" <gi...@apache.org>.
palladium-coder commented on code in PR #14621:
URL: https://github.com/apache/iceberg/pull/14621#discussion_r2576173273
##########
docs/docs/encryption.md:
##########
@@ -0,0 +1,152 @@
+---
+title: "Encryption"
+---
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements. See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License. You may obtain a copy of the License at
+ -
+ - http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+# Encryption
+
+Iceberg table encryption protects confidentiality and integrity of table data in an untrusted storage. The `data`, `delete`, `manifest` and `manifest list` files are encrypted and tamper-proofed before being sent to the storage backend.
+
+The `metadata.json` file does not contain data or stats, and is therefore not encrypted.
+
+Currently, encryption is supported in the Hive and REST catalogs for tables with Parquet and Avro data formats.
+
+Two parameters are required to activate encryption of a table,
+1. Catalog property `encryption.kms-impl`, that specifies the class path for a client of a KMS ("key management service").
+2. Table property `encryption.key-id`, that specifies the ID of a master key used to encrypt and decrypt the table. Master keys are stored and managed in the KMS.
+
+The `encryption.key-id` must be set during the table creation, and never modified or removed during the table lifetime.
Review Comment:
Is there a document/spec which tracks how catalogs should behave .
I would prefer to avoid accidents like these and any custom catalogue to know these behaviours exist .
##########
docs/docs/encryption.md:
##########
@@ -0,0 +1,152 @@
+---
+title: "Encryption"
+---
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements. See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License. You may obtain a copy of the License at
+ -
+ - http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+# Encryption
+
+Iceberg table encryption protects confidentiality and integrity of table data in an untrusted storage. The `data`, `delete`, `manifest` and `manifest list` files are encrypted and tamper-proofed before being sent to the storage backend.
+
+The `metadata.json` file does not contain data or stats, and is therefore not encrypted.
+
+Currently, encryption is supported in the Hive and REST catalogs for tables with Parquet and Avro data formats.
+
+Two parameters are required to activate encryption of a table,
+1. Catalog property `encryption.kms-impl`, that specifies the class path for a client of a KMS ("key management service").
+2. Table property `encryption.key-id`, that specifies the ID of a master key used to encrypt and decrypt the table. Master keys are stored and managed in the KMS.
+
+The `encryption.key-id` must be set during the table creation, and never modified or removed during the table lifetime.
Review Comment:
Is there a document/spec which tracks how catalogs should behave .
I would prefer to avoid accidents like these and any custom catalogue to know these behaviours exist .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Docs: encryption [iceberg]
Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao merged PR #14621:
URL: https://github.com/apache/iceberg/pull/14621
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Docs: encryption [iceberg]
Posted by "ggershinsky (via GitHub)" <gi...@apache.org>.
ggershinsky commented on PR #14621:
URL: https://github.com/apache/iceberg/pull/14621#issuecomment-3551319299
cc @huaxingao
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Docs: encryption [iceberg]
Posted by "palladium-coder (via GitHub)" <gi...@apache.org>.
palladium-coder commented on code in PR #14621:
URL: https://github.com/apache/iceberg/pull/14621#discussion_r2579867296
##########
docs/docs/encryption.md:
##########
@@ -0,0 +1,152 @@
+---
+title: "Encryption"
+---
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements. See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License. You may obtain a copy of the License at
+ -
+ - http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+# Encryption
+
+Iceberg table encryption protects confidentiality and integrity of table data in an untrusted storage. The `data`, `delete`, `manifest` and `manifest list` files are encrypted and tamper-proofed before being sent to the storage backend.
+
+The `metadata.json` file does not contain data or stats, and is therefore not encrypted.
+
+Currently, encryption is supported in the Hive and REST catalogs for tables with Parquet and Avro data formats.
+
+Two parameters are required to activate encryption of a table,
+1. Catalog property `encryption.kms-impl`, that specifies the class path for a client of a KMS ("key management service").
+2. Table property `encryption.key-id`, that specifies the ID of a master key used to encrypt and decrypt the table. Master keys are stored and managed in the KMS.
+
+The `encryption.key-id` must be set during the table creation, and never modified or removed during the table lifetime.
Review Comment:
Thanks for the clarification and updates.
##########
docs/docs/encryption.md:
##########
@@ -0,0 +1,152 @@
+---
+title: "Encryption"
+---
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements. See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License. You may obtain a copy of the License at
+ -
+ - http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+# Encryption
+
+Iceberg table encryption protects confidentiality and integrity of table data in an untrusted storage. The `data`, `delete`, `manifest` and `manifest list` files are encrypted and tamper-proofed before being sent to the storage backend.
+
+The `metadata.json` file does not contain data or stats, and is therefore not encrypted.
+
+Currently, encryption is supported in the Hive and REST catalogs for tables with Parquet and Avro data formats.
+
+Two parameters are required to activate encryption of a table,
+1. Catalog property `encryption.kms-impl`, that specifies the class path for a client of a KMS ("key management service").
+2. Table property `encryption.key-id`, that specifies the ID of a master key used to encrypt and decrypt the table. Master keys are stored and managed in the KMS.
+
+The `encryption.key-id` must be set during the table creation, and never modified or removed during the table lifetime.
Review Comment:
Thanks for the clarification and updates.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Docs: encryption [iceberg]
Posted by "singhpk234 (via GitHub)" <gi...@apache.org>.
singhpk234 commented on code in PR #14621:
URL: https://github.com/apache/iceberg/pull/14621#discussion_r2818009412
##########
docs/docs/encryption.md:
##########
@@ -0,0 +1,153 @@
+---
+title: "Encryption"
+---
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements. See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License. You may obtain a copy of the License at
+ -
+ - http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+# Encryption
+
+Iceberg table encryption protects confidentiality and integrity of table data in an untrusted storage. The `data`, `delete`, `manifest` and `manifest list` files are encrypted and tamper-proofed before being sent to the storage backend.
+
+The `metadata.json` file does not contain data or stats, and is therefore not encrypted.
Review Comment:
Sorry i missed your ping last week @ggershinsky
yes thats mostly true, though think of a protection policy like user is not allowed to see data from country USA, but the country is a partition column so not only does collecting these summary reveal that there exists USA data but you can find how many countries are there, also additionaly you can do reverse eng like he the partition i can see if has x rows means this value how much y rows in partition would mean
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Docs: encryption [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on PR #14621:
URL: https://github.com/apache/iceberg/pull/14621#issuecomment-3609617295
@ggershinsky This doesn't render correctly. Can you check and fix?
<img width="3224" height="844" alt="CleanShot 2025-12-04 at 09 50 52@2x" src="https://github.com/user-attachments/assets/f93a35a0-1eb2-4a48-bd2b-dc89c1937f26" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Docs: encryption [iceberg]
Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on code in PR #14621:
URL: https://github.com/apache/iceberg/pull/14621#discussion_r2573270132
##########
docs/docs/encryption.md:
##########
@@ -0,0 +1,152 @@
+---
+title: "Encryption"
+---
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements. See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License. You may obtain a copy of the License at
+ -
+ - http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+# Encryption
+
+Iceberg table encryption protects confidentiality and integrity of table data in an untrusted storage. The `data`, `delete`, `manifest` and `manifest list` files are encrypted and tamper-proofed before being sent to the storage backend.
+
+The `metadata.json` file does not contain data or stats, and is therefore not encrypted.
+
+Currently, encryption is supported in the Hive and REST catalogs for tables with Parquet and Avro data formats.
+
+Two parameters are required to activate encryption of a table,
+1. Catalog property `encryption.kms-impl`, that specifies the class path for a client of a KMS ("key management service").
+2. Table property `encryption.key-id`, that specifies the ID of a master key used to encrypt and decrypt the table. Master keys are stored and managed in the KMS.
+
+The `encryption.key-id` must be set during the table creation, and never modified or removed during the table lifetime.
+
+For more details on table encryption, see the "Appendix: Internals Overview" [subsection](#appendix-internals-overview).
+
+## Example
+
+```sh
+spark-sql --packages org.apache.iceberg:iceberg-spark-runtime-{{ sparkVersionMajor }}:{{ icebergVersion }}\
+ --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \
+ --conf spark.sql.catalog.spark_catalog=org.apache.iceberg.spark.SparkSessionCatalog \
+ --conf spark.sql.catalog.spark_catalog.type=hive \
+ --conf spark.sql.catalog.local=org.apache.iceberg.spark.SparkCatalog \
+ --conf spark.sql.catalog.local.type=hive \
+ --conf spark.sql.catalog.local.encryption.kms-impl=org.apache.iceberg.aws.AwsKeyManagementClient
+```
+
+```sql
+CREATE TABLE local.db.table (id bigint, data string) USING iceberg
+TBLPROPERTIES ('encryption.key-id'='{{ master key id }}');
+```
+
+Inserted data will be automatically encrypted,
+
+```sql
+INSERT INTO local.db.table VALUES (1, 'a'), (2, 'b'), (3, 'c');
+```
+
+To verify encryption, the contents of data, manifest and manifest list files can be dumped in the command line with
+
+```sh
+hexdump -C {{ /path/to/file }} | more
+```
+
+The Parquet files must start with the "PARE" magic string (PARquet Encrypted footer mode), and manifest/list files must start with "AGS1" magic string (Aes Gcm Stream version 1).
+
+Queried data will be automatically decrypted,
+
+```sql
+SELECT * FROM local.db.table;
+```
+
+## Catalog security requirements
+
+To function properly, Iceberg table encryption requires the catalog implementations not to retrieve the metadata
+directly from metadata.json files, if these file are kept unprotected in a storage vulnerable to tampering.
Review Comment:
nit: `these file` -> `these files`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Docs: encryption [iceberg]
Posted by "ggershinsky (via GitHub)" <gi...@apache.org>.
ggershinsky commented on code in PR #14621:
URL: https://github.com/apache/iceberg/pull/14621#discussion_r2576141429
##########
docs/docs/encryption.md:
##########
@@ -0,0 +1,152 @@
+---
+title: "Encryption"
+---
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements. See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License. You may obtain a copy of the License at
+ -
+ - http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+# Encryption
+
+Iceberg table encryption protects confidentiality and integrity of table data in an untrusted storage. The `data`, `delete`, `manifest` and `manifest list` files are encrypted and tamper-proofed before being sent to the storage backend.
+
+The `metadata.json` file does not contain data or stats, and is therefore not encrypted.
+
+Currently, encryption is supported in the Hive and REST catalogs for tables with Parquet and Avro data formats.
+
+Two parameters are required to activate encryption of a table,
+1. Catalog property `encryption.kms-impl`, that specifies the class path for a client of a KMS ("key management service").
+2. Table property `encryption.key-id`, that specifies the ID of a master key used to encrypt and decrypt the table. Master keys are stored and managed in the KMS.
+
+The `encryption.key-id` must be set during the table creation, and never modified or removed during the table lifetime.
Review Comment:
This requirement is applicable to all catalogs. Per our recent community discussion, the rest encryption updates should be rest-specific.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Docs: encryption [iceberg]
Posted by "ggershinsky (via GitHub)" <gi...@apache.org>.
ggershinsky commented on code in PR #14621:
URL: https://github.com/apache/iceberg/pull/14621#discussion_r2818325362
##########
docs/docs/encryption.md:
##########
@@ -0,0 +1,153 @@
+---
+title: "Encryption"
+---
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements. See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License. You may obtain a copy of the License at
+ -
+ - http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+# Encryption
+
+Iceberg table encryption protects confidentiality and integrity of table data in an untrusted storage. The `data`, `delete`, `manifest` and `manifest list` files are encrypted and tamper-proofed before being sent to the storage backend.
+
+The `metadata.json` file does not contain data or stats, and is therefore not encrypted.
Review Comment:
@singhpk234 can you please point where in the code this information (`country`=`USA`) is recorded in the metadata.json?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Docs: encryption [iceberg]
Posted by "palladium-coder (via GitHub)" <gi...@apache.org>.
palladium-coder commented on code in PR #14621:
URL: https://github.com/apache/iceberg/pull/14621#discussion_r2575923776
##########
docs/docs/encryption.md:
##########
@@ -0,0 +1,152 @@
+---
+title: "Encryption"
+---
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements. See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License. You may obtain a copy of the License at
+ -
+ - http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+# Encryption
+
+Iceberg table encryption protects confidentiality and integrity of table data in an untrusted storage. The `data`, `delete`, `manifest` and `manifest list` files are encrypted and tamper-proofed before being sent to the storage backend.
+
+The `metadata.json` file does not contain data or stats, and is therefore not encrypted.
+
+Currently, encryption is supported in the Hive and REST catalogs for tables with Parquet and Avro data formats.
+
+Two parameters are required to activate encryption of a table,
+1. Catalog property `encryption.kms-impl`, that specifies the class path for a client of a KMS ("key management service").
+2. Table property `encryption.key-id`, that specifies the ID of a master key used to encrypt and decrypt the table. Master keys are stored and managed in the KMS.
+
+The `encryption.key-id` must be set during the table creation, and never modified or removed during the table lifetime.
Review Comment:
Should rest_spec.md also be updated to ensure this requirement to prevent accidents?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Docs: encryption [iceberg]
Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on code in PR #14621:
URL: https://github.com/apache/iceberg/pull/14621#discussion_r2573269120
##########
docs/docs/encryption.md:
##########
@@ -0,0 +1,152 @@
+---
+title: "Encryption"
+---
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements. See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License. You may obtain a copy of the License at
+ -
+ - http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+# Encryption
+
+Iceberg table encryption protects confidentiality and integrity of table data in an untrusted storage. The `data`, `delete`, `manifest` and `manifest list` files are encrypted and tamper-proofed before being sent to the storage backend.
+
+The `metadata.json` file does not contain data or stats, and is therefore not encrypted.
+
+Currently, encryption is supported in the Hive and REST catalogs for tables with Parquet and Avro data formats.
+
+Two parameters are required to activate encryption of a table,
Review Comment:
nit: should we end the sentence with a `:` instead of `,`?
##########
docs/docs/encryption.md:
##########
@@ -0,0 +1,152 @@
+---
+title: "Encryption"
+---
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements. See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License. You may obtain a copy of the License at
+ -
+ - http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+# Encryption
+
+Iceberg table encryption protects confidentiality and integrity of table data in an untrusted storage. The `data`, `delete`, `manifest` and `manifest list` files are encrypted and tamper-proofed before being sent to the storage backend.
+
+The `metadata.json` file does not contain data or stats, and is therefore not encrypted.
+
+Currently, encryption is supported in the Hive and REST catalogs for tables with Parquet and Avro data formats.
+
+Two parameters are required to activate encryption of a table,
+1. Catalog property `encryption.kms-impl`, that specifies the class path for a client of a KMS ("key management service").
+2. Table property `encryption.key-id`, that specifies the ID of a master key used to encrypt and decrypt the table. Master keys are stored and managed in the KMS.
+
+The `encryption.key-id` must be set during the table creation, and never modified or removed during the table lifetime.
Review Comment:
nit:: should this sentence go in section 2?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Docs: encryption [iceberg]
Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on PR #14621:
URL: https://github.com/apache/iceberg/pull/14621#issuecomment-3609115991
Thanks @ggershinsky for the PR! Thanks @palladium-coder @smaheshwar-pltr for the review!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Docs: encryption [iceberg]
Posted by "palladium-coder (via GitHub)" <gi...@apache.org>.
palladium-coder commented on code in PR #14621:
URL: https://github.com/apache/iceberg/pull/14621#discussion_r2579867432
##########
docs/docs/encryption.md:
##########
@@ -0,0 +1,152 @@
+---
+title: "Encryption"
+---
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements. See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License. You may obtain a copy of the License at
+ -
+ - http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+# Encryption
+
+Iceberg table encryption protects confidentiality and integrity of table data in an untrusted storage. The `data`, `delete`, `manifest` and `manifest list` files are encrypted and tamper-proofed before being sent to the storage backend.
+
+The `metadata.json` file does not contain data or stats, and is therefore not encrypted.
+
+Currently, encryption is supported in the Hive and REST catalogs for tables with Parquet and Avro data formats.
+
+Two parameters are required to activate encryption of a table,
+1. Catalog property `encryption.kms-impl`, that specifies the class path for a client of a KMS ("key management service").
+2. Table property `encryption.key-id`, that specifies the ID of a master key used to encrypt and decrypt the table. Master keys are stored and managed in the KMS.
+
+The `encryption.key-id` must be set during the table creation, and never modified or removed during the table lifetime.
Review Comment:
Thanks for the clarification and updates.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Docs: encryption [iceberg]
Posted by "ggershinsky (via GitHub)" <gi...@apache.org>.
ggershinsky commented on PR #14621:
URL: https://github.com/apache/iceberg/pull/14621#issuecomment-3610536886
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Docs: encryption [iceberg]
Posted by "smaheshwar-pltr (via GitHub)" <gi...@apache.org>.
smaheshwar-pltr commented on code in PR #14621:
URL: https://github.com/apache/iceberg/pull/14621#discussion_r2567064275
##########
docs/docs/encryption.md:
##########
@@ -0,0 +1,146 @@
+---
+title: "Encryption"
+---
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements. See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License. You may obtain a copy of the License at
+ -
+ - http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+# Encryption
+
+Iceberg table encryption protects confidentiality and integrity of table data in an untrusted storage. The `data`, `delete`, `manifest` and `manifest list` files are encrypted and tamper-proofed before being sent to the storage backend.
+
+The `metadata.json` file does not contain data or stats, and is therefore not encrypted.
+
+Currently, encryption is supported in the Hive and REST catalogs for tables with Parquet and Avro data formats.
+
+Two parameters are required to activate encryption of a table,
+1. Catalog property `encryption.kms-impl`, that specifies the class path for a client of a KMS ("key management service").
+2. Table property `encryption.key-id`, that specifies the ID of a master key used to encrypt and decrypt the table. Master keys are stored and managed in the KMS.
+
+For more details, see the "Appendix: How It Works" [subsection](#appendix-how-it-works).
+
+## Example
+
+```sh
+spark-sql --packages org.apache.iceberg:iceberg-spark-runtime-{{ sparkVersionMajor }}:{{ icebergVersion }}\
+ --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \
+ --conf spark.sql.catalog.spark_catalog=org.apache.iceberg.spark.SparkSessionCatalog \
+ --conf spark.sql.catalog.spark_catalog.type=hive \
+ --conf spark.sql.catalog.local=org.apache.iceberg.spark.SparkCatalog \
+ --conf spark.sql.catalog.local.type=hive \
+ --conf spark.sql.catalog.local.encryption.kms-impl=org.apache.iceberg.aws.AwsKeyManagementClient
+```
+
+```sql
+CREATE TABLE local.db.table (id bigint, data string) USING iceberg
+TBLPROPERTIES ('encryption.key-id'='{{ master key id }}');
+```
+
+Inserted data will be automatically encrypted,
+
+```sql
+INSERT INTO local.db.table VALUES (1, 'a'), (2, 'b'), (3, 'c');
+```
+
+To verify encryption, the contents of data, manifest and manifest list files can be dumped in the command line with
+
+```sh
+hexdump -C {{ /path/to/file }} | more
+```
+
+The Parquet files must start with the "PARE" magic string (PARquet Encrypted footer mode), and manifest/list files must start with "AGS1" magic string (Aes Gcm Stream version 1).
+
+Queried data will be automatically decrypted,
+
+```sql
+SELECT * FROM local.db.table;
+```
+
+## Security requirements
+
+To function properly, Iceberg table encryption places the following requirements on the catalogs:
+
+1. For protection of table data confidentiality, the table encryption properties (`encryption.key-id` and an optional `encryption.data-key-length`) must be kept in a tamper-proof storage or in a trusted independent database. Catalogs must not retrieve these properties directly from the metadata.json, if this file is kept unprotected in a storage vulnerable to tampering.
+2. For protection of table integrity, the metadata json must be kept in a tamper-proof storage or in a trusted independent object store. Catalogs must not retrieve the metadata.json file directly, if it is kept unprotected in a storage vulnerable to tampering.
Review Comment:
Thank you for this section! It is very helpful in understanding motivations
Realise this may still be in-progress, but just flagging that I'm a bit confused from an IRC implementation perspective about what's required here.
I gather from (2) and [the devlist thread](https://lists.apache.org/thread/0nn11o4xf1nmw68d4px33sxw5tzzmgbo) that for untrusted storage (I'd imagine most IRCs currently write metadata JSONs in the same object storage as data files), an IRC should tamper-check a metadata JSON before returning its `TableMetadata` spec object - however, it then feels that (1) is not necessary anymore.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Docs: encryption [iceberg]
Posted by "singhpk234 (via GitHub)" <gi...@apache.org>.
singhpk234 commented on code in PR #14621:
URL: https://github.com/apache/iceberg/pull/14621#discussion_r2789802262
##########
docs/docs/encryption.md:
##########
@@ -0,0 +1,153 @@
+---
+title: "Encryption"
+---
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements. See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License. You may obtain a copy of the License at
+ -
+ - http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+# Encryption
+
+Iceberg table encryption protects confidentiality and integrity of table data in an untrusted storage. The `data`, `delete`, `manifest` and `manifest list` files are encrypted and tamper-proofed before being sent to the storage backend.
+
+The `metadata.json` file does not contain data or stats, and is therefore not encrypted.
Review Comment:
Thank you for taking look @ggershinsky !
1. Partition summary or even any summary props are supposed to be optional and not defined in spec, when you use iceberg java impl these are collected when `write.summary.partition-limit` is enable (default off) please check this for details : https://iceberg.apache.org/docs/nightly/configuration/#write-properties
2. Yes they are enabled by the writer (user setting this table prop `write.summary.partition-limit` engines such as spark etc collect these when enabled.
3. please check this https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/SnapshotSummary.java#L202
on a high level it can reveal which column values and stats on file counts etc
https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/SnapshotSummary.java#L163
please let me know what do you think about it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Docs: encryption [iceberg]
Posted by "ggershinsky (via GitHub)" <gi...@apache.org>.
ggershinsky commented on code in PR #14621:
URL: https://github.com/apache/iceberg/pull/14621#discussion_r2576783589
##########
docs/docs/encryption.md:
##########
@@ -0,0 +1,152 @@
+---
+title: "Encryption"
+---
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements. See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License. You may obtain a copy of the License at
+ -
+ - http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+# Encryption
+
+Iceberg table encryption protects confidentiality and integrity of table data in an untrusted storage. The `data`, `delete`, `manifest` and `manifest list` files are encrypted and tamper-proofed before being sent to the storage backend.
+
+The `metadata.json` file does not contain data or stats, and is therefore not encrypted.
+
+Currently, encryption is supported in the Hive and REST catalogs for tables with Parquet and Avro data formats.
+
+Two parameters are required to activate encryption of a table,
+1. Catalog property `encryption.kms-impl`, that specifies the class path for a client of a KMS ("key management service").
+2. Table property `encryption.key-id`, that specifies the ID of a master key used to encrypt and decrypt the table. Master keys are stored and managed in the KMS.
+
+The `encryption.key-id` must be set during the table creation, and never modified or removed during the table lifetime.
Review Comment:
we can add a pointer to the catalog security requirements section to the custom-catalogs.md.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Docs: encryption [iceberg]
Posted by "palladium-coder (via GitHub)" <gi...@apache.org>.
palladium-coder commented on code in PR #14621:
URL: https://github.com/apache/iceberg/pull/14621#discussion_r2575923776
##########
docs/docs/encryption.md:
##########
@@ -0,0 +1,152 @@
+---
+title: "Encryption"
+---
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements. See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License. You may obtain a copy of the License at
+ -
+ - http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+# Encryption
+
+Iceberg table encryption protects confidentiality and integrity of table data in an untrusted storage. The `data`, `delete`, `manifest` and `manifest list` files are encrypted and tamper-proofed before being sent to the storage backend.
+
+The `metadata.json` file does not contain data or stats, and is therefore not encrypted.
+
+Currently, encryption is supported in the Hive and REST catalogs for tables with Parquet and Avro data formats.
+
+Two parameters are required to activate encryption of a table,
+1. Catalog property `encryption.kms-impl`, that specifies the class path for a client of a KMS ("key management service").
+2. Table property `encryption.key-id`, that specifies the ID of a master key used to encrypt and decrypt the table. Master keys are stored and managed in the KMS.
+
+The `encryption.key-id` must be set during the table creation, and never modified or removed during the table lifetime.
Review Comment:
Should rest_spec.md also be updated to ensure this requirement in order to prevent accidents?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Docs: encryption [iceberg]
Posted by "ggershinsky (via GitHub)" <gi...@apache.org>.
ggershinsky commented on code in PR #14621:
URL: https://github.com/apache/iceberg/pull/14621#discussion_r2815631246
##########
docs/docs/encryption.md:
##########
@@ -0,0 +1,153 @@
+---
+title: "Encryption"
+---
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements. See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License. You may obtain a copy of the License at
+ -
+ - http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+# Encryption
+
+Iceberg table encryption protects confidentiality and integrity of table data in an untrusted storage. The `data`, `delete`, `manifest` and `manifest list` files are encrypted and tamper-proofed before being sent to the storage backend.
+
+The `metadata.json` file does not contain data or stats, and is therefore not encrypted.
Review Comment:
Since this subject is important, I'll record my current understanding of it. From
https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/SnapshotSummary.java#L223
<img width="763" height="654" alt="image" src="https://github.com/user-attachments/assets/c99ab418-f314-443b-876a-bb39e7c9e72f" />
\- the partition summaries keep int/long counters and a boolean flag. There are no column min/max stats.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Docs: encryption [iceberg]
Posted by "ggershinsky (via GitHub)" <gi...@apache.org>.
ggershinsky commented on code in PR #14621:
URL: https://github.com/apache/iceberg/pull/14621#discussion_r2783880167
##########
docs/docs/encryption.md:
##########
@@ -0,0 +1,153 @@
+---
+title: "Encryption"
+---
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements. See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License. You may obtain a copy of the License at
+ -
+ - http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+# Encryption
+
+Iceberg table encryption protects confidentiality and integrity of table data in an untrusted storage. The `data`, `delete`, `manifest` and `manifest list` files are encrypted and tamper-proofed before being sent to the storage backend.
+
+The `metadata.json` file does not contain data or stats, and is therefore not encrypted.
Review Comment:
Thanks @singhpk234, I'll have a look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Docs: encryption [iceberg]
Posted by "ggershinsky (via GitHub)" <gi...@apache.org>.
ggershinsky commented on code in PR #14621:
URL: https://github.com/apache/iceberg/pull/14621#discussion_r2791725253
##########
docs/docs/encryption.md:
##########
@@ -0,0 +1,153 @@
+---
+title: "Encryption"
+---
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements. See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License. You may obtain a copy of the License at
+ -
+ - http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+# Encryption
+
+Iceberg table encryption protects confidentiality and integrity of table data in an untrusted storage. The `data`, `delete`, `manifest` and `manifest list` files are encrypted and tamper-proofed before being sent to the storage backend.
+
+The `metadata.json` file does not contain data or stats, and is therefore not encrypted.
Review Comment:
Are partition summary fields encapsulated in the SnapshotSummary.UpdateMetrics class?
https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/SnapshotSummary.java#L223
Looks like a set of counters. Are there stats (col min/max) too?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Docs: encryption [iceberg]
Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on code in PR #14621:
URL: https://github.com/apache/iceberg/pull/14621#discussion_r2553778010
##########
docs/docs/encryption.md:
##########
@@ -0,0 +1,112 @@
+---
+title: "Encryption"
+---
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements. See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License. You may obtain a copy of the License at
+ -
+ - http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+# Encryption
+
+Iceberg table encryption protects confidentiality and integrity of table data in an untrusted storage. The `data`, `delete`, `manifest` and `manifest list` files are encrypted and tamper-proofed before being sent to the storage backend.
+
+The `metadata.json` file does not contain confidential information, and is therefore not encrypted.
+
+Currently, table encryption is supported with the Hive and REST catalogs.
+
+Two parameters are required to activate encryption of a table,
+1. Catalog property `encryption.kms-impl`, that specifies the class path for a client of a KMS ("key management service").
+2. Table property `encryption.key-id`, that specifies the ID of a master key used to encrypt and decrypt the table. Master keys are stored and managed in the KMS.
+
Review Comment:
Could we add a brief ‘How it works’ subsection (per‑file AES key → AES‑GCM encrypt → wrapKey + store wrapped key → unwrapKey on read → decrypt
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Docs: encryption [iceberg]
Posted by "ggershinsky (via GitHub)" <gi...@apache.org>.
ggershinsky commented on code in PR #14621:
URL: https://github.com/apache/iceberg/pull/14621#discussion_r2567598614
##########
docs/docs/encryption.md:
##########
@@ -0,0 +1,146 @@
+---
+title: "Encryption"
+---
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements. See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License. You may obtain a copy of the License at
+ -
+ - http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+# Encryption
+
+Iceberg table encryption protects confidentiality and integrity of table data in an untrusted storage. The `data`, `delete`, `manifest` and `manifest list` files are encrypted and tamper-proofed before being sent to the storage backend.
+
+The `metadata.json` file does not contain data or stats, and is therefore not encrypted.
+
+Currently, encryption is supported in the Hive and REST catalogs for tables with Parquet and Avro data formats.
+
+Two parameters are required to activate encryption of a table,
+1. Catalog property `encryption.kms-impl`, that specifies the class path for a client of a KMS ("key management service").
+2. Table property `encryption.key-id`, that specifies the ID of a master key used to encrypt and decrypt the table. Master keys are stored and managed in the KMS.
+
+For more details, see the "Appendix: How It Works" [subsection](#appendix-how-it-works).
+
+## Example
+
+```sh
+spark-sql --packages org.apache.iceberg:iceberg-spark-runtime-{{ sparkVersionMajor }}:{{ icebergVersion }}\
+ --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \
+ --conf spark.sql.catalog.spark_catalog=org.apache.iceberg.spark.SparkSessionCatalog \
+ --conf spark.sql.catalog.spark_catalog.type=hive \
+ --conf spark.sql.catalog.local=org.apache.iceberg.spark.SparkCatalog \
+ --conf spark.sql.catalog.local.type=hive \
+ --conf spark.sql.catalog.local.encryption.kms-impl=org.apache.iceberg.aws.AwsKeyManagementClient
+```
+
+```sql
+CREATE TABLE local.db.table (id bigint, data string) USING iceberg
+TBLPROPERTIES ('encryption.key-id'='{{ master key id }}');
+```
+
+Inserted data will be automatically encrypted,
+
+```sql
+INSERT INTO local.db.table VALUES (1, 'a'), (2, 'b'), (3, 'c');
+```
+
+To verify encryption, the contents of data, manifest and manifest list files can be dumped in the command line with
+
+```sh
+hexdump -C {{ /path/to/file }} | more
+```
+
+The Parquet files must start with the "PARE" magic string (PARquet Encrypted footer mode), and manifest/list files must start with "AGS1" magic string (Aes Gcm Stream version 1).
+
+Queried data will be automatically decrypted,
+
+```sql
+SELECT * FROM local.db.table;
+```
+
+## Security requirements
+
+To function properly, Iceberg table encryption places the following requirements on the catalogs:
+
+1. For protection of table data confidentiality, the table encryption properties (`encryption.key-id` and an optional `encryption.data-key-length`) must be kept in a tamper-proof storage or in a trusted independent database. Catalogs must not retrieve these properties directly from the metadata.json, if this file is kept unprotected in a storage vulnerable to tampering.
+2. For protection of table integrity, the metadata json must be kept in a tamper-proof storage or in a trusted independent object store. Catalogs must not retrieve the metadata.json file directly, if it is kept unprotected in a storage vulnerable to tampering.
Review Comment:
Yep, this can be simplified.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Docs: encryption [iceberg]
Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on code in PR #14621:
URL: https://github.com/apache/iceberg/pull/14621#discussion_r2573269509
##########
docs/docs/encryption.md:
##########
@@ -0,0 +1,152 @@
+---
+title: "Encryption"
+---
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements. See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License. You may obtain a copy of the License at
+ -
+ - http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+# Encryption
+
+Iceberg table encryption protects confidentiality and integrity of table data in an untrusted storage. The `data`, `delete`, `manifest` and `manifest list` files are encrypted and tamper-proofed before being sent to the storage backend.
+
+The `metadata.json` file does not contain data or stats, and is therefore not encrypted.
+
+Currently, encryption is supported in the Hive and REST catalogs for tables with Parquet and Avro data formats.
+
+Two parameters are required to activate encryption of a table,
+1. Catalog property `encryption.kms-impl`, that specifies the class path for a client of a KMS ("key management service").
+2. Table property `encryption.key-id`, that specifies the ID of a master key used to encrypt and decrypt the table. Master keys are stored and managed in the KMS.
+
+The `encryption.key-id` must be set during the table creation, and never modified or removed during the table lifetime.
+
+For more details on table encryption, see the "Appendix: Internals Overview" [subsection](#appendix-internals-overview).
+
+## Example
+
+```sh
+spark-sql --packages org.apache.iceberg:iceberg-spark-runtime-{{ sparkVersionMajor }}:{{ icebergVersion }}\
+ --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \
+ --conf spark.sql.catalog.spark_catalog=org.apache.iceberg.spark.SparkSessionCatalog \
+ --conf spark.sql.catalog.spark_catalog.type=hive \
+ --conf spark.sql.catalog.local=org.apache.iceberg.spark.SparkCatalog \
+ --conf spark.sql.catalog.local.type=hive \
+ --conf spark.sql.catalog.local.encryption.kms-impl=org.apache.iceberg.aws.AwsKeyManagementClient
+```
+
+```sql
+CREATE TABLE local.db.table (id bigint, data string) USING iceberg
+TBLPROPERTIES ('encryption.key-id'='{{ master key id }}');
+```
+
+Inserted data will be automatically encrypted,
+
+```sql
+INSERT INTO local.db.table VALUES (1, 'a'), (2, 'b'), (3, 'c');
+```
+
+To verify encryption, the contents of data, manifest and manifest list files can be dumped in the command line with
+
+```sh
+hexdump -C {{ /path/to/file }} | more
+```
+
+The Parquet files must start with the "PARE" magic string (PARquet Encrypted footer mode), and manifest/list files must start with "AGS1" magic string (Aes Gcm Stream version 1).
+
+Queried data will be automatically decrypted,
+
+```sql
+SELECT * FROM local.db.table;
+```
+
+## Catalog security requirements
+
+To function properly, Iceberg table encryption requires the catalog implementations not to retrieve the metadata
+directly from metadata.json files, if these file are kept unprotected in a storage vulnerable to tampering.
+Catalogs may keep the metadata in a trusted independent object store.
+Catalogs may work with metadata.json files in a tamper-proof storage.
+Catalogs may use checksum techniques to verify integrity of metadata.json files in a storage vulnerable to tampering
+(the checksums must be kept in a separate trusted storage).
Review Comment:
nit: I think these points render as a single paragraph. Could we make them bullets?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Docs: encryption [iceberg]
Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on PR #14621:
URL: https://github.com/apache/iceberg/pull/14621#issuecomment-3610357769
I fixed the rendering problem in https://github.com/apache/iceberg/pull/14756
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Docs: encryption [iceberg]
Posted by "palladium-coder (via GitHub)" <gi...@apache.org>.
palladium-coder commented on code in PR #14621:
URL: https://github.com/apache/iceberg/pull/14621#discussion_r2579867296
##########
docs/docs/encryption.md:
##########
@@ -0,0 +1,152 @@
+---
+title: "Encryption"
+---
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements. See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License. You may obtain a copy of the License at
+ -
+ - http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+# Encryption
+
+Iceberg table encryption protects confidentiality and integrity of table data in an untrusted storage. The `data`, `delete`, `manifest` and `manifest list` files are encrypted and tamper-proofed before being sent to the storage backend.
+
+The `metadata.json` file does not contain data or stats, and is therefore not encrypted.
+
+Currently, encryption is supported in the Hive and REST catalogs for tables with Parquet and Avro data formats.
+
+Two parameters are required to activate encryption of a table,
+1. Catalog property `encryption.kms-impl`, that specifies the class path for a client of a KMS ("key management service").
+2. Table property `encryption.key-id`, that specifies the ID of a master key used to encrypt and decrypt the table. Master keys are stored and managed in the KMS.
+
+The `encryption.key-id` must be set during the table creation, and never modified or removed during the table lifetime.
Review Comment:
Thanks for the clarification and updating the custom catalog doc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org