You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@polaris.apache.org by "flyrain (via GitHub)" <gi...@apache.org> on 2025/10/16 22:44:12 UTC
[I] Iceberg encryption support [polaris]
flyrain opened a new issue, #2829:
URL: https://github.com/apache/polaris/issues/2829
### Is your feature request related to a problem? Please describe.
_No response_
### Describe the solution you'd like
There are interests to bring Iceberg table encryption support in Polaris. Filing this issue to facilitate discussion.
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Iceberg encryption support [polaris]
Posted by "hkwi (via GitHub)" <gi...@apache.org>.
hkwi commented on issue #2829:
URL: https://github.com/apache/polaris/issues/2829#issuecomment-4969530022
# Table-Level Encryption: Hive Catalog Implementation, REST Catalog Specification, and Polaris REST Catalog Implementation
I suggest organizing the current status of table-level encryption support into the following three areas.
The Hive Catalog implementation, the REST Catalog specification, and the Polaris REST Catalog implementation address related but different layers of the Iceberg integration.
## 1. Client-driven encryption support
In this model, the client encrypts and decrypts table data. The catalog server does not need to access plaintext data or connect to the KMS.
### Hive Catalog Implementation
Hive Catalog currently supports:
- catalog-level KMS client configuration;
- `EncryptionManager` and `EncryptingFileIO`;
- encrypted data files, manifests, and manifest lists;
- encrypted table metadata and wrapped encryption keys.
### REST Catalog Specification
The REST Catalog API already defines the metadata needed to represent encrypted tables, including:
- `encryption-keys` in table metadata;
- `add-encryption-key` and `remove-encryption-key` updates;
- `key-metadata` in content files.
Iceberg PR [#13225](https://github.com/apache/iceberg/pull/13225) adds client-side encryption support for REST Catalogs. The PR is currently open.
This is primarily an Iceberg client implementation. It does not require the REST server to own or access the KMS, as long as the server preserves and commits the encryption metadata.
### Polaris REST Catalog Implementation
For client-driven encryption, Polaris should not need to connect to the KMS. The server can preserve and commit the encryption metadata through the normal table update paths.
Polaris does not currently have an explicit table-encryption integration or a dedicated encryption end-to-end test. The client-driven path should therefore be considered separately from server-side encryption operations.
## 2. Full REST Catalog semantics for encrypted tables covered by the existing specification
This area concerns operations where the REST server reads, plans, or deletes table files.
### Hive Catalog Implementation
Hive supports encrypted purge through the normal Catalog API:
```text
HiveCatalog.dropTable(identifier, true)
-> CatalogUtil.dropTableData(ops.io(), metadata)
-> HiveTableOperations.io()
-> EncryptingFileIO
```
This allows encrypted manifest lists and manifests to be read before the referenced files are deleted.
Hive also supports client-side scan planning for encrypted tables through the encrypted table `FileIO`.
### REST Catalog Specification
The REST API defines:
- `purgeRequested` for table deletion;
- server-side scan planning endpoints;
- storage credentials associated with scan results;
- encrypted content-file metadata.
The specification defines the REST contract, while the server implementation is responsible for applying the corresponding table operations.
### Polaris REST Catalog Implementation
The main identified gap is encrypted server-side purge.
Polaris schedules cleanup tasks that create a normal storage `FileIO`. Those tasks then read manifest lists and manifests through that `FileIO`. The current path does not construct an `EncryptionManager` or combine the storage `FileIO` with `EncryptingFileIO`.
Therefore, encrypted server-side purge is not currently handled through the same path as Hive.
Polaris also does not currently implement the general server-side scan planning endpoint. This is a broader REST Catalog implementation gap. If server-side scan planning is added, encrypted tables will need to preserve and return the relevant file encryption metadata and storage credentials required by the client.
The metadata transport itself is a separate matter. Since Polaris applies table updates and persists table metadata through the normal paths, `encryption-keys` and `key-metadata` do not necessarily require a Polaris-specific encryption implementation.
## 3. Functionality not yet specified by the REST Catalog API
The following topics appear to require additional specification:
- how a REST server obtains and configures its KMS client;
- whether KMS ownership belongs to the client, the server, or both;
- how client and server KMS permissions and trust relationships are established;
- whether credential vending also applies to KMS access;
- an equivalent of Hive Metastore metadata hashing and tamper protection;
- automated encryption-key retirement and removal of obsolete keys;
- server-side maintenance operations beyond the existing REST Catalog API.
These topics should be distinguished from the behavior already represented by the current REST API.
## Proposed scope for this issue
The discussion could track the following separately:
- Iceberg client support for REST Catalog encryption, including [PR #13225](https://github.com/apache/iceberg/pull/13225);
- Polaris implementation of existing REST semantics, especially encrypted server-side purge;
- additional specification work for KMS ownership, trust boundaries, and server-side KMS configuration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Iceberg encryption support [polaris]
Posted by "obelix74 (via GitHub)" <gi...@apache.org>.
obelix74 commented on issue #2829:
URL: https://github.com/apache/polaris/issues/2829#issuecomment-3413157319
Broadly speaking, there is a need to store encrypted data in tables for Iceberg. Clients get a table specific key and decrypt the data in the client. This allows storage of sensitive data in public cloud tables.
The iceberg specification specifies `encryption-keys` in the `table-metadata-fields` section. https://iceberg.apache.org/spec/#table-metadata-fields
My understanding of how this works:
* Client-side encryption: Iceberg encrypts data on the client before writing it to storage and decrypts it on the client after reading it. The storage backend, like Amazon S3, never sees the plain text data.
* Key Management Service (KMS): Encryption and decryption keys are managed by a KMS, which handles key access, authentication, and authorization.
* To access an encrypted table, a user must have permissions to both the data and the KMS key used for encryption.
* The user uses the KMS key to decrypt the data.
I think we need to be able to use encrypted data only on certain tables, so that only sensitive data goes through this approach.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Iceberg encryption support [polaris]
Posted by "hkwi (via GitHub)" <gi...@apache.org>.
hkwi commented on issue #2829:
URL: https://github.com/apache/polaris/issues/2829#issuecomment-5030399020
Follow-up based on the implementation and review discussion in [#5060](https://github.com/apache/polaris/pull/5060):
A clarification to my previous comment is needed.
- Iceberg native encryption protects data files, delete files, manifest files, and manifest-list files. The table metadata JSON itself remains plaintext; it carries the wrapped encryption keys needed to open the encrypted files.
- The statement that the REST server does not need KMS access applies to the normal client-written commit path. The engine/client writes the encrypted data, manifests, and manifest lists, while Polaris preserves and commits the resulting metadata updates. Polaris should not initialize writer-side encryption or write manifests.
- Server-side purge is a separate read path. To enumerate the referenced files safely, Polaris must read and decrypt the manifest lists and manifests written by the engine. Therefore, a Polaris cleanup worker does require KMS access and the table's wrapped keys when purging an encrypted table. It does not rewrite or re-encrypt those manifests.
[#5060](https://github.com/apache/polaris/pull/5060) has been updated to reflect this responsibility split:
- writer-side encryption support was removed from `LocalIcebergCatalog`;
- encryption context is preserved only for asynchronous cleanup tasks, which reconstruct an encryption-aware `FileIO` to read encrypted manifest lists and manifests during purge; and
- the test now uses a REST integration path where test-side engine code performs encrypted writes and Polaris handles the purge.
Standard engine-side encryption through `RESTCatalog` still depends on the open Iceberg PR [apache/iceberg#13225](https://github.com/apache/iceberg/pull/13225). Until that support is available, the Polaris integration test uses a test-only adapter to model the client/engine behavior.
This narrows the remaining areas as follows: client-side REST encryption support belongs in Iceberg; encrypted server-side purge is the Polaris implementation work in #5060; and production KMS configuration, credentials, multiple-store routing, and trust boundaries still need separate documentation and community discussion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org