You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "xanderbailey (via GitHub)" <gi...@apache.org> on 2026/06/15 09:03:02 UTC
[PR] feat(encryption): [11/N] KMS factory trait and catalog wiring [iceberg-rust]
xanderbailey opened a new pull request, #2650:
URL: https://github.com/apache/iceberg-rust/pull/2650
## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
-->
- Working towards: #2034
## What changes are included in this PR?
Adds `KmsClientFactory` to the catalog builder to allow a catalog to build one KMS client per catalog, this mirrors [Java's implementation ](https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/encryption/EncryptionUtil.java#L47). Note in rust we don't have reflection so using Java's `encryption.kms-impl` isn't an option for us. This is a very similar pattern to what we do for the `StorageFactory` so I believe the pattern is idiomatic for us.
<!--
Provide a summary of the modifications in this PR. List the main changes such as new features, bug fixes, refactoring, or any other updates.
-->
## Are these changes tested?
<!--
Specify what test covers (unit test, integration test, etc.).
If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(encryption): [11/N] KMS factory trait and catalog wiring [iceberg-rust]
Posted by "blackmwk (via GitHub)" <gi...@apache.org>.
blackmwk commented on code in PR #2650:
URL: https://github.com/apache/iceberg-rust/pull/2650#discussion_r3542768478
##########
crates/iceberg/src/table.rs:
##########
@@ -113,8 +113,15 @@ impl TableBuilder {
/// If the table metadata has the `encryption.key-id` property set, a
/// [`KeyManagementClient`] must be provided here so the table can build
/// an [`EncryptionManager`]; otherwise [`Self::build`] will return an error.
- pub fn kms_client(mut self, kms_client: Arc<dyn KeyManagementClient>) -> Self {
- self.kms_client = Some(kms_client);
+ ///
+ /// Accepts either an `Arc<dyn KeyManagementClient>` or an
+ /// `Option<Arc<dyn KeyManagementClient>>`, so catalogs that hold an
+ /// optional client can forward it directly. Passing `None` is a no-op.
+ pub fn kms_client(
Review Comment:
This change is odd to me, we could just skip this method call if it's None.
##########
crates/iceberg/src/table.rs:
##########
@@ -113,8 +113,15 @@ impl TableBuilder {
/// If the table metadata has the `encryption.key-id` property set, a
/// [`KeyManagementClient`] must be provided here so the table can build
/// an [`EncryptionManager`]; otherwise [`Self::build`] will return an error.
- pub fn kms_client(mut self, kms_client: Arc<dyn KeyManagementClient>) -> Self {
- self.kms_client = Some(kms_client);
+ ///
+ /// Accepts either an `Arc<dyn KeyManagementClient>` or an
+ /// `Option<Arc<dyn KeyManagementClient>>`, so catalogs that hold an
+ /// optional client can forward it directly. Passing `None` is a no-op.
+ pub fn kms_client(
Review Comment:
Also it makes the api inconsistent with other parts.
##########
crates/iceberg/src/encryption/kms/memory.rs:
##########
@@ -142,6 +143,39 @@ impl MemoryKeyManagementClient {
}
}
+/// Factory for creating [`MemoryKeyManagementClient`] instances.
+///
+/// This factory creates a fresh in-memory KMS client for each call.
+/// Useful for testing encryption flows without a real KMS backend.
+#[derive(Debug, Clone)]
+pub struct MemoryKmsClientFactory {
+ master_keys: Arc<RwLock<HashMap<String, SensitiveBytes>>>,
+ master_key_size: AesKeySize,
+}
+
+impl MemoryKmsClientFactory {
+ /// Creates a new factory that will produce clients sharing the given master keys.
+ pub fn new(kms: &MemoryKeyManagementClient) -> Self {
Review Comment:
This doesn't make sense to me to build a client factory from a client. I think the correct behavior should be like `MemoryKmsClientFactory` constructor accepts an `AesKeySize`, and it generate the key table. Then passes it to `MemoryKmsClient`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(encryption): [11/N] KMS factory trait and catalog wiring [iceberg-rust]
Posted by "xanderbailey (via GitHub)" <gi...@apache.org>.
xanderbailey commented on code in PR #2650:
URL: https://github.com/apache/iceberg-rust/pull/2650#discussion_r3542860749
##########
crates/iceberg/src/table.rs:
##########
@@ -113,8 +113,15 @@ impl TableBuilder {
/// If the table metadata has the `encryption.key-id` property set, a
/// [`KeyManagementClient`] must be provided here so the table can build
/// an [`EncryptionManager`]; otherwise [`Self::build`] will return an error.
- pub fn kms_client(mut self, kms_client: Arc<dyn KeyManagementClient>) -> Self {
- self.kms_client = Some(kms_client);
+ ///
+ /// Accepts either an `Arc<dyn KeyManagementClient>` or an
+ /// `Option<Arc<dyn KeyManagementClient>>`, so catalogs that hold an
+ /// optional client can forward it directly. Passing `None` is a no-op.
+ pub fn kms_client(
Review Comment:
I did it like this because else in all call sites you'd need:
```rust
if let Some(c) = &self.kms_client {
builder = builder.kms_client(c.clone());
}
```
which doesn't read very nicely in my opinion.
Would just `Option<...>` be better rather than the `Into`?
Seems like we get away with this for other fields because they're actually always present when we use this builder but KMS client is legitimately optional.
##########
crates/iceberg/src/table.rs:
##########
@@ -113,8 +113,15 @@ impl TableBuilder {
/// If the table metadata has the `encryption.key-id` property set, a
/// [`KeyManagementClient`] must be provided here so the table can build
/// an [`EncryptionManager`]; otherwise [`Self::build`] will return an error.
- pub fn kms_client(mut self, kms_client: Arc<dyn KeyManagementClient>) -> Self {
- self.kms_client = Some(kms_client);
+ ///
+ /// Accepts either an `Arc<dyn KeyManagementClient>` or an
+ /// `Option<Arc<dyn KeyManagementClient>>`, so catalogs that hold an
+ /// optional client can forward it directly. Passing `None` is a no-op.
+ pub fn kms_client(
Review Comment:
I did it like this because else in all call sites you'd need:
```rust
if let Some(c) = &self.kms_client {
builder = builder.kms_client(c.clone());
}
```
which doesn't read very nicely in my opinion.
Would just `Option<...>` be better rather than the `Into`?
Seems like we get away with this for other fields because they're actually always present when we use this builder but KMS client is legitimately optional.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(encryption): [11/N] KMS factory trait and catalog wiring [iceberg-rust]
Posted by "xanderbailey (via GitHub)" <gi...@apache.org>.
xanderbailey commented on code in PR #2650:
URL: https://github.com/apache/iceberg-rust/pull/2650#discussion_r3542793154
##########
crates/iceberg/src/encryption/kms/memory.rs:
##########
@@ -142,6 +143,39 @@ impl MemoryKeyManagementClient {
}
}
+/// Factory for creating [`MemoryKeyManagementClient`] instances.
+///
+/// This factory creates a fresh in-memory KMS client for each call.
+/// Useful for testing encryption flows without a real KMS backend.
+#[derive(Debug, Clone)]
+pub struct MemoryKmsClientFactory {
+ master_keys: Arc<RwLock<HashMap<String, SensitiveBytes>>>,
+ master_key_size: AesKeySize,
+}
+
+impl MemoryKmsClientFactory {
+ /// Creates a new factory that will produce clients sharing the given master keys.
+ pub fn new(kms: &MemoryKeyManagementClient) -> Self {
Review Comment:
That makes it harder to use in tests I think, I did it this way because for this you really just want all the constructor args of the in memory client to be forwarded on, what do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(encryption): [11/N] KMS factory trait and catalog wiring [iceberg-rust]
Posted by "blackmwk (via GitHub)" <gi...@apache.org>.
blackmwk commented on code in PR #2650:
URL: https://github.com/apache/iceberg-rust/pull/2650#discussion_r3550637429
##########
crates/iceberg/src/table.rs:
##########
@@ -113,8 +113,15 @@ impl TableBuilder {
/// If the table metadata has the `encryption.key-id` property set, a
/// [`KeyManagementClient`] must be provided here so the table can build
/// an [`EncryptionManager`]; otherwise [`Self::build`] will return an error.
- pub fn kms_client(mut self, kms_client: Arc<dyn KeyManagementClient>) -> Self {
- self.kms_client = Some(kms_client);
+ ///
+ /// Accepts either an `Arc<dyn KeyManagementClient>` or an
+ /// `Option<Arc<dyn KeyManagementClient>>`, so catalogs that hold an
+ /// optional client can forward it directly. Passing `None` is a no-op.
+ pub fn kms_client(
Review Comment:
Sounds reasonable to me, but should we also change Catalog api? It's also optional in catalog. Also please change all docs to say KmsClient/KmsClientFactory is optional.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(encryption): [11/N] KMS factory trait and catalog wiring [iceberg-rust]
Posted by "blackmwk (via GitHub)" <gi...@apache.org>.
blackmwk commented on code in PR #2650:
URL: https://github.com/apache/iceberg-rust/pull/2650#discussion_r3550601761
##########
crates/iceberg/src/encryption/kms/memory.rs:
##########
@@ -142,6 +143,39 @@ impl MemoryKeyManagementClient {
}
}
+/// Factory for creating [`MemoryKeyManagementClient`] instances.
+///
+/// This factory creates a fresh in-memory KMS client for each call.
+/// Useful for testing encryption flows without a real KMS backend.
+#[derive(Debug, Clone)]
+pub struct MemoryKmsClientFactory {
+ master_keys: Arc<RwLock<HashMap<String, SensitiveBytes>>>,
+ master_key_size: AesKeySize,
+}
+
+impl MemoryKmsClientFactory {
+ /// Creates a new factory that will produce clients sharing the given master keys.
+ pub fn new(kms: &MemoryKeyManagementClient) -> Self {
Review Comment:
Thanks, it looks much better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(encryption): [11/N] KMS factory trait and catalog wiring [iceberg-rust]
Posted by "xanderbailey (via GitHub)" <gi...@apache.org>.
xanderbailey commented on code in PR #2650:
URL: https://github.com/apache/iceberg-rust/pull/2650#discussion_r3542860749
##########
crates/iceberg/src/table.rs:
##########
@@ -113,8 +113,15 @@ impl TableBuilder {
/// If the table metadata has the `encryption.key-id` property set, a
/// [`KeyManagementClient`] must be provided here so the table can build
/// an [`EncryptionManager`]; otherwise [`Self::build`] will return an error.
- pub fn kms_client(mut self, kms_client: Arc<dyn KeyManagementClient>) -> Self {
- self.kms_client = Some(kms_client);
+ ///
+ /// Accepts either an `Arc<dyn KeyManagementClient>` or an
+ /// `Option<Arc<dyn KeyManagementClient>>`, so catalogs that hold an
+ /// optional client can forward it directly. Passing `None` is a no-op.
+ pub fn kms_client(
Review Comment:
I did it like this because else in all call sites you'd need:
```rust
if let Some(c) = &self.kms_client {
builder = builder.kms_client(c.clone());
}
```
which doesn't read very nicely in my opinion.
Would just `Option<...>` be better rather than the `Into`?
Seems like we get away with this for other fields because they're actually always present when we use this builder but KMS client is legitimately optional.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(encryption): [11/N] KMS factory trait and catalog wiring [iceberg-rust]
Posted by "xanderbailey (via GitHub)" <gi...@apache.org>.
xanderbailey commented on code in PR #2650:
URL: https://github.com/apache/iceberg-rust/pull/2650#discussion_r3550921881
##########
crates/iceberg/src/table.rs:
##########
@@ -113,8 +113,15 @@ impl TableBuilder {
/// If the table metadata has the `encryption.key-id` property set, a
/// [`KeyManagementClient`] must be provided here so the table can build
/// an [`EncryptionManager`]; otherwise [`Self::build`] will return an error.
- pub fn kms_client(mut self, kms_client: Arc<dyn KeyManagementClient>) -> Self {
- self.kms_client = Some(kms_client);
+ ///
+ /// Accepts either an `Arc<dyn KeyManagementClient>` or an
+ /// `Option<Arc<dyn KeyManagementClient>>`, so catalogs that hold an
+ /// optional client can forward it directly. Passing `None` is a no-op.
+ pub fn kms_client(
Review Comment:
Sorry maybe I didn't quite follow, what's the recommendation here? You're happy with this API or should we change? To me it doesn't seem crazy for a genuinely optional parameter to have an optional builder.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(encryption): [11/N] KMS factory trait and catalog wiring [iceberg-rust]
Posted by "xanderbailey (via GitHub)" <gi...@apache.org>.
xanderbailey commented on code in PR #2650:
URL: https://github.com/apache/iceberg-rust/pull/2650#discussion_r3545269887
##########
crates/iceberg/src/encryption/kms/memory.rs:
##########
@@ -142,6 +143,39 @@ impl MemoryKeyManagementClient {
}
}
+/// Factory for creating [`MemoryKeyManagementClient`] instances.
+///
+/// This factory creates a fresh in-memory KMS client for each call.
+/// Useful for testing encryption flows without a real KMS backend.
+#[derive(Debug, Clone)]
+pub struct MemoryKmsClientFactory {
+ master_keys: Arc<RwLock<HashMap<String, SensitiveBytes>>>,
+ master_key_size: AesKeySize,
+}
+
+impl MemoryKmsClientFactory {
+ /// Creates a new factory that will produce clients sharing the given master keys.
+ pub fn new(kms: &MemoryKeyManagementClient) -> Self {
Review Comment:
Have updated now to hopefully have better ergonomics for the API [0e47498](https://github.com/apache/iceberg-rust/pull/2650/commits/0e4749802413a12ecc55b0f25ecdf88e398b2f71) Let me know what you think.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(encryption): [11/N] KMS factory trait and catalog wiring [iceberg-rust]
Posted by "blackmwk (via GitHub)" <gi...@apache.org>.
blackmwk commented on code in PR #2650:
URL: https://github.com/apache/iceberg-rust/pull/2650#discussion_r3550637429
##########
crates/iceberg/src/table.rs:
##########
@@ -113,8 +113,15 @@ impl TableBuilder {
/// If the table metadata has the `encryption.key-id` property set, a
/// [`KeyManagementClient`] must be provided here so the table can build
/// an [`EncryptionManager`]; otherwise [`Self::build`] will return an error.
- pub fn kms_client(mut self, kms_client: Arc<dyn KeyManagementClient>) -> Self {
- self.kms_client = Some(kms_client);
+ ///
+ /// Accepts either an `Arc<dyn KeyManagementClient>` or an
+ /// `Option<Arc<dyn KeyManagementClient>>`, so catalogs that hold an
+ /// optional client can forward it directly. Passing `None` is a no-op.
+ pub fn kms_client(
Review Comment:
Sounds reasonable to me, but should we also change Catalog api? It's also optional in catalog. Also please change all docs to say KmsClient/KmsClientFactory is optional.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(encryption): [11/N] KMS factory trait and catalog wiring [iceberg-rust]
Posted by "blackmwk (via GitHub)" <gi...@apache.org>.
blackmwk merged PR #2650:
URL: https://github.com/apache/iceberg-rust/pull/2650
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(encryption): [11/N] KMS factory trait and catalog wiring [iceberg-rust]
Posted by "xanderbailey (via GitHub)" <gi...@apache.org>.
xanderbailey commented on code in PR #2650:
URL: https://github.com/apache/iceberg-rust/pull/2650#discussion_r3539944273
##########
crates/iceberg/src/catalog/mod.rs:
##########
@@ -152,6 +153,28 @@ pub trait CatalogBuilder: Default + Debug + Send + Sync {
/// ```
fn with_storage_factory(self, storage_factory: Arc<dyn StorageFactory>) -> Self;
+ /// Set a [`KmsClientFactory`] to enable table encryption.
+ ///
+ /// When provided, the catalog calls the factory once during
+ /// [`load`](Self::load) with the catalog properties to create a shared
+ /// [`KeyManagementClient`](crate::encryption::KeyManagementClient).
+ /// That client is then passed to each table's `TableBuilder` so tables
+ /// with `encryption.key-id` set can construct an `EncryptionManager`.
+ ///
+ /// # Example
+ ///
+ /// ```rust,ignore
+ /// use iceberg::CatalogBuilder;
+ /// use iceberg::encryption::kms::KmsClientFactory;
+ /// use std::sync::Arc;
+ ///
+ /// let catalog = MyCatalogBuilder::default()
+ /// .with_kms_client_factory(Arc::new(MyKmsClientFactory))
Review Comment:
Yeah I suspect this is desired
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(encryption): [11/N] KMS factory trait and catalog wiring [iceberg-rust]
Posted by "xanderbailey (via GitHub)" <gi...@apache.org>.
xanderbailey commented on code in PR #2650:
URL: https://github.com/apache/iceberg-rust/pull/2650#discussion_r3551859363
##########
crates/iceberg/src/table.rs:
##########
@@ -113,8 +113,15 @@ impl TableBuilder {
/// If the table metadata has the `encryption.key-id` property set, a
/// [`KeyManagementClient`] must be provided here so the table can build
/// an [`EncryptionManager`]; otherwise [`Self::build`] will return an error.
- pub fn kms_client(mut self, kms_client: Arc<dyn KeyManagementClient>) -> Self {
- self.kms_client = Some(kms_client);
+ ///
+ /// Accepts either an `Arc<dyn KeyManagementClient>` or an
+ /// `Option<Arc<dyn KeyManagementClient>>`, so catalogs that hold an
+ /// optional client can forward it directly. Passing `None` is a no-op.
+ pub fn kms_client(
Review Comment:
[remove optional builder](https://github.com/apache/iceberg-rust/pull/2650/commits/797890a35b7659222df3e4308d8a4a8e1c3f6d0a), let me know what you think.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(encryption): [11/N] KMS factory trait and catalog wiring [iceberg-rust]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on code in PR #2650:
URL: https://github.com/apache/iceberg-rust/pull/2650#discussion_r3501435439
##########
crates/iceberg/src/catalog/mod.rs:
##########
@@ -152,6 +153,28 @@ pub trait CatalogBuilder: Default + Debug + Send + Sync {
/// ```
fn with_storage_factory(self, storage_factory: Arc<dyn StorageFactory>) -> Self;
+ /// Set a [`KmsClientFactory`] to enable table encryption.
+ ///
+ /// When provided, the catalog calls the factory once during
+ /// [`load`](Self::load) with the catalog properties to create a shared
+ /// [`KeyManagementClient`](crate::encryption::KeyManagementClient).
+ /// That client is then passed to each table's `TableBuilder` so tables
+ /// with `encryption.key-id` set can construct an `EncryptionManager`.
+ ///
+ /// # Example
+ ///
+ /// ```rust,ignore
+ /// use iceberg::CatalogBuilder;
+ /// use iceberg::encryption::kms::KmsClientFactory;
+ /// use std::sync::Arc;
+ ///
+ /// let catalog = MyCatalogBuilder::default()
+ /// .with_kms_client_factory(Arc::new(MyKmsClientFactory))
+ /// .load("my_catalog", props)
+ /// .await?;
+ /// ```
+ fn with_kms_client_factory(self, kms_client_factory: Arc<dyn KmsClientFactory>) -> Self;
Review Comment:
As [11/N] of a stack, wiring the catalog KMS client to the Table's EncryptionManager is the whole job here. The read path can't consume it until a later part connects the EncryptionManager to file reads (the FileKeyResolver seam from your prototype), so there's nothing to change here. The one thing worth keeping in mind for that later PR: the seam should be feedable by either this KmsClientFactory-built client (native KMS) or an external resolver (a Comet-style JVM key source), since Comet resolves keys JVM-side and would not set a KmsClientFactory. Flagging it now only so the two entry points converge on one EncryptionManager/resolver seam rather than forking.
##########
crates/iceberg/src/catalog/mod.rs:
##########
@@ -152,6 +153,28 @@ pub trait CatalogBuilder: Default + Debug + Send + Sync {
/// ```
fn with_storage_factory(self, storage_factory: Arc<dyn StorageFactory>) -> Self;
+ /// Set a [`KmsClientFactory`] to enable table encryption.
+ ///
+ /// When provided, the catalog calls the factory once during
+ /// [`load`](Self::load) with the catalog properties to create a shared
+ /// [`KeyManagementClient`](crate::encryption::KeyManagementClient).
+ /// That client is then passed to each table's `TableBuilder` so tables
+ /// with `encryption.key-id` set can construct an `EncryptionManager`.
+ ///
+ /// # Example
+ ///
+ /// ```rust,ignore
+ /// use iceberg::CatalogBuilder;
+ /// use iceberg::encryption::kms::KmsClientFactory;
+ /// use std::sync::Arc;
+ ///
+ /// let catalog = MyCatalogBuilder::default()
+ /// .with_kms_client_factory(Arc::new(MyKmsClientFactory))
Review Comment:
Adding this as a required method breaks external CatalogBuilder implementors. A default impl that ignored the factory would silently drop encryption config, which is worse, so requiring it is probably the right call. Just worth calling out the breakage explicitly (changelog) so downstream implementors are not surprised.
##########
crates/iceberg/src/encryption/kms/factory.rs:
##########
@@ -0,0 +1,51 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Factory trait for creating [`KeyManagementClient`] instances.
+
+use std::collections::HashMap;
+use std::fmt::Debug;
+use std::sync::Arc;
+
+use async_trait::async_trait;
+
+use super::KeyManagementClient;
+use crate::Result;
+
+/// Factory for creating a [`KeyManagementClient`] from catalog properties.
+///
+/// Replaces Java's reflection-based `encryption.kms-impl` + `initialize(properties)`
+/// pattern. Users provide an implementation of this trait to the catalog builder via
+/// [`CatalogBuilder::with_kms_client_factory`](crate::CatalogBuilder::with_kms_client_factory).
+///
+/// The catalog calls [`create_kms_client`](Self::create_kms_client) **once** during
+/// catalog initialization with the catalog's properties. The resulting client is
+/// shared across all tables in the catalog and passed to each table's
+/// [`EncryptionManager`](crate::encryption::EncryptionManager) via
+/// `TableBuilder::kms_client(...)`.
+#[async_trait]
+pub trait KmsClientFactory: Debug + Send + Sync {
+ /// Create a [`KeyManagementClient`] from catalog properties.
+ ///
+ /// Called once during catalog initialization. Properties may include
+ /// KMS endpoint, region, credentials, or any backend-specific
+ /// configuration needed to construct the client.
+ async fn create_kms_client(
Review Comment:
I first wondered if this should select the KMS per table, but the spec model is catalog-scoped KMS: per docs/docs/encryption.md the KMS is a catalog property (encryption.kms-type / encryption.kms-impl), and the per-table axis is the encryption.key-id master-key property, not a per-table KMS impl. So one client per catalog is right, and create_kms_client(&self.config.props) matches Java's KeyManagementClient.initialize(properties = catalog properties) exactly. No change on scope. The one thing to confirm is the other axis: that per-table encryption.key-id is honored when the EncryptionManager is built (the PR body implies it is). The recurring "key_metadata is implementation-specific" point (relevant to #2584/#2586, not this PR) is spec-backed too: format/spec.md field 131 and the encryption-keys section.
##########
crates/iceberg/src/encryption/kms/memory.rs:
##########
@@ -142,6 +143,39 @@ impl MemoryKeyManagementClient {
}
}
+/// Factory for creating [`MemoryKeyManagementClient`] instances.
+///
+/// This factory creates a fresh in-memory KMS client for each call.
+/// Useful for testing encryption flows without a real KMS backend.
+#[derive(Debug, Clone)]
+pub struct MemoryKmsClientFactory {
+ master_keys: Arc<RwLock<HashMap<String, SensitiveBytes>>>,
+ master_key_size: AesKeySize,
+}
+
+impl MemoryKmsClientFactory {
Review Comment:
MemoryKmsClientFactory is the natural test seam, but I do not see a test that a factory-configured catalog yields a table whose EncryptionManager actually holds the client. A small test (build catalog with_kms_client_factory(MemoryKmsClientFactory), load a table, assert the client reached the EncryptionManager) would lock the wiring in. Good fit for the loader test suite.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(encryption): [11/N] KMS factory trait and catalog wiring [iceberg-rust]
Posted by "blackmwk (via GitHub)" <gi...@apache.org>.
blackmwk commented on code in PR #2650:
URL: https://github.com/apache/iceberg-rust/pull/2650#discussion_r3550665177
##########
crates/iceberg/src/table.rs:
##########
@@ -113,8 +113,15 @@ impl TableBuilder {
/// If the table metadata has the `encryption.key-id` property set, a
/// [`KeyManagementClient`] must be provided here so the table can build
/// an [`EncryptionManager`]; otherwise [`Self::build`] will return an error.
- pub fn kms_client(mut self, kms_client: Arc<dyn KeyManagementClient>) -> Self {
- self.kms_client = Some(kms_client);
+ ///
+ /// Accepts either an `Arc<dyn KeyManagementClient>` or an
+ /// `Option<Arc<dyn KeyManagementClient>>`, so catalogs that hold an
+ /// optional client can forward it directly. Passing `None` is a no-op.
+ pub fn kms_client(
Review Comment:
It sound reasonable to me. From a user's point, I accept both apis. But from a maintainer's point, I prefer the api without `Option`, since it's easier to maintain api consistency, otherwise we need to remember that for optional fields, we should accept `Option`, and other fields we don't.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(encryption): [11/N] KMS factory trait and catalog wiring [iceberg-rust]
Posted by "blackmwk (via GitHub)" <gi...@apache.org>.
blackmwk commented on code in PR #2650:
URL: https://github.com/apache/iceberg-rust/pull/2650#discussion_r3551271633
##########
crates/iceberg/src/table.rs:
##########
@@ -113,8 +113,15 @@ impl TableBuilder {
/// If the table metadata has the `encryption.key-id` property set, a
/// [`KeyManagementClient`] must be provided here so the table can build
/// an [`EncryptionManager`]; otherwise [`Self::build`] will return an error.
- pub fn kms_client(mut self, kms_client: Arc<dyn KeyManagementClient>) -> Self {
- self.kms_client = Some(kms_client);
+ ///
+ /// Accepts either an `Arc<dyn KeyManagementClient>` or an
+ /// `Option<Arc<dyn KeyManagementClient>>`, so catalogs that hold an
+ /// optional client can forward it directly. Passing `None` is a no-op.
+ pub fn kms_client(
Review Comment:
I mean I'm still in favor of removing the Option, since other apis in this crate adopt this approach, e.g. if a parameter is optional, user just don't call the function to set it. It would be easier to ensure api consistency in this repo.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(encryption): [11/N] KMS factory trait and catalog wiring [iceberg-rust]
Posted by "xanderbailey (via GitHub)" <gi...@apache.org>.
xanderbailey commented on PR #2650:
URL: https://github.com/apache/iceberg-rust/pull/2650#issuecomment-4912576680
@blackmwk I think this is ready for review when you have a moment! Thanks in advance!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(encryption): [11/N] KMS factory trait and catalog wiring [iceberg-rust]
Posted by "xanderbailey (via GitHub)" <gi...@apache.org>.
xanderbailey commented on code in PR #2650:
URL: https://github.com/apache/iceberg-rust/pull/2650#discussion_r3539942308
##########
crates/iceberg/src/encryption/kms/memory.rs:
##########
@@ -142,6 +143,39 @@ impl MemoryKeyManagementClient {
}
}
+/// Factory for creating [`MemoryKeyManagementClient`] instances.
+///
+/// This factory creates a fresh in-memory KMS client for each call.
+/// Useful for testing encryption flows without a real KMS backend.
+#[derive(Debug, Clone)]
+pub struct MemoryKmsClientFactory {
+ master_keys: Arc<RwLock<HashMap<String, SensitiveBytes>>>,
+ master_key_size: AesKeySize,
+}
+
+impl MemoryKmsClientFactory {
Review Comment:
Nice idea to have an end-to-end test for the in-memory catalog / kms. Added [end-to-end test](https://github.com/apache/iceberg-rust/pull/2650/commits/9008d7d34b19465db9b9a4c6e053fd0a35d886f7)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org