You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/08/15 17:11:41 UTC
[GitHub] [iceberg] JiJiTang opened a new pull request, #5539: [Core]Add EncryptionManagerFactory to configure encryption via catalog properties and table metadata.
JiJiTang opened a new pull request, #5539:
URL: https://github.com/apache/iceberg/pull/5539
And also make KmsClient `Closeable` to clean up underlying resources when necessary, ref the design of `FileIO`. For some cases, KmsClient implementation is expensive to be initialised per encryption manager call, when using shared KmsClient for multiple encryption managers we need to close underlying resources to avoid potential leak.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] JiJiTang commented on a diff in pull request #5539: [Core]Add EncryptionManagerFactory to configure encryption via catalog properties and table metadata.
Posted by GitBox <gi...@apache.org>.
JiJiTang commented on code in PR #5539:
URL: https://github.com/apache/iceberg/pull/5539#discussion_r967416409
##########
hive-metastore/src/main/java/org/apache/iceberg/hive/HiveTableOperations.java:
##########
@@ -210,6 +232,11 @@ protected String tableName() {
return fullName;
}
+ @Override
+ protected EncryptionManagerFactory encryptionManagerFactory() {
Review Comment:
@rdblue, very good spot, thanks VM! During my first iteration, I was putting this method in `TableOperations` and use the factory to return the manager, and this will make `TableOperations` have two encryption related methods, some of my colleagues were worrying about breaking change to `TableOperations` as public API. Myself I vote also to move this method to `TableOperations` cc @stevenzwu @ggershinsky for inputs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [Core]Add EncryptionManagerFactory to configure encryption via catalog properties and table metadata. [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5539:
URL: https://github.com/apache/iceberg/pull/5539#issuecomment-2309064371
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] JiJiTang commented on pull request #5539: [Core]Add EncryptionManagerFactory to configure encryption via catalog properties and table metadata.
Posted by GitBox <gi...@apache.org>.
JiJiTang commented on PR #5539:
URL: https://github.com/apache/iceberg/pull/5539#issuecomment-1215410370
cc @jackye1995 for help to review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] JiJiTang commented on a diff in pull request #5539: [Core]Add EncryptionManagerFactory to configure encryption via catalog properties and table metadata.
Posted by GitBox <gi...@apache.org>.
JiJiTang commented on code in PR #5539:
URL: https://github.com/apache/iceberg/pull/5539#discussion_r975739639
##########
hive-metastore/src/main/java/org/apache/iceberg/hive/HiveTableOperations.java:
##########
@@ -210,6 +232,11 @@ protected String tableName() {
return fullName;
}
+ @Override
+ protected EncryptionManagerFactory encryptionManagerFactory() {
Review Comment:
@rdblue , calling encryptionManagerFactory.create() will be a duplicated logic for table operations sub classes. That's why I put the duplicated logic to `BaseMetastoreTableOperations` class and in that class I defined method `protected abstract EncryptionManagerFactory encryptionManagerFactory()`. I would think the factory create logic to `TableOperations` directly and that would require to make `TableOperations` as an abstract class, need your advice on this.
##########
hive-metastore/src/main/java/org/apache/iceberg/hive/HiveTableOperations.java:
##########
@@ -210,6 +232,11 @@ protected String tableName() {
return fullName;
}
+ @Override
+ protected EncryptionManagerFactory encryptionManagerFactory() {
Review Comment:
@rdblue , calling encryptionManagerFactory.create() will be duplicated logic for table operations sub classes. That's why I put the duplicated logic to `BaseMetastoreTableOperations` class and in that class I defined method `protected abstract EncryptionManagerFactory encryptionManagerFactory()`. I would think the factory create logic to `TableOperations` directly and that would require to make `TableOperations` as an abstract class, need your advice on this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] JiJiTang commented on a diff in pull request #5539: [Core]Add EncryptionManagerFactory to configure encryption via catalog properties and table metadata.
Posted by GitBox <gi...@apache.org>.
JiJiTang commented on code in PR #5539:
URL: https://github.com/apache/iceberg/pull/5539#discussion_r967440824
##########
core/src/main/java/org/apache/iceberg/hadoop/HadoopTables.java:
##########
@@ -203,13 +204,31 @@ public boolean dropTable(String location, boolean purge) {
}
}
+ private Map<String, String> encryptionConfigFromConf() {
Review Comment:
@rdblue , not sure I understand this one, need your help to clarify: do you mean for `HadoopTables` we don't need to incorporate `EncryptionManagerFactory` initialisation logic or we shouldn't load the catalog properties from Hadoop conf?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] JiJiTang commented on a diff in pull request #5539: [Core]Add EncryptionManagerFactory to configure encryption via catalog properties and table metadata.
Posted by GitBox <gi...@apache.org>.
JiJiTang commented on code in PR #5539:
URL: https://github.com/apache/iceberg/pull/5539#discussion_r967421120
##########
core/src/main/java/org/apache/iceberg/encryption/EncryptionManagerFactory.java:
##########
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.encryption;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.util.Map;
+import org.apache.iceberg.TableMetadata;
+
+public interface EncryptionManagerFactory extends Closeable {
+
+ /**
+ * Initialize EncryptionManagerFactory from catalog properties.
+ *
+ * @param properties catalog properties
+ */
+ default void initialize(Map<String, String> properties) {}
+ /**
+ * Create encryption manager from table metadata.
+ *
+ * @param tableMetadata table metadata
+ * @return created encryption manager instance.
+ */
+ EncryptionManager create(TableMetadata tableMetadata);
+
+ /**
+ * Close EncryptionManagerFactory to release underlying resources.
+ *
+ * <p>Calling this method is only required when this EncryptionManagerFactory instance is no
+ * longer expected to be used, and the resources it holds need to be explicitly released to avoid
+ * resource leaks.
+ */
+ @Override
+ default void close() throws IOException {}
Review Comment:
@rdblue , very good point, I was referring `FileIO` interface which has a default implementation, will update it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #5539: [Core]Add EncryptionManagerFactory to configure encryption via catalog properties and table metadata.
Posted by GitBox <gi...@apache.org>.
rdblue commented on code in PR #5539:
URL: https://github.com/apache/iceberg/pull/5539#discussion_r950384092
##########
hive-metastore/src/main/java/org/apache/iceberg/hive/HiveTableOperations.java:
##########
@@ -210,6 +232,11 @@ protected String tableName() {
return fullName;
}
+ @Override
+ protected EncryptionManagerFactory encryptionManagerFactory() {
Review Comment:
Why is this exposing the factory? Shouldn't this use the factory to return an encryption manager [here](https://github.com/apache/iceberg/blob/master/core/src/main/java/org/apache/iceberg/TableOperations.java#L72)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] JiJiTang commented on a diff in pull request #5539: [Core]Add EncryptionManagerFactory to configure encryption via catalog properties and table metadata.
Posted by GitBox <gi...@apache.org>.
JiJiTang commented on code in PR #5539:
URL: https://github.com/apache/iceberg/pull/5539#discussion_r975736515
##########
core/src/main/java/org/apache/iceberg/encryption/EncryptionManagerFactory.java:
##########
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.encryption;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.util.Map;
+import org.apache.iceberg.TableMetadata;
+
+public interface EncryptionManagerFactory extends Closeable {
+
+ /**
+ * Initialize EncryptionManagerFactory from catalog properties.
+ *
+ * @param properties catalog properties
+ */
+ default void initialize(Map<String, String> properties) {}
+ /**
+ * Create encryption manager from table metadata.
+ *
+ * @param tableMetadata table metadata
+ * @return created encryption manager instance.
+ */
+ EncryptionManager create(TableMetadata tableMetadata);
+
+ /**
+ * Close EncryptionManagerFactory to release underlying resources.
+ *
+ * <p>Calling this method is only required when this EncryptionManagerFactory instance is no
+ * longer expected to be used, and the resources it holds need to be explicitly released to avoid
+ * resource leaks.
+ */
+ @Override
+ default void close() throws IOException {}
Review Comment:
@rdblue , updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] JiJiTang commented on a diff in pull request #5539: [Core]Add EncryptionManagerFactory to configure encryption via catalog properties and table metadata.
Posted by GitBox <gi...@apache.org>.
JiJiTang commented on code in PR #5539:
URL: https://github.com/apache/iceberg/pull/5539#discussion_r975736119
##########
core/src/main/java/org/apache/iceberg/encryption/EncryptionManagerFactory.java:
##########
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.encryption;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.util.Map;
+import org.apache.iceberg.TableMetadata;
+
+public interface EncryptionManagerFactory extends Closeable {
+
+ /**
+ * Initialize EncryptionManagerFactory from catalog properties.
+ *
+ * @param properties catalog properties
+ */
+ default void initialize(Map<String, String> properties) {}
+ /**
Review Comment:
@rdblue updated, whitespace added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [Core]Add EncryptionManagerFactory to configure encryption via catalog properties and table metadata. [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5539:
URL: https://github.com/apache/iceberg/pull/5539#issuecomment-2295037897
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #5539: [Core]Add EncryptionManagerFactory to configure encryption via catalog properties and table metadata.
Posted by GitBox <gi...@apache.org>.
rdblue commented on code in PR #5539:
URL: https://github.com/apache/iceberg/pull/5539#discussion_r950381409
##########
core/src/main/java/org/apache/iceberg/encryption/EncryptionManagerFactory.java:
##########
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.encryption;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.util.Map;
+import org.apache.iceberg.TableMetadata;
+
+public interface EncryptionManagerFactory extends Closeable {
+
+ /**
+ * Initialize EncryptionManagerFactory from catalog properties.
+ *
+ * @param properties catalog properties
+ */
+ default void initialize(Map<String, String> properties) {}
+ /**
Review Comment:
Missing whitespace.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] JiJiTang commented on a diff in pull request #5539: [Core]Add EncryptionManagerFactory to configure encryption via catalog properties and table metadata.
Posted by GitBox <gi...@apache.org>.
JiJiTang commented on code in PR #5539:
URL: https://github.com/apache/iceberg/pull/5539#discussion_r967416409
##########
hive-metastore/src/main/java/org/apache/iceberg/hive/HiveTableOperations.java:
##########
@@ -210,6 +232,11 @@ protected String tableName() {
return fullName;
}
+ @Override
+ protected EncryptionManagerFactory encryptionManagerFactory() {
Review Comment:
@rdblue, very good spot, thanks VM! During my first iteration, I was putting this method in `TableOperations` and use the factory to return the manager, and this will make `TableOperations` have two encryption related methods, some of my colleagues are worrying about breaking change to `TableOperations` as public API. Myself I vote also to move this method to `TableOperations` cc @stevenzwu @ggershinsky
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #5539: [Core]Add EncryptionManagerFactory to configure encryption via catalog properties and table metadata.
Posted by GitBox <gi...@apache.org>.
rdblue commented on code in PR #5539:
URL: https://github.com/apache/iceberg/pull/5539#discussion_r950382482
##########
core/src/main/java/org/apache/iceberg/encryption/EncryptionManagerFactory.java:
##########
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.encryption;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.util.Map;
+import org.apache.iceberg.TableMetadata;
+
+public interface EncryptionManagerFactory extends Closeable {
+
+ /**
+ * Initialize EncryptionManagerFactory from catalog properties.
+ *
+ * @param properties catalog properties
+ */
+ default void initialize(Map<String, String> properties) {}
+ /**
+ * Create encryption manager from table metadata.
+ *
+ * @param tableMetadata table metadata
+ * @return created encryption manager instance.
+ */
+ EncryptionManager create(TableMetadata tableMetadata);
+
+ /**
+ * Close EncryptionManagerFactory to release underlying resources.
+ *
+ * <p>Calling this method is only required when this EncryptionManagerFactory instance is no
+ * longer expected to be used, and the resources it holds need to be explicitly released to avoid
+ * resource leaks.
+ */
+ @Override
+ default void close() throws IOException {}
Review Comment:
I don't think it is generally a good idea to add a default implementation of `close` because we want people to explicitly add a noop close to make implementers consider what needs to be cleaned up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #5539: [Core]Add EncryptionManagerFactory to configure encryption via catalog properties and table metadata.
Posted by GitBox <gi...@apache.org>.
rdblue commented on code in PR #5539:
URL: https://github.com/apache/iceberg/pull/5539#discussion_r950381845
##########
core/src/main/java/org/apache/iceberg/encryption/EncryptionManagerFactory.java:
##########
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.encryption;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.util.Map;
+import org.apache.iceberg.TableMetadata;
+
+public interface EncryptionManagerFactory extends Closeable {
+
+ /**
+ * Initialize EncryptionManagerFactory from catalog properties.
+ *
+ * @param properties catalog properties
+ */
+ default void initialize(Map<String, String> properties) {}
+ /**
+ * Create encryption manager from table metadata.
+ *
+ * @param tableMetadata table metadata
+ * @return created encryption manager instance.
+ */
+ EncryptionManager create(TableMetadata tableMetadata);
Review Comment:
I think this needs to be more narrow, not `TableMetadata`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] JiJiTang commented on a diff in pull request #5539: [Core]Add EncryptionManagerFactory to configure encryption via catalog properties and table metadata.
Posted by GitBox <gi...@apache.org>.
JiJiTang commented on code in PR #5539:
URL: https://github.com/apache/iceberg/pull/5539#discussion_r975739639
##########
hive-metastore/src/main/java/org/apache/iceberg/hive/HiveTableOperations.java:
##########
@@ -210,6 +232,11 @@ protected String tableName() {
return fullName;
}
+ @Override
+ protected EncryptionManagerFactory encryptionManagerFactory() {
Review Comment:
@rdblue , calling encryptionManagerFactory.create() will be duplicated logic for table operations sub classes. I put the duplicated logic to `BaseMetastoreTableOperations` class and in that class I defined method `protected abstract EncryptionManagerFactory encryptionManagerFactory()`. I would think put the factory create logic to `TableOperations` directly and that would require to make `TableOperations` as an abstract class and add a protected `encryptionManagerFactory` method there, need your advice on this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] JiJiTang commented on a diff in pull request #5539: [Core]Add EncryptionManagerFactory to configure encryption via catalog properties and table metadata.
Posted by GitBox <gi...@apache.org>.
JiJiTang commented on code in PR #5539:
URL: https://github.com/apache/iceberg/pull/5539#discussion_r967419493
##########
core/src/main/java/org/apache/iceberg/encryption/EncryptionManagerFactory.java:
##########
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.encryption;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.util.Map;
+import org.apache.iceberg.TableMetadata;
+
+public interface EncryptionManagerFactory extends Closeable {
+
+ /**
+ * Initialize EncryptionManagerFactory from catalog properties.
+ *
+ * @param properties catalog properties
+ */
+ default void initialize(Map<String, String> properties) {}
+ /**
+ * Create encryption manager from table metadata.
+ *
+ * @param tableMetadata table metadata
+ * @return created encryption manager instance.
+ */
+ EncryptionManager create(TableMetadata tableMetadata);
Review Comment:
@rdblue , thanks a lot and agreed, at this stage we can narrow it to table properties. I am wondering if we will need table schema as well for column level encryption with different keys, cc @ggershinsky for inputs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] JiJiTang commented on a diff in pull request #5539: [Core]Add EncryptionManagerFactory to configure encryption via catalog properties and table metadata.
Posted by GitBox <gi...@apache.org>.
JiJiTang commented on code in PR #5539:
URL: https://github.com/apache/iceberg/pull/5539#discussion_r967416409
##########
hive-metastore/src/main/java/org/apache/iceberg/hive/HiveTableOperations.java:
##########
@@ -210,6 +232,11 @@ protected String tableName() {
return fullName;
}
+ @Override
+ protected EncryptionManagerFactory encryptionManagerFactory() {
Review Comment:
@rdblue, very good spot! During my first iteration, I was putting this method in `TableOperations` and use the factory to return the manager, and this will make `TableOperations` have two encryption related methods, some of my colleagues are worrying about breaking change to `TableOperations` as public API. Myself I vote also to move this method to `TableOperations` cc @stevenzwu @ggershinsky
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] JiJiTang commented on a diff in pull request #5539: [Core]Add EncryptionManagerFactory to configure encryption via catalog properties and table metadata.
Posted by GitBox <gi...@apache.org>.
JiJiTang commented on code in PR #5539:
URL: https://github.com/apache/iceberg/pull/5539#discussion_r976330084
##########
core/src/main/java/org/apache/iceberg/encryption/EncryptionManagerFactory.java:
##########
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.encryption;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.util.Map;
+import org.apache.iceberg.TableMetadata;
+
+public interface EncryptionManagerFactory extends Closeable {
+
+ /**
+ * Initialize EncryptionManagerFactory from catalog properties.
+ *
+ * @param properties catalog properties
+ */
+ default void initialize(Map<String, String> properties) {}
+ /**
+ * Create encryption manager from table metadata.
+ *
+ * @param tableMetadata table metadata
+ * @return created encryption manager instance.
+ */
+ EncryptionManager create(TableMetadata tableMetadata);
Review Comment:
@ggershinsky, the validation totally make sense, thanks a lot for the comment. I will update this method to keep only table properties and schema as params.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] [Core]Add EncryptionManagerFactory to configure encryption via catalog properties and table metadata. [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #5539: [Core]Add EncryptionManagerFactory to configure encryption via catalog properties and table metadata.
URL: https://github.com/apache/iceberg/pull/5539
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] JiJiTang commented on pull request #5539: [Core]Add EncryptionManagerFactory to configure encryption via catalog properties and table metadata.
Posted by GitBox <gi...@apache.org>.
JiJiTang commented on PR #5539:
URL: https://github.com/apache/iceberg/pull/5539#issuecomment-1289454522
cc @flyrain for review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] JiJiTang commented on a diff in pull request #5539: [Core]Add EncryptionManagerFactory to configure encryption via catalog properties and table metadata.
Posted by GitBox <gi...@apache.org>.
JiJiTang commented on code in PR #5539:
URL: https://github.com/apache/iceberg/pull/5539#discussion_r975739639
##########
hive-metastore/src/main/java/org/apache/iceberg/hive/HiveTableOperations.java:
##########
@@ -210,6 +232,11 @@ protected String tableName() {
return fullName;
}
+ @Override
+ protected EncryptionManagerFactory encryptionManagerFactory() {
Review Comment:
@rdblue , calling encryptionManagerFactory.create() will be duplicated logic for table operations sub classes. I put the duplicated logic to `BaseMetastoreTableOperations` class and in that class I defined method `protected abstract EncryptionManagerFactory encryptionManagerFactory()`. I would think the factory create logic to `TableOperations` directly and that would require to make `TableOperations` as an abstract class, need your advice on this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a diff in pull request #5539: [Core]Add EncryptionManagerFactory to configure encryption via catalog properties and table metadata.
Posted by GitBox <gi...@apache.org>.
rdblue commented on code in PR #5539:
URL: https://github.com/apache/iceberg/pull/5539#discussion_r950386181
##########
core/src/main/java/org/apache/iceberg/hadoop/HadoopTables.java:
##########
@@ -203,13 +204,31 @@ public boolean dropTable(String location, boolean purge) {
}
}
+ private Map<String, String> encryptionConfigFromConf() {
Review Comment:
I don't think this should update `HadoopTables`. New features should not be added to that interface.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] JiJiTang commented on a diff in pull request #5539: [Core]Add EncryptionManagerFactory to configure encryption via catalog properties and table metadata.
Posted by GitBox <gi...@apache.org>.
JiJiTang commented on code in PR #5539:
URL: https://github.com/apache/iceberg/pull/5539#discussion_r967421120
##########
core/src/main/java/org/apache/iceberg/encryption/EncryptionManagerFactory.java:
##########
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.encryption;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.util.Map;
+import org.apache.iceberg.TableMetadata;
+
+public interface EncryptionManagerFactory extends Closeable {
+
+ /**
+ * Initialize EncryptionManagerFactory from catalog properties.
+ *
+ * @param properties catalog properties
+ */
+ default void initialize(Map<String, String> properties) {}
+ /**
+ * Create encryption manager from table metadata.
+ *
+ * @param tableMetadata table metadata
+ * @return created encryption manager instance.
+ */
+ EncryptionManager create(TableMetadata tableMetadata);
+
+ /**
+ * Close EncryptionManagerFactory to release underlying resources.
+ *
+ * <p>Calling this method is only required when this EncryptionManagerFactory instance is no
+ * longer expected to be used, and the resources it holds need to be explicitly released to avoid
+ * resource leaks.
+ */
+ @Override
+ default void close() throws IOException {}
Review Comment:
@rdblue , very good point, I was referring `FileIO` interface which has a default implementation, will update it to remove default implementation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] JiJiTang commented on pull request #5539: [Core]Add EncryptionManagerFactory to configure encryption via catalog properties and table metadata.
Posted by GitBox <gi...@apache.org>.
JiJiTang commented on PR #5539:
URL: https://github.com/apache/iceberg/pull/5539#issuecomment-1215409475
cc @ggershinsky @RussellSpitzer @flyrain @stevenzwu
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ggershinsky commented on a diff in pull request #5539: [Core]Add EncryptionManagerFactory to configure encryption via catalog properties and table metadata.
Posted by GitBox <gi...@apache.org>.
ggershinsky commented on code in PR #5539:
URL: https://github.com/apache/iceberg/pull/5539#discussion_r976185861
##########
core/src/main/java/org/apache/iceberg/encryption/EncryptionManagerFactory.java:
##########
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.encryption;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.util.Map;
+import org.apache.iceberg.TableMetadata;
+
+public interface EncryptionManagerFactory extends Closeable {
+
+ /**
+ * Initialize EncryptionManagerFactory from catalog properties.
+ *
+ * @param properties catalog properties
+ */
+ default void initialize(Map<String, String> properties) {}
+ /**
+ * Create encryption manager from table metadata.
+ *
+ * @param tableMetadata table metadata
+ * @return created encryption manager instance.
+ */
+ EncryptionManager create(TableMetadata tableMetadata);
Review Comment:
@JiJiTang In general, we can set up column encryption with table properties only; but it would be good to have the schema too - this will enable us to verify the column encryption properties, making sure nothing is misspelled so we don't get an instruction to encrypt non-existing column (this can also lead to having a sensitive column remaining unencrypted..)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] JiJiTang commented on a diff in pull request #5539: [Core]Add EncryptionManagerFactory to configure encryption via catalog properties and table metadata.
Posted by GitBox <gi...@apache.org>.
JiJiTang commented on code in PR #5539:
URL: https://github.com/apache/iceberg/pull/5539#discussion_r976330084
##########
core/src/main/java/org/apache/iceberg/encryption/EncryptionManagerFactory.java:
##########
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg.encryption;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.util.Map;
+import org.apache.iceberg.TableMetadata;
+
+public interface EncryptionManagerFactory extends Closeable {
+
+ /**
+ * Initialize EncryptionManagerFactory from catalog properties.
+ *
+ * @param properties catalog properties
+ */
+ default void initialize(Map<String, String> properties) {}
+ /**
+ * Create encryption manager from table metadata.
+ *
+ * @param tableMetadata table metadata
+ * @return created encryption manager instance.
+ */
+ EncryptionManager create(TableMetadata tableMetadata);
Review Comment:
@ggershinsky, the validation totally make sense, thanks a lot for the comment. I will update this method to only keep table properties and schema as params.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org