You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/02/09 23:00:41 UTC
[GitHub] [iceberg] andersonm-ibm opened a new pull request #4080: Add VaultKmsClient as an example KMS client implementation
andersonm-ibm opened a new pull request #4080:
URL: https://github.com/apache/iceberg/pull/4080
An implementation of the KmsClient API where master encryption keys are managed in Hashicorp Vault by its transit engine.
The KMS client is configured with Hadoop configuration properties:
```
keystore.kms.client.access.token
keystore.kms.client.instance.url
```
The token is passed as a hadoop property and not as an environment variable because of consistency. Security of the token is less of an issue here than in a file keystore, since it should be a non-root token that has a time-to-live (TTL).
This PR depends on the encryption PRs, in particular the KMS client interface : https://github.com/apache/iceberg/pull/3470/.
@ggershinsky
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] liujinhui1994 commented on pull request #4080: Add VaultKmsClient as an example KMS client implementation
Posted by GitBox <gi...@apache.org>.
liujinhui1994 commented on pull request #4080:
URL: https://github.com/apache/iceberg/pull/4080#issuecomment-1034597105
@andersonm-ibm Please help me understand https://github.com/apache/iceberg/pull/3470 why not use parquet KMSClient here but wrap one myself.
org.apache.parquet.crypto.keytools.KmsClient
thanks~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ggershinsky commented on a change in pull request #4080: Add VaultKmsClient as an example KMS client implementation
Posted by GitBox <gi...@apache.org>.
ggershinsky commented on a change in pull request #4080:
URL: https://github.com/apache/iceberg/pull/4080#discussion_r803649065
##########
File path: core/src/test/java/org/apache/iceberg/encryption/kms/VaultKmsClient.java
##########
@@ -0,0 +1,164 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.encryption.kms;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import java.util.Base64;
+import java.util.HashMap;
+import java.util.Map;
+import okhttp3.ConnectionSpec;
+import okhttp3.MediaType;
+import okhttp3.OkHttpClient;
+import okhttp3.Request;
+import okhttp3.RequestBody;
+import okhttp3.Response;
+import org.apache.iceberg.encryption.KmsClient;
+
+
+/**
+ * Example of a KMS client for testing and demonstrations; not for use in production.
+ * An implementation of {@link KmsClient} that relies on Hashicorp Vault transit engine to
+ * manage encryption keys.
+ * On initialization it is manadatory to set the properties {@link VaultKmsClient#KMS_INSTANCE_URL_PROP}
+ * and {@link VaultKmsClient#ACCESS_TOKEN_PROP}.
+ * Authentication to Vault is done using an access token passed in {@link VaultKmsClient#ACCESS_TOKEN_PROP}.
+ * If token is changed - {@link VaultKmsClient#initialize(Map)} should be called again with the new
+ * token in property {@link VaultKmsClient#ACCESS_TOKEN_PROP}.
+ * Pre-requisite: install Hashicorp Vault and enable transit engine as per
+ * https://www.vaultproject.io/docs/secrets/transit
+ */
+public class VaultKmsClient implements KmsClient {
+ /**
+ * Property name for Vault access token.
+ */
+ public static final String ACCESS_TOKEN_PROP = "keystore.kms.client.access.token";
+
+ /**
+ * Property name for Vault instance URL
+ */
+ public static final String KMS_INSTANCE_URL_PROP = "keystore.kms.client.instance.url";
Review comment:
"keystore" -> "vault" ? or even "kms.client.vault.instance.url"?
##########
File path: core/src/test/java/org/apache/iceberg/encryption/kms/VaultKmsClient.java
##########
@@ -0,0 +1,164 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.encryption.kms;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import java.util.Base64;
+import java.util.HashMap;
+import java.util.Map;
+import okhttp3.ConnectionSpec;
+import okhttp3.MediaType;
+import okhttp3.OkHttpClient;
+import okhttp3.Request;
+import okhttp3.RequestBody;
+import okhttp3.Response;
+import org.apache.iceberg.encryption.KmsClient;
+
+
+/**
+ * Example of a KMS client for testing and demonstrations; not for use in production.
+ * An implementation of {@link KmsClient} that relies on Hashicorp Vault transit engine to
+ * manage encryption keys.
+ * On initialization it is manadatory to set the properties {@link VaultKmsClient#KMS_INSTANCE_URL_PROP}
+ * and {@link VaultKmsClient#ACCESS_TOKEN_PROP}.
+ * Authentication to Vault is done using an access token passed in {@link VaultKmsClient#ACCESS_TOKEN_PROP}.
+ * If token is changed - {@link VaultKmsClient#initialize(Map)} should be called again with the new
+ * token in property {@link VaultKmsClient#ACCESS_TOKEN_PROP}.
+ * Pre-requisite: install Hashicorp Vault and enable transit engine as per
+ * https://www.vaultproject.io/docs/secrets/transit
+ */
+public class VaultKmsClient implements KmsClient {
+ /**
+ * Property name for Vault access token.
+ */
+ public static final String ACCESS_TOKEN_PROP = "keystore.kms.client.access.token";
Review comment:
"keystore" -> "vault" ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ggershinsky commented on pull request #4080: Add VaultKmsClient as an example KMS client implementation
Posted by GitBox <gi...@apache.org>.
ggershinsky commented on pull request #4080:
URL: https://github.com/apache/iceberg/pull/4080#issuecomment-1034901947
> The token is passed as a hadoop property and not as an environment variable because of consistency. Security of the token is less of an issue here than in a file keystore, since it should be a non-root token that has a time-to-live (TTL).
Agreed, though something like a ramdisk file can be better than a hadoop property (easier to refresh the token contents once in TTL, and less dependency on hadoop; the latter though should not be a big issue in a sample)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] andersonm-ibm commented on pull request #4080: Add VaultKmsClient as an example KMS client implementation
Posted by GitBox <gi...@apache.org>.
andersonm-ibm commented on pull request #4080:
URL: https://github.com/apache/iceberg/pull/4080#issuecomment-1034880588
> > Why can't it be used in production, and what if I want to use it in production? Can you help me understand?
> > @andersonm-ibm
>
> Hi @liujinhui1994 . This can be used as the basis for the production-grade KMS client. However, some of the reasons for this not being production-ready, in no specific order:
>
> * We won't be supporting changes in Vault API, which might break this client , or maintaining different versions of this client
>
> * More flexibility might be needed in the definition of the path to the transit engine, which in this example is hardcoded to "/v1/transit"
>
> * The final decision on how to pass the access token might depend on the production environment and company policies
>
> * Error handling can be made more tailored to specific use cases. For example, if you need to differentiate the cases where Vault doesn't grant access to the keys, then you would define a specific exception
>
> * More rigorous testing would be required to cover all the scenarios relevant to a production environment
And, of course, a large and important missing part here is token lifecycle management.
It would be interesting to hear about various usage scenarios in order to understand which token lifecycle events to support.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ggershinsky commented on pull request #4080: Add VaultKmsClient as an example KMS client implementation
Posted by GitBox <gi...@apache.org>.
ggershinsky commented on pull request #4080:
URL: https://github.com/apache/iceberg/pull/4080#issuecomment-1034896512
> The KMS client is configured with Hadoop configuration properties:
I think these two are better set as table properties.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ggershinsky commented on pull request #4080: Add VaultKmsClient as an example KMS client implementation
Posted by GitBox <gi...@apache.org>.
ggershinsky commented on pull request #4080:
URL: https://github.com/apache/iceberg/pull/4080#issuecomment-1034905316
> why do we have a separate KmsClient interface in Iceberg, different from the one defined in parquet.
We build a general key management layer in Iceberg encryption, that will work for all Iceberg data formats, so it should not depend on parquet. Also, this KmsClient is cleaner and more intuitive than what we have in parquet-mr.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] andersonm-ibm commented on pull request #4080: Add VaultKmsClient as an example KMS client implementation
Posted by GitBox <gi...@apache.org>.
andersonm-ibm commented on pull request #4080:
URL: https://github.com/apache/iceberg/pull/4080#issuecomment-1034656040
> @andersonm-ibm Please help me understand #3470 why not use parquet KMSClient here but wrap one myself. org.apache.parquet.crypto.keytools.KmsClient thanks~
If I understand correctly your question, you're asking why do we have a separate KmsClient interface in Iceberg, different from the one defined in parquet.
Then @ggershinsky would be the best person to answer you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ggershinsky edited a comment on pull request #4080: Add VaultKmsClient as an example KMS client implementation
Posted by GitBox <gi...@apache.org>.
ggershinsky edited a comment on pull request #4080:
URL: https://github.com/apache/iceberg/pull/4080#issuecomment-1034896512
> The KMS client is configured with Hadoop configuration properties:
I think the url is better set as a table property
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] liujinhui1994 commented on pull request #4080: Add VaultKmsClient as an example KMS client implementation
Posted by GitBox <gi...@apache.org>.
liujinhui1994 commented on pull request #4080:
URL: https://github.com/apache/iceberg/pull/4080#issuecomment-1034546009
Why can't it be used in production, and what if I want to use it in production? Can you help me understand?
@andersonm-ibm
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] andersonm-ibm commented on pull request #4080: Add VaultKmsClient as an example KMS client implementation
Posted by GitBox <gi...@apache.org>.
andersonm-ibm commented on pull request #4080:
URL: https://github.com/apache/iceberg/pull/4080#issuecomment-1034553840
> Why can't it be used in production, and what if I want to use it in production? Can you help me understand?
>
> @andersonm-ibm
Hi @liujinhui1994 . This can be used as the basis for the production-grade KMS client. However, some of the reasons for this not being production-ready, in no specific order:
- We won't be supporting changes in Vault API, which might break this client , or maintaining different versions of this client
- More flexibility might be needed in the definition of the path to the transit engine, which in this example is hardcoded to "/v1/transit"
- The final decision on how to pass the access token might depend on the production environment and company policies
- Error handling can be made more tailored to specific use cases. For example, if you need to differentiate the cases where Vault doesn't grant access to the keys, then you would define a specific exception
- More rigorous testing would be required to cover all the scenarios relevant to a production environment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org