You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ranger.apache.org by Dhaval Shah <dh...@gmail.com> on 2019/08/01 12:57:30 UTC
Re: Review Request 71176: RANGER-2497 Support Azure Key Vault for
storing master keys of Ranger KMS
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71176/
-----------------------------------------------------------
(Updated Aug. 1, 2019, 12:57 p.m.)
Review request for ranger, Ankita Sinha, Don Bosco Durai, bhavik patel, Gautam Borad, Abhay Kulkarni, Madhan Neethiraj, Mehul Parikh, Nikhil P, Ramesh Mani, and Velmurugan Periasamy.
Bugs: RANGER-2497
https://issues.apache.org/jira/browse/RANGER-2497
Repository: ranger
Description
-------
User story: As a security admin, I want to escrow and manage master encryption keys for securing my Hadoop cluster EZs in Ranger KMS service with Azure Key Vault service.
For Microsoft Azure Key Vault overview refer to: https://docs.microsoft.com/en-us/azure/key-vault/
For REST API guide refer to: https://docs.microsoft.com/en-us/rest/api/keyvault/
Acceptance Criteria:
1.) Ranger KMS has ability to configure AKV service to be used for master key offload
2.) Ranger KMS provides ability to provide key management functions (create keys, manage keys, retrieve keys, rollover) using AKV
Diffs (updated)
-----
kms/config/kms-webapp/dbks-site.xml 05a1a13
kms/pom.xml df46496
kms/scripts/DBMKTOAZUREKEYVAULT.sh PRE-CREATION
kms/scripts/install.properties 798dd8c
kms/scripts/setup.sh c430ef9
kms/src/main/java/org/apache/hadoop/crypto/key/AzureKeyVaultClientAuthenticator.java PRE-CREATION
kms/src/main/java/org/apache/hadoop/crypto/key/DBToAzureKeyVault.java PRE-CREATION
kms/src/main/java/org/apache/hadoop/crypto/key/JKS2RangerUtil.java 5e394de
kms/src/main/java/org/apache/hadoop/crypto/key/Ranger2JKSUtil.java f542364
kms/src/main/java/org/apache/hadoop/crypto/key/RangerKeyStore.java 86f1a29
kms/src/main/java/org/apache/hadoop/crypto/key/RangerKeyStoreProvider.java b280cbf
kms/src/main/java/org/apache/hadoop/crypto/key/RangerKeyVaultKeyGenerator.java PRE-CREATION
pom.xml 7cf134c
src/main/assembly/kms.xml 468bede
Diff: https://reviews.apache.org/r/71176/diff/2/
Changes: https://reviews.apache.org/r/71176/diff/1-2/
Testing
-------
1.) Fresh installation of Ranger KMS with Azure Key Vault.
2.) Export / Import of zone keys from / to keystore file.
3.) Migration of Ranger KMS DB to Azure Key Vault.
Thanks,
Dhaval Shah
Re: Review Request 71176: RANGER-2497 Support Azure Key Vault for
storing master keys of Ranger KMS
Posted by Mehul Parikh <me...@freestoneinfotech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71176/#review217977
-----------------------------------------------------------
Ship it!
Ship It!
- Mehul Parikh
On Aug. 12, 2019, 9:12 a.m., Dhaval Shah wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71176/
> -----------------------------------------------------------
>
> (Updated Aug. 12, 2019, 9:12 a.m.)
>
>
> Review request for ranger, Ankita Sinha, Don Bosco Durai, bhavik patel, Gautam Borad, Abhay Kulkarni, Madhan Neethiraj, Mehul Parikh, Nikhil P, Ramesh Mani, and Velmurugan Periasamy.
>
>
> Bugs: RANGER-2497
> https://issues.apache.org/jira/browse/RANGER-2497
>
>
> Repository: ranger
>
>
> Description
> -------
>
> User story: As a security admin, I want to escrow and manage master encryption keys for securing my Hadoop cluster EZs in Ranger KMS service with Azure Key Vault service.
>
> For Microsoft Azure Key Vault overview refer to: https://docs.microsoft.com/en-us/azure/key-vault/
> For REST API guide refer to: https://docs.microsoft.com/en-us/rest/api/keyvault/
>
> Acceptance Criteria:
>
> 1.) Ranger KMS has ability to configure AKV service to be used for master key offload
> 2.) Ranger KMS provides ability to provide key management functions (create keys, manage keys, retrieve keys, rollover) using AKV
>
>
> Diffs
> -----
>
> LICENSE.txt a424ebe
> NOTICE.txt a82c1f0
> kms/config/kms-webapp/dbks-site.xml 05a1a13
> kms/pom.xml df46496
> kms/scripts/DBMKTOAZUREKEYVAULT.sh PRE-CREATION
> kms/scripts/install.properties 798dd8c
> kms/scripts/setup.sh c430ef9
> kms/src/main/java/org/apache/hadoop/crypto/key/AzureKeyVaultClientAuthenticator.java PRE-CREATION
> kms/src/main/java/org/apache/hadoop/crypto/key/DBToAzureKeyVault.java PRE-CREATION
> kms/src/main/java/org/apache/hadoop/crypto/key/JKS2RangerUtil.java 5e394de
> kms/src/main/java/org/apache/hadoop/crypto/key/Ranger2JKSUtil.java f542364
> kms/src/main/java/org/apache/hadoop/crypto/key/RangerKeyStore.java 86f1a29
> kms/src/main/java/org/apache/hadoop/crypto/key/RangerKeyStoreProvider.java b280cbf
> kms/src/main/java/org/apache/hadoop/crypto/key/RangerKeyVaultKeyGenerator.java PRE-CREATION
> pom.xml 7cf134c
> src/main/assembly/kms.xml 468bede
>
>
> Diff: https://reviews.apache.org/r/71176/diff/3/
>
>
> Testing
> -------
>
> 1.) Fresh installation of Ranger KMS with Azure Key Vault.
> 2.) Export / Import of zone keys from / to keystore file.
> 3.) Migration of Ranger KMS DB to Azure Key Vault.
>
>
> Thanks,
>
> Dhaval Shah
>
>
Re: Review Request 71176: RANGER-2497 Support Azure Key Vault for
storing master keys of Ranger KMS
Posted by Velmurugan Periasamy <vp...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71176/#review217489
-----------------------------------------------------------
Ship it!
Ship It!
- Velmurugan Periasamy
On Aug. 12, 2019, 9:12 a.m., Dhaval Shah wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71176/
> -----------------------------------------------------------
>
> (Updated Aug. 12, 2019, 9:12 a.m.)
>
>
> Review request for ranger, Ankita Sinha, Don Bosco Durai, bhavik patel, Gautam Borad, Abhay Kulkarni, Madhan Neethiraj, Mehul Parikh, Nikhil P, Ramesh Mani, and Velmurugan Periasamy.
>
>
> Bugs: RANGER-2497
> https://issues.apache.org/jira/browse/RANGER-2497
>
>
> Repository: ranger
>
>
> Description
> -------
>
> User story: As a security admin, I want to escrow and manage master encryption keys for securing my Hadoop cluster EZs in Ranger KMS service with Azure Key Vault service.
>
> For Microsoft Azure Key Vault overview refer to: https://docs.microsoft.com/en-us/azure/key-vault/
> For REST API guide refer to: https://docs.microsoft.com/en-us/rest/api/keyvault/
>
> Acceptance Criteria:
>
> 1.) Ranger KMS has ability to configure AKV service to be used for master key offload
> 2.) Ranger KMS provides ability to provide key management functions (create keys, manage keys, retrieve keys, rollover) using AKV
>
>
> Diffs
> -----
>
> LICENSE.txt a424ebe
> NOTICE.txt a82c1f0
> kms/config/kms-webapp/dbks-site.xml 05a1a13
> kms/pom.xml df46496
> kms/scripts/DBMKTOAZUREKEYVAULT.sh PRE-CREATION
> kms/scripts/install.properties 798dd8c
> kms/scripts/setup.sh c430ef9
> kms/src/main/java/org/apache/hadoop/crypto/key/AzureKeyVaultClientAuthenticator.java PRE-CREATION
> kms/src/main/java/org/apache/hadoop/crypto/key/DBToAzureKeyVault.java PRE-CREATION
> kms/src/main/java/org/apache/hadoop/crypto/key/JKS2RangerUtil.java 5e394de
> kms/src/main/java/org/apache/hadoop/crypto/key/Ranger2JKSUtil.java f542364
> kms/src/main/java/org/apache/hadoop/crypto/key/RangerKeyStore.java 86f1a29
> kms/src/main/java/org/apache/hadoop/crypto/key/RangerKeyStoreProvider.java b280cbf
> kms/src/main/java/org/apache/hadoop/crypto/key/RangerKeyVaultKeyGenerator.java PRE-CREATION
> pom.xml 7cf134c
> src/main/assembly/kms.xml 468bede
>
>
> Diff: https://reviews.apache.org/r/71176/diff/3/
>
>
> Testing
> -------
>
> 1.) Fresh installation of Ranger KMS with Azure Key Vault.
> 2.) Export / Import of zone keys from / to keystore file.
> 3.) Migration of Ranger KMS DB to Azure Key Vault.
>
>
> Thanks,
>
> Dhaval Shah
>
>
Re: Review Request 71176: RANGER-2497 Support Azure Key Vault for
storing master keys of Ranger KMS
Posted by Dhaval Shah <dh...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71176/
-----------------------------------------------------------
(Updated Aug. 12, 2019, 9:12 a.m.)
Review request for ranger, Ankita Sinha, Don Bosco Durai, bhavik patel, Gautam Borad, Abhay Kulkarni, Madhan Neethiraj, Mehul Parikh, Nikhil P, Ramesh Mani, and Velmurugan Periasamy.
Bugs: RANGER-2497
https://issues.apache.org/jira/browse/RANGER-2497
Repository: ranger
Description
-------
User story: As a security admin, I want to escrow and manage master encryption keys for securing my Hadoop cluster EZs in Ranger KMS service with Azure Key Vault service.
For Microsoft Azure Key Vault overview refer to: https://docs.microsoft.com/en-us/azure/key-vault/
For REST API guide refer to: https://docs.microsoft.com/en-us/rest/api/keyvault/
Acceptance Criteria:
1.) Ranger KMS has ability to configure AKV service to be used for master key offload
2.) Ranger KMS provides ability to provide key management functions (create keys, manage keys, retrieve keys, rollover) using AKV
Diffs (updated)
-----
LICENSE.txt a424ebe
NOTICE.txt a82c1f0
kms/config/kms-webapp/dbks-site.xml 05a1a13
kms/pom.xml df46496
kms/scripts/DBMKTOAZUREKEYVAULT.sh PRE-CREATION
kms/scripts/install.properties 798dd8c
kms/scripts/setup.sh c430ef9
kms/src/main/java/org/apache/hadoop/crypto/key/AzureKeyVaultClientAuthenticator.java PRE-CREATION
kms/src/main/java/org/apache/hadoop/crypto/key/DBToAzureKeyVault.java PRE-CREATION
kms/src/main/java/org/apache/hadoop/crypto/key/JKS2RangerUtil.java 5e394de
kms/src/main/java/org/apache/hadoop/crypto/key/Ranger2JKSUtil.java f542364
kms/src/main/java/org/apache/hadoop/crypto/key/RangerKeyStore.java 86f1a29
kms/src/main/java/org/apache/hadoop/crypto/key/RangerKeyStoreProvider.java b280cbf
kms/src/main/java/org/apache/hadoop/crypto/key/RangerKeyVaultKeyGenerator.java PRE-CREATION
pom.xml 7cf134c
src/main/assembly/kms.xml 468bede
Diff: https://reviews.apache.org/r/71176/diff/3/
Changes: https://reviews.apache.org/r/71176/diff/2-3/
Testing
-------
1.) Fresh installation of Ranger KMS with Azure Key Vault.
2.) Export / Import of zone keys from / to keystore file.
3.) Migration of Ranger KMS DB to Azure Key Vault.
Thanks,
Dhaval Shah
Re: Review Request 71176: RANGER-2497 Support Azure Key Vault for
storing master keys of Ranger KMS
Posted by Dhaval Shah <dh...@gmail.com>.
> On Aug. 8, 2019, 4:32 p.m., Velmurugan Periasamy wrote:
> > pom.xml
> > Lines 198 (patched)
> > <https://reviews.apache.org/r/71176/diff/1/?file=2158090#file2158090line198>
> >
> > Make sure these new dependencies are added to LICENSE and NOTICE as required
Done.
> On Aug. 8, 2019, 4:32 p.m., Velmurugan Periasamy wrote:
> > kms/src/main/java/org/apache/hadoop/crypto/key/RangerKeyStore.java
> > Lines 92 (patched)
> > <https://reviews.apache.org/r/71176/diff/2/?file=2158714#file2158714line93>
> >
> > Would it be cleaner to isolate this to a separate class?
I have created AzureKeyVaultClientAuthenticator.java which is does the authentication for id /password and id/certificate.
I have created RangerKeyVaultKeyGenerator.java which does all the operation which deals with AKV i.e. Master Key generation, ZoneKey encrypt / decrypt (The way we have for other HSM).
Changes in this class deals more with RangerKMS.
Reason for visibility of huge change is because in this HSM we do the zoneKey encryption / decryption on AKV. So we have to undergo the code change depending upon the o/p and i/p parameters of AKV methods.
> On Aug. 8, 2019, 4:32 p.m., Velmurugan Periasamy wrote:
> > kms/src/main/java/org/apache/hadoop/crypto/key/RangerKeyStoreProvider.java
> > Lines 76 (patched)
> > <https://reviews.apache.org/r/71176/diff/2/?file=2158715#file2158715line82>
> >
> > Would it be better to isolate this to a separate class? I see similar approach is done for other HSM as well, but it might become challenging to maintain as more integrations keep being added.
I have created AzureKeyVaultClientAuthenticator.java which is does the authentication for id /password and id/certificate.
I have created RangerKeyVaultKeyGenerator.java which does all the operation which deals with AKV i.e. Master Key generation, ZoneKey encrypt / decrypt (The way we have for other HSM).
Changes in this class deals more with RangerKMS.
Reason for visibility of huge change is because in this HSM we do the zoneKey encryption / decryption on AKV. So we have to undergo the code change depending upon the o/p and i/p parameters of AKV methods.
- Dhaval
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71176/#review216999
-----------------------------------------------------------
On Aug. 12, 2019, 9:12 a.m., Dhaval Shah wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71176/
> -----------------------------------------------------------
>
> (Updated Aug. 12, 2019, 9:12 a.m.)
>
>
> Review request for ranger, Ankita Sinha, Don Bosco Durai, bhavik patel, Gautam Borad, Abhay Kulkarni, Madhan Neethiraj, Mehul Parikh, Nikhil P, Ramesh Mani, and Velmurugan Periasamy.
>
>
> Bugs: RANGER-2497
> https://issues.apache.org/jira/browse/RANGER-2497
>
>
> Repository: ranger
>
>
> Description
> -------
>
> User story: As a security admin, I want to escrow and manage master encryption keys for securing my Hadoop cluster EZs in Ranger KMS service with Azure Key Vault service.
>
> For Microsoft Azure Key Vault overview refer to: https://docs.microsoft.com/en-us/azure/key-vault/
> For REST API guide refer to: https://docs.microsoft.com/en-us/rest/api/keyvault/
>
> Acceptance Criteria:
>
> 1.) Ranger KMS has ability to configure AKV service to be used for master key offload
> 2.) Ranger KMS provides ability to provide key management functions (create keys, manage keys, retrieve keys, rollover) using AKV
>
>
> Diffs
> -----
>
> LICENSE.txt a424ebe
> NOTICE.txt a82c1f0
> kms/config/kms-webapp/dbks-site.xml 05a1a13
> kms/pom.xml df46496
> kms/scripts/DBMKTOAZUREKEYVAULT.sh PRE-CREATION
> kms/scripts/install.properties 798dd8c
> kms/scripts/setup.sh c430ef9
> kms/src/main/java/org/apache/hadoop/crypto/key/AzureKeyVaultClientAuthenticator.java PRE-CREATION
> kms/src/main/java/org/apache/hadoop/crypto/key/DBToAzureKeyVault.java PRE-CREATION
> kms/src/main/java/org/apache/hadoop/crypto/key/JKS2RangerUtil.java 5e394de
> kms/src/main/java/org/apache/hadoop/crypto/key/Ranger2JKSUtil.java f542364
> kms/src/main/java/org/apache/hadoop/crypto/key/RangerKeyStore.java 86f1a29
> kms/src/main/java/org/apache/hadoop/crypto/key/RangerKeyStoreProvider.java b280cbf
> kms/src/main/java/org/apache/hadoop/crypto/key/RangerKeyVaultKeyGenerator.java PRE-CREATION
> pom.xml 7cf134c
> src/main/assembly/kms.xml 468bede
>
>
> Diff: https://reviews.apache.org/r/71176/diff/3/
>
>
> Testing
> -------
>
> 1.) Fresh installation of Ranger KMS with Azure Key Vault.
> 2.) Export / Import of zone keys from / to keystore file.
> 3.) Migration of Ranger KMS DB to Azure Key Vault.
>
>
> Thanks,
>
> Dhaval Shah
>
>
Re: Review Request 71176: RANGER-2497 Support Azure Key Vault for
storing master keys of Ranger KMS
Posted by Velmurugan Periasamy <vp...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71176/#review216999
-----------------------------------------------------------
pom.xml
Lines 198 (patched)
<https://reviews.apache.org/r/71176/#comment304237>
Make sure these new dependencies are added to LICENSE and NOTICE as required
kms/src/main/java/org/apache/hadoop/crypto/key/RangerKeyStore.java
Lines 92 (patched)
<https://reviews.apache.org/r/71176/#comment304388>
Would it be cleaner to isolate this to a separate class?
kms/src/main/java/org/apache/hadoop/crypto/key/RangerKeyStoreProvider.java
Lines 76 (patched)
<https://reviews.apache.org/r/71176/#comment304389>
Would it be better to isolate this to a separate class? I see similar approach is done for other HSM as well, but it might become challenging to maintain as more integrations keep being added.
- Velmurugan Periasamy
On Aug. 1, 2019, 12:57 p.m., Dhaval Shah wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71176/
> -----------------------------------------------------------
>
> (Updated Aug. 1, 2019, 12:57 p.m.)
>
>
> Review request for ranger, Ankita Sinha, Don Bosco Durai, bhavik patel, Gautam Borad, Abhay Kulkarni, Madhan Neethiraj, Mehul Parikh, Nikhil P, Ramesh Mani, and Velmurugan Periasamy.
>
>
> Bugs: RANGER-2497
> https://issues.apache.org/jira/browse/RANGER-2497
>
>
> Repository: ranger
>
>
> Description
> -------
>
> User story: As a security admin, I want to escrow and manage master encryption keys for securing my Hadoop cluster EZs in Ranger KMS service with Azure Key Vault service.
>
> For Microsoft Azure Key Vault overview refer to: https://docs.microsoft.com/en-us/azure/key-vault/
> For REST API guide refer to: https://docs.microsoft.com/en-us/rest/api/keyvault/
>
> Acceptance Criteria:
>
> 1.) Ranger KMS has ability to configure AKV service to be used for master key offload
> 2.) Ranger KMS provides ability to provide key management functions (create keys, manage keys, retrieve keys, rollover) using AKV
>
>
> Diffs
> -----
>
> kms/config/kms-webapp/dbks-site.xml 05a1a13
> kms/pom.xml df46496
> kms/scripts/DBMKTOAZUREKEYVAULT.sh PRE-CREATION
> kms/scripts/install.properties 798dd8c
> kms/scripts/setup.sh c430ef9
> kms/src/main/java/org/apache/hadoop/crypto/key/AzureKeyVaultClientAuthenticator.java PRE-CREATION
> kms/src/main/java/org/apache/hadoop/crypto/key/DBToAzureKeyVault.java PRE-CREATION
> kms/src/main/java/org/apache/hadoop/crypto/key/JKS2RangerUtil.java 5e394de
> kms/src/main/java/org/apache/hadoop/crypto/key/Ranger2JKSUtil.java f542364
> kms/src/main/java/org/apache/hadoop/crypto/key/RangerKeyStore.java 86f1a29
> kms/src/main/java/org/apache/hadoop/crypto/key/RangerKeyStoreProvider.java b280cbf
> kms/src/main/java/org/apache/hadoop/crypto/key/RangerKeyVaultKeyGenerator.java PRE-CREATION
> pom.xml 7cf134c
> src/main/assembly/kms.xml 468bede
>
>
> Diff: https://reviews.apache.org/r/71176/diff/2/
>
>
> Testing
> -------
>
> 1.) Fresh installation of Ranger KMS with Azure Key Vault.
> 2.) Export / Import of zone keys from / to keystore file.
> 3.) Migration of Ranger KMS DB to Azure Key Vault.
>
>
> Thanks,
>
> Dhaval Shah
>
>