You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Vihang Karajgaonkar <vi...@cloudera.com> on 2017/09/22 15:50:36 UTC
Review Request 62453: HIVE-17371 : Move tokenstores to metastore
module
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62453/
-----------------------------------------------------------
Review request for hive, Aihua Xu, Alan Gates, and Thejas Nair.
Bugs: HIVE-17371
https://issues.apache.org/jira/browse/HIVE-17371
Repository: hive-git
Description
-------
HIVE-17371 : Move tokenstores to metastore module
I think it makes more sense to move the tokenstore implementations to metastore instead of having two copies of each in Hive and metastore. The TokenManager and SecretManager classes can be kept as is to each consuming applications like HS2 or Metastore itself so that future modifications can be done independently.
Here is the approach I followed.
1. Moved the HiveDelegationTokenManager and TokenStoreDelegationTokenSecretManager from shims-common to service module since Metastore anyways has its own copy of TokenManager and Metastore cannot have a dependency on shims-common.
2. Moved the MemoryTokenStore, ZooKeeperTokenStore and DBTokenStore to standalone-metastore module. In order for these tokenstores to be consumable for multiple applications like HS2 and Metastore (and hopefully other applications in the future) we need to operate of higher level abstraction of AbstractDelegationTokenIdentifier and DelegationTokenInformation.
3. Each application can define its DelegationTokenIndentifier like Hive already does in DelegationTokenIdentifier (may be we should rename this to HiveDelegationTokenIdentifier to make it more explicit. Similarly, metastore has its own implementation of DelegationTokenIdentifier in DelegationTokenIdentifier (again we should rename it possibly to say MetastoreDelegationTokenIndentifier)
4. Modified the shims-common classes of DelegationTokenSecretManager and DelegationTokenStore to operate on hadoop level abstractions instead of Hive defined ones.
Diffs
-----
itests/hive-minikdc/src/test/java/org/apache/hive/minikdc/TestHiveAuthFactory.java e3a51909df14d90a538426163f332a6378f35f40
itests/hive-minikdc/src/test/java/org/apache/hive/minikdc/TestJdbcWithDBTokenStore.java d690aaa673a50785561750f4f461ec867b6f0abc
itests/hive-unit-hadoop2/src/test/java/org/apache/hadoop/hive/metastore/security/TestHadoopAuthBridge23.java ef360400a1873ab142cb9cbdce045cbed394af0e
itests/hive-unit/src/test/java/org/apache/hadoop/hive/thrift/TestDBTokenStore.java 4bfa22419cb1ea5abe56773c1c76186b01996e03
itests/hive-unit/src/test/java/org/apache/hadoop/hive/thrift/TestZooKeeperTokenStore.java 7800416a60239246e96497229d592736f2a9d79d
metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java cf33cca24ffc7583f2be62bb79815b5e43b2ff42
service/src/java/org/apache/hive/service/auth/HiveAuthFactory.java 541fe5ec8da9ba66ac44c65e6fe0dcd3c601943f
shims/common/src/main/java/org/apache/hadoop/hive/shims/Utils.java 3c93186082ae363497dc94ca1303db4f746fd8b5
shims/common/src/main/java/org/apache/hadoop/hive/thrift/DBTokenStore.java d6dc0796e77591d3afca8dbd29c3aa0eff255dd0
shims/common/src/main/java/org/apache/hadoop/hive/thrift/DelegationTokenSecretManager.java 5299e18743aa45c539287b335f95e8ce8df0fc35
shims/common/src/main/java/org/apache/hadoop/hive/thrift/DelegationTokenStore.java 867b4ed98859f681770fccfe7478ab519c823924
shims/common/src/main/java/org/apache/hadoop/hive/thrift/HadoopThriftAuthBridge.java 6c8b362270e48cb5641d77b09f1a19d875df0cbb
shims/common/src/main/java/org/apache/hadoop/hive/thrift/HiveDelegationTokenManager.java b3e4a7608282be603e79d1d101679e239a5219b0
shims/common/src/main/java/org/apache/hadoop/hive/thrift/MemoryTokenStore.java 9d837b8fa19ed53546c4a95944484cc1f06a21e6
shims/common/src/main/java/org/apache/hadoop/hive/thrift/TokenStoreDelegationTokenSecretManager.java 4719b85e8d2f9045ae61e0e86c6624b865aa2653
shims/common/src/main/java/org/apache/hadoop/hive/thrift/ZooKeeperTokenStore.java 885ec56d543e1c46e74dc148c73f249803b7604c
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/security/DelegationTokenStore.java 0cafeff89614c94ba551a5f1ba9c4d892ee5720a
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/security/MemoryTokenStore.java c484cd3132d08c8c0493fa98e79fa449e84ef9b8
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/security/MetastoreDelegationTokenManager.java 2b0110fe0badcb571144067539031fb6ac81276b
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/security/TokenStoreDelegationTokenSecretManager.java 4abcec789476982389b3c0bc34d4990c59dbf9da
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/utils/SecurityUtils.java 9f0ca829b1e3a469a3275c54a74d90278fc030fa
Diff: https://reviews.apache.org/r/62453/diff/1/
Testing
-------
HiveQA
Thanks,
Vihang Karajgaonkar
Re: Review Request 62453: HIVE-17371 : Move tokenstores to metastore
module
Posted by Vihang Karajgaonkar <vi...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62453/
-----------------------------------------------------------
(Updated Oct. 6, 2017, 6:33 p.m.)
Review request for hive, Aihua Xu, Alan Gates, and Thejas Nair.
Changes
-------
rebased and resolved conflicts.
Bugs: HIVE-17371
https://issues.apache.org/jira/browse/HIVE-17371
Repository: hive-git
Description
-------
HIVE-17371 : Move tokenstores to metastore module
Diffs (updated)
-----
beeline/src/test/org/apache/hive/beeline/ProxyAuthTest.java 318857dddbff22615fcf660988b7dc9f25fbec65
hcatalog/core/src/main/java/org/apache/hive/hcatalog/common/HCatUtil.java 81804cfb2e309387051fb1762ff1e5959b9f9ce2
hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/Security.java 9b621952e63ab43a0a907e97d451a0d8bb190940
hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/tool/TempletonControllerJob.java f4c4b76c8127c4be02eb1cfb0622ef7a07b08470
itests/hive-minikdc/src/test/java/org/apache/hive/minikdc/TestHiveAuthFactory.java e3a51909df14d90a538426163f332a6378f35f40
itests/hive-minikdc/src/test/java/org/apache/hive/minikdc/TestJdbcWithDBTokenStore.java d690aaa673a50785561750f4f461ec867b6f0abc
itests/hive-minikdc/src/test/java/org/apache/hive/minikdc/TestJdbcWithMiniKdc.java 256262d6e488237950bb42b4b93e72f1fe82efb3
itests/hive-unit/src/test/java/org/apache/hadoop/hive/thrift/TestDBTokenStore.java 4bfa22419cb1ea5abe56773c1c76186b01996e03
itests/hive-unit/src/test/java/org/apache/hadoop/hive/thrift/TestZooKeeperTokenStore.java 7800416a60239246e96497229d592736f2a9d79d
jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java a9a4f2cf66f649c67e09a53f73e04951c1bb9e57
jdbc/src/java/org/apache/hive/jdbc/Utils.java 574fb7e1d2c46d3e51278143b74ca853bbe7769a
metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 5617e1ce5748c85831c570d3e4c09fd6cdc17bc8
metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java 6bc45b6c7b4e209e2214f0aafddc062660b23d77
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java a147a2590d6de1fe161c7b02f043f179243cf83c
service/src/java/org/apache/hive/service/auth/HiveAuthFactory.java 0f4a4d8d545cfaf6869b706db828e768f1d53d4d
service/src/java/org/apache/hive/service/auth/HttpAuthUtils.java 8b5661a0508cf0b68d7040e3de85f4694cb9c3c5
service/src/java/org/apache/hive/service/auth/KerberosSaslHelper.java ea2c689a4fb2e2ba6867b276044cbca0ac6899d3
service/src/java/org/apache/hive/service/cli/session/HiveSessionImplwithUGI.java 00a7e742cabd2fc36faa464b29250b5a6a9b1159
service/src/java/org/apache/hive/service/cli/session/SessionUtils.java PRE-CREATION
shims/0.23/src/main/java/org/apache/hadoop/hive/thrift/HadoopThriftAuthBridge23.java 5e21c9f187977a8bf556f76cbdc0c49f3a531641
shims/common/src/main/java/org/apache/hadoop/hive/shims/ShimLoader.java f15e7ff53fa340ab384db6071146b95529022fed
shims/common/src/main/java/org/apache/hadoop/hive/shims/Utils.java 3c93186082ae363497dc94ca1303db4f746fd8b5
shims/common/src/main/java/org/apache/hadoop/hive/thrift/DBTokenStore.java d6dc0796e77591d3afca8dbd29c3aa0eff255dd0
shims/common/src/main/java/org/apache/hadoop/hive/thrift/DelegationTokenIdentifier.java 4ca3c0beefa6944407f4b1d0aebee79e86b4ab84
shims/common/src/main/java/org/apache/hadoop/hive/thrift/DelegationTokenSecretManager.java 5299e18743aa45c539287b335f95e8ce8df0fc35
shims/common/src/main/java/org/apache/hadoop/hive/thrift/DelegationTokenSelector.java f6e2420eb20f58a22f636976fd53a98c294ddfe2
shims/common/src/main/java/org/apache/hadoop/hive/thrift/DelegationTokenStore.java 867b4ed98859f681770fccfe7478ab519c823924
shims/common/src/main/java/org/apache/hadoop/hive/thrift/HadoopThriftAuthBridge.java 2d39bea4be27d7c99285f53b8288d141b5acedc0
shims/common/src/main/java/org/apache/hadoop/hive/thrift/HiveDelegationTokenManager.java b3e4a7608282be603e79d1d101679e239a5219b0
shims/common/src/main/java/org/apache/hadoop/hive/thrift/MemoryTokenStore.java 9d837b8fa19ed53546c4a95944484cc1f06a21e6
shims/common/src/main/java/org/apache/hadoop/hive/thrift/TokenStoreDelegationTokenSecretManager.java 4719b85e8d2f9045ae61e0e86c6624b865aa2653
shims/common/src/main/java/org/apache/hadoop/hive/thrift/ZooKeeperTokenStore.java 885ec56d543e1c46e74dc148c73f249803b7604c
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/security/DelegationTokenStore.java 0cafeff89614c94ba551a5f1ba9c4d892ee5720a
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/security/MemoryTokenStore.java c484cd3132d08c8c0493fa98e79fa449e84ef9b8
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/security/MetastoreDelegationTokenManager.java 2b0110fe0badcb571144067539031fb6ac81276b
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/security/TokenStoreDelegationTokenSecretManager.java 4abcec789476982389b3c0bc34d4990c59dbf9da
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/utils/SecurityUtils.java 9f0ca829b1e3a469a3275c54a74d90278fc030fa
Diff: https://reviews.apache.org/r/62453/diff/3/
Changes: https://reviews.apache.org/r/62453/diff/2-3/
Testing
-------
HiveQA
Thanks,
Vihang Karajgaonkar
Re: Review Request 62453: HIVE-17371 : Move tokenstores to metastore
module
Posted by Vihang Karajgaonkar <vi...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62453/
-----------------------------------------------------------
(Updated Oct. 6, 2017, 2:13 a.m.)
Review request for hive, Aihua Xu, Alan Gates, and Thejas Nair.
Changes
-------
Moved DelegationTokenManager and associated classes to standalone-metastore module as suggested by Thejas.
Bugs: HIVE-17371
https://issues.apache.org/jira/browse/HIVE-17371
Repository: hive-git
Description (updated)
-------
HIVE-17371 : Move tokenstores to metastore module
Diffs (updated)
-----
beeline/src/test/org/apache/hive/beeline/ProxyAuthTest.java 318857dddbff22615fcf660988b7dc9f25fbec65
hcatalog/core/src/main/java/org/apache/hive/hcatalog/common/HCatUtil.java 8b927af8a2cf7cb06244e5a9ae7b40d0237a1c33
hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/Security.java 9b621952e63ab43a0a907e97d451a0d8bb190940
hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/tool/TempletonControllerJob.java f4c4b76c8127c4be02eb1cfb0622ef7a07b08470
itests/hive-minikdc/src/test/java/org/apache/hive/minikdc/TestHiveAuthFactory.java e3a51909df14d90a538426163f332a6378f35f40
itests/hive-minikdc/src/test/java/org/apache/hive/minikdc/TestJdbcWithDBTokenStore.java d690aaa673a50785561750f4f461ec867b6f0abc
itests/hive-minikdc/src/test/java/org/apache/hive/minikdc/TestJdbcWithMiniKdc.java 256262d6e488237950bb42b4b93e72f1fe82efb3
itests/hive-unit/src/test/java/org/apache/hadoop/hive/thrift/TestDBTokenStore.java 4bfa22419cb1ea5abe56773c1c76186b01996e03
itests/hive-unit/src/test/java/org/apache/hadoop/hive/thrift/TestZooKeeperTokenStore.java 7800416a60239246e96497229d592736f2a9d79d
jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java 1311d2d88c2880991718c5d9f9fb13579b25831a
jdbc/src/java/org/apache/hive/jdbc/Utils.java bfae8b9e41e4d75b6a56a6a20343afc030d9251d
metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java cf33cca24ffc7583f2be62bb79815b5e43b2ff42
metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java 75e418039175f8a578212872fdf167bd40fc70e6
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java bbe13fd77b5a0cfa7a6818f7a9e23985772acf3d
service/src/java/org/apache/hive/service/auth/HiveAuthFactory.java 541fe5ec8da9ba66ac44c65e6fe0dcd3c601943f
service/src/java/org/apache/hive/service/auth/HttpAuthUtils.java 8b5661a0508cf0b68d7040e3de85f4694cb9c3c5
service/src/java/org/apache/hive/service/auth/KerberosSaslHelper.java ea2c689a4fb2e2ba6867b276044cbca0ac6899d3
service/src/java/org/apache/hive/service/cli/session/HiveSessionImplwithUGI.java 00a7e742cabd2fc36faa464b29250b5a6a9b1159
shims/0.23/src/main/java/org/apache/hadoop/hive/thrift/HadoopThriftAuthBridge23.java 5e21c9f187977a8bf556f76cbdc0c49f3a531641
shims/common/src/main/java/org/apache/hadoop/hive/shims/ShimLoader.java f15e7ff53fa340ab384db6071146b95529022fed
shims/common/src/main/java/org/apache/hadoop/hive/shims/Utils.java 3c93186082ae363497dc94ca1303db4f746fd8b5
shims/common/src/main/java/org/apache/hadoop/hive/thrift/DBTokenStore.java d6dc0796e77591d3afca8dbd29c3aa0eff255dd0
shims/common/src/main/java/org/apache/hadoop/hive/thrift/DelegationTokenIdentifier.java 4ca3c0beefa6944407f4b1d0aebee79e86b4ab84
shims/common/src/main/java/org/apache/hadoop/hive/thrift/DelegationTokenSecretManager.java 5299e18743aa45c539287b335f95e8ce8df0fc35
shims/common/src/main/java/org/apache/hadoop/hive/thrift/DelegationTokenSelector.java f6e2420eb20f58a22f636976fd53a98c294ddfe2
shims/common/src/main/java/org/apache/hadoop/hive/thrift/DelegationTokenStore.java 867b4ed98859f681770fccfe7478ab519c823924
shims/common/src/main/java/org/apache/hadoop/hive/thrift/HadoopThriftAuthBridge.java 6c8b362270e48cb5641d77b09f1a19d875df0cbb
shims/common/src/main/java/org/apache/hadoop/hive/thrift/HiveDelegationTokenManager.java b3e4a7608282be603e79d1d101679e239a5219b0
shims/common/src/main/java/org/apache/hadoop/hive/thrift/MemoryTokenStore.java 9d837b8fa19ed53546c4a95944484cc1f06a21e6
shims/common/src/main/java/org/apache/hadoop/hive/thrift/TokenStoreDelegationTokenSecretManager.java 4719b85e8d2f9045ae61e0e86c6624b865aa2653
shims/common/src/main/java/org/apache/hadoop/hive/thrift/ZooKeeperTokenStore.java 885ec56d543e1c46e74dc148c73f249803b7604c
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/security/DelegationTokenStore.java 0cafeff89614c94ba551a5f1ba9c4d892ee5720a
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/security/MemoryTokenStore.java c484cd3132d08c8c0493fa98e79fa449e84ef9b8
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/security/MetastoreDelegationTokenManager.java 2b0110fe0badcb571144067539031fb6ac81276b
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/security/TokenStoreDelegationTokenSecretManager.java 4abcec789476982389b3c0bc34d4990c59dbf9da
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/utils/SecurityUtils.java 9f0ca829b1e3a469a3275c54a74d90278fc030fa
Diff: https://reviews.apache.org/r/62453/diff/2/
Changes: https://reviews.apache.org/r/62453/diff/1-2/
Testing
-------
HiveQA
Thanks,
Vihang Karajgaonkar