You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Mohit Sabharwal <mo...@cloudera.com> on 2014/10/10 01:35:27 UTC
Review Request 26515: HIVE-8424 : Support fair scheduler user queue
mapping in non-impersonation mode
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26515/
-----------------------------------------------------------
Review request for hive.
Bugs: HIVE-8424
https://issues.apache.org/jira/browse/HIVE-8424
Repository: hive-git
Description
-------
HIVE-8424 : Support fair scheduler user queue mapping in non-impersonation mode
Under non-impersonation mode, all MR jobs run as the hive system user.
The default scheduler queue mapping is one queue per user. This is problematic
for users who use the queues to regulate and track their MR resource usage.
This patch uses the Yarn API to set the fair scheduler queue mapping for
the current user, if impersonation is turned off.
Diffs
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 6a1a5f0542ca0723e9e415863ca74c8b43bcc96c
itests/hive-unit/pom.xml 077631a7d809e6d5fd79815fca8882898ee172e9
itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestSchedulerQueue.java PRE-CREATION
pom.xml f5a25e5c86b0801d382abd66597be8d7ac69e426
service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java f021870cf48c55a3d2dea4c825f1f25d0ab008f3
shims/0.20/src/main/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java 52f02c1a7915125fedd7b756e1db4611cf81d966
shims/0.20S/src/main/java/org/apache/hadoop/hive/shims/Hadoop20SShims.java 07946df1b0fc9064925f431fe1f0e2c020b41dfc
shims/0.23/pom.xml ad71fccc2f94f4523cbb3572fecc06e54528d630
shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java 9829c1b54ed2c1cb3068a569ff47d897f6284933
shims/common/src/main/java/org/apache/hadoop/hive/shims/HadoopShims.java d56795f7be1cffa13ea7b3ea2cd50db90730c020
Diff: https://reviews.apache.org/r/26515/diff/
Testing
-------
Added unit test
Thanks,
Mohit Sabharwal
Re: Review Request 26515: HIVE-8424 : Support fair scheduler user
queue mapping in non-impersonation mode
Posted by Brock Noland <br...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26515/#review56425
-----------------------------------------------------------
Ship it!
Ship It!
- Brock Noland
On Oct. 10, 2014, 4:28 a.m., Mohit Sabharwal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/26515/
> -----------------------------------------------------------
>
> (Updated Oct. 10, 2014, 4:28 a.m.)
>
>
> Review request for hive.
>
>
> Bugs: HIVE-8424
> https://issues.apache.org/jira/browse/HIVE-8424
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> HIVE-8424 : Support fair scheduler user queue mapping in non-impersonation mode
>
> Under non-impersonation mode, all MR jobs run as the hive system user.
> The default scheduler queue mapping is one queue per user. This is problematic
> for users who use the queues to regulate and track their MR resource usage.
>
> This patch uses the Yarn API to set the fair scheduler queue mapping for
> the current user, if impersonation is turned off.
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 6a1a5f0542ca0723e9e415863ca74c8b43bcc96c
> itests/hive-unit/pom.xml 077631a7d809e6d5fd79815fca8882898ee172e9
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestSchedulerQueue.java PRE-CREATION
> pom.xml f5a25e5c86b0801d382abd66597be8d7ac69e426
> service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java f021870cf48c55a3d2dea4c825f1f25d0ab008f3
> shims/0.20/src/main/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java 52f02c1a7915125fedd7b756e1db4611cf81d966
> shims/0.20S/src/main/java/org/apache/hadoop/hive/shims/Hadoop20SShims.java 07946df1b0fc9064925f431fe1f0e2c020b41dfc
> shims/0.23/pom.xml ad71fccc2f94f4523cbb3572fecc06e54528d630
> shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java 9829c1b54ed2c1cb3068a569ff47d897f6284933
> shims/common/src/main/java/org/apache/hadoop/hive/shims/HadoopShims.java d56795f7be1cffa13ea7b3ea2cd50db90730c020
>
> Diff: https://reviews.apache.org/r/26515/diff/
>
>
> Testing
> -------
>
> Added unit test
>
>
> Thanks,
>
> Mohit Sabharwal
>
>
Re: Review Request 26515: HIVE-8424 : Support fair scheduler user
queue mapping in non-impersonation mode
Posted by Prasad Mujumdar <pr...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26515/#review56373
-----------------------------------------------------------
Ship it!
Looks fine to me.
- Prasad Mujumdar
On Oct. 10, 2014, 4:28 a.m., Mohit Sabharwal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/26515/
> -----------------------------------------------------------
>
> (Updated Oct. 10, 2014, 4:28 a.m.)
>
>
> Review request for hive.
>
>
> Bugs: HIVE-8424
> https://issues.apache.org/jira/browse/HIVE-8424
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> HIVE-8424 : Support fair scheduler user queue mapping in non-impersonation mode
>
> Under non-impersonation mode, all MR jobs run as the hive system user.
> The default scheduler queue mapping is one queue per user. This is problematic
> for users who use the queues to regulate and track their MR resource usage.
>
> This patch uses the Yarn API to set the fair scheduler queue mapping for
> the current user, if impersonation is turned off.
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 6a1a5f0542ca0723e9e415863ca74c8b43bcc96c
> itests/hive-unit/pom.xml 077631a7d809e6d5fd79815fca8882898ee172e9
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestSchedulerQueue.java PRE-CREATION
> pom.xml f5a25e5c86b0801d382abd66597be8d7ac69e426
> service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java f021870cf48c55a3d2dea4c825f1f25d0ab008f3
> shims/0.20/src/main/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java 52f02c1a7915125fedd7b756e1db4611cf81d966
> shims/0.20S/src/main/java/org/apache/hadoop/hive/shims/Hadoop20SShims.java 07946df1b0fc9064925f431fe1f0e2c020b41dfc
> shims/0.23/pom.xml ad71fccc2f94f4523cbb3572fecc06e54528d630
> shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java 9829c1b54ed2c1cb3068a569ff47d897f6284933
> shims/common/src/main/java/org/apache/hadoop/hive/shims/HadoopShims.java d56795f7be1cffa13ea7b3ea2cd50db90730c020
>
> Diff: https://reviews.apache.org/r/26515/diff/
>
>
> Testing
> -------
>
> Added unit test
>
>
> Thanks,
>
> Mohit Sabharwal
>
>
Re: Review Request 26515: HIVE-8424 : Support fair scheduler user
queue mapping in non-impersonation mode
Posted by Mohit Sabharwal <mo...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26515/
-----------------------------------------------------------
(Updated Oct. 13, 2014, 11:04 p.m.)
Review request for hive.
Changes
-------
Removed extra yarn dependency from hive-unit-hadoop2/pom.xml
Bugs: HIVE-8424
https://issues.apache.org/jira/browse/HIVE-8424
Repository: hive-git
Description
-------
HIVE-8424 : Support fair scheduler user queue mapping in non-impersonation mode
Under non-impersonation mode, all MR jobs run as the hive system user.
The default scheduler queue mapping is one queue per user. This is problematic
for users who use the queues to regulate and track their MR resource usage.
This patch uses the Yarn API to set the fair scheduler queue mapping for
the current user, if impersonation is turned off.
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 018c6035dd557ebb812f4224509bcf17bfdf045a
itests/hive-unit-hadoop2/pom.xml cbdf506f94e2f720543e5932f0af65623fb43dc5
itests/hive-unit-hadoop2/src/test/java/org/apache/hive/jdbc/TestSchedulerQueue.java PRE-CREATION
service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java f021870cf48c55a3d2dea4c825f1f25d0ab008f3
shims/0.20/src/main/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java 52f02c1a7915125fedd7b756e1db4611cf81d966
shims/0.20S/src/main/java/org/apache/hadoop/hive/shims/Hadoop20SShims.java 07946df1b0fc9064925f431fe1f0e2c020b41dfc
shims/0.23/pom.xml ad71fccc2f94f4523cbb3572fecc06e54528d630
shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java 9829c1b54ed2c1cb3068a569ff47d897f6284933
shims/common/src/main/java/org/apache/hadoop/hive/shims/HadoopShims.java d56795f7be1cffa13ea7b3ea2cd50db90730c020
Diff: https://reviews.apache.org/r/26515/diff/
Testing
-------
Added unit test
Thanks,
Mohit Sabharwal
Re: Review Request 26515: HIVE-8424 : Support fair scheduler user
queue mapping in non-impersonation mode
Posted by Mohit Sabharwal <mo...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26515/
-----------------------------------------------------------
(Updated Oct. 13, 2014, 10:39 p.m.)
Review request for hive.
Changes
-------
Inc. feedback:
Improved config doc.
Moved test to hive-unit-hadoop2
Bugs: HIVE-8424
https://issues.apache.org/jira/browse/HIVE-8424
Repository: hive-git
Description
-------
HIVE-8424 : Support fair scheduler user queue mapping in non-impersonation mode
Under non-impersonation mode, all MR jobs run as the hive system user.
The default scheduler queue mapping is one queue per user. This is problematic
for users who use the queues to regulate and track their MR resource usage.
This patch uses the Yarn API to set the fair scheduler queue mapping for
the current user, if impersonation is turned off.
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 018c6035dd557ebb812f4224509bcf17bfdf045a
itests/hive-unit-hadoop2/pom.xml cbdf506f94e2f720543e5932f0af65623fb43dc5
itests/hive-unit-hadoop2/src/test/java/org/apache/hive/jdbc/TestSchedulerQueue.java PRE-CREATION
service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java f021870cf48c55a3d2dea4c825f1f25d0ab008f3
shims/0.20/src/main/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java 52f02c1a7915125fedd7b756e1db4611cf81d966
shims/0.20S/src/main/java/org/apache/hadoop/hive/shims/Hadoop20SShims.java 07946df1b0fc9064925f431fe1f0e2c020b41dfc
shims/0.23/pom.xml ad71fccc2f94f4523cbb3572fecc06e54528d630
shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java 9829c1b54ed2c1cb3068a569ff47d897f6284933
shims/common/src/main/java/org/apache/hadoop/hive/shims/HadoopShims.java d56795f7be1cffa13ea7b3ea2cd50db90730c020
Diff: https://reviews.apache.org/r/26515/diff/
Testing
-------
Added unit test
Thanks,
Mohit Sabharwal
Re: Review Request 26515: HIVE-8424 : Support fair scheduler user
queue mapping in non-impersonation mode
Posted by Mohit Sabharwal <mo...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26515/
-----------------------------------------------------------
(Updated Oct. 10, 2014, 4:28 a.m.)
Review request for hive.
Changes
-------
Inc. Prasad's feedback.
Bugs: HIVE-8424
https://issues.apache.org/jira/browse/HIVE-8424
Repository: hive-git
Description
-------
HIVE-8424 : Support fair scheduler user queue mapping in non-impersonation mode
Under non-impersonation mode, all MR jobs run as the hive system user.
The default scheduler queue mapping is one queue per user. This is problematic
for users who use the queues to regulate and track their MR resource usage.
This patch uses the Yarn API to set the fair scheduler queue mapping for
the current user, if impersonation is turned off.
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 6a1a5f0542ca0723e9e415863ca74c8b43bcc96c
itests/hive-unit/pom.xml 077631a7d809e6d5fd79815fca8882898ee172e9
itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestSchedulerQueue.java PRE-CREATION
pom.xml f5a25e5c86b0801d382abd66597be8d7ac69e426
service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java f021870cf48c55a3d2dea4c825f1f25d0ab008f3
shims/0.20/src/main/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java 52f02c1a7915125fedd7b756e1db4611cf81d966
shims/0.20S/src/main/java/org/apache/hadoop/hive/shims/Hadoop20SShims.java 07946df1b0fc9064925f431fe1f0e2c020b41dfc
shims/0.23/pom.xml ad71fccc2f94f4523cbb3572fecc06e54528d630
shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java 9829c1b54ed2c1cb3068a569ff47d897f6284933
shims/common/src/main/java/org/apache/hadoop/hive/shims/HadoopShims.java d56795f7be1cffa13ea7b3ea2cd50db90730c020
Diff: https://reviews.apache.org/r/26515/diff/
Testing
-------
Added unit test
Thanks,
Mohit Sabharwal
Re: Review Request 26515: HIVE-8424 : Support fair scheduler user
queue mapping in non-impersonation mode
Posted by Prasad Mujumdar <pr...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26515/#review56090
-----------------------------------------------------------
Mostly look fine. Just one comment below.
service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java
<https://reviews.apache.org/r/26515/#comment96443>
If we are calling this by default then the non-23 shims shouldn't thrown an exception. Note that the config property is true by default, that means every HS2 session throw a scary warning message with Hadoop 1.x
IMO the other shim implementations can be no-op.
- Prasad Mujumdar
On Oct. 9, 2014, 11:35 p.m., Mohit Sabharwal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/26515/
> -----------------------------------------------------------
>
> (Updated Oct. 9, 2014, 11:35 p.m.)
>
>
> Review request for hive.
>
>
> Bugs: HIVE-8424
> https://issues.apache.org/jira/browse/HIVE-8424
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> HIVE-8424 : Support fair scheduler user queue mapping in non-impersonation mode
>
> Under non-impersonation mode, all MR jobs run as the hive system user.
> The default scheduler queue mapping is one queue per user. This is problematic
> for users who use the queues to regulate and track their MR resource usage.
>
> This patch uses the Yarn API to set the fair scheduler queue mapping for
> the current user, if impersonation is turned off.
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 6a1a5f0542ca0723e9e415863ca74c8b43bcc96c
> itests/hive-unit/pom.xml 077631a7d809e6d5fd79815fca8882898ee172e9
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestSchedulerQueue.java PRE-CREATION
> pom.xml f5a25e5c86b0801d382abd66597be8d7ac69e426
> service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java f021870cf48c55a3d2dea4c825f1f25d0ab008f3
> shims/0.20/src/main/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java 52f02c1a7915125fedd7b756e1db4611cf81d966
> shims/0.20S/src/main/java/org/apache/hadoop/hive/shims/Hadoop20SShims.java 07946df1b0fc9064925f431fe1f0e2c020b41dfc
> shims/0.23/pom.xml ad71fccc2f94f4523cbb3572fecc06e54528d630
> shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java 9829c1b54ed2c1cb3068a569ff47d897f6284933
> shims/common/src/main/java/org/apache/hadoop/hive/shims/HadoopShims.java d56795f7be1cffa13ea7b3ea2cd50db90730c020
>
> Diff: https://reviews.apache.org/r/26515/diff/
>
>
> Testing
> -------
>
> Added unit test
>
>
> Thanks,
>
> Mohit Sabharwal
>
>