You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Zhiyuan Yang <sj...@gmail.com> on 2017/10/11 21:31:42 UTC
Re: Review Request 62706: HIVE-17473 implement workload management
pools
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62706/#review187692
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java
Lines 250 (patched)
<https://reviews.apache.org/r/62706/#comment264747>
Why add up parallelism of parent node with children node? Shouldn't parent's paralleism be a sum of children's?
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java
Lines 275 (patched)
<https://reviews.apache.org/r/62706/#comment264753>
This piece get really complicated right now. I think there is good chance this can be prettier. Are you going you rewrite this (as you mentioned in jira)?
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java
Lines 322 (patched)
<https://reviews.apache.org/r/62706/#comment264739>
unreachable statement?
ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestWorkloadManager.java
Lines 243 (patched)
<https://reviews.apache.org/r/62706/#comment264751>
Why can user use non-leaf queue? The fact that sum of sub-queue can be less than parent queue's resource looks weird. Is this by design?
- Zhiyuan Yang
On Sept. 30, 2017, 12:57 a.m., Sergey Shelukhin wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62706/
> -----------------------------------------------------------
>
> (Updated Sept. 30, 2017, 12:57 a.m.)
>
>
> Review request for hive, Zhiyuan Yang and Prasanth_J.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> see jira
>
>
> Diffs
> -----
>
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPool.java 4f2997b95b
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/UserPoolMapping.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java 3f621271cc
> ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestWorkloadManager.java 7adf895077
> service/src/java/org/apache/hive/service/server/HiveServer2.java 5cb973ca95
>
>
> Diff: https://reviews.apache.org/r/62706/diff/1/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Sergey Shelukhin
>
>
Re: Review Request 62706: HIVE-17473 implement workload management
pools
Posted by Sergey Shelukhin <se...@hortonworks.com>.
> On Oct. 11, 2017, 9:31 p.m., Zhiyuan Yang wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java
> > Lines 250 (patched)
> > <https://reviews.apache.org/r/62706/diff/1/?file=1840588#file1840588line256>
> >
> > Why add up parallelism of parent node with children node? Shouldn't parent's paralleism be a sum of children's?
No, parallelism are tracked separately, see the doc
> On Oct. 11, 2017, 9:31 p.m., Zhiyuan Yang wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java
> > Lines 275 (patched)
> > <https://reviews.apache.org/r/62706/diff/1/?file=1840588#file1840588line284>
> >
> > This piece get really complicated right now. I think there is good chance this can be prettier. Are you going you rewrite this (as you mentioned in jira)?
Yeah, it's going to be rewritten in the next patch
> On Oct. 11, 2017, 9:31 p.m., Zhiyuan Yang wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java
> > Lines 322 (patched)
> > <https://reviews.apache.org/r/62706/diff/1/?file=1840588#file1840588line331>
> >
> > unreachable statement?
it's reachable if internal versions don't match
> On Oct. 11, 2017, 9:31 p.m., Zhiyuan Yang wrote:
> > ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestWorkloadManager.java
> > Lines 243 (patched)
> > <https://reviews.apache.org/r/62706/diff/1/?file=1840589#file1840589line246>
> >
> > Why can user use non-leaf queue? The fact that sum of sub-queue can be less than parent queue's resource looks weird. Is this by design?
See design doc :) Because percentages are percentages, and parallelisms are absolute (percentages don't map well to whole AMs - hard to reason about how many AMs one will get and rounding can produce weird behavior), percentages are from parent total, and parallelisms are independent.
- Sergey
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/62706/#review187692
-----------------------------------------------------------
On Sept. 30, 2017, 12:57 a.m., Sergey Shelukhin wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/62706/
> -----------------------------------------------------------
>
> (Updated Sept. 30, 2017, 12:57 a.m.)
>
>
> Review request for hive, Zhiyuan Yang and Prasanth_J.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> see jira
>
>
> Diffs
> -----
>
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPool.java 4f2997b95b
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/UserPoolMapping.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java 3f621271cc
> ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestWorkloadManager.java 7adf895077
> service/src/java/org/apache/hive/service/server/HiveServer2.java 5cb973ca95
>
>
> Diff: https://reviews.apache.org/r/62706/diff/1/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Sergey Shelukhin
>
>