You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Sergey Shelukhin <se...@hortonworks.com> on 2017/11/16 03:36:04 UTC
Review Request 63864: HIVE-18072 WM - fix various bugs based on
cluster testing - part 2
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63864/
-----------------------------------------------------------
Review request for hive and Prasanth_J.
Repository: hive-git
Description
-------
see jira
Diffs
-----
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java a02a414a76
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java 9dc521e39e
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/GuaranteedTasksAllocator.java a1775cd6bb
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/UserPoolMapping.java 851245c154
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java 1fe5859490
ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestWorkloadManager.java 5ba6639e0c
Diff: https://reviews.apache.org/r/63864/diff/1/
Testing
-------
Thanks,
Sergey Shelukhin
Re: Review Request 63864: HIVE-18072 WM - fix various bugs based on
cluster testing - part 2
Posted by j....@gmail.com.
> On Nov. 16, 2017, 7:45 p.m., Prasanth_J wrote:
> > llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java
> > Lines 414 (patched)
> > <https://reviews.apache.org/r/63864/diff/1/?file=1894037#file1894037line414>
> >
> > or update log4j2 properties file with WARN default for WM_LOG logger.
>
> Sergey Shelukhin wrote:
> Well, for now I want them to always be logged
You could add the logger at INFO level in properties file and switch to WARN when you don't need it. Changing the source requires recompilation on every change.
- Prasanth_J
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63864/#review191232
-----------------------------------------------------------
On Nov. 16, 2017, 3:36 a.m., Sergey Shelukhin wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63864/
> -----------------------------------------------------------
>
> (Updated Nov. 16, 2017, 3:36 a.m.)
>
>
> Review request for hive and Prasanth_J.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> see jira
>
>
> Diffs
> -----
>
> llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java a02a414a76
> llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java 9dc521e39e
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/GuaranteedTasksAllocator.java a1775cd6bb
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/UserPoolMapping.java 851245c154
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java 1fe5859490
> ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestWorkloadManager.java 5ba6639e0c
>
>
> Diff: https://reviews.apache.org/r/63864/diff/1/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Sergey Shelukhin
>
>
Re: Review Request 63864: HIVE-18072 WM - fix various bugs based on
cluster testing - part 2
Posted by Sergey Shelukhin <se...@hortonworks.com>.
> On Nov. 16, 2017, 7:45 p.m., Prasanth_J wrote:
> > llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java
> > Lines 316 (patched)
> > <https://reviews.apache.org/r/63864/diff/1/?file=1894036#file1894036line316>
> >
> > why would it stack overflow? I don't see a loop here.
it is sometimes called from a callback from a failed call, and in turn invokes a callback again and retries, and since this is a failure on the same thread it does it infinitely and very fast.
The root cause was that entity manager didn't know a node for an attempt even though TaskInfo does... so I just use TaskInfo.
I'm not sure why there's no node in the map, the whole state management in this class seems a bit chaotic.
> On Nov. 16, 2017, 7:45 p.m., Prasanth_J wrote:
> > llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java
> > Lines 414 (patched)
> > <https://reviews.apache.org/r/63864/diff/1/?file=1894037#file1894037line414>
> >
> > or update log4j2 properties file with WARN default for WM_LOG logger.
Well, for now I want them to always be logged
- Sergey
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63864/#review191232
-----------------------------------------------------------
On Nov. 16, 2017, 3:36 a.m., Sergey Shelukhin wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63864/
> -----------------------------------------------------------
>
> (Updated Nov. 16, 2017, 3:36 a.m.)
>
>
> Review request for hive and Prasanth_J.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> see jira
>
>
> Diffs
> -----
>
> llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java a02a414a76
> llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java 9dc521e39e
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/GuaranteedTasksAllocator.java a1775cd6bb
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/UserPoolMapping.java 851245c154
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java 1fe5859490
> ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestWorkloadManager.java 5ba6639e0c
>
>
> Diff: https://reviews.apache.org/r/63864/diff/1/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Sergey Shelukhin
>
>
Re: Review Request 63864: HIVE-18072 WM - fix various bugs based on
cluster testing - part 2
Posted by j....@gmail.com.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63864/#review191232
-----------------------------------------------------------
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java
Lines 316 (patched)
<https://reviews.apache.org/r/63864/#comment268924>
why would it stack overflow? I don't see a loop here.
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java
Lines 168 (patched)
<https://reviews.apache.org/r/63864/#comment268925>
nit: Map
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java
Lines 414 (patched)
<https://reviews.apache.org/r/63864/#comment268928>
or update log4j2 properties file with WARN default for WM_LOG logger.
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java
Line 211 (original), 221 (patched)
<https://reviews.apache.org/r/63864/#comment268930>
Will fix it in HIVE-18025
- Prasanth_J
On Nov. 16, 2017, 3:36 a.m., Sergey Shelukhin wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63864/
> -----------------------------------------------------------
>
> (Updated Nov. 16, 2017, 3:36 a.m.)
>
>
> Review request for hive and Prasanth_J.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> see jira
>
>
> Diffs
> -----
>
> llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java a02a414a76
> llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java 9dc521e39e
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/GuaranteedTasksAllocator.java a1775cd6bb
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/UserPoolMapping.java 851245c154
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java 1fe5859490
> ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestWorkloadManager.java 5ba6639e0c
>
>
> Diff: https://reviews.apache.org/r/63864/diff/1/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Sergey Shelukhin
>
>
Re: Review Request 63864: HIVE-18072 WM - fix various bugs based on
cluster testing - part 2
Posted by j....@gmail.com.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63864/#review191558
-----------------------------------------------------------
Ship it!
Ship It!
- Prasanth_J
On Nov. 17, 2017, 7:55 p.m., Sergey Shelukhin wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/63864/
> -----------------------------------------------------------
>
> (Updated Nov. 17, 2017, 7:55 p.m.)
>
>
> Review request for hive and Prasanth_J.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> see jira
>
>
> Diffs
> -----
>
> llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java a02a414a76
> llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java 9dc521e39e
> llap-tez/src/test/org/apache/hadoop/hive/llap/tezplugins/TestLlapTaskSchedulerService.java 51d2e08393
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/UserPoolMapping.java 851245c154
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java 3990f95334
> ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestWorkloadManager.java 5ba6639e0c
>
>
> Diff: https://reviews.apache.org/r/63864/diff/4/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Sergey Shelukhin
>
>
Re: Review Request 63864: HIVE-18072 WM - fix various bugs based on
cluster testing - part 2
Posted by Sergey Shelukhin <se...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63864/
-----------------------------------------------------------
(Updated Nov. 17, 2017, 7:55 p.m.)
Review request for hive and Prasanth_J.
Repository: hive-git
Description
-------
see jira
Diffs (updated)
-----
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java a02a414a76
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java 9dc521e39e
llap-tez/src/test/org/apache/hadoop/hive/llap/tezplugins/TestLlapTaskSchedulerService.java 51d2e08393
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/UserPoolMapping.java 851245c154
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java 3990f95334
ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestWorkloadManager.java 5ba6639e0c
Diff: https://reviews.apache.org/r/63864/diff/4/
Changes: https://reviews.apache.org/r/63864/diff/3-4/
Testing
-------
Thanks,
Sergey Shelukhin
Re: Review Request 63864: HIVE-18072 WM - fix various bugs based on
cluster testing - part 2
Posted by Sergey Shelukhin <se...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63864/
-----------------------------------------------------------
(Updated Nov. 17, 2017, 2:06 a.m.)
Review request for hive and Prasanth_J.
Repository: hive-git
Description
-------
see jira
Diffs (updated)
-----
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java a02a414a76
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java 9dc521e39e
llap-tez/src/test/org/apache/hadoop/hive/llap/tezplugins/TestLlapTaskSchedulerService.java 51d2e08393
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/UserPoolMapping.java 851245c154
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java 3990f95334
ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestWorkloadManager.java 5ba6639e0c
Diff: https://reviews.apache.org/r/63864/diff/3/
Changes: https://reviews.apache.org/r/63864/diff/2-3/
Testing
-------
Thanks,
Sergey Shelukhin
Re: Review Request 63864: HIVE-18072 WM - fix various bugs based on
cluster testing - part 2
Posted by Sergey Shelukhin <se...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/63864/
-----------------------------------------------------------
(Updated Nov. 16, 2017, 9:30 p.m.)
Review request for hive and Prasanth_J.
Repository: hive-git
Description
-------
see jira
Diffs (updated)
-----
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java a02a414a76
llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java 9dc521e39e
llap-tez/src/test/org/apache/hadoop/hive/llap/tezplugins/TestLlapTaskSchedulerService.java 51d2e08393
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/GuaranteedTasksAllocator.java a1775cd6bb
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/UserPoolMapping.java 851245c154
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java 1fe5859490
ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestWorkloadManager.java 5ba6639e0c
Diff: https://reviews.apache.org/r/63864/diff/2/
Changes: https://reviews.apache.org/r/63864/diff/1-2/
Testing
-------
Thanks,
Sergey Shelukhin