You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@tez.apache.org by "Rajesh Balamohan (JIRA)" <ji...@apache.org> on 2017/04/19 07:31:41 UTC
[jira] [Updated] (TEZ-3688) Add analyzer to provide the set of
machines that participated in the vertex and the number of tasks that ran
on that node
[ https://issues.apache.org/jira/browse/TEZ-3688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rajesh Balamohan updated TEZ-3688:
----------------------------------
Attachment: TEZ-3688.1.patch
Attaching .1 patch for review.
For every vertex, it provides "vertexName", "node", "numOfTasks", "totalTasks", "percentage". This would be useful for quickly checking if there were any allocation skew of tasks on specific nodes for every vertex.
Example console output is given below.. In zeppelin, this can be a separate table.
{noformat}
DEBUG [main] (TezAnalyzerBase.java:204) - --------------------------------------------------------------------------------
DEBUG [main] (TezAnalyzerBase.java:205) - vertexName node numOfTasks totalTasks Percentage
DEBUG [main] (TezAnalyzerBase.java:206) - --------------------------------------------------------------------------------
DEBUG [main] (TezAnalyzerBase.java:211) - Map 8 node_n05 1 1 100
DEBUG [main] (TezAnalyzerBase.java:211) - Map 5 node_n05 1 1 100
DEBUG [main] (TezAnalyzerBase.java:211) - Map 6 node_n06 73 697 10.47
DEBUG [main] (TezAnalyzerBase.java:211) - Map 6 node_n09 71 697 10.19
DEBUG [main] (TezAnalyzerBase.java:211) - Map 6 node_n03 71 697 10.19
DEBUG [main] (TezAnalyzerBase.java:211) - Map 6 node_n04 71 697 10.19
DEBUG [main] (TezAnalyzerBase.java:211) - Map 6 node_n08 71 697 10.19
DEBUG [main] (TezAnalyzerBase.java:211) - Map 6 node_n02 70 697 10.04
DEBUG [main] (TezAnalyzerBase.java:211) - Map 6 node_n01 69 697 9.9
DEBUG [main] (TezAnalyzerBase.java:211) - Map 6 node_n10 69 697 9.9
DEBUG [main] (TezAnalyzerBase.java:211) - Map 6 node_n05 67 697 9.61
DEBUG [main] (TezAnalyzerBase.java:211) - Map 6 node_n07 65 697 9.33
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 7 node_n02 269 760 35.39
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 7 node_n06 61 760 8.03
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 7 node_n03 58 760 7.63
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 7 node_n01 56 760 7.37
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 7 node_n05 55 760 7.24
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 7 node_n08 55 760 7.24
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 7 node_n07 53 760 6.97
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 7 node_n04 53 760 6.97
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 7 node_n10 52 760 6.84
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 7 node_n09 48 760 6.32
DEBUG [main] (TezAnalyzerBase.java:211) - Map 1 node_n07 97 910 10.66
DEBUG [main] (TezAnalyzerBase.java:211) - Map 1 node_n09 94 910 10.33
DEBUG [main] (TezAnalyzerBase.java:211) - Map 1 node_n06 94 910 10.33
DEBUG [main] (TezAnalyzerBase.java:211) - Map 1 node_n02 92 910 10.11
DEBUG [main] (TezAnalyzerBase.java:211) - Map 1 node_n05 91 910 10
DEBUG [main] (TezAnalyzerBase.java:211) - Map 1 node_n04 91 910 10
DEBUG [main] (TezAnalyzerBase.java:211) - Map 1 node_n10 91 910 10
DEBUG [main] (TezAnalyzerBase.java:211) - Map 1 node_n03 89 910 9.78
DEBUG [main] (TezAnalyzerBase.java:211) - Map 1 node_n01 88 910 9.67
DEBUG [main] (TezAnalyzerBase.java:211) - Map 1 node_n08 83 910 9.12
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 2 node_n02 565 1009 56
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 2 node_n09 54 1009 5.35
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 2 node_n04 54 1009 5.35
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 2 node_n08 51 1009 5.05
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 2 node_n05 50 1009 4.96
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 2 node_n07 49 1009 4.86
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 2 node_n03 49 1009 4.86
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 2 node_n10 48 1009 4.76
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 2 node_n01 46 1009 4.56
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 2 node_n06 43 1009 4.26
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 3 node_n07 111 1009 11
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 3 node_n06 109 1009 10.8
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 3 node_n08 108 1009 10.7
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 3 node_n10 107 1009 10.6
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 3 node_n02 106 1009 10.51
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 3 node_n05 102 1009 10.11
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 3 node_n04 96 1009 9.51
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 3 node_n03 93 1009 9.22
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 3 node_n01 89 1009 8.82
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 3 node_n09 88 1009 8.72
DEBUG [main] (TezAnalyzerBase.java:211) - Reducer 4 node_n03 1 1 100
DEBUG [main] (TezAnalyzerBase.java:213) - --------------------------------------------------------------------------------
{noformat}
> Add analyzer to provide the set of machines that participated in the vertex and the number of tasks that ran on that node
> -------------------------------------------------------------------------------------------------------------------------
>
> Key: TEZ-3688
> URL: https://issues.apache.org/jira/browse/TEZ-3688
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Nita Dembla
> Assignee: Rajesh Balamohan
> Attachments: TEZ-3688.1.patch
>
>
> Need an analyzer to parse DAG info and outputs the how many tasks of a vertex ran on a node. This would help in isolating task scheduling issues.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)