[jira] [Updated] (TEZ-3805) Analyzer: Add an analyzer to find out scheduling misses in 1:1 edges

["Rajesh Balamohan (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/TEZ-3805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Rajesh Balamohan updated TEZ-3805:
----------------------------------
    Attachment: TEZ-3805.1.patch

{noformat}
java -cp `hadoop classpath`:./*: org.apache.tez.analyzer.plugins.OneOnOneEdgeAnalyzer --saveResults --outputDir /tmp/test/ --fromSimpleHistory --eventFileName /tmp/q95_1_1_reducer_13.txt --dagId dag_1490656001509_6107_2 

sourceVertex,downstreamVertex,srcTaskId,srcContainerHost,destContainerHost
Reducer 12,Reducer 13,0,mc118-10,mc119-10
Reducer 12,Reducer 13,1,mc118-10,mc120-10
Reducer 12,Reducer 13,3,mc108-10,mc119-10
Reducer 12,Reducer 13,5,mc107-10,mc106-10
Reducer 12,Reducer 13,6,mc119-10,mc107-10
Reducer 12,Reducer 13,7,mc120-10,mc119-10
...
...
Reducer 12,Reducer 13,1003,mc120-10,mc107-10
Reducer 12,Reducer 13,1005,mc107-10,mc106-10
Reducer 12,Reducer 13,1006,mc106-10,mc118-10
Reducer 12,Reducer 13,1008,mc120-10,mc107-10
{noformat}

In this case, it turned out that ~800 tasks out of 1009 tasks were assigned in different machines.

> Analyzer: Add an analyzer to find out scheduling misses in 1:1 edges
> --------------------------------------------------------------------
>
>                 Key: TEZ-3805
>                 URL: https://issues.apache.org/jira/browse/TEZ-3805
>             Project: Apache Tez
>          Issue Type: Improvement
>            Reporter: Rajesh Balamohan
>            Assignee: Rajesh Balamohan
>         Attachments: TEZ-3805.1.patch
>
>
> When 1:1 edge is used, it would be helpful to find out whether downstream tasks ran on the same location provided in the hints by the runtime. 
> One of the recent feature in upstream project (hive) used 1:1 edge. Instead of checking the logs, it would be useful to have an analyzer to churn out the details.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)