[jira] [Commented] (TEZ-2148) Slow container grabbing with Capacity Scheduler in comparision to MapReduce

["Jeff Zhang (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TEZ-2148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14338269#comment-14338269 ] 

Jeff Zhang commented on TEZ-2148:
---------------------------------

Quickly go through the logs, and found that tez use 280 containers while mr only use 201 containers. And tez split it into 4 dags while mr only use 1 job. 


> Slow container grabbing with Capacity Scheduler in comparision to MapReduce
> ---------------------------------------------------------------------------
>
>                 Key: TEZ-2148
>                 URL: https://issues.apache.org/jira/browse/TEZ-2148
>             Project: Apache Tez
>          Issue Type: Task
>    Affects Versions: 0.5.1
>            Reporter: Johannes Zillmann
>         Attachments: applicationLogs.zip, capacity-scheduler.xml, client-mapreduce.log, client-tez.log
>
>
> A customer experienced the following:
> - Setup a CapacityScheduler for user 'company'
> - Same processing job on same data is faster with MapReduce then with Tez with "normal" cluster business. Only if nothing else runs on Hadoop then Tez outperforms MapReduce. (Its hard to give exact data here since we get every information second hand from the customer, but the timings were pretty stable over a dozen of runs. The MapReduce job in about 70 sec and Tez in about 170 sec.)
> So questions is, is there some difference in how Tez is grabbing resources from the capacity scheduler in difference to MapReduce ?
> Looking at the logs it looks like Tez is always very slow in starting the containers where as MapReduce parallelizes very quickly.
> Attached client and application logs for Tez and MapReduce run as well as the scheduler configuration.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)