You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@tez.apache.org by "Rajesh Balamohan (JIRA)" <ji...@apache.org> on 2014/08/29 11:39:53 UTC
[jira] [Moved] (TEZ-1523) Enhance natural order scheduler to
prevent downstream vertex from monopolizing the cluster resources
[ https://issues.apache.org/jira/browse/TEZ-1523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rajesh Balamohan moved HIVE-7910 to TEZ-1523:
---------------------------------------------
Key: TEZ-1523 (was: HIVE-7910)
Project: Apache Tez (was: Hive)
> Enhance natural order scheduler to prevent downstream vertex from monopolizing the cluster resources
> ----------------------------------------------------------------------------------------------------
>
> Key: TEZ-1523
> URL: https://issues.apache.org/jira/browse/TEZ-1523
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Rajesh Balamohan
> Labels: performance
>
> M2 M7
> \ /
> (sg) \ /
> R3 / (b)
> \ /
> (b) \ /
> \ /
> M5
> |
> R6
> Plz refer to the attachment (task runtime SVG). In this case, M5 got scheduled much earlier than R3 (R3 is mentioned as green color in the diagram) and retained lots of containers. R3 got less containers to work with.
> Attaching the output from the status monitor when the job ran; Map_5 has taken up almost all containers, whereas Reducer_3 got fraction of the capacity.
> Map_2: 1/1 Map_5: 0(+373)/1000 Map_7: 1/1 Reducer_3: 0/8000 Reducer_6: 0/1
> Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: 0/8000 Reducer_6: 0/1
> Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: 0(+1)/8000 Reducer_6: 0/1
> ....
> Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: 14(+7)/8000 Reducer_6: 0/1
> Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: 63(+14)/8000 Reducer_6: 0/1
> Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: 159(+22)/8000 Reducer_6: 0/1
> Map_2: 1/1 Map_5: 0(+374)/1000 Map_7: 1/1 Reducer_3: 308(+29)/8000 Reducer_6: 0/1
> ...
> Creating this JIRA as a placeholder for scheduler enhancement. One possibililty could be to
> schedule lesser number of tasks in downstream vertices, based on the information available for the upstream vertex.
--
This message was sent by Atlassian JIRA
(v6.2#6252)