You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Vasco (JIRA)" <ji...@apache.org> on 2012/08/30 23:30:07 UTC
[jira] [Created] (MAPREDUCE-4613) Scheduling of reduce tasks
results in starvation
Vasco created MAPREDUCE-4613:
--------------------------------
Summary: Scheduling of reduce tasks results in starvation
Key: MAPREDUCE-4613
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4613
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: scheduler
Affects Versions: 0.23.1
Environment: 16 machine cluster
Reporter: Vasco
If a job has more reduce tasks than there are containers available, then the reduce tasks can occupy all containers causing starvation.
I understand that the correct behaviour when all containers are taken by reducers while mappers are still pending, is for the running reducers to be "pre-empted". However, pre-emption does not occur.
A work-around is to set the number of reducers < available containers.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4613) Scheduling of reduce tasks
results in starvation
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13446078#comment-13446078 ]
Arun C Murthy commented on MAPREDUCE-4613:
------------------------------------------
Vasco - the fix will show up via 2.1.0, which is very soon. Thanks.
> Scheduling of reduce tasks results in starvation
> ------------------------------------------------
>
> Key: MAPREDUCE-4613
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4613
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: scheduler
> Affects Versions: 0.23.1, 2.0.1-alpha
> Environment: 16 (duo core) machine cluster ==> 32 containers
> namenode and resourcemanager running on separate 17th machine
> Reporter: Vasco
> Attachments: scheduling.png
>
>
> If a job has more reduce tasks than there are containers available, then the reduce tasks can occupy all containers causing starvation. The attached graph illustrates the behaviour. Scheduler used is fifo.
> I understand that the correct behaviour when all containers are taken by reducers while mappers are still pending, is for the running reducers to be "pre-empted". However, pre-emption does not occur.
> A work-around is to set the number of reducers < available containers.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4613) Scheduling of reduce tasks
results in starvation
Posted by "Vasco (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13446085#comment-13446085 ]
Vasco commented on MAPREDUCE-4613:
----------------------------------
I tried in 2.1.0 (misread 4299 to be resolved in 2.0.1). I can confirm starvation issue is fixed there.
With regard to the task scheduling in 2.1.0. If the amount of reducer input is more or less uniform, then what's the use of scheduling some reduce jobs very early in the job? It don't think it decreases total job completion time, since the last reduce job can only be scheduled after all mappers have been scheduled. If there are containers in abundance, then I understand the use of scheduling reducers as early as possible. But, if containers are scares it sound more reasonable to me to give all available resources to the mappers.
> Scheduling of reduce tasks results in starvation
> ------------------------------------------------
>
> Key: MAPREDUCE-4613
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4613
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: scheduler
> Affects Versions: 0.23.1, 2.0.1-alpha
> Environment: 16 (duo core) machine cluster ==> 32 containers
> namenode and resourcemanager running on separate 17th machine
> Reporter: Vasco
> Attachments: scheduling.png
>
>
> If a job has more reduce tasks than there are containers available, then the reduce tasks can occupy all containers causing starvation. The attached graph illustrates the behaviour. Scheduler used is fifo.
> I understand that the correct behaviour when all containers are taken by reducers while mappers are still pending, is for the running reducers to be "pre-empted". However, pre-emption does not occur.
> A work-around is to set the number of reducers < available containers.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (MAPREDUCE-4613) Scheduling of reduce tasks
results in starvation
Posted by "Vasco (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vasco reopened MAPREDUCE-4613:
------------------------------
Starvation still occurring in 2.0.1
> Scheduling of reduce tasks results in starvation
> ------------------------------------------------
>
> Key: MAPREDUCE-4613
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4613
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: scheduler
> Affects Versions: 0.23.1
> Environment: 16 machine cluster
> Reporter: Vasco
>
> If a job has more reduce tasks than there are containers available, then the reduce tasks can occupy all containers causing starvation.
> I understand that the correct behaviour when all containers are taken by reducers while mappers are still pending, is for the running reducers to be "pre-empted". However, pre-emption does not occur.
> A work-around is to set the number of reducers < available containers.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4613) Scheduling of reduce tasks
results in starvation
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13445384#comment-13445384 ]
Arun C Murthy commented on MAPREDUCE-4613:
------------------------------------------
What scheduler are you using? If you are using the default FifoScheduler, you'll run into MAPREDUCE-4299 which isn't committed to hadoop-0.23.1.
> Scheduling of reduce tasks results in starvation
> ------------------------------------------------
>
> Key: MAPREDUCE-4613
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4613
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: scheduler
> Affects Versions: 0.23.1
> Environment: 16 machine cluster
> Reporter: Vasco
>
> If a job has more reduce tasks than there are containers available, then the reduce tasks can occupy all containers causing starvation.
> I understand that the correct behaviour when all containers are taken by reducers while mappers are still pending, is for the running reducers to be "pre-empted". However, pre-emption does not occur.
> A work-around is to set the number of reducers < available containers.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4613) Scheduling of reduce tasks
results in starvation
Posted by "Vasco (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vasco updated MAPREDUCE-4613:
-----------------------------
Affects Version/s: 2.0.1-alpha
> Scheduling of reduce tasks results in starvation
> ------------------------------------------------
>
> Key: MAPREDUCE-4613
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4613
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: scheduler
> Affects Versions: 0.23.1, 2.0.1-alpha
> Environment: 16 machine cluster
> Reporter: Vasco
> Attachments: scheduling.png
>
>
> If a job has more reduce tasks than there are containers available, then the reduce tasks can occupy all containers causing starvation.
> I understand that the correct behaviour when all containers are taken by reducers while mappers are still pending, is for the running reducers to be "pre-empted". However, pre-emption does not occur.
> A work-around is to set the number of reducers < available containers.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4613) Scheduling of reduce tasks
results in starvation
Posted by "Vasco (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vasco updated MAPREDUCE-4613:
-----------------------------
Attachment: scheduling.png
This graph shows the behaviour I am experiencing
> Scheduling of reduce tasks results in starvation
> ------------------------------------------------
>
> Key: MAPREDUCE-4613
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4613
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: scheduler
> Affects Versions: 0.23.1
> Environment: 16 machine cluster
> Reporter: Vasco
> Attachments: scheduling.png
>
>
> If a job has more reduce tasks than there are containers available, then the reduce tasks can occupy all containers causing starvation.
> I understand that the correct behaviour when all containers are taken by reducers while mappers are still pending, is for the running reducers to be "pre-empted". However, pre-emption does not occur.
> A work-around is to set the number of reducers < available containers.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-4613) Scheduling of reduce tasks
results in starvation
Posted by "Vasco (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vasco resolved MAPREDUCE-4613.
------------------------------
Resolution: Duplicate
MAPREDUCE-4299
> Scheduling of reduce tasks results in starvation
> ------------------------------------------------
>
> Key: MAPREDUCE-4613
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4613
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: scheduler
> Affects Versions: 0.23.1
> Environment: 16 machine cluster
> Reporter: Vasco
>
> If a job has more reduce tasks than there are containers available, then the reduce tasks can occupy all containers causing starvation.
> I understand that the correct behaviour when all containers are taken by reducers while mappers are still pending, is for the running reducers to be "pre-empted". However, pre-emption does not occur.
> A work-around is to set the number of reducers < available containers.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4613) Scheduling of reduce tasks
results in starvation
Posted by "Vasco (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vasco updated MAPREDUCE-4613:
-----------------------------
Description:
If a job has more reduce tasks than there are containers available, then the reduce tasks can occupy all containers causing starvation. The attached graph illustrates the behaviour. Scheduler used is fifo.
I understand that the correct behaviour when all containers are taken by reducers while mappers are still pending, is for the running reducers to be "pre-empted". However, pre-emption does not occur.
A work-around is to set the number of reducers < available containers.
was:
If a job has more reduce tasks than there are containers available, then the reduce tasks can occupy all containers causing starvation.
I understand that the correct behaviour when all containers are taken by reducers while mappers are still pending, is for the running reducers to be "pre-empted". However, pre-emption does not occur.
A work-around is to set the number of reducers < available containers.
> Scheduling of reduce tasks results in starvation
> ------------------------------------------------
>
> Key: MAPREDUCE-4613
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4613
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: scheduler
> Affects Versions: 0.23.1, 2.0.1-alpha
> Environment: 16 machine cluster
> Reporter: Vasco
> Attachments: scheduling.png
>
>
> If a job has more reduce tasks than there are containers available, then the reduce tasks can occupy all containers causing starvation. The attached graph illustrates the behaviour. Scheduler used is fifo.
> I understand that the correct behaviour when all containers are taken by reducers while mappers are still pending, is for the running reducers to be "pre-empted". However, pre-emption does not occur.
> A work-around is to set the number of reducers < available containers.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4613) Scheduling of reduce tasks
results in starvation
Posted by "Vasco (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vasco updated MAPREDUCE-4613:
-----------------------------
Environment:
16 (duo core) machine cluster ==> 32 containers
namenode and resourcemanager running on separate 17th machine
was:16 machine cluster
> Scheduling of reduce tasks results in starvation
> ------------------------------------------------
>
> Key: MAPREDUCE-4613
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4613
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: scheduler
> Affects Versions: 0.23.1, 2.0.1-alpha
> Environment: 16 (duo core) machine cluster ==> 32 containers
> namenode and resourcemanager running on separate 17th machine
> Reporter: Vasco
> Attachments: scheduling.png
>
>
> If a job has more reduce tasks than there are containers available, then the reduce tasks can occupy all containers causing starvation. The attached graph illustrates the behaviour. Scheduler used is fifo.
> I understand that the correct behaviour when all containers are taken by reducers while mappers are still pending, is for the running reducers to be "pre-empted". However, pre-emption does not occur.
> A work-around is to set the number of reducers < available containers.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-4613) Scheduling of reduce tasks
results in starvation
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy resolved MAPREDUCE-4613.
--------------------------------------
Resolution: Duplicate
Dup of MAPREDUCE-4299. Thanks Vasco.
> Scheduling of reduce tasks results in starvation
> ------------------------------------------------
>
> Key: MAPREDUCE-4613
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4613
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: scheduler
> Affects Versions: 0.23.1, 2.0.1-alpha
> Environment: 16 (duo core) machine cluster ==> 32 containers
> namenode and resourcemanager running on separate 17th machine
> Reporter: Vasco
> Attachments: scheduling.png
>
>
> If a job has more reduce tasks than there are containers available, then the reduce tasks can occupy all containers causing starvation. The attached graph illustrates the behaviour. Scheduler used is fifo.
> I understand that the correct behaviour when all containers are taken by reducers while mappers are still pending, is for the running reducers to be "pre-empted". However, pre-emption does not occur.
> A work-around is to set the number of reducers < available containers.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4613) Scheduling of reduce tasks
results in starvation
Posted by "Vasco (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13445436#comment-13445436 ]
Vasco commented on MAPREDUCE-4613:
----------------------------------
That is exactly the issue, this bug can be marked duplicate
> Scheduling of reduce tasks results in starvation
> ------------------------------------------------
>
> Key: MAPREDUCE-4613
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4613
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: scheduler
> Affects Versions: 0.23.1
> Environment: 16 machine cluster
> Reporter: Vasco
>
> If a job has more reduce tasks than there are containers available, then the reduce tasks can occupy all containers causing starvation.
> I understand that the correct behaviour when all containers are taken by reducers while mappers are still pending, is for the running reducers to be "pre-empted". However, pre-emption does not occur.
> A work-around is to set the number of reducers < available containers.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira