You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Karam Singh (JIRA)" <ji...@apache.org> on 2009/07/07 10:28:15 UTC
[jira] Created: (MAPREDUCE-722) More slots are getting reserved for
HiRAM job tasks then required
More slots are getting reserved for HiRAM job tasks then required
-----------------------------------------------------------------
Key: MAPREDUCE-722
URL: https://issues.apache.org/jira/browse/MAPREDUCE-722
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/capacity-sched
Environment: Cluster MR capacity=248/248 Map slot size=1500 mb and reduce slot size=2048 mb. Total number of nodes=124
4 queues each having Capacity=25% User Limit=100%.
Reporter: Karam Singh
Submitted a normal job with map=124=reduces
After submitted High RAM with maps=31=reduces map.memory=1800 reduce.memory=2800
Again 3 job maps=124=reduces
total of 248 slots were reserved for both maps and reduces for High Job which much higher then required.
Is observed in Hadoop 0.20.0
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (MAPREDUCE-722) More slots are getting reserved
for HiRAM job tasks then required
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala resolved MAPREDUCE-722.
----------------------------------------
Resolution: Fixed
Fix Version/s: 0.21.0
Hadoop Flags: [Reviewed]
I just committed this. Thanks, Vinod !
> More slots are getting reserved for HiRAM job tasks then required
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-722
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-722
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/capacity-sched
> Environment: Cluster MR capacity=248/248 Map slot size=1500 mb and reduce slot size=2048 mb. Total number of nodes=124
> 4 queues each having Capacity=25% User Limit=100%.
> Reporter: Karam Singh
> Assignee: Vinod K V
> Fix For: 0.21.0
>
> Attachments: MAPREDUCE-722.1.txt, MAPREDUCE-722.txt
>
>
> Submitted a normal job with map=124=reduces
> After submitted High RAM with maps=31=reduces map.memory=1800 reduce.memory=2800
> Again 3 job maps=124=reduces
> total of 248 slots were reserved for both maps and reduces for High Job which much higher then required.
> Is observed in Hadoop 0.20.0
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-722) More slots are getting reserved for
HiRAM job tasks then required
Posted by "Vinod K V (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod K V updated MAPREDUCE-722:
--------------------------------
Attachment: MAPREDUCE-722.txt
Hemanth found out the reason for this - a faulty conditional in getTaskFromQueue (CapacityTaskScheduler.java +538) :
{code}
if (memory requirement match for this job on this TT) {
Go ahead and give a task
} else {
if (getPendingTasks(j) != 0 || hasSpeculativeTask(j, taskTrackerStatus) ||
!hasSufficientReservedTaskTrackers(j)) {
Reserve this TaskTracker.
}
}
{code}
Even when enough reservations are already made, because all the conditions are OR'ed instead of AND'ed, reservations continue to be made till all nodes in the cluster get reserved for the job.
I am attaching a patch for this. Changing the conditional to be:
{code}
if ((getPendingTasks(j) != 0 && !hasSufficientReservedTaskTrackers(j))
|| hasSpeculativeTask(j, taskTrackerStatus) {
Reserve the taskTracker.
}
{code}
Added a new test case that fails without the code changes and succeeds with. Also fixed two other tests that were buggy and so didn't catch the problems found in this issue.
This patch is still *incomplete* w.r.t speculative tasks. We need more thought regarding this, as TaskTrackers reserved for one speculative task T-1 may not be usable by another task T-2 of the same job.
> More slots are getting reserved for HiRAM job tasks then required
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-722
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-722
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/capacity-sched
> Environment: Cluster MR capacity=248/248 Map slot size=1500 mb and reduce slot size=2048 mb. Total number of nodes=124
> 4 queues each having Capacity=25% User Limit=100%.
> Reporter: Karam Singh
> Assignee: Vinod K V
> Attachments: MAPREDUCE-722.txt
>
>
> Submitted a normal job with map=124=reduces
> After submitted High RAM with maps=31=reduces map.memory=1800 reduce.memory=2800
> Again 3 job maps=124=reduces
> total of 248 slots were reserved for both maps and reduces for High Job which much higher then required.
> Is observed in Hadoop 0.20.0
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-722) More slots are getting reserved
for HiRAM job tasks then required
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12730105#action_12730105 ]
Hudson commented on MAPREDUCE-722:
----------------------------------
Integrated in Hadoop-Mapreduce-trunk #20 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/20/])
> More slots are getting reserved for HiRAM job tasks then required
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-722
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-722
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/capacity-sched
> Environment: Cluster MR capacity=248/248 Map slot size=1500 mb and reduce slot size=2048 mb. Total number of nodes=124
> 4 queues each having Capacity=25% User Limit=100%.
> Reporter: Karam Singh
> Assignee: Vinod K V
> Fix For: 0.21.0
>
> Attachments: MAPREDUCE-722.1.txt, MAPREDUCE-722.txt
>
>
> Submitted a normal job with map=124=reduces
> After submitted High RAM with maps=31=reduces map.memory=1800 reduce.memory=2800
> Again 3 job maps=124=reduces
> total of 248 slots were reserved for both maps and reduces for High Job which much higher then required.
> Is observed in Hadoop 0.20.0
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-722) More slots are getting reserved
for HiRAM job tasks then required
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12728145#action_12728145 ]
Hemanth Yamijala commented on MAPREDUCE-722:
--------------------------------------------
Results of test patch
{noformat}
[exec] -1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 3 new or modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] -1 Eclipse classpath. The patch causes the Eclipse classpath to differ from the contents of the lib directories.
[exec]
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
{noformat}
The eclipse classpath problem can be ignored, as it does not play well with IVY.
The patch only touches capacity scheduler, whose tests pass locally.
> More slots are getting reserved for HiRAM job tasks then required
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-722
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-722
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/capacity-sched
> Environment: Cluster MR capacity=248/248 Map slot size=1500 mb and reduce slot size=2048 mb. Total number of nodes=124
> 4 queues each having Capacity=25% User Limit=100%.
> Reporter: Karam Singh
> Assignee: Vinod K V
> Attachments: MAPREDUCE-722.1.txt, MAPREDUCE-722.txt
>
>
> Submitted a normal job with map=124=reduces
> After submitted High RAM with maps=31=reduces map.memory=1800 reduce.memory=2800
> Again 3 job maps=124=reduces
> total of 248 slots were reserved for both maps and reduces for High Job which much higher then required.
> Is observed in Hadoop 0.20.0
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-722) More slots are getting reserved
for HiRAM job tasks then required
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12728192#action_12728192 ]
Arun C Murthy commented on MAPREDUCE-722:
-----------------------------------------
+1 for the patch and not relying on CapacityScheduler.TaskSchedulingMgr.hasSpeculativeTask which needs to be fixed anyway (MAPREDUCE-725).
> More slots are getting reserved for HiRAM job tasks then required
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-722
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-722
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/capacity-sched
> Environment: Cluster MR capacity=248/248 Map slot size=1500 mb and reduce slot size=2048 mb. Total number of nodes=124
> 4 queues each having Capacity=25% User Limit=100%.
> Reporter: Karam Singh
> Assignee: Vinod K V
> Attachments: MAPREDUCE-722.1.txt, MAPREDUCE-722.txt
>
>
> Submitted a normal job with map=124=reduces
> After submitted High RAM with maps=31=reduces map.memory=1800 reduce.memory=2800
> Again 3 job maps=124=reduces
> total of 248 slots were reserved for both maps and reduces for High Job which much higher then required.
> Is observed in Hadoop 0.20.0
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (MAPREDUCE-722) More slots are getting reserved
for HiRAM job tasks then required
Posted by "Vinod K V (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod K V reassigned MAPREDUCE-722:
-----------------------------------
Assignee: Vinod K V
> More slots are getting reserved for HiRAM job tasks then required
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-722
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-722
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/capacity-sched
> Environment: Cluster MR capacity=248/248 Map slot size=1500 mb and reduce slot size=2048 mb. Total number of nodes=124
> 4 queues each having Capacity=25% User Limit=100%.
> Reporter: Karam Singh
> Assignee: Vinod K V
>
> Submitted a normal job with map=124=reduces
> After submitted High RAM with maps=31=reduces map.memory=1800 reduce.memory=2800
> Again 3 job maps=124=reduces
> total of 248 slots were reserved for both maps and reduces for High Job which much higher then required.
> Is observed in Hadoop 0.20.0
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-722) More slots are getting reserved
for HiRAM job tasks then required
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12728125#action_12728125 ]
Hemanth Yamijala commented on MAPREDUCE-722:
--------------------------------------------
I am not entirely sure this is a bad decision at all. I think we can assume that at the point the high RAM job decides there are speculative tasks to execute, there will be certain good tasks that will not need speculation, that are still running and will come back to the job tracker. At that point, we can certainly run the speculative tasks of high RAM jobs, and there will be no starvation. I certainly don't believe there is going to be guaranteed starvation in any case.
Also, our speculation heuristics at this point are only now improving (with HADOOP-2141) and there is a good chance users of older versions of Hadoop (0.20 and before) will not rely on speculation to work that nicely anyway.
Also, in the extreme case that this condition of starvation is indeed hit, there is the workaround that the user could kill slow running tasks if he thinks the job is taking too long to finish. This is not the greatest workaround available, but will work in the short - medium term.
So, I would definitely recommend we favor cluster utilization as I suppose is done in the last patch.
> More slots are getting reserved for HiRAM job tasks then required
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-722
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-722
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/capacity-sched
> Environment: Cluster MR capacity=248/248 Map slot size=1500 mb and reduce slot size=2048 mb. Total number of nodes=124
> 4 queues each having Capacity=25% User Limit=100%.
> Reporter: Karam Singh
> Assignee: Vinod K V
> Attachments: MAPREDUCE-722.1.txt, MAPREDUCE-722.txt
>
>
> Submitted a normal job with map=124=reduces
> After submitted High RAM with maps=31=reduces map.memory=1800 reduce.memory=2800
> Again 3 job maps=124=reduces
> total of 248 slots were reserved for both maps and reduces for High Job which much higher then required.
> Is observed in Hadoop 0.20.0
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-722) More slots are getting reserved for
HiRAM job tasks then required
Posted by "Vinod K V (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod K V updated MAPREDUCE-722:
--------------------------------
Attachment: MAPREDUCE-722.1.txt
Thought about it a bit more and discussed with Hemanth and Devaraj. This does look like a complicated issue and needs a larger discussion.
For fixing the issue at hand, I am reverting changes w.r.t speculative execution. So, the attached patch fixed cluster drain problem in general for high memory jobs. But if a high memory job enables speculative execution, it might starve. Will track the fix for speculative execution in a follow up JIRA.
> More slots are getting reserved for HiRAM job tasks then required
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-722
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-722
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/capacity-sched
> Environment: Cluster MR capacity=248/248 Map slot size=1500 mb and reduce slot size=2048 mb. Total number of nodes=124
> 4 queues each having Capacity=25% User Limit=100%.
> Reporter: Karam Singh
> Assignee: Vinod K V
> Attachments: MAPREDUCE-722.1.txt, MAPREDUCE-722.txt
>
>
> Submitted a normal job with map=124=reduces
> After submitted High RAM with maps=31=reduces map.memory=1800 reduce.memory=2800
> Again 3 job maps=124=reduces
> total of 248 slots were reserved for both maps and reduces for High Job which much higher then required.
> Is observed in Hadoop 0.20.0
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-722) More slots are getting reserved
for HiRAM job tasks then required
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12728137#action_12728137 ]
Hemanth Yamijala commented on MAPREDUCE-722:
--------------------------------------------
Code changes look good to me. +1.
> More slots are getting reserved for HiRAM job tasks then required
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-722
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-722
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/capacity-sched
> Environment: Cluster MR capacity=248/248 Map slot size=1500 mb and reduce slot size=2048 mb. Total number of nodes=124
> 4 queues each having Capacity=25% User Limit=100%.
> Reporter: Karam Singh
> Assignee: Vinod K V
> Attachments: MAPREDUCE-722.1.txt, MAPREDUCE-722.txt
>
>
> Submitted a normal job with map=124=reduces
> After submitted High RAM with maps=31=reduces map.memory=1800 reduce.memory=2800
> Again 3 job maps=124=reduces
> total of 248 slots were reserved for both maps and reduces for High Job which much higher then required.
> Is observed in Hadoop 0.20.0
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.