You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Allen Wittenauer (JIRA)" <ji...@apache.org> on 2011/05/12 01:11:47 UTC
[jira] [Commented] (MAPREDUCE-2441) regression: maximum limit of -1
+ user-lmit math appears to be off
[ https://issues.apache.org/jira/browse/MAPREDUCE-2441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13032165#comment-13032165 ]
Allen Wittenauer commented on MAPREDUCE-2441:
---------------------------------------------
Actually, it looks like queue spillage/task stealing doesn't work at all, whether it is -1 or not. The problem code appears to be in assignSlotsToJob which appears to have replaced the two-phase system in previous versions with a single phase. This single phase does this check to determine the limit:
{code}
int limit =
Math.min(
Math.max(divideAndCeil(currentCapacity, activeUsers),
divideAndCeil(ulMin*currentCapacity, 100)),
(int)(queueCapacity * ulMinFactor)
);
{code}
In a two queue system where one is -1 and the other is a number, the maximum queue capacity ends up being set to the remainder. Without a second pass, any additional slots from other queues are essentially ignored.
> regression: maximum limit of -1 + user-lmit math appears to be off
> ------------------------------------------------------------------
>
> Key: MAPREDUCE-2441
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2441
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/capacity-sched
> Affects Versions: 0.20.203.0
> Reporter: Allen Wittenauer
> Priority: Blocker
> Attachments: capsched.xml
>
>
> The math around the slot usage when maximum-capacity=-1 appears to be faulty. See comments.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira