You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Hong Tang (JIRA)" <ji...@apache.org> on 2010/04/20 19:20:50 UTC
[jira] Created: (MAPREDUCE-1715)
TaskSchedulingContext.updateNoOfSlotsOccupiedByUser does not update slots
used by a new user
TaskSchedulingContext.updateNoOfSlotsOccupiedByUser does not update slots used by a new user
--------------------------------------------------------------------------------------------
Key: MAPREDUCE-1715
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1715
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Hong Tang
TaskSchedulingContext.updateNoOfSlotsOccupiedByUser does not update slots used by a new user.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1715)
TaskSchedulingContext.updateNoOfSlotsOccupiedByUser does not update slots
used by a new user
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12858999#action_12858999 ]
Hemanth Yamijala commented on MAPREDUCE-1715:
---------------------------------------------
I originally thought this will completely break user limits with hierarchical queues. But referring to the documentation, I see that user limits are checked only for JobQueues, where the details are properly updated.
Still, this problem remains at least for reporting. When we report aggregate statistics for container queues, we also show the per user usage information, served out of the numSlotsOccupiedByUser data structure. Hence, this should still be fixed.
It is another matter if we want to have user limits checked on container queues as well. I can imagine use cases where this makes sense. But I don't recollect if this was ignored for a specific reason.
> TaskSchedulingContext.updateNoOfSlotsOccupiedByUser does not update slots used by a new user
> --------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1715
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1715
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Hong Tang
>
> TaskSchedulingContext.updateNoOfSlotsOccupiedByUser does not update slots used by a new user.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1715)
TaskSchedulingContext.updateNoOfSlotsOccupiedByUser does not update slots
used by a new user
Posted by "Hong Tang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12858974#action_12858974 ]
Hong Tang commented on MAPREDUCE-1715:
--------------------------------------
Found this bug when working on MAPREDUCE-1695. The code (after MAPREDUCE-1695) reads as follows:
{code}
private void updateNoOfSlotsOccupiedByUser(Map<String, Integer> nou) {
for (Iterator<Map.Entry<String, Integer>> it = nou.entrySet().iterator(); it.hasNext(); ) {
Map.Entry<String, Integer> entry = it.next();
String key = entry.getKey();
Integer currentVal = numSlotsOccupiedByUser.get(key);
if (currentVal != null) {
this.numSlotsOccupiedByUser.put(key, currentVal + entry.getValue());
}
}
}
{code}
What this method does is to add the incremental update of slot usage by users to the global data structure. However, if a user is not existent in the global map numSlotsOccupiedByUser, the update would be ignored. Since this is the only place where the global map is updated and that it starts empty, this method has no effect (namely, currentVal will always be null).
The change is rather simple:
{code}
private void updateNoOfSlotsOccupiedByUser(Map<String, Integer> nou) {
for (Iterator<Map.Entry<String, Integer>> it = nou.entrySet().iterator(); it.hasNext(); ) {
Map.Entry<String, Integer> entry = it.next();
String key = entry.getKey();
Integer currentVal = numSlotsOccupiedByUser.get(key);
if (currentVal != null) {
this.numSlotsOccupiedByUser.put(key, currentVal + entry.getValue());
} else {
this.numSlotsOccupiedByUser.put(key, entry.getValue());
}
}
}
{code}
> TaskSchedulingContext.updateNoOfSlotsOccupiedByUser does not update slots used by a new user
> --------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1715
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1715
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Hong Tang
>
> TaskSchedulingContext.updateNoOfSlotsOccupiedByUser does not update slots used by a new user.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.