You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Bikas Saha (JIRA)" <ji...@apache.org> on 2012/09/21 01:01:17 UTC
[jira] [Created] (MAPREDUCE-4671) AM does not tell the RM about
container requests that are no longer needed
Bikas Saha created MAPREDUCE-4671:
-------------------------------------
Summary: AM does not tell the RM about container requests that are no longer needed
Key: MAPREDUCE-4671
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4671
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Bikas Saha
Assignee: Bikas Saha
Say the AM wanted a container at hosts h1, h2, h3. After getting a container at h1 it should tell RM that it no longer needs containers at h2, h3. Otherwise on the RM h2, h3 remain valid allocation locations.
The AM RMContainerAllocator does remove these resource requests internally. When the resource request container count drops to 0 then it drops the resource request from its tables but forgets to send the 0 sized request to the RM.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4671) AM does not tell the RM about
container requests that are no longer needed
Posted by "Bikas Saha (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bikas Saha updated MAPREDUCE-4671:
----------------------------------
Attachment: MAPREDUCE-4671.2.patch
New patch ensures that there wont be duplicates in the ask set.
I have left a question in comments for the committers.
> AM does not tell the RM about container requests that are no longer needed
> --------------------------------------------------------------------------
>
> Key: MAPREDUCE-4671
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4671
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Attachments: MAPREDUCE-4671.1.patch, MAPREDUCE-4671.2.patch
>
>
> Say the AM wanted a container at hosts h1, h2, h3. After getting a container at h1 it should tell RM that it no longer needs containers at h2, h3. Otherwise on the RM h2, h3 remain valid allocation locations.
> The AM RMContainerAllocator does remove these resource requests internally. When the resource request container count drops to 0 then it drops the resource request from its tables but forgets to send the 0 sized request to the RM.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4671) AM does not tell the RM about
container requests that are no longer needed
Posted by "Bikas Saha (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bikas Saha updated MAPREDUCE-4671:
----------------------------------
Status: Patch Available (was: Open)
> AM does not tell the RM about container requests that are no longer needed
> --------------------------------------------------------------------------
>
> Key: MAPREDUCE-4671
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4671
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 2.0.0-alpha, 0.23.3
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Attachments: MAPREDUCE-4671.1.patch, MAPREDUCE-4671.2.patch, MAPREDUCE-4671.3.patch
>
>
> Say the AM wanted a container at hosts h1, h2, h3. After getting a container at h1 it should tell RM that it no longer needs containers at h2, h3. Otherwise on the RM h2, h3 remain valid allocation locations.
> The AM RMContainerAllocator does remove these resource requests internally. When the resource request container count drops to 0 then it drops the resource request from its tables but forgets to send the 0 sized request to the RM.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4671) AM does not tell the RM about
container requests that are no longer needed
Posted by "Bikas Saha (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13471346#comment-13471346 ]
Bikas Saha commented on MAPREDUCE-4671:
---------------------------------------
I think we can assert that the value should never go below 0 because that would mean we are trying to remove more containers than we requested, which would be an app bug.
> AM does not tell the RM about container requests that are no longer needed
> --------------------------------------------------------------------------
>
> Key: MAPREDUCE-4671
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4671
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.23.3, 2.0.0-alpha
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Attachments: MAPREDUCE-4671.1.patch, MAPREDUCE-4671.2.patch
>
>
> Say the AM wanted a container at hosts h1, h2, h3. After getting a container at h1 it should tell RM that it no longer needs containers at h2, h3. Otherwise on the RM h2, h3 remain valid allocation locations.
> The AM RMContainerAllocator does remove these resource requests internally. When the resource request container count drops to 0 then it drops the resource request from its tables but forgets to send the 0 sized request to the RM.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4671) AM does not tell the RM about
container requests that are no longer needed
Posted by "Bikas Saha (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bikas Saha updated MAPREDUCE-4671:
----------------------------------
Attachment: MAPREDUCE-4671.3.patch
> AM does not tell the RM about container requests that are no longer needed
> --------------------------------------------------------------------------
>
> Key: MAPREDUCE-4671
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4671
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.23.3, 2.0.0-alpha
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Attachments: MAPREDUCE-4671.1.patch, MAPREDUCE-4671.2.patch, MAPREDUCE-4671.3.patch
>
>
> Say the AM wanted a container at hosts h1, h2, h3. After getting a container at h1 it should tell RM that it no longer needs containers at h2, h3. Otherwise on the RM h2, h3 remain valid allocation locations.
> The AM RMContainerAllocator does remove these resource requests internally. When the resource request container count drops to 0 then it drops the resource request from its tables but forgets to send the 0 sized request to the RM.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4671) AM does not tell the RM about
container requests that are no longer needed
Posted by "Bo Wang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13460037#comment-13460037 ]
Bo Wang commented on MAPREDUCE-4671:
------------------------------------
This is a problem I'd questioned. Good to see the JIRA.
> AM does not tell the RM about container requests that are no longer needed
> --------------------------------------------------------------------------
>
> Key: MAPREDUCE-4671
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4671
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Bikas Saha
> Assignee: Bikas Saha
>
> Say the AM wanted a container at hosts h1, h2, h3. After getting a container at h1 it should tell RM that it no longer needs containers at h2, h3. Otherwise on the RM h2, h3 remain valid allocation locations.
> The AM RMContainerAllocator does remove these resource requests internally. When the resource request container count drops to 0 then it drops the resource request from its tables but forgets to send the 0 sized request to the RM.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4671) AM does not tell the RM about
container requests that are no longer needed
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13460059#comment-13460059 ]
Hadoop QA commented on MAPREDUCE-4671:
--------------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12545978/MAPREDUCE-4671.1.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 1 new or modified test files.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app.
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2866//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2866//console
This message is automatically generated.
> AM does not tell the RM about container requests that are no longer needed
> --------------------------------------------------------------------------
>
> Key: MAPREDUCE-4671
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4671
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Attachments: MAPREDUCE-4671.1.patch
>
>
> Say the AM wanted a container at hosts h1, h2, h3. After getting a container at h1 it should tell RM that it no longer needs containers at h2, h3. Otherwise on the RM h2, h3 remain valid allocation locations.
> The AM RMContainerAllocator does remove these resource requests internally. When the resource request container count drops to 0 then it drops the resource request from its tables but forgets to send the 0 sized request to the RM.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4671) AM does not tell the RM about
container requests that are no longer needed
Posted by "Siddharth Seth (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Siddharth Seth updated MAPREDUCE-4671:
--------------------------------------
Target Version/s: 2.0.3-alpha
Affects Version/s: 0.23.3
2.0.0-alpha
Status: Open (was: Patch Available)
bq. // ??? shoudl this assert that the new value >= 0 ???
That does need to be checked before the decrement. There's a comment further up in the code which mentions a decrement coming in after this value has gone to 0. See {{containerFailedOnHost}}.
Looking at this and YARN-103 again, we're not really gaining much by keeping entries in the main request table - (remoteRequestsTable). To avoid duplicate requests, does it make sense to change ResourceRequest to ignore containerCount when computing {{equals}} and {{hashCode}}, or defining a local structure with the required implementation.
> AM does not tell the RM about container requests that are no longer needed
> --------------------------------------------------------------------------
>
> Key: MAPREDUCE-4671
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4671
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 2.0.0-alpha, 0.23.3
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Attachments: MAPREDUCE-4671.1.patch, MAPREDUCE-4671.2.patch
>
>
> Say the AM wanted a container at hosts h1, h2, h3. After getting a container at h1 it should tell RM that it no longer needs containers at h2, h3. Otherwise on the RM h2, h3 remain valid allocation locations.
> The AM RMContainerAllocator does remove these resource requests internally. When the resource request container count drops to 0 then it drops the resource request from its tables but forgets to send the 0 sized request to the RM.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4671) AM does not tell the RM about
container requests that are no longer needed
Posted by "Bikas Saha (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bikas Saha updated MAPREDUCE-4671:
----------------------------------
Attachment: MAPREDUCE-4671.4.patch
Attaching rebased patch.
> AM does not tell the RM about container requests that are no longer needed
> --------------------------------------------------------------------------
>
> Key: MAPREDUCE-4671
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4671
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.23.3, 2.0.0-alpha
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Attachments: MAPREDUCE-4671.1.patch, MAPREDUCE-4671.2.patch, MAPREDUCE-4671.3.patch, MAPREDUCE-4671.4.patch
>
>
> Say the AM wanted a container at hosts h1, h2, h3. After getting a container at h1 it should tell RM that it no longer needs containers at h2, h3. Otherwise on the RM h2, h3 remain valid allocation locations.
> The AM RMContainerAllocator does remove these resource requests internally. When the resource request container count drops to 0 then it drops the resource request from its tables but forgets to send the 0 sized request to the RM.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4671) AM does not tell the RM about
container requests that are no longer needed
Posted by "Bikas Saha (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13460115#comment-13460115 ]
Bikas Saha commented on MAPREDUCE-4671:
---------------------------------------
Saying I need x out of y is done by saying I need x because the protocol is built around setting absolute ask values. Since the protocol is lossy wrt relationships between container requests, in general it would be hard (perhaps impossible in some cases) for the RM to figure out what the other requests were that it now needs to cancel unless that relationship information is passed in by the AM.
>From what I see, communication overhead would be similar in both approaches (1 allocate message to send the information). I am not sure how RM side work would be less. Approach 1 seems simpler to me given that RM code does not need to be changed :) Though I might be wrong :P
Perhaps I might be missing something in your comments. A few examples would help.
> AM does not tell the RM about container requests that are no longer needed
> --------------------------------------------------------------------------
>
> Key: MAPREDUCE-4671
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4671
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Attachments: MAPREDUCE-4671.1.patch
>
>
> Say the AM wanted a container at hosts h1, h2, h3. After getting a container at h1 it should tell RM that it no longer needs containers at h2, h3. Otherwise on the RM h2, h3 remain valid allocation locations.
> The AM RMContainerAllocator does remove these resource requests internally. When the resource request container count drops to 0 then it drops the resource request from its tables but forgets to send the 0 sized request to the RM.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4671) AM does not tell the RM about
container requests that are no longer needed
Posted by "Bikas Saha (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bikas Saha updated MAPREDUCE-4671:
----------------------------------
Status: Patch Available (was: Open)
> AM does not tell the RM about container requests that are no longer needed
> --------------------------------------------------------------------------
>
> Key: MAPREDUCE-4671
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4671
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 2.0.0-alpha, 0.23.3
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Attachments: MAPREDUCE-4671.1.patch, MAPREDUCE-4671.2.patch, MAPREDUCE-4671.3.patch, MAPREDUCE-4671.4.patch
>
>
> Say the AM wanted a container at hosts h1, h2, h3. After getting a container at h1 it should tell RM that it no longer needs containers at h2, h3. Otherwise on the RM h2, h3 remain valid allocation locations.
> The AM RMContainerAllocator does remove these resource requests internally. When the resource request container count drops to 0 then it drops the resource request from its tables but forgets to send the 0 sized request to the RM.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4671) AM does not tell the RM about
container requests that are no longer needed
Posted by "Karthik Kambatla (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13460050#comment-13460050 ]
Karthik Kambatla commented on MAPREDUCE-4671:
---------------------------------------------
Thanks Bikas for filing this JIRA.
I think this issue can be addressed in two ways:
# As you suggest, the AM can inform the RM about the container requests that are no longer needed.
# Alternatively, we can augment the ResourceRequest API to allow the AM to convey that it needs (1) 'x' of the 'y' containers that it is requesting, and (2) if the RM can invalidate the remaining ('y' - 'x' ) container requests. This way, we can avoid (1) the communication overhead, and (2) redundant work on the RM side.
What do you think?
> AM does not tell the RM about container requests that are no longer needed
> --------------------------------------------------------------------------
>
> Key: MAPREDUCE-4671
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4671
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Attachments: MAPREDUCE-4671.1.patch
>
>
> Say the AM wanted a container at hosts h1, h2, h3. After getting a container at h1 it should tell RM that it no longer needs containers at h2, h3. Otherwise on the RM h2, h3 remain valid allocation locations.
> The AM RMContainerAllocator does remove these resource requests internally. When the resource request container count drops to 0 then it drops the resource request from its tables but forgets to send the 0 sized request to the RM.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4671) AM does not tell the RM about
container requests that are no longer needed
Posted by "Siddharth Seth (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13472865#comment-13472865 ]
Siddharth Seth commented on MAPREDUCE-4671:
-------------------------------------------
iirc, blacklisting the node changes the request directly in the RMContainerRequestor. When the RMContainerAllocator removes a previous ask (assigned task, etc) - it includes the blacklisted host in this request.
Asserting the value never goes below 0 - will need to change this behaviour. Otherwise a simple check and ignore is sufficient.
> AM does not tell the RM about container requests that are no longer needed
> --------------------------------------------------------------------------
>
> Key: MAPREDUCE-4671
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4671
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.23.3, 2.0.0-alpha
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Attachments: MAPREDUCE-4671.1.patch, MAPREDUCE-4671.2.patch
>
>
> Say the AM wanted a container at hosts h1, h2, h3. After getting a container at h1 it should tell RM that it no longer needs containers at h2, h3. Otherwise on the RM h2, h3 remain valid allocation locations.
> The AM RMContainerAllocator does remove these resource requests internally. When the resource request container count drops to 0 then it drops the resource request from its tables but forgets to send the 0 sized request to the RM.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4671) AM does not tell the RM about
container requests that are no longer needed
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13460125#comment-13460125 ]
Hadoop QA commented on MAPREDUCE-4671:
--------------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12545987/MAPREDUCE-4671.2.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 1 new or modified test files.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app.
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2867//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2867//console
This message is automatically generated.
> AM does not tell the RM about container requests that are no longer needed
> --------------------------------------------------------------------------
>
> Key: MAPREDUCE-4671
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4671
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Attachments: MAPREDUCE-4671.1.patch, MAPREDUCE-4671.2.patch
>
>
> Say the AM wanted a container at hosts h1, h2, h3. After getting a container at h1 it should tell RM that it no longer needs containers at h2, h3. Otherwise on the RM h2, h3 remain valid allocation locations.
> The AM RMContainerAllocator does remove these resource requests internally. When the resource request container count drops to 0 then it drops the resource request from its tables but forgets to send the 0 sized request to the RM.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4671) AM does not tell the RM about
container requests that are no longer needed
Posted by "Bikas Saha (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bikas Saha updated MAPREDUCE-4671:
----------------------------------
Status: Open (was: Patch Available)
> AM does not tell the RM about container requests that are no longer needed
> --------------------------------------------------------------------------
>
> Key: MAPREDUCE-4671
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4671
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 2.0.0-alpha, 0.23.3
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Attachments: MAPREDUCE-4671.1.patch, MAPREDUCE-4671.2.patch, MAPREDUCE-4671.3.patch, MAPREDUCE-4671.4.patch
>
>
> Say the AM wanted a container at hosts h1, h2, h3. After getting a container at h1 it should tell RM that it no longer needs containers at h2, h3. Otherwise on the RM h2, h3 remain valid allocation locations.
> The AM RMContainerAllocator does remove these resource requests internally. When the resource request container count drops to 0 then it drops the resource request from its tables but forgets to send the 0 sized request to the RM.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4671) AM does not tell the RM about
container requests that are no longer needed
Posted by "Bikas Saha (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bikas Saha updated MAPREDUCE-4671:
----------------------------------
Status: Open (was: Patch Available)
> AM does not tell the RM about container requests that are no longer needed
> --------------------------------------------------------------------------
>
> Key: MAPREDUCE-4671
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4671
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Attachments: MAPREDUCE-4671.1.patch, MAPREDUCE-4671.2.patch
>
>
> Say the AM wanted a container at hosts h1, h2, h3. After getting a container at h1 it should tell RM that it no longer needs containers at h2, h3. Otherwise on the RM h2, h3 remain valid allocation locations.
> The AM RMContainerAllocator does remove these resource requests internally. When the resource request container count drops to 0 then it drops the resource request from its tables but forgets to send the 0 sized request to the RM.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4671) AM does not tell the RM about
container requests that are no longer needed
Posted by "Bikas Saha (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bikas Saha updated MAPREDUCE-4671:
----------------------------------
Status: Patch Available (was: Open)
> AM does not tell the RM about container requests that are no longer needed
> --------------------------------------------------------------------------
>
> Key: MAPREDUCE-4671
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4671
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Attachments: MAPREDUCE-4671.1.patch, MAPREDUCE-4671.2.patch
>
>
> Say the AM wanted a container at hosts h1, h2, h3. After getting a container at h1 it should tell RM that it no longer needs containers at h2, h3. Otherwise on the RM h2, h3 remain valid allocation locations.
> The AM RMContainerAllocator does remove these resource requests internally. When the resource request container count drops to 0 then it drops the resource request from its tables but forgets to send the 0 sized request to the RM.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4671) AM does not tell the RM about
container requests that are no longer needed
Posted by "Bikas Saha (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bikas Saha updated MAPREDUCE-4671:
----------------------------------
Status: Patch Available (was: Open)
> AM does not tell the RM about container requests that are no longer needed
> --------------------------------------------------------------------------
>
> Key: MAPREDUCE-4671
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4671
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Attachments: MAPREDUCE-4671.1.patch
>
>
> Say the AM wanted a container at hosts h1, h2, h3. After getting a container at h1 it should tell RM that it no longer needs containers at h2, h3. Otherwise on the RM h2, h3 remain valid allocation locations.
> The AM RMContainerAllocator does remove these resource requests internally. When the resource request container count drops to 0 then it drops the resource request from its tables but forgets to send the 0 sized request to the RM.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4671) AM does not tell the RM about
container requests that are no longer needed
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13473035#comment-13473035 ]
Hadoop QA commented on MAPREDUCE-4671:
--------------------------------------
{color:green}+1 overall{color}. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12548529/MAPREDUCE-4671.3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2923//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2923//console
This message is automatically generated.
> AM does not tell the RM about container requests that are no longer needed
> --------------------------------------------------------------------------
>
> Key: MAPREDUCE-4671
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4671
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.23.3, 2.0.0-alpha
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Attachments: MAPREDUCE-4671.1.patch, MAPREDUCE-4671.2.patch, MAPREDUCE-4671.3.patch
>
>
> Say the AM wanted a container at hosts h1, h2, h3. After getting a container at h1 it should tell RM that it no longer needs containers at h2, h3. Otherwise on the RM h2, h3 remain valid allocation locations.
> The AM RMContainerAllocator does remove these resource requests internally. When the resource request container count drops to 0 then it drops the resource request from its tables but forgets to send the 0 sized request to the RM.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4671) AM does not tell the RM about
container requests that are no longer needed
Posted by "Bikas Saha (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bikas Saha updated MAPREDUCE-4671:
----------------------------------
Attachment: MAPREDUCE-4671.1.patch
Attaching a test that shows the problem along with a quick fix that passes the test.
> AM does not tell the RM about container requests that are no longer needed
> --------------------------------------------------------------------------
>
> Key: MAPREDUCE-4671
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4671
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Attachments: MAPREDUCE-4671.1.patch
>
>
> Say the AM wanted a container at hosts h1, h2, h3. After getting a container at h1 it should tell RM that it no longer needs containers at h2, h3. Otherwise on the RM h2, h3 remain valid allocation locations.
> The AM RMContainerAllocator does remove these resource requests internally. When the resource request container count drops to 0 then it drops the resource request from its tables but forgets to send the 0 sized request to the RM.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4671) AM does not tell the RM about
container requests that are no longer needed
Posted by "Bikas Saha (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13473028#comment-13473028 ]
Bikas Saha commented on MAPREDUCE-4671:
---------------------------------------
New patch.
Checking for resource request falling below zero.
Changing the ask list to have custom comparator to avoid duplication of resource requests.
> AM does not tell the RM about container requests that are no longer needed
> --------------------------------------------------------------------------
>
> Key: MAPREDUCE-4671
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4671
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.23.3, 2.0.0-alpha
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Attachments: MAPREDUCE-4671.1.patch, MAPREDUCE-4671.2.patch, MAPREDUCE-4671.3.patch
>
>
> Say the AM wanted a container at hosts h1, h2, h3. After getting a container at h1 it should tell RM that it no longer needs containers at h2, h3. Otherwise on the RM h2, h3 remain valid allocation locations.
> The AM RMContainerAllocator does remove these resource requests internally. When the resource request container count drops to 0 then it drops the resource request from its tables but forgets to send the 0 sized request to the RM.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4671) AM does not tell the RM about
container requests that are no longer needed
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13486743#comment-13486743 ]
Hadoop QA commented on MAPREDUCE-4671:
--------------------------------------
{color:green}+1 overall{color}. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12551323/MAPREDUCE-4671.4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2973//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2973//console
This message is automatically generated.
> AM does not tell the RM about container requests that are no longer needed
> --------------------------------------------------------------------------
>
> Key: MAPREDUCE-4671
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4671
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.23.3, 2.0.0-alpha
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Attachments: MAPREDUCE-4671.1.patch, MAPREDUCE-4671.2.patch, MAPREDUCE-4671.3.patch, MAPREDUCE-4671.4.patch
>
>
> Say the AM wanted a container at hosts h1, h2, h3. After getting a container at h1 it should tell RM that it no longer needs containers at h2, h3. Otherwise on the RM h2, h3 remain valid allocation locations.
> The AM RMContainerAllocator does remove these resource requests internally. When the resource request container count drops to 0 then it drops the resource request from its tables but forgets to send the 0 sized request to the RM.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4671) AM does not tell the RM about
container requests that are no longer needed
Posted by "Bikas Saha (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484868#comment-13484868 ]
Bikas Saha commented on MAPREDUCE-4671:
---------------------------------------
Ping
> AM does not tell the RM about container requests that are no longer needed
> --------------------------------------------------------------------------
>
> Key: MAPREDUCE-4671
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4671
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.23.3, 2.0.0-alpha
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Attachments: MAPREDUCE-4671.1.patch, MAPREDUCE-4671.2.patch, MAPREDUCE-4671.3.patch
>
>
> Say the AM wanted a container at hosts h1, h2, h3. After getting a container at h1 it should tell RM that it no longer needs containers at h2, h3. Otherwise on the RM h2, h3 remain valid allocation locations.
> The AM RMContainerAllocator does remove these resource requests internally. When the resource request container count drops to 0 then it drops the resource request from its tables but forgets to send the 0 sized request to the RM.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira