You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Vinod Kumar Vavilapalli (Created) (JIRA)" <ji...@apache.org> on 2011/12/16 02:09:30 UTC
[jira] [Created] (MAPREDUCE-3572) MR AM's dispatcher is blocked by
heartbeats to ResourceManager
MR AM's dispatcher is blocked by heartbeats to ResourceManager
--------------------------------------------------------------
Key: MAPREDUCE-3572
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3572
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Reporter: Vinod Kumar Vavilapalli
Assignee: Vinod Kumar Vavilapalli
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3572) MR AM's dispatcher is blocked by
heartbeats to ResourceManager
Posted by "Vinod Kumar Vavilapalli (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-3572:
-----------------------------------------------
Status: Patch Available (was: Open)
> MR AM's dispatcher is blocked by heartbeats to ResourceManager
> --------------------------------------------------------------
>
> Key: MAPREDUCE-3572
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3572
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am, mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Vinod Kumar Vavilapalli
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3572-20111215.txt, MAPREDUCE-3572-20120104.txt
>
>
> All the heartbeat processing is done in {{RMContainerAllocator}} locking the object. The event processing is also locked on this, causing the dispatcher to be blocked and the rest of the AM getting stalled.
> The event processing should be in a separate thread.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3572) MR AM's dispatcher is blocked
by heartbeats to ResourceManager
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13180385#comment-13180385 ]
Hudson commented on MAPREDUCE-3572:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk #949 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/949/])
MAPREDUCE-3572. Moved AM event dispatcher to a separate thread for performance reasons. Contributed by Vinod K V.
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1227426
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMCommunicator.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java
> MR AM's dispatcher is blocked by heartbeats to ResourceManager
> --------------------------------------------------------------
>
> Key: MAPREDUCE-3572
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3572
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am, mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Vinod Kumar Vavilapalli
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3572-20111215.txt, MAPREDUCE-3572-20120104.txt
>
>
> All the heartbeat processing is done in {{RMContainerAllocator}} locking the object. The event processing is also locked on this, causing the dispatcher to be blocked and the rest of the AM getting stalled.
> The event processing should be in a separate thread.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3572) MR AM's dispatcher is blocked by
heartbeats to ResourceManager
Posted by "Mahadev konar (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar updated MAPREDUCE-3572:
-------------------------------------
Priority: Critical (was: Major)
> MR AM's dispatcher is blocked by heartbeats to ResourceManager
> --------------------------------------------------------------
>
> Key: MAPREDUCE-3572
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3572
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am, mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Vinod Kumar Vavilapalli
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3572-20111215.txt
>
>
> All the heartbeat processing is done in {{RMContainerAllocator}} locking the object. The event processing is also locked on this, causing the dispatcher to be blocked and the rest of the AM getting stalled.
> The event processing should be in a separate thread.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3572) MR AM's dispatcher is blocked
by heartbeats to ResourceManager
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13180102#comment-13180102 ]
Hudson commented on MAPREDUCE-3572:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #1570 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1570/])
MAPREDUCE-3572. Moved AM event dispatcher to a separate thread for performance reasons. Contributed by Vinod K V.
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1227426
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMCommunicator.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java
> MR AM's dispatcher is blocked by heartbeats to ResourceManager
> --------------------------------------------------------------
>
> Key: MAPREDUCE-3572
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3572
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am, mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Vinod Kumar Vavilapalli
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3572-20111215.txt, MAPREDUCE-3572-20120104.txt
>
>
> All the heartbeat processing is done in {{RMContainerAllocator}} locking the object. The event processing is also locked on this, causing the dispatcher to be blocked and the rest of the AM getting stalled.
> The event processing should be in a separate thread.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3572) MR AM's dispatcher is blocked
by heartbeats to ResourceManager
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13179747#comment-13179747 ]
Hadoop QA commented on MAPREDUCE-3572:
--------------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12509433/MAPREDUCE-3572-20120104.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1533//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1533//console
This message is automatically generated.
> MR AM's dispatcher is blocked by heartbeats to ResourceManager
> --------------------------------------------------------------
>
> Key: MAPREDUCE-3572
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3572
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am, mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Vinod Kumar Vavilapalli
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3572-20111215.txt, MAPREDUCE-3572-20120104.txt
>
>
> All the heartbeat processing is done in {{RMContainerAllocator}} locking the object. The event processing is also locked on this, causing the dispatcher to be blocked and the rest of the AM getting stalled.
> The event processing should be in a separate thread.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3572) MR AM's dispatcher is blocked
by heartbeats to ResourceManager
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13180155#comment-13180155 ]
Hudson commented on MAPREDUCE-3572:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #1518 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1518/])
MAPREDUCE-3572. Moved AM event dispatcher to a separate thread for performance reasons. Contributed by Vinod K V.
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1227426
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMCommunicator.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java
> MR AM's dispatcher is blocked by heartbeats to ResourceManager
> --------------------------------------------------------------
>
> Key: MAPREDUCE-3572
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3572
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am, mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Vinod Kumar Vavilapalli
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3572-20111215.txt, MAPREDUCE-3572-20120104.txt
>
>
> All the heartbeat processing is done in {{RMContainerAllocator}} locking the object. The event processing is also locked on this, causing the dispatcher to be blocked and the rest of the AM getting stalled.
> The event processing should be in a separate thread.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3572) MR AM's dispatcher is blocked
by heartbeats to ResourceManager
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13180105#comment-13180105 ]
Hudson commented on MAPREDUCE-3572:
-----------------------------------
Integrated in Hadoop-Common-trunk-Commit #1498 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1498/])
MAPREDUCE-3572. Moved AM event dispatcher to a separate thread for performance reasons. Contributed by Vinod K V.
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1227426
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMCommunicator.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java
> MR AM's dispatcher is blocked by heartbeats to ResourceManager
> --------------------------------------------------------------
>
> Key: MAPREDUCE-3572
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3572
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am, mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Vinod Kumar Vavilapalli
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3572-20111215.txt, MAPREDUCE-3572-20120104.txt
>
>
> All the heartbeat processing is done in {{RMContainerAllocator}} locking the object. The event processing is also locked on this, causing the dispatcher to be blocked and the rest of the AM getting stalled.
> The event processing should be in a separate thread.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3572) MR AM's dispatcher is blocked
by heartbeats to ResourceManager
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13180127#comment-13180127 ]
Hudson commented on MAPREDUCE-3572:
-----------------------------------
Integrated in Hadoop-Mapreduce-0.23-Commit #351 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/351/])
Merge -c 1227426 from trunk to branch-0.23 to fix MAPREDUCE-3572. Moved AM event dispatcher to a separate thread for performance reasons.
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1227428
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMCommunicator.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java
> MR AM's dispatcher is blocked by heartbeats to ResourceManager
> --------------------------------------------------------------
>
> Key: MAPREDUCE-3572
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3572
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am, mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Vinod Kumar Vavilapalli
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3572-20111215.txt, MAPREDUCE-3572-20120104.txt
>
>
> All the heartbeat processing is done in {{RMContainerAllocator}} locking the object. The event processing is also locked on this, causing the dispatcher to be blocked and the rest of the AM getting stalled.
> The event processing should be in a separate thread.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3572) MR AM's dispatcher is blocked
by heartbeats to ResourceManager
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13180328#comment-13180328 ]
Hudson commented on MAPREDUCE-3572:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk #916 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/916/])
MAPREDUCE-3572. Moved AM event dispatcher to a separate thread for performance reasons. Contributed by Vinod K V.
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1227426
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMCommunicator.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java
> MR AM's dispatcher is blocked by heartbeats to ResourceManager
> --------------------------------------------------------------
>
> Key: MAPREDUCE-3572
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3572
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am, mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Vinod Kumar Vavilapalli
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3572-20111215.txt, MAPREDUCE-3572-20120104.txt
>
>
> All the heartbeat processing is done in {{RMContainerAllocator}} locking the object. The event processing is also locked on this, causing the dispatcher to be blocked and the rest of the AM getting stalled.
> The event processing should be in a separate thread.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3572) MR AM's dispatcher is blocked by
heartbeats to ResourceManager
Posted by "Vinod Kumar Vavilapalli (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-3572:
-----------------------------------------------
Component/s: mr-am
Description:
All the heartbeat processing is done in {{RMContainerAllocator}} locking the object. The event processing is also locked on this, causing the dispatcher to be blocked and the rest of the AM getting stalled.
The event processing should be in a separate thread.
Affects Version/s: 0.23.0
Fix Version/s: 0.23.1
> MR AM's dispatcher is blocked by heartbeats to ResourceManager
> --------------------------------------------------------------
>
> Key: MAPREDUCE-3572
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3572
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am, mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Vinod Kumar Vavilapalli
> Fix For: 0.23.1
>
>
> All the heartbeat processing is done in {{RMContainerAllocator}} locking the object. The event processing is also locked on this, causing the dispatcher to be blocked and the rest of the AM getting stalled.
> The event processing should be in a separate thread.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3572) MR AM's dispatcher is blocked by
heartbeats to ResourceManager
Posted by "Vinod Kumar Vavilapalli (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-3572:
-----------------------------------------------
Attachment: MAPREDUCE-3572-20111215.txt
Raw patch moving the event processing in RMContainerAllocator into a separate thread so that the rest of the AM can do its work. The patch may need a bit of cleanup and tests.
> MR AM's dispatcher is blocked by heartbeats to ResourceManager
> --------------------------------------------------------------
>
> Key: MAPREDUCE-3572
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3572
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am, mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Vinod Kumar Vavilapalli
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3572-20111215.txt
>
>
> All the heartbeat processing is done in {{RMContainerAllocator}} locking the object. The event processing is also locked on this, causing the dispatcher to be blocked and the rest of the AM getting stalled.
> The event processing should be in a separate thread.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3572) MR AM's dispatcher is blocked by
heartbeats to ResourceManager
Posted by "Vinod Kumar Vavilapalli (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-3572:
-----------------------------------------------
Attachment: MAPREDUCE-3572-20120104.txt
Patch updated to latest code-base.
No new tests as it is about simply moving the event handling to a separate thread.
> MR AM's dispatcher is blocked by heartbeats to ResourceManager
> --------------------------------------------------------------
>
> Key: MAPREDUCE-3572
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3572
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am, mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Vinod Kumar Vavilapalli
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3572-20111215.txt, MAPREDUCE-3572-20120104.txt
>
>
> All the heartbeat processing is done in {{RMContainerAllocator}} locking the object. The event processing is also locked on this, causing the dispatcher to be blocked and the rest of the AM getting stalled.
> The event processing should be in a separate thread.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3572) MR AM's dispatcher is blocked
by heartbeats to ResourceManager
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13180338#comment-13180338 ]
Hudson commented on MAPREDUCE-3572:
-----------------------------------
Integrated in Hadoop-Hdfs-0.23-Build #129 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/129/])
Merge -c 1227426 from trunk to branch-0.23 to fix MAPREDUCE-3572. Moved AM event dispatcher to a separate thread for performance reasons.
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1227428
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMCommunicator.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java
> MR AM's dispatcher is blocked by heartbeats to ResourceManager
> --------------------------------------------------------------
>
> Key: MAPREDUCE-3572
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3572
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am, mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Vinod Kumar Vavilapalli
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3572-20111215.txt, MAPREDUCE-3572-20120104.txt
>
>
> All the heartbeat processing is done in {{RMContainerAllocator}} locking the object. The event processing is also locked on this, causing the dispatcher to be blocked and the rest of the AM getting stalled.
> The event processing should be in a separate thread.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3572) MR AM's dispatcher is blocked
by heartbeats to ResourceManager
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13180369#comment-13180369 ]
Hudson commented on MAPREDUCE-3572:
-----------------------------------
Integrated in Hadoop-Mapreduce-0.23-Build #151 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/151/])
Merge -c 1227426 from trunk to branch-0.23 to fix MAPREDUCE-3572. Moved AM event dispatcher to a separate thread for performance reasons.
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1227428
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMCommunicator.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java
> MR AM's dispatcher is blocked by heartbeats to ResourceManager
> --------------------------------------------------------------
>
> Key: MAPREDUCE-3572
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3572
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am, mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Vinod Kumar Vavilapalli
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3572-20111215.txt, MAPREDUCE-3572-20120104.txt
>
>
> All the heartbeat processing is done in {{RMContainerAllocator}} locking the object. The event processing is also locked on this, causing the dispatcher to be blocked and the rest of the AM getting stalled.
> The event processing should be in a separate thread.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3572) MR AM's dispatcher is blocked
by heartbeats to ResourceManager
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13180107#comment-13180107 ]
Hudson commented on MAPREDUCE-3572:
-----------------------------------
Integrated in Hadoop-Hdfs-0.23-Commit #329 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/329/])
Merge -c 1227426 from trunk to branch-0.23 to fix MAPREDUCE-3572. Moved AM event dispatcher to a separate thread for performance reasons.
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1227428
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMCommunicator.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java
> MR AM's dispatcher is blocked by heartbeats to ResourceManager
> --------------------------------------------------------------
>
> Key: MAPREDUCE-3572
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3572
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am, mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Vinod Kumar Vavilapalli
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3572-20111215.txt, MAPREDUCE-3572-20120104.txt
>
>
> All the heartbeat processing is done in {{RMContainerAllocator}} locking the object. The event processing is also locked on this, causing the dispatcher to be blocked and the rest of the AM getting stalled.
> The event processing should be in a separate thread.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3572) MR AM's dispatcher is blocked by
heartbeats to ResourceManager
Posted by "Arun C Murthy (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-3572:
-------------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this. Thanks Vinod!
> MR AM's dispatcher is blocked by heartbeats to ResourceManager
> --------------------------------------------------------------
>
> Key: MAPREDUCE-3572
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3572
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am, mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Vinod Kumar Vavilapalli
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3572-20111215.txt, MAPREDUCE-3572-20120104.txt
>
>
> All the heartbeat processing is done in {{RMContainerAllocator}} locking the object. The event processing is also locked on this, causing the dispatcher to be blocked and the rest of the AM getting stalled.
> The event processing should be in a separate thread.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3572) MR AM's dispatcher is blocked
by heartbeats to ResourceManager
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13180112#comment-13180112 ]
Hudson commented on MAPREDUCE-3572:
-----------------------------------
Integrated in Hadoop-Common-0.23-Commit #340 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/340/])
Merge -c 1227426 from trunk to branch-0.23 to fix MAPREDUCE-3572. Moved AM event dispatcher to a separate thread for performance reasons.
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1227428
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMCommunicator.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java
> MR AM's dispatcher is blocked by heartbeats to ResourceManager
> --------------------------------------------------------------
>
> Key: MAPREDUCE-3572
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3572
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am, mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Vinod Kumar Vavilapalli
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3572-20111215.txt, MAPREDUCE-3572-20120104.txt
>
>
> All the heartbeat processing is done in {{RMContainerAllocator}} locking the object. The event processing is also locked on this, causing the dispatcher to be blocked and the rest of the AM getting stalled.
> The event processing should be in a separate thread.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira