You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Tom White (JIRA)" <ji...@apache.org> on 2012/05/31 22:42:23 UTC
[jira] [Created] (MAPREDUCE-4299) Terasort hangs with MR2
FifoScheduler
Tom White created MAPREDUCE-4299:
------------------------------------
Summary: Terasort hangs with MR2 FifoScheduler
Key: MAPREDUCE-4299
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4299
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 2.0.0-alpha
Reporter: Tom White
What happens is that the number of reducers ramp up until they occupy all of the job's containers, at which point the maps no longer make any progress and the job hangs.
When the same job is run with the CapacityScheduler it succeeds, so this looks like a FifoScheduler bug.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4299) Terasort hangs with MR2
FifoScheduler
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13414406#comment-13414406 ]
Hudson commented on MAPREDUCE-4299:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk #1136 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1136/])
MAPREDUCE-4299. Terasort hangs with MR2 FifoScheduler (Tom White via bobby) (Revision 1361397)
Result = SUCCESS
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1361397
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestFifoScheduler.java
> Terasort hangs with MR2 FifoScheduler
> -------------------------------------
>
> Key: MAPREDUCE-4299
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4299
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 2.0.0-alpha
> Reporter: Tom White
> Assignee: Tom White
> Fix For: 0.23.3, 2.0.1-alpha, 3.0.0
>
> Attachments: MAPREDUCE-4299.patch, MAPREDUCE-4299.patch, MAPREDUCE-4299.patch
>
>
> What happens is that the number of reducers ramp up until they occupy all of the job's containers, at which point the maps no longer make any progress and the job hangs.
> When the same job is run with the CapacityScheduler it succeeds, so this looks like a FifoScheduler bug.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4299) Terasort hangs with MR2
FifoScheduler
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13411992#comment-13411992 ]
Hadoop QA commented on MAPREDUCE-4299:
--------------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12536108/MAPREDUCE-4299.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 1 new or modified test files.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2574//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2574//console
This message is automatically generated.
> Terasort hangs with MR2 FifoScheduler
> -------------------------------------
>
> Key: MAPREDUCE-4299
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4299
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 2.0.0-alpha
> Reporter: Tom White
> Assignee: Tom White
> Attachments: MAPREDUCE-4299.patch, MAPREDUCE-4299.patch, MAPREDUCE-4299.patch
>
>
> What happens is that the number of reducers ramp up until they occupy all of the job's containers, at which point the maps no longer make any progress and the job hangs.
> When the same job is run with the CapacityScheduler it succeeds, so this looks like a FifoScheduler bug.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (MAPREDUCE-4299) Terasort hangs with MR2
FifoScheduler
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy closed MAPREDUCE-4299.
------------------------------------
> Terasort hangs with MR2 FifoScheduler
> -------------------------------------
>
> Key: MAPREDUCE-4299
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4299
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 2.0.0-alpha
> Reporter: Tom White
> Assignee: Tom White
> Fix For: 0.23.3, 2.0.2-alpha
>
> Attachments: MAPREDUCE-4299.patch, MAPREDUCE-4299.patch, MAPREDUCE-4299.patch
>
>
> What happens is that the number of reducers ramp up until they occupy all of the job's containers, at which point the maps no longer make any progress and the job hangs.
> When the same job is run with the CapacityScheduler it succeeds, so this looks like a FifoScheduler bug.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4299) Terasort hangs with MR2
FifoScheduler
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13414056#comment-13414056 ]
Hudson commented on MAPREDUCE-4299:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #2531 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2531/])
MAPREDUCE-4299. Terasort hangs with MR2 FifoScheduler (Tom White via bobby) (Revision 1361397)
Result = SUCCESS
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1361397
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestFifoScheduler.java
> Terasort hangs with MR2 FifoScheduler
> -------------------------------------
>
> Key: MAPREDUCE-4299
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4299
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 2.0.0-alpha
> Reporter: Tom White
> Assignee: Tom White
> Fix For: 0.23.3, 2.0.1-alpha, 3.0.0
>
> Attachments: MAPREDUCE-4299.patch, MAPREDUCE-4299.patch, MAPREDUCE-4299.patch
>
>
> What happens is that the number of reducers ramp up until they occupy all of the job's containers, at which point the maps no longer make any progress and the job hangs.
> When the same job is run with the CapacityScheduler it succeeds, so this looks like a FifoScheduler bug.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4299) Terasort hangs with MR2
FifoScheduler
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13414352#comment-13414352 ]
Hudson commented on MAPREDUCE-4299:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk #1103 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1103/])
MAPREDUCE-4299. Terasort hangs with MR2 FifoScheduler (Tom White via bobby) (Revision 1361397)
Result = FAILURE
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1361397
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestFifoScheduler.java
> Terasort hangs with MR2 FifoScheduler
> -------------------------------------
>
> Key: MAPREDUCE-4299
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4299
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 2.0.0-alpha
> Reporter: Tom White
> Assignee: Tom White
> Fix For: 0.23.3, 2.0.1-alpha, 3.0.0
>
> Attachments: MAPREDUCE-4299.patch, MAPREDUCE-4299.patch, MAPREDUCE-4299.patch
>
>
> What happens is that the number of reducers ramp up until they occupy all of the job's containers, at which point the maps no longer make any progress and the job hangs.
> When the same job is run with the CapacityScheduler it succeeds, so this looks like a FifoScheduler bug.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4299) Terasort hangs with MR2
FifoScheduler
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13411629#comment-13411629 ]
Hadoop QA commented on MAPREDUCE-4299:
--------------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12536039/MAPREDUCE-4299.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 1 new or modified test files.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2572//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2572//console
This message is automatically generated.
> Terasort hangs with MR2 FifoScheduler
> -------------------------------------
>
> Key: MAPREDUCE-4299
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4299
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 2.0.0-alpha
> Reporter: Tom White
> Assignee: Tom White
> Attachments: MAPREDUCE-4299.patch, MAPREDUCE-4299.patch
>
>
> What happens is that the number of reducers ramp up until they occupy all of the job's containers, at which point the maps no longer make any progress and the job hangs.
> When the same job is run with the CapacityScheduler it succeeds, so this looks like a FifoScheduler bug.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4299) Terasort hangs with MR2
FifoScheduler
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13414389#comment-13414389 ]
Hudson commented on MAPREDUCE-4299:
-----------------------------------
Integrated in Hadoop-Hdfs-0.23-Build #313 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/313/])
svn merge -c 1361397 FIXES: MAPREDUCE-4299. Terasort hangs with MR2 FifoScheduler (Tom White via bobby) (Revision 1361399)
Result = SUCCESS
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1361399
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestFifoScheduler.java
> Terasort hangs with MR2 FifoScheduler
> -------------------------------------
>
> Key: MAPREDUCE-4299
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4299
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 2.0.0-alpha
> Reporter: Tom White
> Assignee: Tom White
> Fix For: 0.23.3, 2.0.1-alpha, 3.0.0
>
> Attachments: MAPREDUCE-4299.patch, MAPREDUCE-4299.patch, MAPREDUCE-4299.patch
>
>
> What happens is that the number of reducers ramp up until they occupy all of the job's containers, at which point the maps no longer make any progress and the job hangs.
> When the same job is run with the CapacityScheduler it succeeds, so this looks like a FifoScheduler bug.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4299) Terasort hangs with MR2
FifoScheduler
Posted by "Karthik Kambatla (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13411754#comment-13411754 ]
Karthik Kambatla commented on MAPREDUCE-4299:
---------------------------------------------
Tom,
Thanks for the patch.
I had a couple of questions though, please correct me if I understand it wrong.
- Just realized that the application headroom is being set after the for loop; shouldn't we update it within the for loop itself so that subsequent iterations (applications) would see the updated resources.
- If number of applications makes a difference, the test should probably have more applications before we trigger node update.
> Terasort hangs with MR2 FifoScheduler
> -------------------------------------
>
> Key: MAPREDUCE-4299
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4299
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 2.0.0-alpha
> Reporter: Tom White
> Assignee: Tom White
> Attachments: MAPREDUCE-4299.patch, MAPREDUCE-4299.patch
>
>
> What happens is that the number of reducers ramp up until they occupy all of the job's containers, at which point the maps no longer make any progress and the job hangs.
> When the same job is run with the CapacityScheduler it succeeds, so this looks like a FifoScheduler bug.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4299) Terasort hangs with MR2
FifoScheduler
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated MAPREDUCE-4299:
---------------------------------
Attachment: MAPREDUCE-4299.patch
The problem is that FifoScheduler always sets the application headroom to be the entire set of cluster resources, without taking into account any containers that have been assigned. In some cases, like the terasort case mentioned in the JIRA, this leads to the reducer tasks using all the cluster resources before the map tasks have finished, resulting in deadlock.
Attached is a fix with a unit test.
> Terasort hangs with MR2 FifoScheduler
> -------------------------------------
>
> Key: MAPREDUCE-4299
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4299
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 2.0.0-alpha
> Reporter: Tom White
> Attachments: MAPREDUCE-4299.patch
>
>
> What happens is that the number of reducers ramp up until they occupy all of the job's containers, at which point the maps no longer make any progress and the job hangs.
> When the same job is run with the CapacityScheduler it succeeds, so this looks like a FifoScheduler bug.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4299) Terasort hangs with MR2
FifoScheduler
Posted by "Karthik Kambatla (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13412000#comment-13412000 ]
Karthik Kambatla commented on MAPREDUCE-4299:
---------------------------------------------
+1
> Terasort hangs with MR2 FifoScheduler
> -------------------------------------
>
> Key: MAPREDUCE-4299
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4299
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 2.0.0-alpha
> Reporter: Tom White
> Assignee: Tom White
> Attachments: MAPREDUCE-4299.patch, MAPREDUCE-4299.patch, MAPREDUCE-4299.patch
>
>
> What happens is that the number of reducers ramp up until they occupy all of the job's containers, at which point the maps no longer make any progress and the job hangs.
> When the same job is run with the CapacityScheduler it succeeds, so this looks like a FifoScheduler bug.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4299) Terasort hangs with MR2
FifoScheduler
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13414117#comment-13414117 ]
Hudson commented on MAPREDUCE-4299:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #2485 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2485/])
MAPREDUCE-4299. Terasort hangs with MR2 FifoScheduler (Tom White via bobby) (Revision 1361397)
Result = FAILURE
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1361397
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestFifoScheduler.java
> Terasort hangs with MR2 FifoScheduler
> -------------------------------------
>
> Key: MAPREDUCE-4299
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4299
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 2.0.0-alpha
> Reporter: Tom White
> Assignee: Tom White
> Fix For: 0.23.3, 2.0.1-alpha, 3.0.0
>
> Attachments: MAPREDUCE-4299.patch, MAPREDUCE-4299.patch, MAPREDUCE-4299.patch
>
>
> What happens is that the number of reducers ramp up until they occupy all of the job's containers, at which point the maps no longer make any progress and the job hangs.
> When the same job is run with the CapacityScheduler it succeeds, so this looks like a FifoScheduler bug.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4299) Terasort hangs with MR2
FifoScheduler
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13411590#comment-13411590 ]
Hadoop QA commented on MAPREDUCE-4299:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12535965/MAPREDUCE-4299.patch
against trunk revision .
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2571//console
This message is automatically generated.
> Terasort hangs with MR2 FifoScheduler
> -------------------------------------
>
> Key: MAPREDUCE-4299
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4299
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 2.0.0-alpha
> Reporter: Tom White
> Assignee: Tom White
> Attachments: MAPREDUCE-4299.patch
>
>
> What happens is that the number of reducers ramp up until they occupy all of the job's containers, at which point the maps no longer make any progress and the job hangs.
> When the same job is run with the CapacityScheduler it succeeds, so this looks like a FifoScheduler bug.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4299) Terasort hangs with MR2
FifoScheduler
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated MAPREDUCE-4299:
---------------------------------
Attachment: MAPREDUCE-4299.patch
The patch was out of date with trunk - here's an updated version.
> Terasort hangs with MR2 FifoScheduler
> -------------------------------------
>
> Key: MAPREDUCE-4299
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4299
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 2.0.0-alpha
> Reporter: Tom White
> Assignee: Tom White
> Attachments: MAPREDUCE-4299.patch, MAPREDUCE-4299.patch
>
>
> What happens is that the number of reducers ramp up until they occupy all of the job's containers, at which point the maps no longer make any progress and the job hangs.
> When the same job is run with the CapacityScheduler it succeeds, so this looks like a FifoScheduler bug.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4299) Terasort hangs with MR2
FifoScheduler
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated MAPREDUCE-4299:
-------------------------------------------
Resolution: Fixed
Fix Version/s: 3.0.0
2.0.1-alpha
0.23.3
Status: Resolved (was: Patch Available)
Thanks for the fix Tom. I merge this into trunk, branch-2, and branch-0.23
> Terasort hangs with MR2 FifoScheduler
> -------------------------------------
>
> Key: MAPREDUCE-4299
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4299
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 2.0.0-alpha
> Reporter: Tom White
> Assignee: Tom White
> Fix For: 0.23.3, 2.0.1-alpha, 3.0.0
>
> Attachments: MAPREDUCE-4299.patch, MAPREDUCE-4299.patch, MAPREDUCE-4299.patch
>
>
> What happens is that the number of reducers ramp up until they occupy all of the job's containers, at which point the maps no longer make any progress and the job hangs.
> When the same job is run with the CapacityScheduler it succeeds, so this looks like a FifoScheduler bug.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4299) Terasort hangs with MR2
FifoScheduler
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13411231#comment-13411231 ]
Hadoop QA commented on MAPREDUCE-4299:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12535965/MAPREDUCE-4299.patch
against trunk revision .
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2570//console
This message is automatically generated.
> Terasort hangs with MR2 FifoScheduler
> -------------------------------------
>
> Key: MAPREDUCE-4299
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4299
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 2.0.0-alpha
> Reporter: Tom White
> Assignee: Tom White
> Attachments: MAPREDUCE-4299.patch
>
>
> What happens is that the number of reducers ramp up until they occupy all of the job's containers, at which point the maps no longer make any progress and the job hangs.
> When the same job is run with the CapacityScheduler it succeeds, so this looks like a FifoScheduler bug.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4299) Terasort hangs with MR2
FifoScheduler
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated MAPREDUCE-4299:
---------------------------------
Assignee: Tom White
Status: Patch Available (was: Open)
> Terasort hangs with MR2 FifoScheduler
> -------------------------------------
>
> Key: MAPREDUCE-4299
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4299
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 2.0.0-alpha
> Reporter: Tom White
> Assignee: Tom White
> Attachments: MAPREDUCE-4299.patch
>
>
> What happens is that the number of reducers ramp up until they occupy all of the job's containers, at which point the maps no longer make any progress and the job hangs.
> When the same job is run with the CapacityScheduler it succeeds, so this looks like a FifoScheduler bug.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4299) Terasort hangs with MR2
FifoScheduler
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13414037#comment-13414037 ]
Robert Joseph Evans commented on MAPREDUCE-4299:
------------------------------------------------
I am a +1 too. I'll check this in.
> Terasort hangs with MR2 FifoScheduler
> -------------------------------------
>
> Key: MAPREDUCE-4299
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4299
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 2.0.0-alpha
> Reporter: Tom White
> Assignee: Tom White
> Attachments: MAPREDUCE-4299.patch, MAPREDUCE-4299.patch, MAPREDUCE-4299.patch
>
>
> What happens is that the number of reducers ramp up until they occupy all of the job's containers, at which point the maps no longer make any progress and the job hangs.
> When the same job is run with the CapacityScheduler it succeeds, so this looks like a FifoScheduler bug.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4299) Terasort hangs with MR2
FifoScheduler
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13414058#comment-13414058 ]
Hudson commented on MAPREDUCE-4299:
-----------------------------------
Integrated in Hadoop-Common-trunk-Commit #2465 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2465/])
MAPREDUCE-4299. Terasort hangs with MR2 FifoScheduler (Tom White via bobby) (Revision 1361397)
Result = SUCCESS
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1361397
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestFifoScheduler.java
> Terasort hangs with MR2 FifoScheduler
> -------------------------------------
>
> Key: MAPREDUCE-4299
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4299
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 2.0.0-alpha
> Reporter: Tom White
> Assignee: Tom White
> Fix For: 0.23.3, 2.0.1-alpha, 3.0.0
>
> Attachments: MAPREDUCE-4299.patch, MAPREDUCE-4299.patch, MAPREDUCE-4299.patch
>
>
> What happens is that the number of reducers ramp up until they occupy all of the job's containers, at which point the maps no longer make any progress and the job hangs.
> When the same job is run with the CapacityScheduler it succeeds, so this looks like a FifoScheduler bug.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4299) Terasort hangs with MR2
FifoScheduler
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated MAPREDUCE-4299:
---------------------------------
Attachment: MAPREDUCE-4299.patch
Thanks for taking a look at the patch Karthik.
> Just realized that the application headroom is being set after the for loop; shouldn't we update it within the for loop itself so that subsequent iterations (applications) would see the updated resources.
It's the other way round: the headroom should be set after the assignments have been made in the loop (in the call to assignContainersOnNode).
> If number of applications makes a difference, the test should probably have more applications before we trigger node update.
Good idea - I updated the test to have two applications.
> Terasort hangs with MR2 FifoScheduler
> -------------------------------------
>
> Key: MAPREDUCE-4299
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4299
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 2.0.0-alpha
> Reporter: Tom White
> Assignee: Tom White
> Attachments: MAPREDUCE-4299.patch, MAPREDUCE-4299.patch, MAPREDUCE-4299.patch
>
>
> What happens is that the number of reducers ramp up until they occupy all of the job's containers, at which point the maps no longer make any progress and the job hangs.
> When the same job is run with the CapacityScheduler it succeeds, so this looks like a FifoScheduler bug.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4299) Terasort hangs with MR2
FifoScheduler
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13286915#comment-13286915 ]
Tom White commented on MAPREDUCE-4299:
--------------------------------------
The code in RMContainerAllocator is meant to handle this case by ramping up the number reducers as maps finish. However, there seems to be something fishy about the total amount of memory available to the job. Compare
2012-05-24 16:47:25,803 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: completedMapPercent 0.3 totalMemLimit:63488 finalMapMemLimit:44442 finalReduceMemLimit:19046 netScheduledMapMem:117760 netScheduledReduceMem:15360
to
2012-05-24 16:47:07,521 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReduces:30 ScheduledMaps:160 ScheduledReduces:0 AssignedMaps:0 AssignedReduces:0 completedMaps:0 completedReduces:0 containersAllocated:0 containersReleased:0 hostLocalAssigned:0 rackLocalAssigned:0 availableResources(headroom):memory: 32768
The first says that there is 63488 MB of memory, the second 32768 MB (these numbers stay the same throughout the job). So what could be happening is that the allocator slowly ramps up the number of reducers until they use up 32768 MB (32 slots at 1024MB apiece) thinking that there is still memory available when there isn't. The code has some confusion between the terms 'available resource', 'headroom', and 'cluster resource' - i.e. it's not clear if available resource is a total, or just what's not in use. RMContainerAllocator.getMemLimit() suggests the latter, while the FifoScheduler has the line {{application.setHeadroom(clusterResource)}} which suggests that it's a fixed total.
> Terasort hangs with MR2 FifoScheduler
> -------------------------------------
>
> Key: MAPREDUCE-4299
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4299
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 2.0.0-alpha
> Reporter: Tom White
>
> What happens is that the number of reducers ramp up until they occupy all of the job's containers, at which point the maps no longer make any progress and the job hangs.
> When the same job is run with the CapacityScheduler it succeeds, so this looks like a FifoScheduler bug.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira