You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Vinod Kumar Vavilapalli (Created) (JIRA)" <ji...@apache.org> on 2011/12/15 23:11:32 UTC
[jira] [Created] (MAPREDUCE-3568) Optimize Job's progress
calculations in MR AM
Optimize Job's progress calculations in MR AM
---------------------------------------------
Key: MAPREDUCE-3568
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3568
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Reporter: Vinod Kumar Vavilapalli
Assignee: Vinod Kumar Vavilapalli
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3568) Optimize Job's progress
calculations in MR AM
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13176645#comment-13176645 ]
Hudson commented on MAPREDUCE-3568:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk #941 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/941/])
MAPREDUCE-3568. Optimized Job's progress calculations in MR AM. (vinodkv)
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1224995
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/Job.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMCommunicator.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerRequestor.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockJobs.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRuntimeEstimators.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestJobImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/PartialJob.java
> Optimize Job's progress calculations in MR AM
> ---------------------------------------------
>
> Key: MAPREDUCE-3568
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3568
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am, mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Vinod Kumar Vavilapalli
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3568-20111215.1.txt, MAPREDUCE-3568-20111220.txt, MAPREDUCE-3568-20111222.txt, MAPREDUCE-3568-20111227.txt
>
>
> Besides catering to client requests, Job progress is calculated in every heartbeat to the RM so as to print the MR AM's progress. Today the map and reduce progresses are calculated by looking up of each task in a big map while we can simply make do with a scan and aggregate. With large number of tasks, this can make a difference.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3568) Optimize Job's progress
calculations in MR AM
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13176298#comment-13176298 ]
Hudson commented on MAPREDUCE-3568:
-----------------------------------
Integrated in Hadoop-Hdfs-0.23-Commit #312 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/312/])
MAPREDUCE-3568. Optimized Job's progress calculations in MR AM. (vinodkv)
svn merge -c 1224995 --ignore-ancestry ../../trunk/
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1224996
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/Job.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskImpl.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMCommunicator.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerRequestor.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockJobs.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRuntimeEstimators.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestJobImpl.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/PartialJob.java
> Optimize Job's progress calculations in MR AM
> ---------------------------------------------
>
> Key: MAPREDUCE-3568
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3568
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am, mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Vinod Kumar Vavilapalli
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3568-20111215.1.txt, MAPREDUCE-3568-20111220.txt, MAPREDUCE-3568-20111222.txt, MAPREDUCE-3568-20111227.txt
>
>
> Besides catering to client requests, Job progress is calculated in every heartbeat to the RM so as to print the MR AM's progress. Today the map and reduce progresses are calculated by looking up of each task in a big map while we can simply make do with a scan and aggregate. With large number of tasks, this can make a difference.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3568) Optimize Job's progress
calculations in MR AM
Posted by "Vinod Kumar Vavilapalli (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-3568:
-----------------------------------------------
Status: Patch Available (was: Open)
> Optimize Job's progress calculations in MR AM
> ---------------------------------------------
>
> Key: MAPREDUCE-3568
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3568
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am, mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Vinod Kumar Vavilapalli
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3568-20111215.1.txt, MAPREDUCE-3568-20111220.txt
>
>
> Besides catering to client requests, Job progress is calculated in every heartbeat to the RM so as to print the MR AM's progress. Today the map and reduce progresses are calculated by looking up of each task in a big map while we can simply make do with a scan and aggregate. With large number of tasks, this can make a difference.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3568) Optimize Job's progress
calculations in MR AM
Posted by "Vinod Kumar Vavilapalli (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-3568:
-----------------------------------------------
Attachment: MAPREDUCE-3568-20111215.1.txt
Raw patch. Will clean it up and perhaps add tests in a bit of time.
> Optimize Job's progress calculations in MR AM
> ---------------------------------------------
>
> Key: MAPREDUCE-3568
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3568
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am, mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Vinod Kumar Vavilapalli
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3568-20111215.1.txt
>
>
> Besides catering to client requests, Job progress is calculated in every heartbeat to the RM so as to print the MR AM's progress. Today the map and reduce progresses are calculated by looking up of each task in a big map while we can simply make do with a scan and aggregate. With large number of tasks, this can make a difference.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3568) Optimize Job's progress
calculations in MR AM
Posted by "Vinod Kumar Vavilapalli (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-3568:
-----------------------------------------------
Status: Open (was: Patch Available)
> Optimize Job's progress calculations in MR AM
> ---------------------------------------------
>
> Key: MAPREDUCE-3568
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3568
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am, mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Vinod Kumar Vavilapalli
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3568-20111215.1.txt, MAPREDUCE-3568-20111220.txt, MAPREDUCE-3568-20111222.txt
>
>
> Besides catering to client requests, Job progress is calculated in every heartbeat to the RM so as to print the MR AM's progress. Today the map and reduce progresses are calculated by looking up of each task in a big map while we can simply make do with a scan and aggregate. With large number of tasks, this can make a difference.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3568) Optimize Job's progress
calculations in MR AM
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13173738#comment-13173738 ]
Hadoop QA commented on MAPREDUCE-3568:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12508170/MAPREDUCE-3568-20111220.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 9 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator
org.apache.hadoop.mapreduce.v2.app.job.impl.TestJobImpl
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1487//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1487//console
This message is automatically generated.
> Optimize Job's progress calculations in MR AM
> ---------------------------------------------
>
> Key: MAPREDUCE-3568
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3568
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am, mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Vinod Kumar Vavilapalli
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3568-20111215.1.txt, MAPREDUCE-3568-20111220.txt
>
>
> Besides catering to client requests, Job progress is calculated in every heartbeat to the RM so as to print the MR AM's progress. Today the map and reduce progresses are calculated by looking up of each task in a big map while we can simply make do with a scan and aggregate. With large number of tasks, this can make a difference.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3568) Optimize Job's progress
calculations in MR AM
Posted by "Vinod Kumar Vavilapalli (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-3568:
-----------------------------------------------
Status: Open (was: Patch Available)
> Optimize Job's progress calculations in MR AM
> ---------------------------------------------
>
> Key: MAPREDUCE-3568
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3568
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am, mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Vinod Kumar Vavilapalli
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3568-20111215.1.txt, MAPREDUCE-3568-20111220.txt, MAPREDUCE-3568-20111222.txt, MAPREDUCE-3568-20111227.txt
>
>
> Besides catering to client requests, Job progress is calculated in every heartbeat to the RM so as to print the MR AM's progress. Today the map and reduce progresses are calculated by looking up of each task in a big map while we can simply make do with a scan and aggregate. With large number of tasks, this can make a difference.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3568) Optimize Job's progress
calculations in MR AM
Posted by "Vinod Kumar Vavilapalli (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-3568:
-----------------------------------------------
Component/s: mr-am
Description: Besides catering to client requests, Job progress is calculated in every heartbeat to the RM so as to print the MR AM's progress. Today the map and reduce progresses are calculated by looking up of each task in a big map while we can simply make do with a scan and aggregate. With large number of tasks, this can make a difference.
Affects Version/s: 0.23.0
Fix Version/s: 0.23.1
> Optimize Job's progress calculations in MR AM
> ---------------------------------------------
>
> Key: MAPREDUCE-3568
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3568
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am, mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Vinod Kumar Vavilapalli
> Fix For: 0.23.1
>
>
> Besides catering to client requests, Job progress is calculated in every heartbeat to the RM so as to print the MR AM's progress. Today the map and reduce progresses are calculated by looking up of each task in a big map while we can simply make do with a scan and aggregate. With large number of tasks, this can make a difference.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Work started] (MAPREDUCE-3568) Optimize Job's progress
calculations in MR AM
Posted by "Vinod Kumar Vavilapalli (Work started) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Work on MAPREDUCE-3568 started by Vinod Kumar Vavilapalli.
> Optimize Job's progress calculations in MR AM
> ---------------------------------------------
>
> Key: MAPREDUCE-3568
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3568
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am, mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Vinod Kumar Vavilapalli
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3568-20111215.1.txt, MAPREDUCE-3568-20111220.txt
>
>
> Besides catering to client requests, Job progress is calculated in every heartbeat to the RM so as to print the MR AM's progress. Today the map and reduce progresses are calculated by looking up of each task in a big map while we can simply make do with a scan and aggregate. With large number of tasks, this can make a difference.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3568) Optimize Job's progress
calculations in MR AM
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13175028#comment-13175028 ]
Hadoop QA commented on MAPREDUCE-3568:
--------------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12508419/MAPREDUCE-3568-20111222.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 15 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1495//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1495//console
This message is automatically generated.
> Optimize Job's progress calculations in MR AM
> ---------------------------------------------
>
> Key: MAPREDUCE-3568
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3568
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am, mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Vinod Kumar Vavilapalli
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3568-20111215.1.txt, MAPREDUCE-3568-20111220.txt, MAPREDUCE-3568-20111222.txt
>
>
> Besides catering to client requests, Job progress is calculated in every heartbeat to the RM so as to print the MR AM's progress. Today the map and reduce progresses are calculated by looking up of each task in a big map while we can simply make do with a scan and aggregate. With large number of tasks, this can make a difference.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3568) Optimize Job's progress
calculations in MR AM
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13176297#comment-13176297 ]
Hudson commented on MAPREDUCE-3568:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #1543 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1543/])
MAPREDUCE-3568. Optimized Job's progress calculations in MR AM. (vinodkv)
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1224995
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/Job.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMCommunicator.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerRequestor.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockJobs.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRuntimeEstimators.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestJobImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/PartialJob.java
> Optimize Job's progress calculations in MR AM
> ---------------------------------------------
>
> Key: MAPREDUCE-3568
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3568
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am, mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Vinod Kumar Vavilapalli
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3568-20111215.1.txt, MAPREDUCE-3568-20111220.txt, MAPREDUCE-3568-20111222.txt, MAPREDUCE-3568-20111227.txt
>
>
> Besides catering to client requests, Job progress is calculated in every heartbeat to the RM so as to print the MR AM's progress. Today the map and reduce progresses are calculated by looking up of each task in a big map while we can simply make do with a scan and aggregate. With large number of tasks, this can make a difference.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3568) Optimize Job's progress
calculations in MR AM
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13176632#comment-13176632 ]
Hudson commented on MAPREDUCE-3568:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk #908 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/908/])
MAPREDUCE-3568. Optimized Job's progress calculations in MR AM. (vinodkv)
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1224995
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/Job.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMCommunicator.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerRequestor.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockJobs.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRuntimeEstimators.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestJobImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/PartialJob.java
> Optimize Job's progress calculations in MR AM
> ---------------------------------------------
>
> Key: MAPREDUCE-3568
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3568
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am, mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Vinod Kumar Vavilapalli
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3568-20111215.1.txt, MAPREDUCE-3568-20111220.txt, MAPREDUCE-3568-20111222.txt, MAPREDUCE-3568-20111227.txt
>
>
> Besides catering to client requests, Job progress is calculated in every heartbeat to the RM so as to print the MR AM's progress. Today the map and reduce progresses are calculated by looking up of each task in a big map while we can simply make do with a scan and aggregate. With large number of tasks, this can make a difference.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3568) Optimize Job's progress
calculations in MR AM
Posted by "Siddharth Seth (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13175238#comment-13175238 ]
Siddharth Seth commented on MAPREDUCE-3568:
-------------------------------------------
+1, Looks good. One minor change though - TestRMContainerAllocator.FakeJob isn't used any more and can be removed.
> Optimize Job's progress calculations in MR AM
> ---------------------------------------------
>
> Key: MAPREDUCE-3568
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3568
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am, mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Vinod Kumar Vavilapalli
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3568-20111215.1.txt, MAPREDUCE-3568-20111220.txt, MAPREDUCE-3568-20111222.txt
>
>
> Besides catering to client requests, Job progress is calculated in every heartbeat to the RM so as to print the MR AM's progress. Today the map and reduce progresses are calculated by looking up of each task in a big map while we can simply make do with a scan and aggregate. With large number of tasks, this can make a difference.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3568) Optimize Job's progress
calculations in MR AM
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13176318#comment-13176318 ]
Hudson commented on MAPREDUCE-3568:
-----------------------------------
Integrated in Hadoop-Mapreduce-0.23-Commit #334 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/334/])
MAPREDUCE-3568. Optimized Job's progress calculations in MR AM. (vinodkv)
svn merge -c 1224995 --ignore-ancestry ../../trunk/
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1224996
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/Job.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskImpl.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMCommunicator.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerRequestor.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockJobs.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRuntimeEstimators.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestJobImpl.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/PartialJob.java
> Optimize Job's progress calculations in MR AM
> ---------------------------------------------
>
> Key: MAPREDUCE-3568
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3568
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am, mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Vinod Kumar Vavilapalli
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3568-20111215.1.txt, MAPREDUCE-3568-20111220.txt, MAPREDUCE-3568-20111222.txt, MAPREDUCE-3568-20111227.txt
>
>
> Besides catering to client requests, Job progress is calculated in every heartbeat to the RM so as to print the MR AM's progress. Today the map and reduce progresses are calculated by looking up of each task in a big map while we can simply make do with a scan and aggregate. With large number of tasks, this can make a difference.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3568) Optimize Job's progress
calculations in MR AM
Posted by "Vinod Kumar Vavilapalli (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-3568:
-----------------------------------------------
Status: Patch Available (was: Open)
> Optimize Job's progress calculations in MR AM
> ---------------------------------------------
>
> Key: MAPREDUCE-3568
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3568
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am, mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Vinod Kumar Vavilapalli
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3568-20111215.1.txt, MAPREDUCE-3568-20111220.txt, MAPREDUCE-3568-20111222.txt
>
>
> Besides catering to client requests, Job progress is calculated in every heartbeat to the RM so as to print the MR AM's progress. Today the map and reduce progresses are calculated by looking up of each task in a big map while we can simply make do with a scan and aggregate. With large number of tasks, this can make a difference.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3568) Optimize Job's progress
calculations in MR AM
Posted by "Vinod Kumar Vavilapalli (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-3568:
-----------------------------------------------
Attachment: MAPREDUCE-3568-20111222.txt
Patch fixing tests. Took longer than I anticipated.
> Optimize Job's progress calculations in MR AM
> ---------------------------------------------
>
> Key: MAPREDUCE-3568
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3568
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am, mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Vinod Kumar Vavilapalli
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3568-20111215.1.txt, MAPREDUCE-3568-20111220.txt, MAPREDUCE-3568-20111222.txt
>
>
> Besides catering to client requests, Job progress is calculated in every heartbeat to the RM so as to print the MR AM's progress. Today the map and reduce progresses are calculated by looking up of each task in a big map while we can simply make do with a scan and aggregate. With large number of tasks, this can make a difference.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3568) Optimize Job's progress
calculations in MR AM
Posted by "Vinod Kumar Vavilapalli (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-3568:
-----------------------------------------------
Attachment: MAPREDUCE-3568-20111227.txt
Thanks for the review, Sid!
Here's an updated patch which removes the unused class.
> Optimize Job's progress calculations in MR AM
> ---------------------------------------------
>
> Key: MAPREDUCE-3568
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3568
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am, mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Vinod Kumar Vavilapalli
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3568-20111215.1.txt, MAPREDUCE-3568-20111220.txt, MAPREDUCE-3568-20111222.txt, MAPREDUCE-3568-20111227.txt
>
>
> Besides catering to client requests, Job progress is calculated in every heartbeat to the RM so as to print the MR AM's progress. Today the map and reduce progresses are calculated by looking up of each task in a big map while we can simply make do with a scan and aggregate. With large number of tasks, this can make a difference.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3568) Optimize Job's progress
calculations in MR AM
Posted by "Vinod Kumar Vavilapalli (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-3568:
-----------------------------------------------
Attachment: MAPREDUCE-3568-20111220.txt
Clean patch with simple test changes to validate the progress from the job directly (besides validating app progress at RM).
> Optimize Job's progress calculations in MR AM
> ---------------------------------------------
>
> Key: MAPREDUCE-3568
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3568
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am, mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Vinod Kumar Vavilapalli
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3568-20111215.1.txt, MAPREDUCE-3568-20111220.txt
>
>
> Besides catering to client requests, Job progress is calculated in every heartbeat to the RM so as to print the MR AM's progress. Today the map and reduce progresses are calculated by looking up of each task in a big map while we can simply make do with a scan and aggregate. With large number of tasks, this can make a difference.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3568) Optimize Job's progress
calculations in MR AM
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13176628#comment-13176628 ]
Hudson commented on MAPREDUCE-3568:
-----------------------------------
Integrated in Hadoop-Hdfs-0.23-Build #121 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/121/])
MAPREDUCE-3568. Optimized Job's progress calculations in MR AM. (vinodkv)
svn merge -c 1224995 --ignore-ancestry ../../trunk/
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1224996
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/Job.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskImpl.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMCommunicator.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerRequestor.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockJobs.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRuntimeEstimators.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestJobImpl.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/PartialJob.java
> Optimize Job's progress calculations in MR AM
> ---------------------------------------------
>
> Key: MAPREDUCE-3568
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3568
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am, mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Vinod Kumar Vavilapalli
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3568-20111215.1.txt, MAPREDUCE-3568-20111220.txt, MAPREDUCE-3568-20111222.txt, MAPREDUCE-3568-20111227.txt
>
>
> Besides catering to client requests, Job progress is calculated in every heartbeat to the RM so as to print the MR AM's progress. Today the map and reduce progresses are calculated by looking up of each task in a big map while we can simply make do with a scan and aggregate. With large number of tasks, this can make a difference.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3568) Optimize Job's progress
calculations in MR AM
Posted by "Vinod Kumar Vavilapalli (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-3568:
-----------------------------------------------
Resolution: Fixed
Release Note: Optimized Job's progress calculations in MR AM.
Status: Resolved (was: Patch Available)
Just committed this to trunk and branch-0.23.
> Optimize Job's progress calculations in MR AM
> ---------------------------------------------
>
> Key: MAPREDUCE-3568
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3568
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am, mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Vinod Kumar Vavilapalli
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3568-20111215.1.txt, MAPREDUCE-3568-20111220.txt, MAPREDUCE-3568-20111222.txt, MAPREDUCE-3568-20111227.txt
>
>
> Besides catering to client requests, Job progress is calculated in every heartbeat to the RM so as to print the MR AM's progress. Today the map and reduce progresses are calculated by looking up of each task in a big map while we can simply make do with a scan and aggregate. With large number of tasks, this can make a difference.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Work stopped] (MAPREDUCE-3568) Optimize Job's progress
calculations in MR AM
Posted by "Vinod Kumar Vavilapalli (Work stopped) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Work on MAPREDUCE-3568 stopped by Vinod Kumar Vavilapalli.
> Optimize Job's progress calculations in MR AM
> ---------------------------------------------
>
> Key: MAPREDUCE-3568
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3568
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am, mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Vinod Kumar Vavilapalli
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3568-20111215.1.txt, MAPREDUCE-3568-20111220.txt
>
>
> Besides catering to client requests, Job progress is calculated in every heartbeat to the RM so as to print the MR AM's progress. Today the map and reduce progresses are calculated by looking up of each task in a big map while we can simply make do with a scan and aggregate. With large number of tasks, this can make a difference.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3568) Optimize Job's progress
calculations in MR AM
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13176639#comment-13176639 ]
Hudson commented on MAPREDUCE-3568:
-----------------------------------
Integrated in Hadoop-Mapreduce-0.23-Build #142 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/142/])
MAPREDUCE-3568. Optimized Job's progress calculations in MR AM. (vinodkv)
svn merge -c 1224995 --ignore-ancestry ../../trunk/
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1224996
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/Job.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskImpl.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMCommunicator.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerRequestor.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockJobs.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRuntimeEstimators.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestJobImpl.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/PartialJob.java
> Optimize Job's progress calculations in MR AM
> ---------------------------------------------
>
> Key: MAPREDUCE-3568
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3568
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am, mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Vinod Kumar Vavilapalli
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3568-20111215.1.txt, MAPREDUCE-3568-20111220.txt, MAPREDUCE-3568-20111222.txt, MAPREDUCE-3568-20111227.txt
>
>
> Besides catering to client requests, Job progress is calculated in every heartbeat to the RM so as to print the MR AM's progress. Today the map and reduce progresses are calculated by looking up of each task in a big map while we can simply make do with a scan and aggregate. With large number of tasks, this can make a difference.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3568) Optimize Job's progress
calculations in MR AM
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13176302#comment-13176302 ]
Hudson commented on MAPREDUCE-3568:
-----------------------------------
Integrated in Hadoop-Common-trunk-Commit #1471 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1471/])
MAPREDUCE-3568. Optimized Job's progress calculations in MR AM. (vinodkv)
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1224995
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/Job.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMCommunicator.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerRequestor.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockJobs.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRuntimeEstimators.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestJobImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/PartialJob.java
> Optimize Job's progress calculations in MR AM
> ---------------------------------------------
>
> Key: MAPREDUCE-3568
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3568
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am, mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Vinod Kumar Vavilapalli
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3568-20111215.1.txt, MAPREDUCE-3568-20111220.txt, MAPREDUCE-3568-20111222.txt, MAPREDUCE-3568-20111227.txt
>
>
> Besides catering to client requests, Job progress is calculated in every heartbeat to the RM so as to print the MR AM's progress. Today the map and reduce progresses are calculated by looking up of each task in a big map while we can simply make do with a scan and aggregate. With large number of tasks, this can make a difference.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3568) Optimize Job's progress
calculations in MR AM
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13176263#comment-13176263 ]
Hadoop QA commented on MAPREDUCE-3568:
--------------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12508693/MAPREDUCE-3568-20111227.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 15 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1510//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1510//console
This message is automatically generated.
> Optimize Job's progress calculations in MR AM
> ---------------------------------------------
>
> Key: MAPREDUCE-3568
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3568
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am, mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Vinod Kumar Vavilapalli
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3568-20111215.1.txt, MAPREDUCE-3568-20111220.txt, MAPREDUCE-3568-20111222.txt, MAPREDUCE-3568-20111227.txt
>
>
> Besides catering to client requests, Job progress is calculated in every heartbeat to the RM so as to print the MR AM's progress. Today the map and reduce progresses are calculated by looking up of each task in a big map while we can simply make do with a scan and aggregate. With large number of tasks, this can make a difference.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3568) Optimize Job's progress
calculations in MR AM
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13176317#comment-13176317 ]
Hudson commented on MAPREDUCE-3568:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #1492 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1492/])
MAPREDUCE-3568. Optimized Job's progress calculations in MR AM. (vinodkv)
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1224995
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/Job.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMCommunicator.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerRequestor.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockJobs.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRuntimeEstimators.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestJobImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/PartialJob.java
> Optimize Job's progress calculations in MR AM
> ---------------------------------------------
>
> Key: MAPREDUCE-3568
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3568
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am, mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Vinod Kumar Vavilapalli
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3568-20111215.1.txt, MAPREDUCE-3568-20111220.txt, MAPREDUCE-3568-20111222.txt, MAPREDUCE-3568-20111227.txt
>
>
> Besides catering to client requests, Job progress is calculated in every heartbeat to the RM so as to print the MR AM's progress. Today the map and reduce progresses are calculated by looking up of each task in a big map while we can simply make do with a scan and aggregate. With large number of tasks, this can make a difference.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3568) Optimize Job's progress
calculations in MR AM
Posted by "Vinod Kumar Vavilapalli (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-3568:
-----------------------------------------------
Hadoop Flags: Reviewed
Status: Patch Available (was: Open)
> Optimize Job's progress calculations in MR AM
> ---------------------------------------------
>
> Key: MAPREDUCE-3568
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3568
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am, mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Vinod Kumar Vavilapalli
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3568-20111215.1.txt, MAPREDUCE-3568-20111220.txt, MAPREDUCE-3568-20111222.txt, MAPREDUCE-3568-20111227.txt
>
>
> Besides catering to client requests, Job progress is calculated in every heartbeat to the RM so as to print the MR AM's progress. Today the map and reduce progresses are calculated by looking up of each task in a big map while we can simply make do with a scan and aggregate. With large number of tasks, this can make a difference.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3568) Optimize Job's progress
calculations in MR AM
Posted by "Vinod Kumar Vavilapalli (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-3568:
-----------------------------------------------
Priority: Critical (was: Major)
> Optimize Job's progress calculations in MR AM
> ---------------------------------------------
>
> Key: MAPREDUCE-3568
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3568
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am, mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Vinod Kumar Vavilapalli
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3568-20111215.1.txt, MAPREDUCE-3568-20111220.txt, MAPREDUCE-3568-20111222.txt
>
>
> Besides catering to client requests, Job progress is calculated in every heartbeat to the RM so as to print the MR AM's progress. Today the map and reduce progresses are calculated by looking up of each task in a big map while we can simply make do with a scan and aggregate. With large number of tasks, this can make a difference.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3568) Optimize Job's progress
calculations in MR AM
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13176305#comment-13176305 ]
Hudson commented on MAPREDUCE-3568:
-----------------------------------
Integrated in Hadoop-Common-0.23-Commit #323 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/323/])
MAPREDUCE-3568. Optimized Job's progress calculations in MR AM. (vinodkv)
svn merge -c 1224995 --ignore-ancestry ../../trunk/
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1224996
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/Job.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskImpl.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMCommunicator.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerRequestor.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockJobs.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRuntimeEstimators.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestJobImpl.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/PartialJob.java
> Optimize Job's progress calculations in MR AM
> ---------------------------------------------
>
> Key: MAPREDUCE-3568
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3568
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mr-am, mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Vinod Kumar Vavilapalli
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3568-20111215.1.txt, MAPREDUCE-3568-20111220.txt, MAPREDUCE-3568-20111222.txt, MAPREDUCE-3568-20111227.txt
>
>
> Besides catering to client requests, Job progress is calculated in every heartbeat to the RM so as to print the MR AM's progress. Today the map and reduce progresses are calculated by looking up of each task in a big map while we can simply make do with a scan and aggregate. With large number of tasks, this can make a difference.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira