You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Karthik Kambatla (JIRA)" <ji...@apache.org> on 2012/08/06 11:05:02 UTC
[jira] [Created] (MAPREDUCE-4518) FairScheduler:
PoolSchedulable#updateDemand() - potential redundant aggregation
Karthik Kambatla created MAPREDUCE-4518:
-------------------------------------------
Summary: FairScheduler: PoolSchedulable#updateDemand() - potential redundant aggregation
Key: MAPREDUCE-4518
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4518
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: contrib/fair-share
Affects Versions: 1.0.3
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
In FS, PoolSchedulable#updateDemand() limits the demand to maxTasks only after iterating though all the pools and computing the final demand.
By checking if the demand has reached maxTasks in every iteration, we can avoid redundant work, at the expense of one condition check every iteration.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4518) FairScheduler:
PoolSchedulable#updateDemand() - potential redundant aggregation
Posted by "Karthik Kambatla (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Karthik Kambatla updated MAPREDUCE-4518:
----------------------------------------
Status: In Progress (was: Patch Available)
> FairScheduler: PoolSchedulable#updateDemand() - potential redundant aggregation
> -------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4518
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4518
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/fair-share
> Affects Versions: 1.0.3
> Reporter: Karthik Kambatla
> Assignee: Karthik Kambatla
> Attachments: MR-4518_branch1.patch, trunk-MR-4518.patch
>
>
> In FS, PoolSchedulable#updateDemand() limits the demand to maxTasks only after iterating though all the pools and computing the final demand.
> By checking if the demand has reached maxTasks in every iteration, we can avoid redundant work, at the expense of one condition check every iteration.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4518) FairScheduler:
PoolSchedulable#updateDemand() - potential redundant aggregation
Posted by "Karthik Kambatla (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Karthik Kambatla updated MAPREDUCE-4518:
----------------------------------------
Attachment: (was: trunk-MR-4518.patch)
> FairScheduler: PoolSchedulable#updateDemand() - potential redundant aggregation
> -------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4518
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4518
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/fair-share
> Affects Versions: 1.0.3
> Reporter: Karthik Kambatla
> Assignee: Karthik Kambatla
> Attachments: MR-4518_branch1.patch, trunk-MR-4518.patch, trunk-MR-4518.patch
>
>
> In FS, PoolSchedulable#updateDemand() limits the demand to maxTasks only after iterating though all the pools and computing the final demand.
> By checking if the demand has reached maxTasks in every iteration, we can avoid redundant work, at the expense of one condition check every iteration.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4518) FairScheduler:
PoolSchedulable#updateDemand() - potential redundant aggregation
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13431891#comment-13431891 ]
Hadoop QA commented on MAPREDUCE-4518:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12540039/trunk-MR-4518.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2720//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2720//console
This message is automatically generated.
> FairScheduler: PoolSchedulable#updateDemand() - potential redundant aggregation
> -------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4518
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4518
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/fair-share
> Affects Versions: 1.0.3
> Reporter: Karthik Kambatla
> Assignee: Karthik Kambatla
> Attachments: MR-4518_branch1.patch, trunk-MR-4518.patch
>
>
> In FS, PoolSchedulable#updateDemand() limits the demand to maxTasks only after iterating though all the pools and computing the final demand.
> By checking if the demand has reached maxTasks in every iteration, we can avoid redundant work, at the expense of one condition check every iteration.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4518) FairScheduler:
PoolSchedulable#updateDemand() - potential redundant aggregation
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13429341#comment-13429341 ]
Hadoop QA commented on MAPREDUCE-4518:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12539254/MR-4518_branch1.patch
against trunk revision .
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2712//console
This message is automatically generated.
> FairScheduler: PoolSchedulable#updateDemand() - potential redundant aggregation
> -------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4518
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4518
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/fair-share
> Affects Versions: 1.0.3
> Reporter: Karthik Kambatla
> Assignee: Karthik Kambatla
> Attachments: MR-4518_branch1.patch
>
>
> In FS, PoolSchedulable#updateDemand() limits the demand to maxTasks only after iterating though all the pools and computing the final demand.
> By checking if the demand has reached maxTasks in every iteration, we can avoid redundant work, at the expense of one condition check every iteration.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4518) FairScheduler:
PoolSchedulable#updateDemand() - potential redundant aggregation
Posted by "Karthik Kambatla (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Karthik Kambatla updated MAPREDUCE-4518:
----------------------------------------
Attachment: MR-4518_branch1.patch
The patch implements the proposed optimization.
> FairScheduler: PoolSchedulable#updateDemand() - potential redundant aggregation
> -------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4518
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4518
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/fair-share
> Affects Versions: 1.0.3
> Reporter: Karthik Kambatla
> Assignee: Karthik Kambatla
> Attachments: MR-4518_branch1.patch
>
>
> In FS, PoolSchedulable#updateDemand() limits the demand to maxTasks only after iterating though all the pools and computing the final demand.
> By checking if the demand has reached maxTasks in every iteration, we can avoid redundant work, at the expense of one condition check every iteration.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4518) FairScheduler:
PoolSchedulable#updateDemand() - potential redundant aggregation
Posted by "Karthik Kambatla (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Karthik Kambatla updated MAPREDUCE-4518:
----------------------------------------
Attachment: trunk-MR-4518.patch
> FairScheduler: PoolSchedulable#updateDemand() - potential redundant aggregation
> -------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4518
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4518
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/fair-share
> Affects Versions: 1.0.3
> Reporter: Karthik Kambatla
> Assignee: Karthik Kambatla
> Attachments: MR-4518_branch1.patch, trunk-MR-4518.patch, trunk-MR-4518.patch
>
>
> In FS, PoolSchedulable#updateDemand() limits the demand to maxTasks only after iterating though all the pools and computing the final demand.
> By checking if the demand has reached maxTasks in every iteration, we can avoid redundant work, at the expense of one condition check every iteration.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4518) FairScheduler:
PoolSchedulable#updateDemand() - potential redundant aggregation
Posted by "Karthik Kambatla (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Karthik Kambatla updated MAPREDUCE-4518:
----------------------------------------
Attachment: trunk-MR-4518.patch
Updated the patch for trunk
- Added constructor to FSQueueSchedulable for testing purposes
- Test checks if the demand is less than or equal to maxResources
- Verified right number of iterations via the logs in the loop in updateDemand()
> FairScheduler: PoolSchedulable#updateDemand() - potential redundant aggregation
> -------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4518
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4518
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/fair-share
> Affects Versions: 1.0.3
> Reporter: Karthik Kambatla
> Assignee: Karthik Kambatla
> Attachments: MR-4518_branch1.patch, trunk-MR-4518.patch, trunk-MR-4518.patch
>
>
> In FS, PoolSchedulable#updateDemand() limits the demand to maxTasks only after iterating though all the pools and computing the final demand.
> By checking if the demand has reached maxTasks in every iteration, we can avoid redundant work, at the expense of one condition check every iteration.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4518) FairScheduler:
PoolSchedulable#updateDemand() - potential redundant aggregation
Posted by "Karthik Kambatla (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Karthik Kambatla updated MAPREDUCE-4518:
----------------------------------------
Attachment: trunk-MR-4518.patch
Uploading patch for trunk.
I couldn't think of a way to test the patch. Can someone suggest a way to test this?
> FairScheduler: PoolSchedulable#updateDemand() - potential redundant aggregation
> -------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4518
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4518
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/fair-share
> Affects Versions: 1.0.3
> Reporter: Karthik Kambatla
> Assignee: Karthik Kambatla
> Attachments: MR-4518_branch1.patch, trunk-MR-4518.patch
>
>
> In FS, PoolSchedulable#updateDemand() limits the demand to maxTasks only after iterating though all the pools and computing the final demand.
> By checking if the demand has reached maxTasks in every iteration, we can avoid redundant work, at the expense of one condition check every iteration.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4518) FairScheduler:
PoolSchedulable#updateDemand() - potential redundant aggregation
Posted by "Alejandro Abdelnur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13429939#comment-13429939 ]
Alejandro Abdelnur commented on MAPREDUCE-4518:
-----------------------------------------------
+1. does this apply to trunk as well? if so we need a patch for it first.
> FairScheduler: PoolSchedulable#updateDemand() - potential redundant aggregation
> -------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4518
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4518
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/fair-share
> Affects Versions: 1.0.3
> Reporter: Karthik Kambatla
> Assignee: Karthik Kambatla
> Attachments: MR-4518_branch1.patch
>
>
> In FS, PoolSchedulable#updateDemand() limits the demand to maxTasks only after iterating though all the pools and computing the final demand.
> By checking if the demand has reached maxTasks in every iteration, we can avoid redundant work, at the expense of one condition check every iteration.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4518) FairScheduler:
PoolSchedulable#updateDemand() - potential redundant aggregation
Posted by "Karthik Kambatla (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Karthik Kambatla updated MAPREDUCE-4518:
----------------------------------------
Status: Patch Available (was: Open)
> FairScheduler: PoolSchedulable#updateDemand() - potential redundant aggregation
> -------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4518
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4518
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/fair-share
> Affects Versions: 1.0.3
> Reporter: Karthik Kambatla
> Assignee: Karthik Kambatla
> Attachments: MR-4518_branch1.patch
>
>
> In FS, PoolSchedulable#updateDemand() limits the demand to maxTasks only after iterating though all the pools and computing the final demand.
> By checking if the demand has reached maxTasks in every iteration, we can avoid redundant work, at the expense of one condition check every iteration.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4518) FairScheduler:
PoolSchedulable#updateDemand() - potential redundant aggregation
Posted by "Karthik Kambatla (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13433510#comment-13433510 ]
Karthik Kambatla commented on MAPREDUCE-4518:
---------------------------------------------
Given that this concerns YARN, should I convert this into a YARN issue?
> FairScheduler: PoolSchedulable#updateDemand() - potential redundant aggregation
> -------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4518
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4518
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/fair-share
> Affects Versions: 1.0.3
> Reporter: Karthik Kambatla
> Assignee: Karthik Kambatla
> Attachments: MR-4518_branch1.patch, trunk-MR-4518.patch, trunk-MR-4518.patch
>
>
> In FS, PoolSchedulable#updateDemand() limits the demand to maxTasks only after iterating though all the pools and computing the final demand.
> By checking if the demand has reached maxTasks in every iteration, we can avoid redundant work, at the expense of one condition check every iteration.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira