You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "dhruba borthakur (JIRA)" <ji...@apache.org> on 2009/04/26 10:14:30 UTC
[jira] Created: (HADOOP-5745) Allow setting the default value of
maxRunningJobs for all pools
Allow setting the default value of maxRunningJobs for all pools
---------------------------------------------------------------
Key: HADOOP-5745
URL: https://issues.apache.org/jira/browse/HADOOP-5745
Project: Hadoop Core
Issue Type: Improvement
Components: contrib/fair-share
Reporter: dhruba borthakur
Assignee: dhruba borthakur
The <pool> element allows setting the maxRunningJobs for that pool. It wold be nice to be able to set a default value for all pools.
In out configuration, pools are autocreated.. every new uesre gets his own pool. We would like to allow each user to be able to run a max of 5 jobs at a time. For the etl pool, this limit will be set to a greater value,
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5745) Allow setting the default value of
maxRunningJobs for all pools
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-5745:
-------------------------------------
Status: Patch Available (was: Open)
> Allow setting the default value of maxRunningJobs for all pools
> ---------------------------------------------------------------
>
> Key: HADOOP-5745
> URL: https://issues.apache.org/jira/browse/HADOOP-5745
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/fair-share
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Attachments: fairShareMaxJobs.patch, fairShareMaxJobs.patch, fairShareMaxJobs.patch
>
>
> The <pool> element allows setting the maxRunningJobs for that pool. It wold be nice to be able to set a default value for all pools.
> In out configuration, pools are autocreated.. every new uesre gets his own pool. We would like to allow each user to be able to run a max of 5 jobs at a time. For the etl pool, this limit will be set to a greater value,
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5745) Allow setting the default value of
maxRunningJobs for all pools
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-5745:
-------------------------------------
Attachment: fairShareMaxJobs.patch
Added unit test.
> Allow setting the default value of maxRunningJobs for all pools
> ---------------------------------------------------------------
>
> Key: HADOOP-5745
> URL: https://issues.apache.org/jira/browse/HADOOP-5745
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/fair-share
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Attachments: fairShareMaxJobs.patch, fairShareMaxJobs.patch, fairShareMaxJobs.patch
>
>
> The <pool> element allows setting the maxRunningJobs for that pool. It wold be nice to be able to set a default value for all pools.
> In out configuration, pools are autocreated.. every new uesre gets his own pool. We would like to allow each user to be able to run a max of 5 jobs at a time. For the etl pool, this limit will be set to a greater value,
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5745) Allow setting the default value of
maxRunningJobs for all pools
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-5745:
-------------------------------------
Attachment: fairShareMaxJobs.patch
Thanks for the review Matei. Please let me know if this patch looks good.
> Allow setting the default value of maxRunningJobs for all pools
> ---------------------------------------------------------------
>
> Key: HADOOP-5745
> URL: https://issues.apache.org/jira/browse/HADOOP-5745
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/fair-share
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Attachments: fairShareMaxJobs.patch, fairShareMaxJobs.patch
>
>
> The <pool> element allows setting the maxRunningJobs for that pool. It wold be nice to be able to set a default value for all pools.
> In out configuration, pools are autocreated.. every new uesre gets his own pool. We would like to allow each user to be able to run a max of 5 jobs at a time. For the etl pool, this limit will be set to a greater value,
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5745) Allow setting the default value of
maxRunningJobs for all pools
Posted by "Matei Zaharia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12704508#action_12704508 ]
Matei Zaharia commented on HADOOP-5745:
---------------------------------------
+1 Looks good to me. I'll wait for Hudson to run and commit it.
> Allow setting the default value of maxRunningJobs for all pools
> ---------------------------------------------------------------
>
> Key: HADOOP-5745
> URL: https://issues.apache.org/jira/browse/HADOOP-5745
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/fair-share
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Attachments: fairShareMaxJobs.patch, fairShareMaxJobs.patch
>
>
> The <pool> element allows setting the maxRunningJobs for that pool. It wold be nice to be able to set a default value for all pools.
> In out configuration, pools are autocreated.. every new uesre gets his own pool. We would like to allow each user to be able to run a max of 5 jobs at a time. For the etl pool, this limit will be set to a greater value,
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5745) Allow setting the default value of
maxRunningJobs for all pools
Posted by "Matei Zaharia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matei Zaharia updated HADOOP-5745:
----------------------------------
Issue Type: New Feature (was: Improvement)
Release Note: Added a configuration parameter for setting a default limit on number of running jobs for all pools in the fair scheduler.
Hadoop Flags: [Reviewed]
> Allow setting the default value of maxRunningJobs for all pools
> ---------------------------------------------------------------
>
> Key: HADOOP-5745
> URL: https://issues.apache.org/jira/browse/HADOOP-5745
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/fair-share
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Attachments: fairShareMaxJobs.patch, fairShareMaxJobs.patch, fairShareMaxJobs.patch
>
>
> The <pool> element allows setting the maxRunningJobs for that pool. It wold be nice to be able to set a default value for all pools.
> In out configuration, pools are autocreated.. every new uesre gets his own pool. We would like to allow each user to be able to run a max of 5 jobs at a time. For the etl pool, this limit will be set to a greater value,
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5745) Allow setting the default value of
maxRunningJobs for all pools
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12706151#action_12706151 ]
Hudson commented on HADOOP-5745:
--------------------------------
Integrated in Hadoop-trunk #827 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/827/])
. Allow setting the default value of maxRunningJobs for all
pools. Contributed by Dhruba Borthakur.
> Allow setting the default value of maxRunningJobs for all pools
> ---------------------------------------------------------------
>
> Key: HADOOP-5745
> URL: https://issues.apache.org/jira/browse/HADOOP-5745
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/fair-share
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Fix For: 0.21.0
>
> Attachments: fairShareMaxJobs.patch, fairShareMaxJobs.patch, fairShareMaxJobs.patch
>
>
> The <pool> element allows setting the maxRunningJobs for that pool. It wold be nice to be able to set a default value for all pools.
> In out configuration, pools are autocreated.. every new uesre gets his own pool. We would like to allow each user to be able to run a max of 5 jobs at a time. For the etl pool, this limit will be set to a greater value,
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5745) Allow setting the default value of
maxRunningJobs for all pools
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12705537#action_12705537 ]
dhruba borthakur commented on HADOOP-5745:
------------------------------------------
It is imposible that the unit test that failed (testDFSRestart) is caused by this fix in the fair-share scheduler.
> Allow setting the default value of maxRunningJobs for all pools
> ---------------------------------------------------------------
>
> Key: HADOOP-5745
> URL: https://issues.apache.org/jira/browse/HADOOP-5745
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/fair-share
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Attachments: fairShareMaxJobs.patch, fairShareMaxJobs.patch, fairShareMaxJobs.patch
>
>
> The <pool> element allows setting the maxRunningJobs for that pool. It wold be nice to be able to set a default value for all pools.
> In out configuration, pools are autocreated.. every new uesre gets his own pool. We would like to allow each user to be able to run a max of 5 jobs at a time. For the etl pool, this limit will be set to a greater value,
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5745) Allow setting the default value of
maxRunningJobs for all pools
Posted by "Matei Zaharia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12702955#action_12702955 ]
Matei Zaharia commented on HADOOP-5745:
---------------------------------------
Dhruba, I think you have a bug in that reloadAllocs doesn't define poolMaxJobsDefault as a local variable. This means that it assigns directly to the class field when it reads that element in the XML file. This is inconsistent with how reloadAllocs handles other settings because it means that if reloadAllocs encounters an error in the file later, it will have set the poolMaxJobsDefault but not the other settings. You just want to define a local variable poolMaxJobsDefault in reloadAllocs the same way it has a local userMaxJobsDefault.
> Allow setting the default value of maxRunningJobs for all pools
> ---------------------------------------------------------------
>
> Key: HADOOP-5745
> URL: https://issues.apache.org/jira/browse/HADOOP-5745
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/fair-share
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Attachments: fairShareMaxJobs.patch
>
>
> The <pool> element allows setting the maxRunningJobs for that pool. It wold be nice to be able to set a default value for all pools.
> In out configuration, pools are autocreated.. every new uesre gets his own pool. We would like to allow each user to be able to run a max of 5 jobs at a time. For the etl pool, this limit will be set to a greater value,
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5745) Allow setting the default value of
maxRunningJobs for all pools
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-5745:
-------------------------------------
Status: Patch Available (was: Open)
> Allow setting the default value of maxRunningJobs for all pools
> ---------------------------------------------------------------
>
> Key: HADOOP-5745
> URL: https://issues.apache.org/jira/browse/HADOOP-5745
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/fair-share
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Attachments: fairShareMaxJobs.patch, fairShareMaxJobs.patch
>
>
> The <pool> element allows setting the maxRunningJobs for that pool. It wold be nice to be able to set a default value for all pools.
> In out configuration, pools are autocreated.. every new uesre gets his own pool. We would like to allow each user to be able to run a max of 5 jobs at a time. For the etl pool, this limit will be set to a greater value,
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5745) Allow setting the default value of
maxRunningJobs for all pools
Posted by "Matei Zaharia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matei Zaharia updated HADOOP-5745:
----------------------------------
Resolution: Fixed
Fix Version/s: 0.21.0
Status: Resolved (was: Patch Available)
I committed this. Thanks Dhruba.
> Allow setting the default value of maxRunningJobs for all pools
> ---------------------------------------------------------------
>
> Key: HADOOP-5745
> URL: https://issues.apache.org/jira/browse/HADOOP-5745
> Project: Hadoop Core
> Issue Type: New Feature
> Components: contrib/fair-share
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Fix For: 0.21.0
>
> Attachments: fairShareMaxJobs.patch, fairShareMaxJobs.patch, fairShareMaxJobs.patch
>
>
> The <pool> element allows setting the maxRunningJobs for that pool. It wold be nice to be able to set a default value for all pools.
> In out configuration, pools are autocreated.. every new uesre gets his own pool. We would like to allow each user to be able to run a max of 5 jobs at a time. For the etl pool, this limit will be set to a greater value,
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5745) Allow setting the default value of
maxRunningJobs for all pools
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-5745:
-------------------------------------
Attachment: fairShareMaxJobs.patch
Allow setting the default value for the maxRunningJobs per pool.
> Allow setting the default value of maxRunningJobs for all pools
> ---------------------------------------------------------------
>
> Key: HADOOP-5745
> URL: https://issues.apache.org/jira/browse/HADOOP-5745
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/fair-share
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Attachments: fairShareMaxJobs.patch
>
>
> The <pool> element allows setting the maxRunningJobs for that pool. It wold be nice to be able to set a default value for all pools.
> In out configuration, pools are autocreated.. every new uesre gets his own pool. We would like to allow each user to be able to run a max of 5 jobs at a time. For the etl pool, this limit will be set to a greater value,
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5745) Allow setting the default value of
maxRunningJobs for all pools
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12705533#action_12705533 ]
Hadoop QA commented on HADOOP-5745:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12407129/fairShareMaxJobs.patch
against trunk revision 771179.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 Eclipse classpath. The patch retains Eclipse classpath integrity.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/285/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/285/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/285/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/285/console
This message is automatically generated.
> Allow setting the default value of maxRunningJobs for all pools
> ---------------------------------------------------------------
>
> Key: HADOOP-5745
> URL: https://issues.apache.org/jira/browse/HADOOP-5745
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/fair-share
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Attachments: fairShareMaxJobs.patch, fairShareMaxJobs.patch, fairShareMaxJobs.patch
>
>
> The <pool> element allows setting the maxRunningJobs for that pool. It wold be nice to be able to set a default value for all pools.
> In out configuration, pools are autocreated.. every new uesre gets his own pool. We would like to allow each user to be able to run a max of 5 jobs at a time. For the etl pool, this limit will be set to a greater value,
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5745) Allow setting the default value of
maxRunningJobs for all pools
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-5745:
-------------------------------------
Status: Open (was: Patch Available)
have to write unit test.
> Allow setting the default value of maxRunningJobs for all pools
> ---------------------------------------------------------------
>
> Key: HADOOP-5745
> URL: https://issues.apache.org/jira/browse/HADOOP-5745
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/fair-share
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Attachments: fairShareMaxJobs.patch, fairShareMaxJobs.patch, fairShareMaxJobs.patch
>
>
> The <pool> element allows setting the maxRunningJobs for that pool. It wold be nice to be able to set a default value for all pools.
> In out configuration, pools are autocreated.. every new uesre gets his own pool. We would like to allow each user to be able to run a max of 5 jobs at a time. For the etl pool, this limit will be set to a greater value,
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5745) Allow setting the default value of
maxRunningJobs for all pools
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12704923#action_12704923 ]
Hadoop QA commented on HADOOP-5745:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12406574/fairShareMaxJobs.patch
against trunk revision 770321.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 Eclipse classpath. The patch retains Eclipse classpath integrity.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/267/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/267/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/267/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/267/console
This message is automatically generated.
> Allow setting the default value of maxRunningJobs for all pools
> ---------------------------------------------------------------
>
> Key: HADOOP-5745
> URL: https://issues.apache.org/jira/browse/HADOOP-5745
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/fair-share
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Attachments: fairShareMaxJobs.patch, fairShareMaxJobs.patch
>
>
> The <pool> element allows setting the maxRunningJobs for that pool. It wold be nice to be able to set a default value for all pools.
> In out configuration, pools are autocreated.. every new uesre gets his own pool. We would like to allow each user to be able to run a max of 5 jobs at a time. For the etl pool, this limit will be set to a greater value,
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.