You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Rajiv Chittajallu (JIRA)" <ji...@apache.org> on 2009/04/08 20:45:12 UTC
[jira] Created: (HADOOP-5643) Ability to blacklist tasktracker
Ability to blacklist tasktracker
--------------------------------
Key: HADOOP-5643
URL: https://issues.apache.org/jira/browse/HADOOP-5643
Project: Hadoop Core
Issue Type: New Feature
Affects Versions: 0.20.0
Reporter: Rajiv Chittajallu
Its not always possible to shutdown the tasktracker to stop scheduling tasks on the node. (eg you can't login to the node but the TT is up).
This can be via
* mapred.exclude and should be refreshed with out restarting the tasktracker
* hadoop job -fail-tracker <tracker id>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5643) Ability to blacklist tasktracker
Posted by "eric baldeschwieler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12703482#action_12703482 ]
eric baldeschwieler commented on HADOOP-5643:
---------------------------------------------
Do we have a good security story for actions taken through the web UI? Absent that, I'd suggest we don't enable this there.
Being able to modify the excludes file and hup the server is probably good enough for an operator.
> Ability to blacklist tasktracker
> --------------------------------
>
> Key: HADOOP-5643
> URL: https://issues.apache.org/jira/browse/HADOOP-5643
> Project: Hadoop Core
> Issue Type: New Feature
> Affects Versions: 0.20.0
> Reporter: Rajiv Chittajallu
> Assignee: Amar Kamat
>
> Its not always possible to shutdown the tasktracker to stop scheduling tasks on the node. (eg you can't login to the node but the TT is up).
> This can be via
> * mapred.exclude and should be refreshed with out restarting the tasktracker
> * hadoop job -fail-tracker <tracker id>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5643) Ability to blacklist tasktracker
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12708187#action_12708187 ]
Hudson commented on HADOOP-5643:
--------------------------------
Integrated in Hadoop-trunk #833 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/833/])
. Adding the file src/test/mapred/org/apache/hadoop/mapred/TestNodeRefresh.java that got missed earlier.
. Adds a way to decommission TaskTrackers while the JobTracker is running. Contributed by Amar Kamat.
> Ability to blacklist tasktracker
> --------------------------------
>
> Key: HADOOP-5643
> URL: https://issues.apache.org/jira/browse/HADOOP-5643
> Project: Hadoop Core
> Issue Type: New Feature
> Affects Versions: 0.20.0
> Reporter: Rajiv Chittajallu
> Assignee: Amar Kamat
> Fix For: 0.21.0
>
> Attachments: HADOOP-5643-v3.4.patch, HADOOP-5643-v4.0.patch, HADOOP-5643-v4.6.patch, HADOOP-5643-v5.1.patch, HADOOP-5643-v5.12-testcase.patch, HADOOP-5643-v5.12.patch, HADOOP-5643-v5.5.patch, HADOOP-5643-v5.9.patch
>
>
> Its not always possible to shutdown the tasktracker to stop scheduling tasks on the node. (eg you can't login to the node but the TT is up).
> This can be via
> * mapred.exclude and should be refreshed with out restarting the tasktracker
> * hadoop job -fail-tracker <tracker id>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5643) Ability to blacklist tasktracker
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12708318#action_12708318 ]
Amar Kamat commented on HADOOP-5643:
------------------------------------
bq. src/hdfs/org/apache/hadoop/hdfs/server/namenode/PermissionChecker.java becomes an empty file in trunk.
We should do a svn delete of this file.
bq. Why renaming the FSNamesystem.checkSuperuserPrivilege() method to checkAccess() even though it is still for checking superuser privilege?
I feel checkSuperuserPrivilege() should be used for simply checking superuser privilege (without permission switch which is just in HDFS for now) and checkAccess() for making a guarded call to checkSuperuserPrivilege(). The reason for doing this was to keep both the MR and HDFS consistent wrt superuser checks.
> Ability to blacklist tasktracker
> --------------------------------
>
> Key: HADOOP-5643
> URL: https://issues.apache.org/jira/browse/HADOOP-5643
> Project: Hadoop Core
> Issue Type: New Feature
> Affects Versions: 0.20.0
> Reporter: Rajiv Chittajallu
> Assignee: Amar Kamat
> Fix For: 0.21.0
>
> Attachments: HADOOP-5643-v3.4.patch, HADOOP-5643-v4.0.patch, HADOOP-5643-v4.6.patch, HADOOP-5643-v5.1.patch, HADOOP-5643-v5.12-testcase.patch, HADOOP-5643-v5.12.patch, HADOOP-5643-v5.5.patch, HADOOP-5643-v5.9.patch
>
>
> Its not always possible to shutdown the tasktracker to stop scheduling tasks on the node. (eg you can't login to the node but the TT is up).
> This can be via
> * mapred.exclude and should be refreshed with out restarting the tasktracker
> * hadoop job -fail-tracker <tracker id>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5643) Ability to blacklist tasktracker
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amar Kamat updated HADOOP-5643:
-------------------------------
Attachment: HADOOP-5643-v3.4.patch
Attaching a patch implementing the above discussed approach. Result of test patch
{code}
[exec] -1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 9 new or modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
[exec]
[exec] -1 release audit. The applied patch generated 472 release audit warnings (more than the trunk's current 469 warnings).
{code}
Not clear why release audit warnings are there. This patch is tested on local box and testing is in progress. Will upload a new patch with fixed warnings and testcases.
> Ability to blacklist tasktracker
> --------------------------------
>
> Key: HADOOP-5643
> URL: https://issues.apache.org/jira/browse/HADOOP-5643
> Project: Hadoop Core
> Issue Type: New Feature
> Affects Versions: 0.20.0
> Reporter: Rajiv Chittajallu
> Assignee: Amar Kamat
> Attachments: HADOOP-5643-v3.4.patch
>
>
> Its not always possible to shutdown the tasktracker to stop scheduling tasks on the node. (eg you can't login to the node but the TT is up).
> This can be via
> * mapred.exclude and should be refreshed with out restarting the tasktracker
> * hadoop job -fail-tracker <tracker id>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5643) Ability to blacklist tasktracker
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amar Kamat updated HADOOP-5643:
-------------------------------
Attachment: Fixed+5643-0.20-final
Attaching a new patch for branch 0.20 merging the 2 patches. Note that this is an example patch for 20 and not to be committed.
> Ability to blacklist tasktracker
> --------------------------------
>
> Key: HADOOP-5643
> URL: https://issues.apache.org/jira/browse/HADOOP-5643
> Project: Hadoop Core
> Issue Type: New Feature
> Affects Versions: 0.20.0
> Reporter: Rajiv Chittajallu
> Assignee: Amar Kamat
> Fix For: 0.21.0
>
> Attachments: Fixed 5643-0.20, Fixed+5643-0.20-final, Fixed+5643-0.20-part2, HADOOP-5643-v3.4.patch, HADOOP-5643-v4.0.patch, HADOOP-5643-v4.6.patch, HADOOP-5643-v5.1.patch, HADOOP-5643-v5.12-testcase.patch, HADOOP-5643-v5.12.patch, HADOOP-5643-v5.5.patch, HADOOP-5643-v5.9.patch
>
>
> Its not always possible to shutdown the tasktracker to stop scheduling tasks on the node. (eg you can't login to the node but the TT is up).
> This can be via
> * mapred.exclude and should be refreshed with out restarting the tasktracker
> * hadoop job -fail-tracker <tracker id>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5643) Ability to blacklist tasktracker
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amar Kamat updated HADOOP-5643:
-------------------------------
Attachment: HADOOP-5643-v5.12.patch
Attaching a patch incorporating Devaraj's offline comments.
# Permission checks are now factored out into a common class
# Replaced decommissioned in some cases with excluded.
Result of test-patch
{code}
[exec] +1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 9 new or modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
[exec]
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
{code}
Running ant test now.
> Ability to blacklist tasktracker
> --------------------------------
>
> Key: HADOOP-5643
> URL: https://issues.apache.org/jira/browse/HADOOP-5643
> Project: Hadoop Core
> Issue Type: New Feature
> Affects Versions: 0.20.0
> Reporter: Rajiv Chittajallu
> Assignee: Amar Kamat
> Attachments: HADOOP-5643-v3.4.patch, HADOOP-5643-v4.0.patch, HADOOP-5643-v4.6.patch, HADOOP-5643-v5.1.patch, HADOOP-5643-v5.12.patch, HADOOP-5643-v5.5.patch, HADOOP-5643-v5.9.patch
>
>
> Its not always possible to shutdown the tasktracker to stop scheduling tasks on the node. (eg you can't login to the node but the TT is up).
> This can be via
> * mapred.exclude and should be refreshed with out restarting the tasktracker
> * hadoop job -fail-tracker <tracker id>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-5643) Ability to blacklist tasktracker
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amar Kamat reassigned HADOOP-5643:
----------------------------------
Assignee: Amar Kamat
> Ability to blacklist tasktracker
> --------------------------------
>
> Key: HADOOP-5643
> URL: https://issues.apache.org/jira/browse/HADOOP-5643
> Project: Hadoop Core
> Issue Type: New Feature
> Affects Versions: 0.20.0
> Reporter: Rajiv Chittajallu
> Assignee: Amar Kamat
>
> Its not always possible to shutdown the tasktracker to stop scheduling tasks on the node. (eg you can't login to the node but the TT is up).
> This can be via
> * mapred.exclude and should be refreshed with out restarting the tasktracker
> * hadoop job -fail-tracker <tracker id>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5643) Ability to blacklist tasktracker
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amar Kamat updated HADOOP-5643:
-------------------------------
Attachment: HADOOP-5643-v5.5.patch
Attaching a patch the tried to provide the refresh facility similar to HDFS. Result of test-patch
{code}
[exec] +1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 21 new or modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
[exec]
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
{code}
Ant test passed on my box.
> Ability to blacklist tasktracker
> --------------------------------
>
> Key: HADOOP-5643
> URL: https://issues.apache.org/jira/browse/HADOOP-5643
> Project: Hadoop Core
> Issue Type: New Feature
> Affects Versions: 0.20.0
> Reporter: Rajiv Chittajallu
> Assignee: Amar Kamat
> Attachments: HADOOP-5643-v3.4.patch, HADOOP-5643-v4.0.patch, HADOOP-5643-v4.6.patch, HADOOP-5643-v5.1.patch, HADOOP-5643-v5.5.patch
>
>
> Its not always possible to shutdown the tasktracker to stop scheduling tasks on the node. (eg you can't login to the node but the TT is up).
> This can be via
> * mapred.exclude and should be refreshed with out restarting the tasktracker
> * hadoop job -fail-tracker <tracker id>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5643) Ability to blacklist tasktracker
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12702325#action_12702325 ]
Amar Kamat commented on HADOOP-5643:
------------------------------------
One more thing i forgot to add is that the jobtracker already reads the hosts file and the exclude file but just once. There is no refresh facility to it. I think we can add that to MR too. So here is the sequence of things :
# files to include can be specified via mapred.hosts and will be read by the jobtracker upon init
# files to exclude can be specified via mapred.hosts.exclude and will also be read by the jobtracker upon init
# Admins can change these files and invoke a _refresh_ from the command line (maybe from the webui too). These files will be loaded back.
# If the hosts are added via command line or webui, it will be appended to the include/exclude files. So that upon next restart, the previously included/excluded host info is re-used.
> Ability to blacklist tasktracker
> --------------------------------
>
> Key: HADOOP-5643
> URL: https://issues.apache.org/jira/browse/HADOOP-5643
> Project: Hadoop Core
> Issue Type: New Feature
> Affects Versions: 0.20.0
> Reporter: Rajiv Chittajallu
> Assignee: Amar Kamat
>
> Its not always possible to shutdown the tasktracker to stop scheduling tasks on the node. (eg you can't login to the node but the TT is up).
> This can be via
> * mapred.exclude and should be refreshed with out restarting the tasktracker
> * hadoop job -fail-tracker <tracker id>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5643) Ability to blacklist tasktracker
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amar Kamat updated HADOOP-5643:
-------------------------------
Attachment: HADOOP-5643-v5.1.patch
Attaching a patch that does what HDFS does. Testing the patch.
> Ability to blacklist tasktracker
> --------------------------------
>
> Key: HADOOP-5643
> URL: https://issues.apache.org/jira/browse/HADOOP-5643
> Project: Hadoop Core
> Issue Type: New Feature
> Affects Versions: 0.20.0
> Reporter: Rajiv Chittajallu
> Assignee: Amar Kamat
> Attachments: HADOOP-5643-v3.4.patch, HADOOP-5643-v4.0.patch, HADOOP-5643-v4.6.patch, HADOOP-5643-v5.1.patch
>
>
> Its not always possible to shutdown the tasktracker to stop scheduling tasks on the node. (eg you can't login to the node but the TT is up).
> This can be via
> * mapred.exclude and should be refreshed with out restarting the tasktracker
> * hadoop job -fail-tracker <tracker id>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5643) Ability to blacklist tasktracker
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12703821#action_12703821 ]
Amar Kamat commented on HADOOP-5643:
------------------------------------
Also added a new parameter {{mapred.permissions.supergroup}} to allow admins specify supergroups. Either the user running the jobtracker or user in the supergroup can issue admin commands.
> Ability to blacklist tasktracker
> --------------------------------
>
> Key: HADOOP-5643
> URL: https://issues.apache.org/jira/browse/HADOOP-5643
> Project: Hadoop Core
> Issue Type: New Feature
> Affects Versions: 0.20.0
> Reporter: Rajiv Chittajallu
> Assignee: Amar Kamat
> Attachments: HADOOP-5643-v3.4.patch
>
>
> Its not always possible to shutdown the tasktracker to stop scheduling tasks on the node. (eg you can't login to the node but the TT is up).
> This can be via
> * mapred.exclude and should be refreshed with out restarting the tasktracker
> * hadoop job -fail-tracker <tracker id>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5643) Ability to blacklist tasktracker
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amar Kamat updated HADOOP-5643:
-------------------------------
Attachment: Fixed+5643-0.20-part2
Example patch not to be committed.
> Ability to blacklist tasktracker
> --------------------------------
>
> Key: HADOOP-5643
> URL: https://issues.apache.org/jira/browse/HADOOP-5643
> Project: Hadoop Core
> Issue Type: New Feature
> Affects Versions: 0.20.0
> Reporter: Rajiv Chittajallu
> Assignee: Amar Kamat
> Fix For: 0.21.0
>
> Attachments: Fixed 5643-0.20, Fixed+5643-0.20-part2, HADOOP-5643-v3.4.patch, HADOOP-5643-v4.0.patch, HADOOP-5643-v4.6.patch, HADOOP-5643-v5.1.patch, HADOOP-5643-v5.12-testcase.patch, HADOOP-5643-v5.12.patch, HADOOP-5643-v5.5.patch, HADOOP-5643-v5.9.patch
>
>
> Its not always possible to shutdown the tasktracker to stop scheduling tasks on the node. (eg you can't login to the node but the TT is up).
> This can be via
> * mapred.exclude and should be refreshed with out restarting the tasktracker
> * hadoop job -fail-tracker <tracker id>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5643) Ability to blacklist tasktracker
Posted by "eric baldeschwieler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12704894#action_12704894 ]
eric baldeschwieler commented on HADOOP-5643:
---------------------------------------------
I think we are going through an expensive process of reinventing the wheel here. We should think about solving this sort of issue once by maintain such lists in a plugable source of configuration and supporting the ability to "hup" the service.
We should then implement config in LDAP / SQL / or some other service via plugins and then we can modify these configurations in an environment with lots of tools to support this stuff. Adding ad hock commands and odd side files that will be lost if we need to swap hardware is awkward.
> Ability to blacklist tasktracker
> --------------------------------
>
> Key: HADOOP-5643
> URL: https://issues.apache.org/jira/browse/HADOOP-5643
> Project: Hadoop Core
> Issue Type: New Feature
> Affects Versions: 0.20.0
> Reporter: Rajiv Chittajallu
> Assignee: Amar Kamat
> Attachments: HADOOP-5643-v3.4.patch, HADOOP-5643-v4.0.patch
>
>
> Its not always possible to shutdown the tasktracker to stop scheduling tasks on the node. (eg you can't login to the node but the TT is up).
> This can be via
> * mapred.exclude and should be refreshed with out restarting the tasktracker
> * hadoop job -fail-tracker <tracker id>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5643) Ability to blacklist tasktracker
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amar Kamat updated HADOOP-5643:
-------------------------------
Attachment: HADOOP-5643-v5.12-testcase.patch
Missed the testcase.
> Ability to blacklist tasktracker
> --------------------------------
>
> Key: HADOOP-5643
> URL: https://issues.apache.org/jira/browse/HADOOP-5643
> Project: Hadoop Core
> Issue Type: New Feature
> Affects Versions: 0.20.0
> Reporter: Rajiv Chittajallu
> Assignee: Amar Kamat
> Attachments: HADOOP-5643-v3.4.patch, HADOOP-5643-v4.0.patch, HADOOP-5643-v4.6.patch, HADOOP-5643-v5.1.patch, HADOOP-5643-v5.12-testcase.patch, HADOOP-5643-v5.12.patch, HADOOP-5643-v5.5.patch, HADOOP-5643-v5.9.patch
>
>
> Its not always possible to shutdown the tasktracker to stop scheduling tasks on the node. (eg you can't login to the node but the TT is up).
> This can be via
> * mapred.exclude and should be refreshed with out restarting the tasktracker
> * hadoop job -fail-tracker <tracker id>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5643) Ability to blacklist tasktracker
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12706114#action_12706114 ]
Amar Kamat commented on HADOOP-5643:
------------------------------------
Result of test patch.
{code}
[exec] +1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 15 new or modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
[exec]
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
{code}
Running ant test
> Ability to blacklist tasktracker
> --------------------------------
>
> Key: HADOOP-5643
> URL: https://issues.apache.org/jira/browse/HADOOP-5643
> Project: Hadoop Core
> Issue Type: New Feature
> Affects Versions: 0.20.0
> Reporter: Rajiv Chittajallu
> Assignee: Amar Kamat
> Attachments: HADOOP-5643-v3.4.patch, HADOOP-5643-v4.0.patch, HADOOP-5643-v4.6.patch, HADOOP-5643-v5.1.patch
>
>
> Its not always possible to shutdown the tasktracker to stop scheduling tasks on the node. (eg you can't login to the node but the TT is up).
> This can be via
> * mapred.exclude and should be refreshed with out restarting the tasktracker
> * hadoop job -fail-tracker <tracker id>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5643) Ability to blacklist tasktracker
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12708233#action_12708233 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-5643:
------------------------------------------------
- src/hdfs/org/apache/hadoop/hdfs/server/namenode/PermissionChecker.java becomes an empty file in trunk.
- Why renaming the FSNamesystem.checkSuperuserPrivilege() method to checkAccess() even though it is still for checking superuser privilege?
> Ability to blacklist tasktracker
> --------------------------------
>
> Key: HADOOP-5643
> URL: https://issues.apache.org/jira/browse/HADOOP-5643
> Project: Hadoop Core
> Issue Type: New Feature
> Affects Versions: 0.20.0
> Reporter: Rajiv Chittajallu
> Assignee: Amar Kamat
> Fix For: 0.21.0
>
> Attachments: HADOOP-5643-v3.4.patch, HADOOP-5643-v4.0.patch, HADOOP-5643-v4.6.patch, HADOOP-5643-v5.1.patch, HADOOP-5643-v5.12-testcase.patch, HADOOP-5643-v5.12.patch, HADOOP-5643-v5.5.patch, HADOOP-5643-v5.9.patch
>
>
> Its not always possible to shutdown the tasktracker to stop scheduling tasks on the node. (eg you can't login to the node but the TT is up).
> This can be via
> * mapred.exclude and should be refreshed with out restarting the tasktracker
> * hadoop job -fail-tracker <tracker id>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5643) Ability to blacklist tasktracker
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amar Kamat updated HADOOP-5643:
-------------------------------
Attachment: HADOOP-5643-v5.9.patch
Attaching a patch incorporating Devaraj's offline comment. Result of test-patch
{code}
[exec] +1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 15 new or modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
[exec]
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
{code}
Ant tests passed on my box.
> Ability to blacklist tasktracker
> --------------------------------
>
> Key: HADOOP-5643
> URL: https://issues.apache.org/jira/browse/HADOOP-5643
> Project: Hadoop Core
> Issue Type: New Feature
> Affects Versions: 0.20.0
> Reporter: Rajiv Chittajallu
> Assignee: Amar Kamat
> Attachments: HADOOP-5643-v3.4.patch, HADOOP-5643-v4.0.patch, HADOOP-5643-v4.6.patch, HADOOP-5643-v5.1.patch, HADOOP-5643-v5.5.patch, HADOOP-5643-v5.9.patch
>
>
> Its not always possible to shutdown the tasktracker to stop scheduling tasks on the node. (eg you can't login to the node but the TT is up).
> This can be via
> * mapred.exclude and should be refreshed with out restarting the tasktracker
> * hadoop job -fail-tracker <tracker id>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5643) Ability to blacklist tasktracker
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12704577#action_12704577 ]
Amar Kamat commented on HADOOP-5643:
------------------------------------
While testing the patch, we found that manually changing the excludes file (maintained by the jobtracker) results in checksum error. So, for now we think keeping it as it is (i.e using java to write/read) makes more sense. Also here are some of the test bugs :
# documentation for command line is not sufficient
# passing a wrong file to hosts reader causes the earlier data to be lost. So the correct way would be to first load the new file and on success replace the internal structures.
# web ui doesnt work as expected.
Will upload a new patch soon.
> Ability to blacklist tasktracker
> --------------------------------
>
> Key: HADOOP-5643
> URL: https://issues.apache.org/jira/browse/HADOOP-5643
> Project: Hadoop Core
> Issue Type: New Feature
> Affects Versions: 0.20.0
> Reporter: Rajiv Chittajallu
> Assignee: Amar Kamat
> Attachments: HADOOP-5643-v3.4.patch, HADOOP-5643-v4.0.patch
>
>
> Its not always possible to shutdown the tasktracker to stop scheduling tasks on the node. (eg you can't login to the node but the TT is up).
> This can be via
> * mapred.exclude and should be refreshed with out restarting the tasktracker
> * hadoop job -fail-tracker <tracker id>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5643) Ability to blacklist tasktracker
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12706192#action_12706192 ]
Amar Kamat commented on HADOOP-5643:
------------------------------------
Ant tests passed on my box.
> Ability to blacklist tasktracker
> --------------------------------
>
> Key: HADOOP-5643
> URL: https://issues.apache.org/jira/browse/HADOOP-5643
> Project: Hadoop Core
> Issue Type: New Feature
> Affects Versions: 0.20.0
> Reporter: Rajiv Chittajallu
> Assignee: Amar Kamat
> Attachments: HADOOP-5643-v3.4.patch, HADOOP-5643-v4.0.patch, HADOOP-5643-v4.6.patch, HADOOP-5643-v5.1.patch
>
>
> Its not always possible to shutdown the tasktracker to stop scheduling tasks on the node. (eg you can't login to the node but the TT is up).
> This can be via
> * mapred.exclude and should be refreshed with out restarting the tasktracker
> * hadoop job -fail-tracker <tracker id>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5643) Ability to blacklist tasktracker
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12701383#action_12701383 ]
Owen O'Malley commented on HADOOP-5643:
---------------------------------------
I think we should also be able to do this via the web ui, which is very convenient.
There should be a way to make it not black listed any more.
It should be persistent across job tracker restarts.
It probably should be decommissioning instead of black listing.
It should probably start rerunning all of the running and stored tasks. (including the map outputs that are stored there)
> Ability to blacklist tasktracker
> --------------------------------
>
> Key: HADOOP-5643
> URL: https://issues.apache.org/jira/browse/HADOOP-5643
> Project: Hadoop Core
> Issue Type: New Feature
> Affects Versions: 0.20.0
> Reporter: Rajiv Chittajallu
>
> Its not always possible to shutdown the tasktracker to stop scheduling tasks on the node. (eg you can't login to the node but the TT is up).
> This can be via
> * mapred.exclude and should be refreshed with out restarting the tasktracker
> * hadoop job -fail-tracker <tracker id>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5643) Ability to blacklist tasktracker
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12703512#action_12703512 ]
Amar Kamat commented on HADOOP-5643:
------------------------------------
Had an offline discussion with Devaraj and we think it makes sense to provide a default location for mapred.hosts.exclude. The purpose of doing this is to provide persistence. The default file would be something like ${hadoop.log.dir}/history/hosts.exclude. By default the jobtracker persists the decommission/recommission host info in this file.
@Eric
I think we should do what we do for job killing i.e private actions thingy. There is always the option to do it either via {{refresh}} or {{-decommission/-recommission}} command line option. All the admin operations at the jobtracker will be checked for owner access. For now I think only the user who runs the jobtracker should be allowed to fire these commands. Thoughts?
> Ability to blacklist tasktracker
> --------------------------------
>
> Key: HADOOP-5643
> URL: https://issues.apache.org/jira/browse/HADOOP-5643
> Project: Hadoop Core
> Issue Type: New Feature
> Affects Versions: 0.20.0
> Reporter: Rajiv Chittajallu
> Assignee: Amar Kamat
>
> Its not always possible to shutdown the tasktracker to stop scheduling tasks on the node. (eg you can't login to the node but the TT is up).
> This can be via
> * mapred.exclude and should be refreshed with out restarting the tasktracker
> * hadoop job -fail-tracker <tracker id>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5643) Ability to blacklist tasktracker
Posted by "Robert Chansler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Chansler updated HADOOP-5643:
------------------------------------
Attachment: Fixed 5643-0.20
Example patch for 0.20 not to be committed.
> Ability to blacklist tasktracker
> --------------------------------
>
> Key: HADOOP-5643
> URL: https://issues.apache.org/jira/browse/HADOOP-5643
> Project: Hadoop Core
> Issue Type: New Feature
> Affects Versions: 0.20.0
> Reporter: Rajiv Chittajallu
> Assignee: Amar Kamat
> Fix For: 0.21.0
>
> Attachments: Fixed 5643-0.20, HADOOP-5643-v3.4.patch, HADOOP-5643-v4.0.patch, HADOOP-5643-v4.6.patch, HADOOP-5643-v5.1.patch, HADOOP-5643-v5.12-testcase.patch, HADOOP-5643-v5.12.patch, HADOOP-5643-v5.5.patch, HADOOP-5643-v5.9.patch
>
>
> Its not always possible to shutdown the tasktracker to stop scheduling tasks on the node. (eg you can't login to the node but the TT is up).
> This can be via
> * mapred.exclude and should be refreshed with out restarting the tasktracker
> * hadoop job -fail-tracker <tracker id>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5643) Ability to blacklist tasktracker
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amar Kamat updated HADOOP-5643:
-------------------------------
Attachment: HADOOP-5643-v4.6.patch
Attaching a patch fixing some bugs. Result of test-patch
{code}
[exec] +1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 15 new or modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
[exec]
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings
{code}
Running ant test now. Will post the results of ant test and cluster run.
> Ability to blacklist tasktracker
> --------------------------------
>
> Key: HADOOP-5643
> URL: https://issues.apache.org/jira/browse/HADOOP-5643
> Project: Hadoop Core
> Issue Type: New Feature
> Affects Versions: 0.20.0
> Reporter: Rajiv Chittajallu
> Assignee: Amar Kamat
> Attachments: HADOOP-5643-v3.4.patch, HADOOP-5643-v4.0.patch, HADOOP-5643-v4.6.patch
>
>
> Its not always possible to shutdown the tasktracker to stop scheduling tasks on the node. (eg you can't login to the node but the TT is up).
> This can be via
> * mapred.exclude and should be refreshed with out restarting the tasktracker
> * hadoop job -fail-tracker <tracker id>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HADOOP-5643) Ability to blacklist tasktracker
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devaraj Das resolved HADOOP-5643.
---------------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
I just committed this. Thanks, Amar! (Please add a release note describing the way to run the command for decommissioning TTs)
> Ability to blacklist tasktracker
> --------------------------------
>
> Key: HADOOP-5643
> URL: https://issues.apache.org/jira/browse/HADOOP-5643
> Project: Hadoop Core
> Issue Type: New Feature
> Affects Versions: 0.20.0
> Reporter: Rajiv Chittajallu
> Assignee: Amar Kamat
> Attachments: HADOOP-5643-v3.4.patch, HADOOP-5643-v4.0.patch, HADOOP-5643-v4.6.patch, HADOOP-5643-v5.1.patch, HADOOP-5643-v5.12-testcase.patch, HADOOP-5643-v5.12.patch, HADOOP-5643-v5.5.patch, HADOOP-5643-v5.9.patch
>
>
> Its not always possible to shutdown the tasktracker to stop scheduling tasks on the node. (eg you can't login to the node but the TT is up).
> This can be via
> * mapred.exclude and should be refreshed with out restarting the tasktracker
> * hadoop job -fail-tracker <tracker id>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5643) Ability to blacklist tasktracker
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12707637#action_12707637 ]
Amar Kamat commented on HADOOP-5643:
------------------------------------
Ant tests passed on my box.
> Ability to blacklist tasktracker
> --------------------------------
>
> Key: HADOOP-5643
> URL: https://issues.apache.org/jira/browse/HADOOP-5643
> Project: Hadoop Core
> Issue Type: New Feature
> Affects Versions: 0.20.0
> Reporter: Rajiv Chittajallu
> Assignee: Amar Kamat
> Attachments: HADOOP-5643-v3.4.patch, HADOOP-5643-v4.0.patch, HADOOP-5643-v4.6.patch, HADOOP-5643-v5.1.patch, HADOOP-5643-v5.12.patch, HADOOP-5643-v5.5.patch, HADOOP-5643-v5.9.patch
>
>
> Its not always possible to shutdown the tasktracker to stop scheduling tasks on the node. (eg you can't login to the node but the TT is up).
> This can be via
> * mapred.exclude and should be refreshed with out restarting the tasktracker
> * hadoop job -fail-tracker <tracker id>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5643) Ability to blacklist tasktracker
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12708749#action_12708749 ]
Amar Kamat commented on HADOOP-5643:
------------------------------------
Nicholas,
If checkAccess() adds to confusion then we better revert the renaming.
> Ability to blacklist tasktracker
> --------------------------------
>
> Key: HADOOP-5643
> URL: https://issues.apache.org/jira/browse/HADOOP-5643
> Project: Hadoop Core
> Issue Type: New Feature
> Affects Versions: 0.20.0
> Reporter: Rajiv Chittajallu
> Assignee: Amar Kamat
> Fix For: 0.21.0
>
> Attachments: HADOOP-5643-v3.4.patch, HADOOP-5643-v4.0.patch, HADOOP-5643-v4.6.patch, HADOOP-5643-v5.1.patch, HADOOP-5643-v5.12-testcase.patch, HADOOP-5643-v5.12.patch, HADOOP-5643-v5.5.patch, HADOOP-5643-v5.9.patch
>
>
> Its not always possible to shutdown the tasktracker to stop scheduling tasks on the node. (eg you can't login to the node but the TT is up).
> This can be via
> * mapred.exclude and should be refreshed with out restarting the tasktracker
> * hadoop job -fail-tracker <tracker id>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5643) Ability to blacklist tasktracker
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amar Kamat updated HADOOP-5643:
-------------------------------
Attachment: HADOOP-5643-v4.0.patch
Attaching a patch with the test case. Result of test-patch
{code}
[exec] +1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 15 new or modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
[exec]
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
{code}
Testing in progress.
> Ability to blacklist tasktracker
> --------------------------------
>
> Key: HADOOP-5643
> URL: https://issues.apache.org/jira/browse/HADOOP-5643
> Project: Hadoop Core
> Issue Type: New Feature
> Affects Versions: 0.20.0
> Reporter: Rajiv Chittajallu
> Assignee: Amar Kamat
> Attachments: HADOOP-5643-v3.4.patch, HADOOP-5643-v4.0.patch
>
>
> Its not always possible to shutdown the tasktracker to stop scheduling tasks on the node. (eg you can't login to the node but the TT is up).
> This can be via
> * mapred.exclude and should be refreshed with out restarting the tasktracker
> * hadoop job -fail-tracker <tracker id>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5643) Ability to blacklist tasktracker
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12706135#action_12706135 ]
Arun C Murthy commented on HADOOP-5643:
---------------------------------------
bq.I think we are going through an expensive process of reinventing the wheel here. We should think about solving this sort of issue once by maintain such lists in a plugable source of configuration and supporting the ability to "hup" the service.
+1. I've opened HADOOP-5772 for the same.
> Ability to blacklist tasktracker
> --------------------------------
>
> Key: HADOOP-5643
> URL: https://issues.apache.org/jira/browse/HADOOP-5643
> Project: Hadoop Core
> Issue Type: New Feature
> Affects Versions: 0.20.0
> Reporter: Rajiv Chittajallu
> Assignee: Amar Kamat
> Attachments: HADOOP-5643-v3.4.patch, HADOOP-5643-v4.0.patch, HADOOP-5643-v4.6.patch, HADOOP-5643-v5.1.patch
>
>
> Its not always possible to shutdown the tasktracker to stop scheduling tasks on the node. (eg you can't login to the node but the TT is up).
> This can be via
> * mapred.exclude and should be refreshed with out restarting the tasktracker
> * hadoop job -fail-tracker <tracker id>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5643) Ability to blacklist tasktracker
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12701438#action_12701438 ]
Amar Kamat commented on HADOOP-5643:
------------------------------------
I think calling this as blacklisting will lead to more confusion. As Owen suggested we can call it as *decommissioning/recommissioning* of trackers which would essentially mean that irrespective of what state the tracker is, the jobtracker is asked to decommission(rerun+ignore)/recommission(add back) it. So the command would be
_bin/hadoop jobtracker -decommission tracker1,tracker2...._ and _bin/hadoop jobtracker -recommission tracker1,tracker2...._.
All the running tasks (also completed maps) that were launched on that machine will be killed and rerun. We can reuse the lost-tracker code for doing this. Maybe a thread should be started on demand (similar to cleanup queue thread) for a decommissioning request. Also these tracker will be added to the ignore list (i.e issue a 'shutdown' upon contact). So a decommission request is equivalent to lost-tracker + add-to-ignore-list.
Upon a recommission, the trackers will be removed from the ignore list. This can be done inline.
>From the webui, a simple checkbox against all the trackers can be provided and an action named 'Decommission' can be provided (similar to actions for jobs on jobtracker.jsp). On the trackers page, we can provide another section for decommissioned trackers and there we can provide a checkbox for recommissioning it.
Note :
1) Acls check should be done before decommissioning and recommissioning.
2) This info needs to be persisted. Upon every decommission/recommission, persist this info to system.dir/jobtracker.info
3) Upon restart, the ignore list will also be recovered and loaded (i.e invoke jobtracker.decommission(recovered-list) from recovery-manager)
4) These new apis can be added to the TaskTrackerManager interface as there really are tasktracker level actions.
----
Thoughts?
> Ability to blacklist tasktracker
> --------------------------------
>
> Key: HADOOP-5643
> URL: https://issues.apache.org/jira/browse/HADOOP-5643
> Project: Hadoop Core
> Issue Type: New Feature
> Affects Versions: 0.20.0
> Reporter: Rajiv Chittajallu
> Assignee: Amar Kamat
>
> Its not always possible to shutdown the tasktracker to stop scheduling tasks on the node. (eg you can't login to the node but the TT is up).
> This can be via
> * mapred.exclude and should be refreshed with out restarting the tasktracker
> * hadoop job -fail-tracker <tracker id>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5643) Ability to blacklist tasktracker
Posted by "Rajiv Chittajallu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12704691#action_12704691 ]
Rajiv Chittajallu commented on HADOOP-5643:
-------------------------------------------
>While testing the patch, we found that manually changing the excludes file (maintained by the jobtracker) results in checksum error.
For the name NameNode, this is allowed. JT should do the same.
> Ability to blacklist tasktracker
> --------------------------------
>
> Key: HADOOP-5643
> URL: https://issues.apache.org/jira/browse/HADOOP-5643
> Project: Hadoop Core
> Issue Type: New Feature
> Affects Versions: 0.20.0
> Reporter: Rajiv Chittajallu
> Assignee: Amar Kamat
> Attachments: HADOOP-5643-v3.4.patch, HADOOP-5643-v4.0.patch
>
>
> Its not always possible to shutdown the tasktracker to stop scheduling tasks on the node. (eg you can't login to the node but the TT is up).
> This can be via
> * mapred.exclude and should be refreshed with out restarting the tasktracker
> * hadoop job -fail-tracker <tracker id>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5643) Ability to blacklist tasktracker
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amar Kamat updated HADOOP-5643:
-------------------------------
Fix Version/s: 0.21.0
Release Note: Added the functionality to refresh jobtrackers node list via command line (bin/hadoop mradmin -refreshNodes). The command should be run as the jobtracker owner (jobtracker process owner) or from a super group (mapred.permissions.supergroup).
> Ability to blacklist tasktracker
> --------------------------------
>
> Key: HADOOP-5643
> URL: https://issues.apache.org/jira/browse/HADOOP-5643
> Project: Hadoop Core
> Issue Type: New Feature
> Affects Versions: 0.20.0
> Reporter: Rajiv Chittajallu
> Assignee: Amar Kamat
> Fix For: 0.21.0
>
> Attachments: HADOOP-5643-v3.4.patch, HADOOP-5643-v4.0.patch, HADOOP-5643-v4.6.patch, HADOOP-5643-v5.1.patch, HADOOP-5643-v5.12-testcase.patch, HADOOP-5643-v5.12.patch, HADOOP-5643-v5.5.patch, HADOOP-5643-v5.9.patch
>
>
> Its not always possible to shutdown the tasktracker to stop scheduling tasks on the node. (eg you can't login to the node but the TT is up).
> This can be via
> * mapred.exclude and should be refreshed with out restarting the tasktracker
> * hadoop job -fail-tracker <tracker id>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5643) Ability to blacklist tasktracker
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12708638#action_12708638 ]
Hudson commented on HADOOP-5643:
--------------------------------
Integrated in Hadoop-trunk #834 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/834/])
. Removing the empty file src/hdfs/org/apache/hadoop/hdfs/server/namenode/PermissionChecker.java.
> Ability to blacklist tasktracker
> --------------------------------
>
> Key: HADOOP-5643
> URL: https://issues.apache.org/jira/browse/HADOOP-5643
> Project: Hadoop Core
> Issue Type: New Feature
> Affects Versions: 0.20.0
> Reporter: Rajiv Chittajallu
> Assignee: Amar Kamat
> Fix For: 0.21.0
>
> Attachments: HADOOP-5643-v3.4.patch, HADOOP-5643-v4.0.patch, HADOOP-5643-v4.6.patch, HADOOP-5643-v5.1.patch, HADOOP-5643-v5.12-testcase.patch, HADOOP-5643-v5.12.patch, HADOOP-5643-v5.5.patch, HADOOP-5643-v5.9.patch
>
>
> Its not always possible to shutdown the tasktracker to stop scheduling tasks on the node. (eg you can't login to the node but the TT is up).
> This can be via
> * mapred.exclude and should be refreshed with out restarting the tasktracker
> * hadoop job -fail-tracker <tracker id>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5643) Ability to blacklist tasktracker
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12708549#action_12708549 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-5643:
------------------------------------------------
bq. I feel checkSuperuserPrivilege() should be used for simply checking superuser privilege (without permission switch which is just in HDFS for now) and checkAccess() for making a guarded call to checkSuperuserPrivilege(). The reason for doing this was to keep both the MR and HDFS consistent wrt superuser checks.
Then, why not using the name "checkSuperuserPrivilege" for superuser checks in both HDFS and MR? "checkAccess" does not seem to mean "check superuser".
Also, "checkAccess" seems to be confusing in HDFS. In FSNamesystem, there are other methods checkPathAccess(..), checkParentAccess(..) and checkAncestorAccess(..) which are nothing to do with superuser.
> Ability to blacklist tasktracker
> --------------------------------
>
> Key: HADOOP-5643
> URL: https://issues.apache.org/jira/browse/HADOOP-5643
> Project: Hadoop Core
> Issue Type: New Feature
> Affects Versions: 0.20.0
> Reporter: Rajiv Chittajallu
> Assignee: Amar Kamat
> Fix For: 0.21.0
>
> Attachments: HADOOP-5643-v3.4.patch, HADOOP-5643-v4.0.patch, HADOOP-5643-v4.6.patch, HADOOP-5643-v5.1.patch, HADOOP-5643-v5.12-testcase.patch, HADOOP-5643-v5.12.patch, HADOOP-5643-v5.5.patch, HADOOP-5643-v5.9.patch
>
>
> Its not always possible to shutdown the tasktracker to stop scheduling tasks on the node. (eg you can't login to the node but the TT is up).
> This can be via
> * mapred.exclude and should be refreshed with out restarting the tasktracker
> * hadoop job -fail-tracker <tracker id>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.