You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Todd Lipcon (JIRA)" <ji...@apache.org> on 2012/09/25 21:30:09 UTC
[jira] [Created] (HDFS-3972) Trash emptier fails in secure cluster
Todd Lipcon created HDFS-3972:
---------------------------------
Summary: Trash emptier fails in secure cluster
Key: HDFS-3972
URL: https://issues.apache.org/jira/browse/HDFS-3972
Project: Hadoop HDFS
Issue Type: Bug
Components: name-node
Affects Versions: 2.0.1-alpha, 3.0.0
Reporter: Todd Lipcon
Assignee: Todd Lipcon
Priority: Critical
In a secure cluster, we're seeing the following issue on the NN when the trash emptier tries to run:
WARN org.apache.hadoop.fs.TrashPolicyDefault: Trash can't list homes: java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host \
is: "xxxxx"; destination host is: "xxxx":8020; Sleeping.
The issue seems to be that the trash emptier thread sends RPCs back to itself, but isn't wrapped in a doAs. Credit goes to Stephen Chu for discovering this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3972) Trash emptier fails in secure HA
cluster
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13464693#comment-13464693 ]
Hudson commented on HDFS-3972:
------------------------------
Integrated in Hadoop-Hdfs-trunk #1178 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1178/])
HDFS-3972. Trash emptier fails in secure HA cluster. Contributed by Todd Lipcon (Revision 1390729)
Result = SUCCESS
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1390729
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/TrashPolicyDefault.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
> Trash emptier fails in secure HA cluster
> ----------------------------------------
>
> Key: HDFS-3972
> URL: https://issues.apache.org/jira/browse/HDFS-3972
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
> Affects Versions: 3.0.0, 2.0.1-alpha
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 2.0.2-alpha
>
> Attachments: hdfs-3972.txt
>
>
> In a secure HA cluster, we're seeing the following issue on the NN when the trash emptier tries to run:
> WARN org.apache.hadoop.fs.TrashPolicyDefault: Trash can't list homes: java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host \
> is: "xxxxx"; destination host is: "xxxx":8020; Sleeping.
> The issue seems to be that the trash emptier thread sends RPCs back to itself, but isn't wrapped in a doAs. Credit goes to Stephen Chu for discovering this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3972) Trash emptier fails in secure HA
cluster
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13464169#comment-13464169 ]
Eli Collins commented on HDFS-3972:
-----------------------------------
Test failures are unrelated:
HDFS-3811 - TestPersistBlocks#TestRestartDfsWithFlush appears to be flaky
HDFS-2434 - TestNameNodeMetrics.testCorruptBlock fails intermittently
HDFS-3948 - TestWebHDFS#testNamenodeRestart is racy
HADOOP-8591- TestZKFailoverController tests time out
> Trash emptier fails in secure HA cluster
> ----------------------------------------
>
> Key: HDFS-3972
> URL: https://issues.apache.org/jira/browse/HDFS-3972
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
> Affects Versions: 3.0.0, 2.0.1-alpha
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Attachments: hdfs-3972.txt
>
>
> In a secure HA cluster, we're seeing the following issue on the NN when the trash emptier tries to run:
> WARN org.apache.hadoop.fs.TrashPolicyDefault: Trash can't list homes: java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host \
> is: "xxxxx"; destination host is: "xxxx":8020; Sleeping.
> The issue seems to be that the trash emptier thread sends RPCs back to itself, but isn't wrapped in a doAs. Credit goes to Stephen Chu for discovering this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3972) Trash emptier fails in secure HA
cluster
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13464747#comment-13464747 ]
Hudson commented on HDFS-3972:
------------------------------
Integrated in Hadoop-Mapreduce-trunk #1209 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1209/])
HDFS-3972. Trash emptier fails in secure HA cluster. Contributed by Todd Lipcon (Revision 1390729)
Result = SUCCESS
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1390729
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/TrashPolicyDefault.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
> Trash emptier fails in secure HA cluster
> ----------------------------------------
>
> Key: HDFS-3972
> URL: https://issues.apache.org/jira/browse/HDFS-3972
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
> Affects Versions: 3.0.0, 2.0.1-alpha
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 2.0.2-alpha
>
> Attachments: hdfs-3972.txt
>
>
> In a secure HA cluster, we're seeing the following issue on the NN when the trash emptier tries to run:
> WARN org.apache.hadoop.fs.TrashPolicyDefault: Trash can't list homes: java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host \
> is: "xxxxx"; destination host is: "xxxx":8020; Sleeping.
> The issue seems to be that the trash emptier thread sends RPCs back to itself, but isn't wrapped in a doAs. Credit goes to Stephen Chu for discovering this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3972) Trash emptier fails in secure HA
cluster
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HDFS-3972:
------------------------------
Attachment: hdfs-3972.txt
The root cause of the issue is that, in HA, the TrashEmptier is started from the context of an RPC, rather than the context of the NN startup code. So, it was running as the administrator who called transitionToActive (remote non-keytab user) rather than the context of the NN login user.
Attached patch fixes the issue. It doesn't include a new unit test, since the issue is only on secure clusters, which aren't automatable.
I tested manually as follows:
- configure a secure HA cluster
- configure fs.trash.interval and fs.trash.checkpoint.interval = 1 in core-site.xml
- started NN and transitioned to active
-- without the patch, I saw the error mentioned above. with the patch, no error
- created and deleted a file, which got moved to trash
- waited a minute, and saw that the trash checkpointer moved the file correctly
-- without the patch, I saw the error again. With the patch, it worked.
> Trash emptier fails in secure HA cluster
> ----------------------------------------
>
> Key: HDFS-3972
> URL: https://issues.apache.org/jira/browse/HDFS-3972
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
> Affects Versions: 3.0.0, 2.0.1-alpha
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Attachments: hdfs-3972.txt
>
>
> In a secure HA cluster, we're seeing the following issue on the NN when the trash emptier tries to run:
> WARN org.apache.hadoop.fs.TrashPolicyDefault: Trash can't list homes: java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host \
> is: "xxxxx"; destination host is: "xxxx":8020; Sleeping.
> The issue seems to be that the trash emptier thread sends RPCs back to itself, but isn't wrapped in a doAs. Credit goes to Stephen Chu for discovering this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3972) Trash emptier fails in secure HA
cluster
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13463335#comment-13463335 ]
Eli Collins commented on HDFS-3972:
-----------------------------------
+1 pending jenkins, looks great
> Trash emptier fails in secure HA cluster
> ----------------------------------------
>
> Key: HDFS-3972
> URL: https://issues.apache.org/jira/browse/HDFS-3972
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
> Affects Versions: 3.0.0, 2.0.1-alpha
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Attachments: hdfs-3972.txt
>
>
> In a secure HA cluster, we're seeing the following issue on the NN when the trash emptier tries to run:
> WARN org.apache.hadoop.fs.TrashPolicyDefault: Trash can't list homes: java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host \
> is: "xxxxx"; destination host is: "xxxx":8020; Sleeping.
> The issue seems to be that the trash emptier thread sends RPCs back to itself, but isn't wrapped in a doAs. Credit goes to Stephen Chu for discovering this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3972) Trash emptier fails in secure HA
cluster
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HDFS-3972:
------------------------------
Description:
In a secure HA cluster, we're seeing the following issue on the NN when the trash emptier tries to run:
WARN org.apache.hadoop.fs.TrashPolicyDefault: Trash can't list homes: java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host \
is: "xxxxx"; destination host is: "xxxx":8020; Sleeping.
The issue seems to be that the trash emptier thread sends RPCs back to itself, but isn't wrapped in a doAs. Credit goes to Stephen Chu for discovering this.
was:
In a secure cluster, we're seeing the following issue on the NN when the trash emptier tries to run:
WARN org.apache.hadoop.fs.TrashPolicyDefault: Trash can't list homes: java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host \
is: "xxxxx"; destination host is: "xxxx":8020; Sleeping.
The issue seems to be that the trash emptier thread sends RPCs back to itself, but isn't wrapped in a doAs. Credit goes to Stephen Chu for discovering this.
Summary: Trash emptier fails in secure HA cluster (was: Trash emptier fails in secure cluster)
edit: updated Summary and Description to indicate this only happens when HA is enabled on the secure cluster
> Trash emptier fails in secure HA cluster
> ----------------------------------------
>
> Key: HDFS-3972
> URL: https://issues.apache.org/jira/browse/HDFS-3972
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
> Affects Versions: 3.0.0, 2.0.1-alpha
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
>
> In a secure HA cluster, we're seeing the following issue on the NN when the trash emptier tries to run:
> WARN org.apache.hadoop.fs.TrashPolicyDefault: Trash can't list homes: java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host \
> is: "xxxxx"; destination host is: "xxxx":8020; Sleeping.
> The issue seems to be that the trash emptier thread sends RPCs back to itself, but isn't wrapped in a doAs. Credit goes to Stephen Chu for discovering this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (HDFS-3972) Trash emptier fails in secure HA
cluster
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy closed HDFS-3972.
-------------------------------
> Trash emptier fails in secure HA cluster
> ----------------------------------------
>
> Key: HDFS-3972
> URL: https://issues.apache.org/jira/browse/HDFS-3972
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
> Affects Versions: 3.0.0, 2.0.1-alpha
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 2.0.2-alpha
>
> Attachments: hdfs-3972.txt
>
>
> In a secure HA cluster, we're seeing the following issue on the NN when the trash emptier tries to run:
> WARN org.apache.hadoop.fs.TrashPolicyDefault: Trash can't list homes: java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host \
> is: "xxxxx"; destination host is: "xxxx":8020; Sleeping.
> The issue seems to be that the trash emptier thread sends RPCs back to itself, but isn't wrapped in a doAs. Credit goes to Stephen Chu for discovering this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3972) Trash emptier fails in secure HA
cluster
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13464246#comment-13464246 ]
Hudson commented on HDFS-3972:
------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #2799 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2799/])
HDFS-3972. Trash emptier fails in secure HA cluster. Contributed by Todd Lipcon (Revision 1390729)
Result = ABORTED
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1390729
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/TrashPolicyDefault.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
> Trash emptier fails in secure HA cluster
> ----------------------------------------
>
> Key: HDFS-3972
> URL: https://issues.apache.org/jira/browse/HDFS-3972
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
> Affects Versions: 3.0.0, 2.0.1-alpha
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 2.0.2-alpha
>
> Attachments: hdfs-3972.txt
>
>
> In a secure HA cluster, we're seeing the following issue on the NN when the trash emptier tries to run:
> WARN org.apache.hadoop.fs.TrashPolicyDefault: Trash can't list homes: java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host \
> is: "xxxxx"; destination host is: "xxxx":8020; Sleeping.
> The issue seems to be that the trash emptier thread sends RPCs back to itself, but isn't wrapped in a doAs. Credit goes to Stephen Chu for discovering this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3972) Trash emptier fails in secure HA
cluster
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13464214#comment-13464214 ]
Hudson commented on HDFS-3972:
------------------------------
Integrated in Hadoop-Common-trunk-Commit #2777 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2777/])
HDFS-3972. Trash emptier fails in secure HA cluster. Contributed by Todd Lipcon (Revision 1390729)
Result = SUCCESS
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1390729
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/TrashPolicyDefault.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
> Trash emptier fails in secure HA cluster
> ----------------------------------------
>
> Key: HDFS-3972
> URL: https://issues.apache.org/jira/browse/HDFS-3972
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
> Affects Versions: 3.0.0, 2.0.1-alpha
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 2.0.2-alpha
>
> Attachments: hdfs-3972.txt
>
>
> In a secure HA cluster, we're seeing the following issue on the NN when the trash emptier tries to run:
> WARN org.apache.hadoop.fs.TrashPolicyDefault: Trash can't list homes: java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host \
> is: "xxxxx"; destination host is: "xxxx":8020; Sleeping.
> The issue seems to be that the trash emptier thread sends RPCs back to itself, but isn't wrapped in a doAs. Credit goes to Stephen Chu for discovering this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3972) Trash emptier fails in secure HA
cluster
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13464221#comment-13464221 ]
Hudson commented on HDFS-3972:
------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #2840 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2840/])
HDFS-3972. Trash emptier fails in secure HA cluster. Contributed by Todd Lipcon (Revision 1390729)
Result = SUCCESS
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1390729
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/TrashPolicyDefault.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
> Trash emptier fails in secure HA cluster
> ----------------------------------------
>
> Key: HDFS-3972
> URL: https://issues.apache.org/jira/browse/HDFS-3972
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
> Affects Versions: 3.0.0, 2.0.1-alpha
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 2.0.2-alpha
>
> Attachments: hdfs-3972.txt
>
>
> In a secure HA cluster, we're seeing the following issue on the NN when the trash emptier tries to run:
> WARN org.apache.hadoop.fs.TrashPolicyDefault: Trash can't list homes: java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host \
> is: "xxxxx"; destination host is: "xxxx":8020; Sleeping.
> The issue seems to be that the trash emptier thread sends RPCs back to itself, but isn't wrapped in a doAs. Credit goes to Stephen Chu for discovering this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3972) Trash emptier fails in secure HA
cluster
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13463422#comment-13463422 ]
Hadoop QA commented on HDFS-3972:
---------------------------------
{color:red}-1 overall{color}. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12546574/hdfs-3972.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.ha.TestZKFailoverController
org.apache.hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics
org.apache.hadoop.hdfs.web.TestWebHDFS
org.apache.hadoop.hdfs.TestPersistBlocks
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3231//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3231//console
This message is automatically generated.
> Trash emptier fails in secure HA cluster
> ----------------------------------------
>
> Key: HDFS-3972
> URL: https://issues.apache.org/jira/browse/HDFS-3972
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
> Affects Versions: 3.0.0, 2.0.1-alpha
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Attachments: hdfs-3972.txt
>
>
> In a secure HA cluster, we're seeing the following issue on the NN when the trash emptier tries to run:
> WARN org.apache.hadoop.fs.TrashPolicyDefault: Trash can't list homes: java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host \
> is: "xxxxx"; destination host is: "xxxx":8020; Sleeping.
> The issue seems to be that the trash emptier thread sends RPCs back to itself, but isn't wrapped in a doAs. Credit goes to Stephen Chu for discovering this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3972) Trash emptier fails in secure HA
cluster
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated HDFS-3972:
------------------------------
Resolution: Fixed
Fix Version/s: 2.0.2-alpha
Target Version/s: (was: 3.0.0, 2.0.2-alpha)
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
I've committed this and merged to branch-2 and branch-2.0.2-alpha.
> Trash emptier fails in secure HA cluster
> ----------------------------------------
>
> Key: HDFS-3972
> URL: https://issues.apache.org/jira/browse/HDFS-3972
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
> Affects Versions: 3.0.0, 2.0.1-alpha
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Fix For: 2.0.2-alpha
>
> Attachments: hdfs-3972.txt
>
>
> In a secure HA cluster, we're seeing the following issue on the NN when the trash emptier tries to run:
> WARN org.apache.hadoop.fs.TrashPolicyDefault: Trash can't list homes: java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host \
> is: "xxxxx"; destination host is: "xxxx":8020; Sleeping.
> The issue seems to be that the trash emptier thread sends RPCs back to itself, but isn't wrapped in a doAs. Credit goes to Stephen Chu for discovering this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3972) Trash emptier fails in secure HA
cluster
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HDFS-3972:
------------------------------
Status: Patch Available (was: Open)
> Trash emptier fails in secure HA cluster
> ----------------------------------------
>
> Key: HDFS-3972
> URL: https://issues.apache.org/jira/browse/HDFS-3972
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
> Affects Versions: 2.0.1-alpha, 3.0.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Critical
> Attachments: hdfs-3972.txt
>
>
> In a secure HA cluster, we're seeing the following issue on the NN when the trash emptier tries to run:
> WARN org.apache.hadoop.fs.TrashPolicyDefault: Trash can't list homes: java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host \
> is: "xxxxx"; destination host is: "xxxx":8020; Sleeping.
> The issue seems to be that the trash emptier thread sends RPCs back to itself, but isn't wrapped in a doAs. Credit goes to Stephen Chu for discovering this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira