You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Todd Lipcon (JIRA)" <ji...@apache.org> on 2012/09/25 21:30:09 UTC

[jira] [Created] (HDFS-3972) Trash emptier fails in secure cluster

Todd Lipcon created HDFS-3972:
---------------------------------

             Summary: Trash emptier fails in secure cluster
                 Key: HDFS-3972
                 URL: https://issues.apache.org/jira/browse/HDFS-3972
             Project: Hadoop HDFS
          Issue Type: Bug
          Components: name-node
    Affects Versions: 2.0.1-alpha, 3.0.0
            Reporter: Todd Lipcon
            Assignee: Todd Lipcon
            Priority: Critical


In a secure cluster, we're seeing the following issue on the NN when the trash emptier tries to run:

WARN org.apache.hadoop.fs.TrashPolicyDefault: Trash can't list homes: java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host \
is: "xxxxx"; destination host is: "xxxx":8020;  Sleeping.

The issue seems to be that the trash emptier thread sends RPCs back to itself, but isn't wrapped in a doAs. Credit goes to Stephen Chu for discovering this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-3972) Trash emptier fails in secure HA cluster

Posted by "Hudson (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-3972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13464693#comment-13464693 ] 

Hudson commented on HDFS-3972:
------------------------------

Integrated in Hadoop-Hdfs-trunk #1178 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1178/])
    HDFS-3972. Trash emptier fails in secure HA cluster. Contributed by Todd Lipcon (Revision 1390729)

     Result = SUCCESS
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1390729
Files : 
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/TrashPolicyDefault.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java

                
> Trash emptier fails in secure HA cluster
> ----------------------------------------
>
>                 Key: HDFS-3972
>                 URL: https://issues.apache.org/jira/browse/HDFS-3972
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: name-node
>    Affects Versions: 3.0.0, 2.0.1-alpha
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>             Fix For: 2.0.2-alpha
>
>         Attachments: hdfs-3972.txt
>
>
> In a secure HA cluster, we're seeing the following issue on the NN when the trash emptier tries to run:
> WARN org.apache.hadoop.fs.TrashPolicyDefault: Trash can't list homes: java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host \
> is: "xxxxx"; destination host is: "xxxx":8020;  Sleeping.
> The issue seems to be that the trash emptier thread sends RPCs back to itself, but isn't wrapped in a doAs. Credit goes to Stephen Chu for discovering this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-3972) Trash emptier fails in secure HA cluster

Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-3972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13464169#comment-13464169 ] 

Eli Collins commented on HDFS-3972:
-----------------------------------

Test failures are unrelated:

HDFS-3811 - TestPersistBlocks#TestRestartDfsWithFlush appears to be flaky
HDFS-2434 - TestNameNodeMetrics.testCorruptBlock fails intermittently
HDFS-3948 - TestWebHDFS#testNamenodeRestart is racy
HADOOP-8591- TestZKFailoverController tests time out
                
> Trash emptier fails in secure HA cluster
> ----------------------------------------
>
>                 Key: HDFS-3972
>                 URL: https://issues.apache.org/jira/browse/HDFS-3972
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: name-node
>    Affects Versions: 3.0.0, 2.0.1-alpha
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>         Attachments: hdfs-3972.txt
>
>
> In a secure HA cluster, we're seeing the following issue on the NN when the trash emptier tries to run:
> WARN org.apache.hadoop.fs.TrashPolicyDefault: Trash can't list homes: java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host \
> is: "xxxxx"; destination host is: "xxxx":8020;  Sleeping.
> The issue seems to be that the trash emptier thread sends RPCs back to itself, but isn't wrapped in a doAs. Credit goes to Stephen Chu for discovering this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-3972) Trash emptier fails in secure HA cluster

Posted by "Hudson (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-3972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13464747#comment-13464747 ] 

Hudson commented on HDFS-3972:
------------------------------

Integrated in Hadoop-Mapreduce-trunk #1209 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1209/])
    HDFS-3972. Trash emptier fails in secure HA cluster. Contributed by Todd Lipcon (Revision 1390729)

     Result = SUCCESS
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1390729
Files : 
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/TrashPolicyDefault.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java

                
> Trash emptier fails in secure HA cluster
> ----------------------------------------
>
>                 Key: HDFS-3972
>                 URL: https://issues.apache.org/jira/browse/HDFS-3972
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: name-node
>    Affects Versions: 3.0.0, 2.0.1-alpha
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>             Fix For: 2.0.2-alpha
>
>         Attachments: hdfs-3972.txt
>
>
> In a secure HA cluster, we're seeing the following issue on the NN when the trash emptier tries to run:
> WARN org.apache.hadoop.fs.TrashPolicyDefault: Trash can't list homes: java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host \
> is: "xxxxx"; destination host is: "xxxx":8020;  Sleeping.
> The issue seems to be that the trash emptier thread sends RPCs back to itself, but isn't wrapped in a doAs. Credit goes to Stephen Chu for discovering this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HDFS-3972) Trash emptier fails in secure HA cluster

Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-3972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HDFS-3972:
------------------------------

    Attachment: hdfs-3972.txt

The root cause of the issue is that, in HA, the TrashEmptier is started from the context of an RPC, rather than the context of the NN startup code. So, it was running as the administrator who called transitionToActive (remote non-keytab user) rather than the context of the NN login user.

Attached patch fixes the issue. It doesn't include a new unit test, since the issue is only on secure clusters, which aren't automatable.

I tested manually as follows:
- configure a secure HA cluster
- configure fs.trash.interval and fs.trash.checkpoint.interval = 1 in core-site.xml
- started NN and transitioned to active
-- without the patch, I saw the error mentioned above. with the patch, no error
- created and deleted a file, which got moved to trash
- waited a minute, and saw that the trash checkpointer moved the file correctly
-- without the patch, I saw the error again. With the patch, it worked.


                
> Trash emptier fails in secure HA cluster
> ----------------------------------------
>
>                 Key: HDFS-3972
>                 URL: https://issues.apache.org/jira/browse/HDFS-3972
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: name-node
>    Affects Versions: 3.0.0, 2.0.1-alpha
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>         Attachments: hdfs-3972.txt
>
>
> In a secure HA cluster, we're seeing the following issue on the NN when the trash emptier tries to run:
> WARN org.apache.hadoop.fs.TrashPolicyDefault: Trash can't list homes: java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host \
> is: "xxxxx"; destination host is: "xxxx":8020;  Sleeping.
> The issue seems to be that the trash emptier thread sends RPCs back to itself, but isn't wrapped in a doAs. Credit goes to Stephen Chu for discovering this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-3972) Trash emptier fails in secure HA cluster

Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-3972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13463335#comment-13463335 ] 

Eli Collins commented on HDFS-3972:
-----------------------------------

+1 pending jenkins, looks great
                
> Trash emptier fails in secure HA cluster
> ----------------------------------------
>
>                 Key: HDFS-3972
>                 URL: https://issues.apache.org/jira/browse/HDFS-3972
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: name-node
>    Affects Versions: 3.0.0, 2.0.1-alpha
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>         Attachments: hdfs-3972.txt
>
>
> In a secure HA cluster, we're seeing the following issue on the NN when the trash emptier tries to run:
> WARN org.apache.hadoop.fs.TrashPolicyDefault: Trash can't list homes: java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host \
> is: "xxxxx"; destination host is: "xxxx":8020;  Sleeping.
> The issue seems to be that the trash emptier thread sends RPCs back to itself, but isn't wrapped in a doAs. Credit goes to Stephen Chu for discovering this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HDFS-3972) Trash emptier fails in secure HA cluster

Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-3972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HDFS-3972:
------------------------------

    Description: 
In a secure HA cluster, we're seeing the following issue on the NN when the trash emptier tries to run:

WARN org.apache.hadoop.fs.TrashPolicyDefault: Trash can't list homes: java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host \
is: "xxxxx"; destination host is: "xxxx":8020;  Sleeping.

The issue seems to be that the trash emptier thread sends RPCs back to itself, but isn't wrapped in a doAs. Credit goes to Stephen Chu for discovering this.

  was:
In a secure cluster, we're seeing the following issue on the NN when the trash emptier tries to run:

WARN org.apache.hadoop.fs.TrashPolicyDefault: Trash can't list homes: java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host \
is: "xxxxx"; destination host is: "xxxx":8020;  Sleeping.

The issue seems to be that the trash emptier thread sends RPCs back to itself, but isn't wrapped in a doAs. Credit goes to Stephen Chu for discovering this.

        Summary: Trash emptier fails in secure HA cluster  (was: Trash emptier fails in secure cluster)

edit: updated Summary and Description to indicate this only happens when HA is enabled on the secure cluster
                
> Trash emptier fails in secure HA cluster
> ----------------------------------------
>
>                 Key: HDFS-3972
>                 URL: https://issues.apache.org/jira/browse/HDFS-3972
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: name-node
>    Affects Versions: 3.0.0, 2.0.1-alpha
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>
> In a secure HA cluster, we're seeing the following issue on the NN when the trash emptier tries to run:
> WARN org.apache.hadoop.fs.TrashPolicyDefault: Trash can't list homes: java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host \
> is: "xxxxx"; destination host is: "xxxx":8020;  Sleeping.
> The issue seems to be that the trash emptier thread sends RPCs back to itself, but isn't wrapped in a doAs. Credit goes to Stephen Chu for discovering this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Closed] (HDFS-3972) Trash emptier fails in secure HA cluster

Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-3972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Arun C Murthy closed HDFS-3972.
-------------------------------

    
> Trash emptier fails in secure HA cluster
> ----------------------------------------
>
>                 Key: HDFS-3972
>                 URL: https://issues.apache.org/jira/browse/HDFS-3972
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: name-node
>    Affects Versions: 3.0.0, 2.0.1-alpha
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>             Fix For: 2.0.2-alpha
>
>         Attachments: hdfs-3972.txt
>
>
> In a secure HA cluster, we're seeing the following issue on the NN when the trash emptier tries to run:
> WARN org.apache.hadoop.fs.TrashPolicyDefault: Trash can't list homes: java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host \
> is: "xxxxx"; destination host is: "xxxx":8020;  Sleeping.
> The issue seems to be that the trash emptier thread sends RPCs back to itself, but isn't wrapped in a doAs. Credit goes to Stephen Chu for discovering this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-3972) Trash emptier fails in secure HA cluster

Posted by "Hudson (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-3972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13464246#comment-13464246 ] 

Hudson commented on HDFS-3972:
------------------------------

Integrated in Hadoop-Mapreduce-trunk-Commit #2799 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2799/])
    HDFS-3972. Trash emptier fails in secure HA cluster. Contributed by Todd Lipcon (Revision 1390729)

     Result = ABORTED
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1390729
Files : 
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/TrashPolicyDefault.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java

                
> Trash emptier fails in secure HA cluster
> ----------------------------------------
>
>                 Key: HDFS-3972
>                 URL: https://issues.apache.org/jira/browse/HDFS-3972
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: name-node
>    Affects Versions: 3.0.0, 2.0.1-alpha
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>             Fix For: 2.0.2-alpha
>
>         Attachments: hdfs-3972.txt
>
>
> In a secure HA cluster, we're seeing the following issue on the NN when the trash emptier tries to run:
> WARN org.apache.hadoop.fs.TrashPolicyDefault: Trash can't list homes: java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host \
> is: "xxxxx"; destination host is: "xxxx":8020;  Sleeping.
> The issue seems to be that the trash emptier thread sends RPCs back to itself, but isn't wrapped in a doAs. Credit goes to Stephen Chu for discovering this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-3972) Trash emptier fails in secure HA cluster

Posted by "Hudson (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-3972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13464214#comment-13464214 ] 

Hudson commented on HDFS-3972:
------------------------------

Integrated in Hadoop-Common-trunk-Commit #2777 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2777/])
    HDFS-3972. Trash emptier fails in secure HA cluster. Contributed by Todd Lipcon (Revision 1390729)

     Result = SUCCESS
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1390729
Files : 
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/TrashPolicyDefault.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java

                
> Trash emptier fails in secure HA cluster
> ----------------------------------------
>
>                 Key: HDFS-3972
>                 URL: https://issues.apache.org/jira/browse/HDFS-3972
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: name-node
>    Affects Versions: 3.0.0, 2.0.1-alpha
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>             Fix For: 2.0.2-alpha
>
>         Attachments: hdfs-3972.txt
>
>
> In a secure HA cluster, we're seeing the following issue on the NN when the trash emptier tries to run:
> WARN org.apache.hadoop.fs.TrashPolicyDefault: Trash can't list homes: java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host \
> is: "xxxxx"; destination host is: "xxxx":8020;  Sleeping.
> The issue seems to be that the trash emptier thread sends RPCs back to itself, but isn't wrapped in a doAs. Credit goes to Stephen Chu for discovering this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-3972) Trash emptier fails in secure HA cluster

Posted by "Hudson (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-3972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13464221#comment-13464221 ] 

Hudson commented on HDFS-3972:
------------------------------

Integrated in Hadoop-Hdfs-trunk-Commit #2840 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2840/])
    HDFS-3972. Trash emptier fails in secure HA cluster. Contributed by Todd Lipcon (Revision 1390729)

     Result = SUCCESS
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1390729
Files : 
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/TrashPolicyDefault.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java

                
> Trash emptier fails in secure HA cluster
> ----------------------------------------
>
>                 Key: HDFS-3972
>                 URL: https://issues.apache.org/jira/browse/HDFS-3972
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: name-node
>    Affects Versions: 3.0.0, 2.0.1-alpha
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>             Fix For: 2.0.2-alpha
>
>         Attachments: hdfs-3972.txt
>
>
> In a secure HA cluster, we're seeing the following issue on the NN when the trash emptier tries to run:
> WARN org.apache.hadoop.fs.TrashPolicyDefault: Trash can't list homes: java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host \
> is: "xxxxx"; destination host is: "xxxx":8020;  Sleeping.
> The issue seems to be that the trash emptier thread sends RPCs back to itself, but isn't wrapped in a doAs. Credit goes to Stephen Chu for discovering this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-3972) Trash emptier fails in secure HA cluster

Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-3972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13463422#comment-13463422 ] 

Hadoop QA commented on HDFS-3972:
---------------------------------

{color:red}-1 overall{color}.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12546574/hdfs-3972.txt
  against trunk revision .

    {color:green}+1 @author{color}.  The patch does not contain any @author tags.

    {color:red}-1 tests included{color}.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    {color:green}+1 javac{color}.  The applied patch does not increase the total number of javac compiler warnings.

    {color:green}+1 javadoc{color}.  The javadoc tool did not generate any warning messages.

    {color:green}+1 eclipse:eclipse{color}.  The patch built with eclipse:eclipse.

    {color:green}+1 findbugs{color}.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    {color:green}+1 release audit{color}.  The applied patch does not increase the total number of release audit warnings.

    {color:red}-1 core tests{color}.  The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:

                  org.apache.hadoop.ha.TestZKFailoverController
                  org.apache.hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics
                  org.apache.hadoop.hdfs.web.TestWebHDFS
                  org.apache.hadoop.hdfs.TestPersistBlocks

    {color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3231//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3231//console

This message is automatically generated.
                
> Trash emptier fails in secure HA cluster
> ----------------------------------------
>
>                 Key: HDFS-3972
>                 URL: https://issues.apache.org/jira/browse/HDFS-3972
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: name-node
>    Affects Versions: 3.0.0, 2.0.1-alpha
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>         Attachments: hdfs-3972.txt
>
>
> In a secure HA cluster, we're seeing the following issue on the NN when the trash emptier tries to run:
> WARN org.apache.hadoop.fs.TrashPolicyDefault: Trash can't list homes: java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host \
> is: "xxxxx"; destination host is: "xxxx":8020;  Sleeping.
> The issue seems to be that the trash emptier thread sends RPCs back to itself, but isn't wrapped in a doAs. Credit goes to Stephen Chu for discovering this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HDFS-3972) Trash emptier fails in secure HA cluster

Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-3972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eli Collins updated HDFS-3972:
------------------------------

          Resolution: Fixed
       Fix Version/s: 2.0.2-alpha
    Target Version/s:   (was: 3.0.0, 2.0.2-alpha)
        Hadoop Flags: Reviewed
              Status: Resolved  (was: Patch Available)

I've committed this and merged to branch-2 and branch-2.0.2-alpha.
                
> Trash emptier fails in secure HA cluster
> ----------------------------------------
>
>                 Key: HDFS-3972
>                 URL: https://issues.apache.org/jira/browse/HDFS-3972
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: name-node
>    Affects Versions: 3.0.0, 2.0.1-alpha
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>             Fix For: 2.0.2-alpha
>
>         Attachments: hdfs-3972.txt
>
>
> In a secure HA cluster, we're seeing the following issue on the NN when the trash emptier tries to run:
> WARN org.apache.hadoop.fs.TrashPolicyDefault: Trash can't list homes: java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host \
> is: "xxxxx"; destination host is: "xxxx":8020;  Sleeping.
> The issue seems to be that the trash emptier thread sends RPCs back to itself, but isn't wrapped in a doAs. Credit goes to Stephen Chu for discovering this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HDFS-3972) Trash emptier fails in secure HA cluster

Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-3972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HDFS-3972:
------------------------------

    Status: Patch Available  (was: Open)
    
> Trash emptier fails in secure HA cluster
> ----------------------------------------
>
>                 Key: HDFS-3972
>                 URL: https://issues.apache.org/jira/browse/HDFS-3972
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: name-node
>    Affects Versions: 2.0.1-alpha, 3.0.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>         Attachments: hdfs-3972.txt
>
>
> In a secure HA cluster, we're seeing the following issue on the NN when the trash emptier tries to run:
> WARN org.apache.hadoop.fs.TrashPolicyDefault: Trash can't list homes: java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host \
> is: "xxxxx"; destination host is: "xxxx":8020;  Sleeping.
> The issue seems to be that the trash emptier thread sends RPCs back to itself, but isn't wrapped in a doAs. Credit goes to Stephen Chu for discovering this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira