You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by "Jonathan Hurley (JIRA)" <ji...@apache.org> on 2015/06/16 05:57:00 UTC

[jira] [Created] (AMBARI-11941) RU: Pre Upgrade HDFS Fails Due To Kerberos Security Exception

Jonathan Hurley created AMBARI-11941:
----------------------------------------

             Summary: RU: Pre Upgrade HDFS Fails Due To Kerberos Security Exception
                 Key: AMBARI-11941
                 URL: https://issues.apache.org/jira/browse/AMBARI-11941
             Project: Ambari
          Issue Type: Bug
    Affects Versions: 2.1.0
            Reporter: Jonathan Hurley
            Assignee: Jonathan Hurley
            Priority: Blocker
             Fix For: 2.1.0


During an upgrade from HDP 2.2 to HDP 2.3, the pre-upgrade of the NameNode fails:

{code:title=cat ip-172-31-41-15.ec2.internal/var/lib/ambari-agent/data/output-3113.txt}
2015-06-13 14:50:17,703 - call['hdfs dfsadmin -safemode get'] {'user': 'hdfs'}
2015-06-13 14:50:22,142 - call returned (255, '15/06/13 14:50:21 WARN ipc.Client: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]\nsafemode: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "ip-172-31-41-15.ec2.internal/172.31.41.15"; destination host is: "ip-172-31-41-15.ec2.internal":8020; ')
2015-06-13 14:50:22,143 - Command: hdfs dfsadmin -safemode get
Code: 255.
Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py", line 311, in <module>
    NameNode().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 216, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py", line 110, in prepare_rolling_upgrade
    namenode_upgrade.prepare_rolling_upgrade()
  File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/namenode_upgrade.py", line 100, in prepare_rolling_upgrade
    raise Fail("Could not transition to safemode state %s. Please check logs to make sure namenode is up." % str(SafeMode.OFF))
resource_management.core.exceptions.Fail: Could not transition to safemode state OFF. Please check logs to make sure namenode is up.
{code}

With the heart of the issue being a kerberos issue:

{code}
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]\nsafemode: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)];
{code}

It looks like kinit was called right before this:

{code}
2015-06-13 14:50:17,473 - Execute['/usr/bin/kinit -kt /etc/security/keytabs/hdfs.headless.keytab hdfs@EXAMPLE.COM'] {}2015-06-13 14:50:17,702 - Prepare to transition into safemode state OFF
2015-06-13 14:50:17,703 - call['hdfs dfsadmin -safemode get'] {'user': 'hdfs'}
2015-06-13 14:50:22,142 - call returned (255, '15/06/13 14:50:21 WARN ipc.Client: Exception encountered while connecting to the server : javax.security.sasl.SaslException:...
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)