You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Akash Mishra <ak...@gmail.com> on 2016/08/19 10:45:49 UTC
Namenode Unable to Authenticate to QJM in Secure mode.
Hi *,
I am trying to run Hadoop cluster [ 2.7.1] in Secure mode. In my cluster
Namenode is failing while restart with
2016-08-19 10:34:49,754 DEBUG
org.apache.hadoop.security.authentication.client.KerberosAuthenticator:
Using fallback authenticator sequence.
2016-08-19 10:34:49,774 DEBUG
org.apache.hadoop.security.UserGroupInformation: PrivilegedActionException
as:hdfs/hadoopdev1.mlan@HADOOPDEV.MLAN (auth:KERBEROS)
cause:java.io.IOException:
org.apache.hadoop.security.authentication.client.AuthenticationException:
Authentication failed, status: 403, message: GSSException: No valid
credentials provided (Mechanism level: Failed to find any Kerberos
credentails)
2016-08-19 10:34:49,775 ERROR
org.apache.hadoop.hdfs.server.namenode.EditLogInputStream: caught exception
initializing
http://hadoopdev1:8480/getJournal?jid=hadoopdev&segmentTxId=2275460&storageInfo=-63%3A1455401088%3A1444912570574%3ACID-f748dfef-c174-4d19-8d18-43b74552c8e6
java.io.IOException:
org.apache.hadoop.security.authentication.client.AuthenticationException:
Authentication failed, status: 403, message: GSSException: No valid
credentials provided (Mechanism level: Failed to find any Kerberos
credentails)
at
org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:464)
at
org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:456)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at
org.apache.hadoop.security.SecurityUtil.doAsUser(SecurityUtil.java:448)
at
org.apache.hadoop.security.SecurityUtil.doAsCurrentUser(SecurityUtil.java:442)
at
org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog.getInputStream(EditLogFileInputStream.java:455)
at
org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.init(EditLogFileInputStream.java:141)
at
org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.nextOpImpl(EditLogFileInputStream.java:192)
at
org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.nextOp(EditLogFileInputStream.java:250)
at
org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
at
org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151)
at
org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178)
at
org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
at
org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151)
at
org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178)
at
org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
I am using MIT 5 Kerberos. I am able to successfully kinit using keytab
file. I have DEBUG log enabled and attaching log from Namenode [nn.log]
and one of QJM [ qjm.log]
Thanks.
--
Regards,
Akash Mishra.
"It's not our abilities that make us, but our decisions."--Albus Dumbledore
Re: Namenode Unable to Authenticate to QJM in Secure mode.
Posted by Rakesh Radhakrishnan <ra...@apache.org>.
Hi Akash,
In general "GSSException: No valid credentials provided" means you don’t
have valid Kerberos credentials. I'm suspecting some issues related to
spnego, could you please revisit all of your kerb related configurations,
probably you can start from the below configuration. Please share
*-site.xml configurations of JN and NNs. Also, please check any unexpected
exceptions in KDC server logs.
I've filtered out "REQUEST /getJournal on org.mortbay.jetty.HttpConnection"
in your "qjm.log" log file and I could see this has came immediately after
your restart, few has succeeded and few others failed with this exception.
2016-08-19 10:34:14,345 DEBUG org.mortbay.log: RESPONSE /getJournal 401
2016-08-19 10:34:14,374 DEBUG org.mortbay.log: RESPONSE /getJournal 403
2016-08-19 10:34:14,382 DEBUG org.mortbay.log: RESPONSE /getJournal 401
2016-08-19 10:34:14,398 DEBUG org.mortbay.log: RESPONSE /getJournal 403
2016-08-19 10:34:49,679 DEBUG org.mortbay.log: RESPONSE /getJournal 401
<property>
<name>dfs.journalnode.kerberos.internal.spnego.principal</name>
<value></value>
<description>
The server principal used by the JournalNode HTTP Server for
SPNEGO authentication when Kerberos security is enabled. This is
typically set to HTTP/_HOST@REALM.TLD. The SPNEGO server principal
begins with the prefix HTTP/ by convention.
If the value is '*', the web server will attempt to login with
every principal specified in the keytab file
dfs.web.authentication.kerberos.keytab.
For most deployments this can be set to
${dfs.web.authentication.kerberos.principal}
i.e use the value of dfs.web.authentication.kerberos.principal.
</description>
</property>
Rakesh,
Intel
On Fri, Aug 19, 2016 at 4:15 PM, Akash Mishra <ak...@gmail.com>
wrote:
> Hi *,
>
> I am trying to run Hadoop cluster [ 2.7.1] in Secure mode. In my cluster
> Namenode is failing while restart with
>
> 2016-08-19 10:34:49,754 DEBUG org.apache.hadoop.security.
> authentication.client.KerberosAuthenticator: Using fallback authenticator
> sequence.
> 2016-08-19 10:34:49,774 DEBUG org.apache.hadoop.security.UserGroupInformation:
> PrivilegedActionException as:hdfs/hadoopdev1.mlan@HADOOPDEV.MLAN
> (auth:KERBEROS) cause:java.io.IOException: org.apache.hadoop.security.
> authentication.client.AuthenticationException: Authentication failed,
> status: 403, message: GSSException: No valid credentials provided
> (Mechanism level: Failed to find any Kerberos credentails)
> 2016-08-19 10:34:49,775 ERROR org.apache.hadoop.hdfs.server.namenode.EditLogInputStream:
> caught exception initializing http://hadoopdev1:8480/
> getJournal?jid=hadoopdev&segmentTxId=2275460&storageInfo=-63%3A1455401088%
> 3A1444912570574%3ACID-f748dfef-c174-4d19-8d18-43b74552c8e6
> java.io.IOException: org.apache.hadoop.security.authentication.client.AuthenticationException:
> Authentication failed, status: 403, message: GSSException: No valid
> credentials provided (Mechanism level: Failed to find any Kerberos
> credentails)
> at org.apache.hadoop.hdfs.server.namenode.
> EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:464)
> at org.apache.hadoop.hdfs.server.namenode.
> EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:456)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(
> UserGroupInformation.java:1657)
> at org.apache.hadoop.security.SecurityUtil.doAsUser(
> SecurityUtil.java:448)
> at org.apache.hadoop.security.SecurityUtil.doAsCurrentUser(
> SecurityUtil.java:442)
> at org.apache.hadoop.hdfs.server.namenode.
> EditLogFileInputStream$URLLog.getInputStream(EditLogFileInputStream.java:
> 455)
> at org.apache.hadoop.hdfs.server.namenode.
> EditLogFileInputStream.init(EditLogFileInputStream.java:141)
> at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.
> nextOpImpl(EditLogFileInputStream.java:192)
> at org.apache.hadoop.hdfs.server.namenode.
> EditLogFileInputStream.nextOp(EditLogFileInputStream.java:250)
> at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.
> readOp(EditLogInputStream.java:85)
> at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.
> skipUntil(EditLogInputStream.java:151)
> at org.apache.hadoop.hdfs.server.namenode.
> RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178)
> at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.
> readOp(EditLogInputStream.java:85)
> at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.
> skipUntil(EditLogInputStream.java:151)
> at org.apache.hadoop.hdfs.server.namenode.
> RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178)
> at org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.
> readOp(EditLogInputStream.java:85)
>
>
> I am using MIT 5 Kerberos. I am able to successfully kinit using keytab
> file. I have DEBUG log enabled and attaching log from Namenode [nn.log]
> and one of QJM [ qjm.log]
>
>
>
> Thanks.
>
>
>
>
> --
>
> Regards,
> Akash Mishra.
>
>
> "It's not our abilities that make us, but our decisions."--Albus Dumbledore
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: user-help@hadoop.apache.org
>