You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Ahad Rana (JIRA)" <ji...@apache.org> on 2012/10/12 02:45:03 UTC
[jira] [Created] (HDFS-4043) Namenode Kerberos Login does not use
proper hostname for host qualified hdfs principal name.
Ahad Rana created HDFS-4043:
-------------------------------
Summary: Namenode Kerberos Login does not use proper hostname for host qualified hdfs principal name.
Key: HDFS-4043
URL: https://issues.apache.org/jira/browse/HDFS-4043
Project: Hadoop HDFS
Issue Type: Bug
Components: security
Affects Versions: 2.0.2-alpha, 2.0.1-alpha, 2.0.0-alpha, 2.0.3-alpha
Environment: CDH4U1 on Ubuntu 12.04
Reporter: Ahad Rana
The Namenode uses the loginAsNameNodeUser method in NameNode.java to login using the hdfs principal. This method in turn invokes SecurityUtil.login with a hostname (last parameter) obtained via a call to InetAddress.getHostName. This call does not always return the fully qualified host name, and thus causes the namenode to login to fail due to kerberos's inability to find a matching hdfs principal in the hdfs.keytab file. Instead it should use InetAddress.getCanonicalHostName. This is consistent with what is used internally by SecurityUtil.java to login in other services, such as the DataNode.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4043) Namenode Kerberos Login does not use
proper hostname for host qualified hdfs principal name.
Posted by "Brahma Reddy Battula (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-4043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13475956#comment-13475956 ]
Brahma Reddy Battula commented on HDFS-4043:
--------------------------------------------
Hi Ahad,
thanks for reply..
{quote}
Have you verified that
the generated service principal name does not match the one stored in your
kdc's database ?
{quote}
generated principal is HTTP/hostname which is not added in the KDC(I mean not stored in kdc database) and not configured..
> Namenode Kerberos Login does not use proper hostname for host qualified hdfs principal name.
> --------------------------------------------------------------------------------------------
>
> Key: HDFS-4043
> URL: https://issues.apache.org/jira/browse/HDFS-4043
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: security
> Affects Versions: 2.0.0-alpha, 2.0.1-alpha, 2.0.2-alpha, 2.0.3-alpha
> Environment: CDH4U1 on Ubuntu 12.04
> Reporter: Ahad Rana
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> The Namenode uses the loginAsNameNodeUser method in NameNode.java to login using the hdfs principal. This method in turn invokes SecurityUtil.login with a hostname (last parameter) obtained via a call to InetAddress.getHostName. This call does not always return the fully qualified host name, and thus causes the namenode to login to fail due to kerberos's inability to find a matching hdfs principal in the hdfs.keytab file. Instead it should use InetAddress.getCanonicalHostName. This is consistent with what is used internally by SecurityUtil.java to login in other services, such as the DataNode.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4043) Namenode Kerberos Login does not use
proper hostname for host qualified hdfs principal name.
Posted by "Brahma Reddy Battula (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-4043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478670#comment-13478670 ]
Brahma Reddy Battula commented on HDFS-4043:
--------------------------------------------
HI Ahad thanks a lot for reply..
{quote}
<property>
<name>dfs.namenode.kerberos.principal</name>
<value>hdfs/n01.prod.company.com@company.com</value>
</property>
<property>
<name>dfs.namenode.kerberos.internal.spnego.principal</name>
<value>HTTP/n01.prod.company.com@company.com</value>
</property>
{quote}
two properties I had configured as
hdfs/hostname of machine@HADOOP.COM(RELAM)
HTTP/hostname of machine@HADOOP.COM
I added both principal's (hdfs/hostname of machine@HADOOP.COM and HTTP/hostname of machine@HADOOP.COM) to KDC and generated keytab
but it's failing for HTTP/(IP of machine) which is not added in the KDC and then ticket is failing like following while doing checkpoint..
Oct 04 03:07:43 host-*******-168 krb5kdc[24598](info): TGS_REQ (6 etypes {3 1 23 16 17 18}) *******.168: ISSUE: authtime 1349300202, etypes {rep=23 tkt=18 ses=23}, hdfs/hadoop@HADOOP.COM for hdfs/hadoop@HADOOP.COM
Oct 04 03:07:43 host-*******-168 krb5kdc[24598](info): TGS_REQ (6 etypes {3 1 23 16 17 18}) *******.168: ISSUE: authtime 1349300202, etypes {rep=23 tkt=18 ses=23}, hdfs/hadoop@HADOOP.COM for hdfs/hadoop@HADOOP.COM
Oct 04 03:07:43 host-*******-168 krb5kdc[24598](info): TGS_REQ (6 etypes {3 1 23 16 17 18}) *******.168: ISSUE: authtime 1349300202, etypes {rep=23 tkt=18 ses=23}, hdfs/hadoop@HADOOP.COM for HTTP/*******.168@HADOOP.COM
Oct 04 03:07:43 host-*******-168 krb5kdc[24598](info): TGS_REQ (6 etypes {3 1 23 16 17 18}) *******.168: ISSUE: authtime 1349300202, etypes {rep=23 tkt=18 ses=23}, hdfs/hadoop@HADOOP.COM for HTTP/*******.168@HADOOP.COM
Mostly your's and mine(HDP-3980) is same,I think...Since KerberosAuthenticator.this*.url.getHost() is always retunring IP of the machine and then principal coming like HTTP/(ip of the machine)..
Please correct me If I am wrong...
> Namenode Kerberos Login does not use proper hostname for host qualified hdfs principal name.
> --------------------------------------------------------------------------------------------
>
> Key: HDFS-4043
> URL: https://issues.apache.org/jira/browse/HDFS-4043
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: security
> Affects Versions: 2.0.0-alpha, 2.0.1-alpha, 2.0.2-alpha, 2.0.3-alpha
> Environment: CDH4U1 on Ubuntu 12.04
> Reporter: Ahad Rana
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> The Namenode uses the loginAsNameNodeUser method in NameNode.java to login using the hdfs principal. This method in turn invokes SecurityUtil.login with a hostname (last parameter) obtained via a call to InetAddress.getHostName. This call does not always return the fully qualified host name, and thus causes the namenode to login to fail due to kerberos's inability to find a matching hdfs principal in the hdfs.keytab file. Instead it should use InetAddress.getCanonicalHostName. This is consistent with what is used internally by SecurityUtil.java to login in other services, such as the DataNode.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4043) Namenode Kerberos Login does not use
proper hostname for host qualified hdfs principal name.
Posted by "Brahma Reddy Battula (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-4043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13479954#comment-13479954 ]
Brahma Reddy Battula commented on HDFS-4043:
--------------------------------------------
[~ahadr]
Let's go ahead and close this JIRA.
{quote}
You are right however, that your problems are a manifestation of the same general bug (inconsistent resolution of canonical principal name via different code paths). Most definitely, incoming IP based connections need to use getCanonicalName to get back a host name that can be used to form the proper principal name. Otherwise you will need to probably go with IP based principal names ?
{quote}
can we discuss this point in HDF-3980..?
> Namenode Kerberos Login does not use proper hostname for host qualified hdfs principal name.
> --------------------------------------------------------------------------------------------
>
> Key: HDFS-4043
> URL: https://issues.apache.org/jira/browse/HDFS-4043
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: security
> Affects Versions: 2.0.0-alpha, 2.0.1-alpha, 2.0.2-alpha, 2.0.3-alpha
> Environment: CDH4U1 on Ubuntu 12.04
> Reporter: Ahad Rana
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> The Namenode uses the loginAsNameNodeUser method in NameNode.java to login using the hdfs principal. This method in turn invokes SecurityUtil.login with a hostname (last parameter) obtained via a call to InetAddress.getHostName. This call does not always return the fully qualified host name, and thus causes the namenode to login to fail due to kerberos's inability to find a matching hdfs principal in the hdfs.keytab file. Instead it should use InetAddress.getCanonicalHostName. This is consistent with what is used internally by SecurityUtil.java to login in other services, such as the DataNode.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4043) Namenode Kerberos Login does not use
proper hostname for host qualified hdfs principal name.
Posted by "Ahad Rana (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-4043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478067#comment-13478067 ]
Ahad Rana commented on HDFS-4043:
---------------------------------
Hi,
Can you explicitly set the namenode principal name (to the proper canonical
name) in your hdfs-site via the following properties and see if this
resolves your issues ?
<property>
<name>dfs.namenode.kerberos.principal</name>
<value>hdfs/n01.prod.company.com@company.com</value>
</property>
<property>
<name>dfs.namenode.kerberos.internal.spnego.principal</name>
<value>HTTP/n01.prod.company.com@company.com</value>
</property>
Best,
Ahad.
On Sun, Oct 14, 2012 at 9:35 PM, Brahma Reddy Battula (JIRA) <
> Namenode Kerberos Login does not use proper hostname for host qualified hdfs principal name.
> --------------------------------------------------------------------------------------------
>
> Key: HDFS-4043
> URL: https://issues.apache.org/jira/browse/HDFS-4043
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: security
> Affects Versions: 2.0.0-alpha, 2.0.1-alpha, 2.0.2-alpha, 2.0.3-alpha
> Environment: CDH4U1 on Ubuntu 12.04
> Reporter: Ahad Rana
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> The Namenode uses the loginAsNameNodeUser method in NameNode.java to login using the hdfs principal. This method in turn invokes SecurityUtil.login with a hostname (last parameter) obtained via a call to InetAddress.getHostName. This call does not always return the fully qualified host name, and thus causes the namenode to login to fail due to kerberos's inability to find a matching hdfs principal in the hdfs.keytab file. Instead it should use InetAddress.getCanonicalHostName. This is consistent with what is used internally by SecurityUtil.java to login in other services, such as the DataNode.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4043) Namenode Kerberos Login does not use
proper hostname for host qualified hdfs principal name.
Posted by "Ahad Rana (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-4043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13479668#comment-13479668 ]
Ahad Rana commented on HDFS-4043:
---------------------------------
Hi Brahma,
Please disregard my last suggestion. Setting dfs.namenode.kerberos.principal or dfs.namenode.kerberos.internal.spnego.principal to and explicit principal name (instead of a pattern name with _HOST in it) triggers other bugs (see HDFS-4081). The bottom line is that it is probably best to set the hostname of the namenode to match exactly the name returned via a reverse-dns query (getCanonicalName). You are right however, that your problems are a manifestation of the same general bug (inconsistent resolution of canonical principal name via different code paths). Most definitely, incoming IP based connections need to use getCanonicalName to get back a host name that can be used to form the proper principal name. Otherwise you will need to probably go with IP based principal names ?
As mentioned above, I have reverted to setting the internal hostname for the namenodes/secondary namenodes to exactly match the fully qualified hostname returned via reverse-dns. And so far, things seems to be working properly now.
> Namenode Kerberos Login does not use proper hostname for host qualified hdfs principal name.
> --------------------------------------------------------------------------------------------
>
> Key: HDFS-4043
> URL: https://issues.apache.org/jira/browse/HDFS-4043
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: security
> Affects Versions: 2.0.0-alpha, 2.0.1-alpha, 2.0.2-alpha, 2.0.3-alpha
> Environment: CDH4U1 on Ubuntu 12.04
> Reporter: Ahad Rana
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> The Namenode uses the loginAsNameNodeUser method in NameNode.java to login using the hdfs principal. This method in turn invokes SecurityUtil.login with a hostname (last parameter) obtained via a call to InetAddress.getHostName. This call does not always return the fully qualified host name, and thus causes the namenode to login to fail due to kerberos's inability to find a matching hdfs principal in the hdfs.keytab file. Instead it should use InetAddress.getCanonicalHostName. This is consistent with what is used internally by SecurityUtil.java to login in other services, such as the DataNode.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4043) Namenode Kerberos Login does not use
proper hostname for host qualified hdfs principal name.
Posted by "Brahma Reddy Battula (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-4043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13475524#comment-13475524 ]
Brahma Reddy Battula commented on HDFS-4043:
--------------------------------------------
Hi Ahad Rana,
I think,,this is same as HDFS-3980..Please refer following comment..
https://issues.apache.org/jira/browse/HDFS-3980?focusedCommentId=13469267&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13469267..
Can I duplicate this...?
Please correct me If I am wrong..
> Namenode Kerberos Login does not use proper hostname for host qualified hdfs principal name.
> --------------------------------------------------------------------------------------------
>
> Key: HDFS-4043
> URL: https://issues.apache.org/jira/browse/HDFS-4043
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: security
> Affects Versions: 2.0.0-alpha, 2.0.1-alpha, 2.0.2-alpha, 2.0.3-alpha
> Environment: CDH4U1 on Ubuntu 12.04
> Reporter: Ahad Rana
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> The Namenode uses the loginAsNameNodeUser method in NameNode.java to login using the hdfs principal. This method in turn invokes SecurityUtil.login with a hostname (last parameter) obtained via a call to InetAddress.getHostName. This call does not always return the fully qualified host name, and thus causes the namenode to login to fail due to kerberos's inability to find a matching hdfs principal in the hdfs.keytab file. Instead it should use InetAddress.getCanonicalHostName. This is consistent with what is used internally by SecurityUtil.java to login in other services, such as the DataNode.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4043) Namenode Kerberos Login does not use
proper hostname for host qualified hdfs principal name.
Posted by "Ahad Rana (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-4043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13475717#comment-13475717 ]
Ahad Rana commented on HDFS-4043:
---------------------------------
Hi Brahma,
Not sure if this is exactly the same bug. It could definitely be the case
that the host name derived via KerberosAuthenticator.this*.url.getHost() *is
not the Canonical Hostname used to generate the server principal, so it is
definitely similar in nature to the bug I reported. Have you verified that
the generated service principal name does not match the one stored in your
kdc's database ?
Ahad.
On Fri, Oct 12, 2012 at 9:25 PM, Brahma Reddy Battula (JIRA) <
> Namenode Kerberos Login does not use proper hostname for host qualified hdfs principal name.
> --------------------------------------------------------------------------------------------
>
> Key: HDFS-4043
> URL: https://issues.apache.org/jira/browse/HDFS-4043
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: security
> Affects Versions: 2.0.0-alpha, 2.0.1-alpha, 2.0.2-alpha, 2.0.3-alpha
> Environment: CDH4U1 on Ubuntu 12.04
> Reporter: Ahad Rana
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> The Namenode uses the loginAsNameNodeUser method in NameNode.java to login using the hdfs principal. This method in turn invokes SecurityUtil.login with a hostname (last parameter) obtained via a call to InetAddress.getHostName. This call does not always return the fully qualified host name, and thus causes the namenode to login to fail due to kerberos's inability to find a matching hdfs principal in the hdfs.keytab file. Instead it should use InetAddress.getCanonicalHostName. This is consistent with what is used internally by SecurityUtil.java to login in other services, such as the DataNode.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira