[jira] [Created] (HADOOP-17447) Exception "Server has invalid Kerberos principal" when try to connect using reverseDNS hostname by IP resolving

["Stanislav Matytsin (Jira)" <ji...@apache.org>]

Stanislav Matytsin created HADOOP-17447:
-------------------------------------------

             Summary: Exception "Server has invalid Kerberos principal" when try to connect using reverseDNS hostname by IP resolving
                 Key: HADOOP-17447
                 URL: https://issues.apache.org/jira/browse/HADOOP-17447
             Project: Hadoop Common
          Issue Type: Bug
          Components: common
    Affects Versions: 3.1.3
            Reporter: Stanislav Matytsin


I try to create filesystem object to operate with remote kerberised cluster but there is an exception because of fail verification NameNode server principal with principal from config

The reason of exception:
In org.apache.hadoop.security.SaslRpcClient#getServerPrincipal there is verifying process of NameNode server principal and principal which gets from config. In config principal keeps in format nn/_HOST@EXAMPLE.COM where _HOST is a placeholder for real NameNode host
Then there is replacing of _HOST placeholder with real host name of name node. And real host name gets as result of InetAddress.getCanonicalHostName() method (look /Users/a16689075/.m2/repository/org/apache/hadoop/hadoop-common/3.1.3/hadoop-common-3.1.3-sources.jar!/org/apache/hadoop/security/SecurityUtil.java:211)

But if there is some reverse DNS in infrastructure where I run this code, reverse DNS resolves host by IP and return host with dot at the end. So InetAddress.getCanonicalHostName() returns real host of NameNode but with dot at the end: nn/ex1.example.com. (the same is actual when you execute for example nslookup ip-addr in command line)
So principal from config after placeholder replacement looks like nn/ex1.example.com.@EXAMPLE.COM (where hostname keeps dot at the end)

Then there is checking if nameNode server pricipal is equals with principal from config. And of course they are different because of dot exists at the end of hostname in config principal

At the same time we can look at constructor sun.security.krb5.PrincipalName#PrincipalName(java.lang.String, int, java.lang.String where we can see similar logic for processing InetAddress.getCanonicalHostName();
If InetAddress.getCanonicalHostName() received host with dot at the end, there it cuts this dot and keeps only hostname without dot

 

Cluster has Kerberos Auth and configs with HA mode for HDFS with 2 namenodes

My code to reproduce this issue is:

...

HdfsConfiguration conf = new HdfsConfiguration(false);
 conf.addResource(hdfsSiteXmlInputStream);
 conf.addResource(coreSiteXmlInputStream)));

Path hdfsKeytabPath = Paths.get("./hdfs.keytab");


 PrincipalName hdfsPrincipalName = KerberosUtils.getPrincipalFromKeytab(hdfsKeytabPath.toFile());
 UserGroupInformation.setConfiguration(conf);
 UserGroupInformation.loginUserFromKeytab(hdfsPrincipalName.getName(), hdfsKeytabPath.toAbsolutePath().toString());

fileSystem = FileSystem.get(conf);

 

Exception:

java.lang.IllegalArgumentException: Server has invalid Kerberos principal: nn/ex1.example.com@EXAMPLE.COM, expecting: nn/ex1.example.com.@EXAMPLE.COM
at org.apache.hadoop.security.SaslRpcClient.getServerPrincipal(SaslRpcClient.java:337)
at org.apache.hadoop.security.SaslRpcClient.createSaslClient(SaslRpcClient.java:234)
at org.apache.hadoop.security.SaslRpcClient.selectSaslClient(SaslRpcClient.java:160)
at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:390)
at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:627)
at org.apache.hadoop.ipc.Client$Connection.access$2300(Client.java:421)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:814)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:810)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:810)
... 73 common frames omitted

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org