webhdfs kerberos checksum failed

["Lanati, Matteo" <Ma...@lrz.de>]

Hi all,

I'm trying to setup webhdfs on Hadoop 1.20 with security.
I added the following to hdfs-site.xml

  <property>
    <name>dfs.webhdfs.enabled</name>
    <value>true</value>
  </property>

  <property>
    <name>dfs.web.authentication.kerberos.principal</name>
    <value>HTTP/master.hadoop.local@HADOOP.LRZ.DE</value>
  </property>

  <property>
    <name>dfs.web.authentication.kerberos.keytab</name>
    <value>/home/hadoop-user/hadoop/conf/keytabs/http.keytab</value>
  </property>

The keytab is valid and I can  get credentials if I try to use it.

When I test WebHDFS with curl

 curl --negotiate -u:anyUser -b ~/cookies.txt -c ~/cookies.txt http://master.hadoop.local:50070/webhdfs/v1/tmp?op=LISTSTATUS

I see the following error in the logfile of the namenode

2013-06-16 17:17:00,170 WARN org.apache.hadoop.security.authentication.server.AuthenticationFilter: Authentication exception: GSSException: Failure unspecified at GSS-API level (Mechanism level: Checksum failed)
org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: Failure unspecified at GSS-API level (Mechanism level: Checksum failed)
        at org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.authenticate(KerberosAuthenticationHandler.java:301)
        at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:348)
        at org.apache.hadoop.hdfs.web.AuthFilter.doFilter(AuthFilter.java:85)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
        at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:914)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
        at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
        at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
        at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
        at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
        at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
        at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
        at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
        at org.mortbay.jetty.Server.handle(Server.java:326)
        at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
        at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
        at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
        at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
        at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
        at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
        at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Caused by: GSSException: Failure unspecified at GSS-API level (Mechanism level: Checksum failed)
        at sun.security.jgss.krb5.Krb5Context.acceptSecContext(Unknown Source)
        at sun.security.jgss.GSSContextImpl.acceptSecContext(Unknown Source)
        at sun.security.jgss.GSSContextImpl.acceptSecContext(Unknown Source)
        at org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler$2.run(KerberosAuthenticationHandler.java:271)
        at org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler$2.run(KerberosAuthenticationHandler.java:263)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Unknown Source)
        at org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.authenticate(KerberosAuthenticationHandler.java:263)
        ... 20 more
Caused by: KrbException: Checksum failed
        at sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType.decrypt(Unknown Source)
        at sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType.decrypt(Unknown Source)
        at sun.security.krb5.EncryptedData.decrypt(Unknown Source)
        at sun.security.krb5.KrbApReq.authenticate(Unknown Source)
        at sun.security.krb5.KrbApReq.<init>(Unknown Source)
        at sun.security.jgss.krb5.InitSecContextToken.<init>(Unknown Source)
        ... 28 more
Caused by: java.security.GeneralSecurityException: Checksum failed
        at sun.security.krb5.internal.crypto.dk.AesDkCrypto.decryptCTS(Unknown Source)
        at sun.security.krb5.internal.crypto.dk.AesDkCrypto.decrypt(Unknown Source)
         at sun.security.krb5.internal.crypto.Aes256.decrypt(Unknown Source)
        ... 34 more

I'm using Java 7 with JCE extensions. I also tried to force a different encryption algorithm for Kerberos (i.e. arcfour-hmac instead of aes256-cts-hmac-sha1-96) but the error message is the same (of course it refers to arcfour-hmac).

Does anyone have an idea of what could be the root of this?

Thanks,

Matteo



Matteo Lanati
Distributed Resources Group
Leibniz-Rechenzentrum (LRZ)
Boltzmannstrasse 1
85748 Garching b. München (Germany)
Phone: +49 89 35831 8724