You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Frank Eisenhauer <Fr...@mhp.com> on 2015/03/15 12:39:01 UTC
WG: STORM HBase Bolt: javax.security.sasl.SaslException: GSS
initiate failed
Hi All,
i've written a STORM topology with a HDFS and a HBase Bolt, which runs absolutely fine with kerberos turned off in my hadoop cluster.
With activated security in the cluster, the HDFS Bolts works like a charm, but I encounter authentication errors with the HBase Bolt.
Accessing HBase shell with a Kerberos ticket works also absolutely fine.
The java exception in the Storm topology log is as follows:
2015-03-15 12:21:11 o.a.h.h.i.RpcClient [WARN] Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSExcept$
2015-03-15 12:21:11 o.a.h.h.i.RpcClient [ERROR] SASL authentication failed. The most likely cause is missing or invalid credentials. Consider 'kinit'.
javax.security.sasl.SaslException: GSS initiate failed
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212) ~[na:1.7.0_45]
at org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:177) ~[stormjar.jar:na]
at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupSaslConnection(RpcClient.java:815) [stormjar.jar:na]
at org.apache.hadoop.hbase.ipc.RpcClient$Connection.access$800(RpcClient.java:349) [stormjar.jar:na]
at org.apache.hadoop.hbase.ipc.RpcClient$Connection$2.run(RpcClient.java:943) ~[stormjar.jar:na]
at org.apache.hadoop.hbase.ipc.RpcClient$Connection$2.run(RpcClient.java:940) ~[stormjar.jar:na]
at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_45]
at javax.security.auth.Subject.doAs(Subject.java:415) ~[na:1.7.0_45]
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) ~[hadoop-common-2.6.0.2.2.0.0-2041.jar:na]
at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:940) [stormjar.jar:na]
at org.apache.hadoop.hbase.ipc.RpcClient$Connection.writeRequest(RpcClient.java:1094) [stormjar.jar:na]
at org.apache.hadoop.hbase.ipc.RpcClient$Connection.tracedWriteRequest(RpcClient.java:1061) [stormjar.jar:na]
at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1516) [stormjar.jar:na]
at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1724) [stormjar.jar:na]
at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1777) [stormjar.jar:na]
at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.get(ClientProtos.java:30373) [stormjar.jar:na]
at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRowOrBefore(ProtobufUtil.java:1604) [stormjar.jar:na]
at org.apache.hadoop.hbase.client.HTable$2.call(HTable.java:768) [stormjar.jar:na]
at org.apache.hadoop.hbase.client.HTable$2.call(HTable.java:766) [stormjar.jar:na]
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126) [stormjar.jar:na]
at org.apache.hadoop.hbase.client.HTable.getRowOrBefore(HTable.java:772) [stormjar.jar:na]
at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:160) [stormjar.jar:na]
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.prefetchRegionCache(ConnectionManager.java:1222) [stormjar.jar:na]
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1286) [stormjar.jar:na]
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1135) [stormjar.jar:na]
at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:362) [stormjar.jar:na]
at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:313) [stormjar.jar:na]
at org.apache.hadoop.hbase.client.HTable.backgroundFlushCommits(HTable.java:1066) [stormjar.jar:na]
at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1344) [stormjar.jar:na]
at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1001) [stormjar.jar:na]
at com.mhp.bigdata.storm.gt6.MHPHBaseBolt2.execute(MHPHBaseBolt2.java:159) [stormjar.jar:na]
at backtype.storm.daemon.executor$fn__5697$tuple_action_fn__5699.invoke(executor.clj:659) [storm-core-0.9.3.2.2.0.0-2041.jar:0.9.3.2.2.0.0-2041]
at backtype.storm.daemon.executor$mk_task_receiver$fn__5620.invoke(executor.clj:415) [storm-core-0.9.3.2.2.0.0-2041.jar:0.9.3.2.2.0.0-2041]
at backtype.storm.disruptor$clojure_handler$reify__1741.onEvent(disruptor.clj:58) [storm-core-0.9.3.2.2.0.0-2041.jar:0.9.3.2.2.0.0-2041]
at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:120) [storm-core-0.9.3.2.2.0.0-2041.jar:0.9.3.2.2.0.0-2041]
at backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99) [storm-core-0.9.3.2.2.0.0-2041.jar:0.9.3.2.2.0.0-2041]
at backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80) [storm-core-0.9.3.2.2.0.0-2041.jar:0.9.3.2.2.0.0-2041]
at backtype.storm.daemon.executor$fn__5697$fn__5710$fn__5761.invoke(executor.clj:794) [storm-core-0.9.3.2.2.0.0-2041.jar:0.9.3.2.2.0.0-2041]
at backtype.storm.util$async_loop$fn__452.invoke(util.clj:465) [storm-core-0.9.3.2.2.0.0-2041.jar:0.9.3.2.2.0.0-2041]
at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
Caused by: org.ietf.jgss.GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147) ~[na:1.7.0_45]
at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121) ~[na:1.7.0_45]
at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187) ~[na:1.7.0_45]
at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223) ~[na:1.7.0_45]
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212) ~[na:1.7.0_45]
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179) ~[na:1.7.0_45]
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193) ~[na:1.7.0_45]
... 40 common frames omitted
My Storm Version is: 0.9.3.2.2.0.0
My HBase Version is: 0.98.4.2.2.0.0
Java JCE is installed. Kerberos is working fine for all other components of the cluster.
Is there any known bug? Or any chance to trace this issue down to its roots?
I'm looking forward to your responses.
Best regards
Frank