You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@metron.apache.org by "Nick Allen (JIRA)" <ji...@apache.org> on 2017/10/19 19:47:00 UTC
[jira] [Comment Edited] (METRON-1266) Profiler - No valid
credentials provided
[ https://issues.apache.org/jira/browse/METRON-1266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16211617#comment-16211617 ]
Nick Allen edited comment on METRON-1266 at 10/19/17 7:46 PM:
--------------------------------------------------------------
I have found that the following property needs to be set on the Profiler topology when running in a Kerberized environment. This is similar to how the other topologies, like Enrichment, are already configured.
{code}
topology.auto-credentials: ['org.apache.storm.security.auth.kerberos.AutoTGT']
{code}
After finding this, the big mystery for me was why the configuration miss did not cause this issue in all kerberized environments, all the time. Surely this miss should break the Profiler when running in any Kerberized environment and so should have been caught sooner.
I think I understand why now. The problem is obviously that a ticket cannot be found to authenticate when attempting to flush profile measurements to HBase. Due to this configuration miss, the Profiler topology itself is not able to generate Kerberos tickets for authentication. At the same time, if the ticket cache on the worker node is already populated with a valid ticket, then this issue will not occur. The ticket cache can be populated if another process generates a ticket or a user manually kinits on the same node.
This explains why the problem occurs sporadically and only in some environments. This issue is less likely to occur in an environment, like Full Dev, where there are fewer, more active nodes. In this case, it is likely that some other process or user already pre-populated the ticket cache. In a larger, multi-node cluster, the ticket cache is less likely to be populated.
was (Author: nickwallen):
I have found that the following property needs to be set on the Profiler topology when running in a Kerberized environment. This is similar to how the other topologies, like Enrichment, are already configured.
{code}
topology.auto-credentials: ['org.apache.storm.security.auth.kerberos.AutoTGT']
{code}
After finding this, the big mystery for me was why the configuration miss did not cause this issue in all kerberized environments, all the time. I think I understand why now. The problem is obviously that a ticket cannot be found to authenticate when attempting to flush profile measurements to HBase. Due to this configuration miss, the Profiler topology itself is not able to generate Kerberos tickets for authentication. At the same time, if the ticket cache on the worker node is already populated with a valid ticket, then this issue will not occur. The ticket cache can be populated if another process generates a ticket or a user manually kinits on the same node.
This explains why the problem occurs sporadically and only in some environments. This issue is less likely to occur in an environment, like Full Dev, where there are fewer, more active nodes. In this case, it is likely that some other process or user already pre-populated the ticket cache. In a larger, multi-node cluster, the ticket cache is less likely to be populated.
> Profiler - No valid credentials provided
> ----------------------------------------
>
> Key: METRON-1266
> URL: https://issues.apache.org/jira/browse/METRON-1266
> Project: Metron
> Issue Type: Bug
> Affects Versions: 0.4.1
> Reporter: Nick Allen
> Assignee: Nick Allen
> Fix For: Next + 1
>
>
> When running the Profiler on a cluster that has multiple nodes and is secured by Kerberos, it was observed that the HBaseBolt was unable to write to HBase. The Storm worker running the HBaseBolt logged the following exception. This does not occur all the time and does not occur in all environments.
> {code}
> 2017-10-19 14:51:00.146 o.a.h.h.i.AbstractRpcClient [ERROR] SASL authentication failed. The most likely cause is missing or invalid credentials. Consider 'kinit'.
> javax.security.sasl.SaslException: GSS initiate failed
> at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211) ~[?:1.8.0_144]
> at org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:179) ~[stormjar.jar:?]
> at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupSaslConnection(RpcClientImpl.java:609) ~[stormjar.jar:?]
> at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.access$600(RpcClientImpl.java:154) [stormjar.jar:?]
> at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:735) ~[stormjar.jar:?]
> at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:732) ~[stormjar.jar:?]
> at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_144]
> at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_144]
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) ~[stormjar.jar:?]
> at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:732) [stormjar.jar:?]
> at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.writeRequest(RpcClientImpl.java:885) [stormjar.jar:?]
> at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.tracedWriteRequest(RpcClientImpl.java:854) [stormjar.jar:?]
> at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1180) [stormjar.jar:?]
> at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:213) [stormjar.jar:?]
> at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:287) [stormjar.jar:?]
> at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:32651) [stormjar.jar:?]
> at org.apache.hadoop.hbase.client.ClientSmallScanner$SmallScannerCallable.call(ClientSmallScanner.java:201) [stormjar.jar:?]
> at org.apache.hadoop.hbase.client.ClientSmallScanner$SmallScannerCallable.call(ClientSmallScanner.java:180) [stormjar.jar:?]
> at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200) [stormjar.jar:?]
> at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:369) [stormjar.jar:?]
> at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:343) [stormjar.jar:?]
> at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126) [stormjar.jar:?]
> at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:64) [stormjar.jar:?]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_144]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_144]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144]
> Caused by: org.ietf.jgss.GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
> at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147) ~[?:1.8.0_144]
> at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122) ~[?:1.8.0_144]
> at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187) ~[?:1.8.0_144]
> at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224) ~[?:1.8.0_144]
> at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212) ~[?:1.8.0_144]
> at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179) ~[?:1.8.0_144]
> at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192) ~[?:1.8.0_144]
> ... 25 more}}
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)