You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@phoenix.apache.org by "MariaCarrie (JIRA)" <ji...@apache.org> on 2019/02/19 07:58:00 UTC
[jira] [Updated] (PHOENIX-5145) GSSException: No valid credentials
provided (Mechanism level: Failed to find any Kerberos tgt)
[ https://issues.apache.org/jira/browse/PHOENIX-5145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
MariaCarrie updated PHOENIX-5145:
---------------------------------
Description:
I can successfully read the data using the local mode. Here is my code:
^val sqlContext: SQLContext = missionSession.app.ss.sqlContext^
^System.setProperty("sun.security.krb5.debug", "true")^
^System.setProperty("sun.security.spnego.debug", "true")^
^UserGroupInformation.loginUserFromKeytab("dmp@DEVDIP.ORG", "devdmp.keytab")^
^// Load as a DataFrame directly using a Configuration object^
^val df: DataFrame = sqlContext.phoenixTableAsDataFrame(missionSession.config.tableName, Seq("ID"), zkUrl = Some(missionSession.config.zkUrl))^
^df.show(5)^
But when I submit this to YARN for execution, an exception will be thrown. The following is the exception information:
^Tue Feb 19 13:07:53 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: java.io.IOException: Can not send request because relogin is in progress.^
^Tue Feb 19 13:07:53 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: java.io.IOException: Can not send request because relogin is in progress.^
^Tue Feb 19 13:07:53 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: java.io.IOException: Can not send request because relogin is in progress.^
^Tue Feb 19 13:07:54 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: java.io.IOException: Can not send request because relogin is in progress.^
^Tue Feb 19 13:07:54 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: java.io.IOException: Can not send request because relogin is in progress.^
^Tue Feb 19 13:07:55 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: java.io.IOException: Can not send request because relogin is in progress.^
^Tue Feb 19 13:07:57 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: java.io.IOException: Can not send request because relogin is in progress.^
^Tue Feb 19 13:08:01 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
^Tue Feb 19 13:08:11 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
^Tue Feb 19 13:08:21 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
^Tue Feb 19 13:08:31 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
^Tue Feb 19 13:08:41 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
^Tue Feb 19 13:09:01 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
^Tue Feb 19 13:09:22 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
^Tue Feb 19 13:09:42 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
^Tue Feb 19 13:10:02 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
^Tue Feb 19 13:10:22 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
^Tue Feb 19 13:10:42 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
^Tue Feb 19 13:11:02 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
^Tue Feb 19 13:11:22 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
^Tue Feb 19 13:11:42 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
^Tue Feb 19 13:12:02 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
^Tue Feb 19 13:12:23 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
^Tue Feb 19 13:12:43 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
^Tue Feb 19 13:13:03 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
^Tue Feb 19 13:13:23 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
^Tue Feb 19 13:13:43 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
^Tue Feb 19 13:14:03 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
^Tue Feb 19 13:14:23 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
^Tue Feb 19 13:14:43 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
^Tue Feb 19 13:15:03 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
^Tue Feb 19 13:15:23 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
^Tue Feb 19 13:15:44 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
^Tue Feb 19 13:16:04 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
^Tue Feb 19 13:16:24 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
^Tue Feb 19 13:16:44 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
^at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:145)^
^at org.apache.hadoop.hbase.client.HTable.get(HTable.java:386)^
^at org.apache.hadoop.hbase.client.HTable.get(HTable.java:360)^
^at org.apache.hadoop.hbase.MetaTableAccessor.getTableState(MetaTableAccessor.java:1076)^
^at org.apache.hadoop.hbase.MetaTableAccessor.tableExists(MetaTableAccessor.java:399)^
^at org.apache.hadoop.hbase.client.HBaseAdmin$6.rpcCall(HBaseAdmin.java:437)^
^at org.apache.hadoop.hbase.client.HBaseAdmin$6.rpcCall(HBaseAdmin.java:434)^
^at org.apache.hadoop.hbase.client.RpcRetryingCallable.call(RpcRetryingCallable.java:58)^
^at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:107)^
^... 55 more^
^Caused by: java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
^at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:180)^
^at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:390)^
^at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:95)^
^at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410)^
^at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:406)^
^at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103)^
^at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118)^
^at org.apache.hadoop.hbase.ipc.BufferCallBeforeInitHandler.userEventTriggered(BufferCallBeforeInitHandler.java:92)^
^at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:329)^
^at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:315)^
^at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:307)^
^at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:108)^
^at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:329)^
^at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:315)^
^at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:307)^
^at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:108)^
^at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:353)^
^at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:329)^
^at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:315)^
^at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:307)^
^at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.userEventTriggered(DefaultChannelPipeline.java:1377)^
^at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:329)^
^at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:315)^
^at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireUserEventTriggered(DefaultChannelPipeline.java:929)^
^at org.apache.hadoop.hbase.ipc.NettyRpcConnection.failInit(NettyRpcConnection.java:179)^
^at org.apache.hadoop.hbase.ipc.NettyRpcConnection.access$500(NettyRpcConnection.java:71)^
^at org.apache.hadoop.hbase.ipc.NettyRpcConnection$2.operationComplete(NettyRpcConnection.java:247)^
^at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)^
^at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:481)^
^at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420)^
^at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:163)^
^at org.apache.hadoop.hbase.ipc.NettyRpcConnection.saslNegotiate(NettyRpcConnection.java:201)^
^at org.apache.hadoop.hbase.ipc.NettyRpcConnection.access$800(NettyRpcConnection.java:71)^
^at org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:273)^
^at org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:261)^
^at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)^
^at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:500)^
^at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:479)^
^at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420)^
^at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104)^
^at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82)^
^at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:306)^
^at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:341)^
^at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:633)^
^at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)^
^at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)^
^at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)^
^at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)^
^at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)^
^... 1 more^
^Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
^at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)^
^at org.apache.hadoop.hbase.security.AbstractHBaseSaslRpcClient.getInitialResponse(AbstractHBaseSaslRpcClient.java:131)^
^at org.apache.hadoop.hbase.security.NettyHBaseSaslRpcClientHandler$1.run(NettyHBaseSaslRpcClientHandler.java:110)^
^at org.apache.hadoop.hbase.security.NettyHBaseSaslRpcClientHandler$1.run(NettyHBaseSaslRpcClientHandler.java:106)^
^at java.security.AccessController.doPrivileged(Native Method)^
^at javax.security.auth.Subject.doAs(Subject.java:422)^
^at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1688)^
^at org.apache.hadoop.hbase.security.NettyHBaseSaslRpcClientHandler.handlerAdded(NettyHBaseSaslRpcClientHandler.java:106)^
^at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.callHandlerAdded0(DefaultChannelPipeline.java:606)^
^at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.addFirst(DefaultChannelPipeline.java:187)^
^at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.addFirst(DefaultChannelPipeline.java:380)^
^at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.addFirst(DefaultChannelPipeline.java:359)^
^at org.apache.hadoop.hbase.ipc.NettyRpcConnection.saslNegotiate(NettyRpcConnection.java:200)^
^... 18 more^
^Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)^
^at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)^
^at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)^
^at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)^
^at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224)^
^at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)^
^at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)^
^at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192)^
^... 30 more^
The following are my submission parameters:
^spark-submit^
^--master yarn^
^--name PHOENIX_SPARK_PLUGIN^
^--deploy-mode cluster^
^--keytab /tesdmp/keytabs/nnjKorRc37PPPjLf/dmp/testdmp.keytab^
^--principal dmp@TESTDIP.ORG^
^--jars /usr/hdp/3.0.0.0-1634/phoenix/phoenix-5.0.0.3.0.0.0-1634-client.jar^
^--conf spark.yarn.maxAppAttempts=1^
^/path/phoenix-spark-plugin.jar^
This program runs on YARN. Do I need to do something extra? Can anyone give me some advice?
was:
I can successfully read the data using the local mode. Here is my code:
^val sqlContext: SQLContext = missionSession.app.ss.sqlContext^
^System.setProperty("sun.security.krb5.debug", "true")^
^System.setProperty("sun.security.spnego.debug", "true")^
^UserGroupInformation.loginUserFromKeytab("dmp@DEVDIP.ORG", "devdmp.keytab")^
^// Load as a DataFrame directly using a Configuration object^
^val df: DataFrame = sqlContext.phoenixTableAsDataFrame(missionSession.config.tableName, Seq("ID"), zkUrl = Some(missionSession.config.zkUrl))^
^df.show(5)^
But when I submit this to YARN for execution, an exception will be thrown. The following is the exception information:
^Tue Feb 19 13:07:53 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: java.io.IOException: Can not send request because relogin is in progress.^
^Tue Feb 19 13:07:53 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: java.io.IOException: Can not send request because relogin is in progress.^
^Tue Feb 19 13:07:53 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: java.io.IOException: Can not send request because relogin is in progress.^
^Tue Feb 19 13:07:54 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: java.io.IOException: Can not send request because relogin is in progress.^
^Tue Feb 19 13:07:54 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: java.io.IOException: Can not send request because relogin is in progress.^
^Tue Feb 19 13:07:55 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: java.io.IOException: Can not send request because relogin is in progress.^
^Tue Feb 19 13:07:57 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: java.io.IOException: Can not send request because relogin is in progress.^
^Tue Feb 19 13:08:01 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
^Tue Feb 19 13:08:11 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
^Tue Feb 19 13:08:21 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
^Tue Feb 19 13:08:31 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
^Tue Feb 19 13:08:41 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
^Tue Feb 19 13:09:01 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
^Tue Feb 19 13:09:22 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
^Tue Feb 19 13:09:42 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
^Tue Feb 19 13:10:02 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
^Tue Feb 19 13:10:22 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
^Tue Feb 19 13:10:42 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
^Tue Feb 19 13:11:02 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
^Tue Feb 19 13:11:22 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
^Tue Feb 19 13:11:42 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
^Tue Feb 19 13:12:02 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
^Tue Feb 19 13:12:23 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
^Tue Feb 19 13:12:43 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
^Tue Feb 19 13:13:03 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
^Tue Feb 19 13:13:23 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
^Tue Feb 19 13:13:43 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
^Tue Feb 19 13:14:03 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
^Tue Feb 19 13:14:23 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
^Tue Feb 19 13:14:43 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
^Tue Feb 19 13:15:03 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
^Tue Feb 19 13:15:23 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
^Tue Feb 19 13:15:44 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
^Tue Feb 19 13:16:04 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
^Tue Feb 19 13:16:24 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
^Tue Feb 19 13:16:44 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
^at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:145)^
^at org.apache.hadoop.hbase.client.HTable.get(HTable.java:386)^
^at org.apache.hadoop.hbase.client.HTable.get(HTable.java:360)^
^at org.apache.hadoop.hbase.MetaTableAccessor.getTableState(MetaTableAccessor.java:1076)^
^at org.apache.hadoop.hbase.MetaTableAccessor.tableExists(MetaTableAccessor.java:399)^
^at org.apache.hadoop.hbase.client.HBaseAdmin$6.rpcCall(HBaseAdmin.java:437)^
^at org.apache.hadoop.hbase.client.HBaseAdmin$6.rpcCall(HBaseAdmin.java:434)^
^at org.apache.hadoop.hbase.client.RpcRetryingCallable.call(RpcRetryingCallable.java:58)^
^at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:107)^
^... 55 more^
^Caused by: java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
^at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:180)^
^at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:390)^
^at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:95)^
^at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410)^
^at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:406)^
^at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103)^
^at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118)^
^at org.apache.hadoop.hbase.ipc.BufferCallBeforeInitHandler.userEventTriggered(BufferCallBeforeInitHandler.java:92)^
^at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:329)^
^at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:315)^
^at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:307)^
^at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:108)^
^at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:329)^
^at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:315)^
^at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:307)^
^at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:108)^
^at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:353)^
^at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:329)^
^at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:315)^
^at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:307)^
^at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.userEventTriggered(DefaultChannelPipeline.java:1377)^
^at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:329)^
^at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:315)^
^at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireUserEventTriggered(DefaultChannelPipeline.java:929)^
^at org.apache.hadoop.hbase.ipc.NettyRpcConnection.failInit(NettyRpcConnection.java:179)^
^at org.apache.hadoop.hbase.ipc.NettyRpcConnection.access$500(NettyRpcConnection.java:71)^
^at org.apache.hadoop.hbase.ipc.NettyRpcConnection$2.operationComplete(NettyRpcConnection.java:247)^
^at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)^
^at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:481)^
^at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420)^
^at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:163)^
^at org.apache.hadoop.hbase.ipc.NettyRpcConnection.saslNegotiate(NettyRpcConnection.java:201)^
^at org.apache.hadoop.hbase.ipc.NettyRpcConnection.access$800(NettyRpcConnection.java:71)^
^at org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:273)^
^at org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:261)^
^at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)^
^at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:500)^
^at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:479)^
^at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420)^
^at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104)^
^at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82)^
^at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:306)^
^at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:341)^
^at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:633)^
^at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)^
^at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)^
^at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)^
^at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)^
^at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)^
^... 1 more^
^Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
^at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)^
^at org.apache.hadoop.hbase.security.AbstractHBaseSaslRpcClient.getInitialResponse(AbstractHBaseSaslRpcClient.java:131)^
^at org.apache.hadoop.hbase.security.NettyHBaseSaslRpcClientHandler$1.run(NettyHBaseSaslRpcClientHandler.java:110)^
^at org.apache.hadoop.hbase.security.NettyHBaseSaslRpcClientHandler$1.run(NettyHBaseSaslRpcClientHandler.java:106)^
^at java.security.AccessController.doPrivileged(Native Method)^
^at javax.security.auth.Subject.doAs(Subject.java:422)^
^at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1688)^
^at org.apache.hadoop.hbase.security.NettyHBaseSaslRpcClientHandler.handlerAdded(NettyHBaseSaslRpcClientHandler.java:106)^
^at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.callHandlerAdded0(DefaultChannelPipeline.java:606)^
^at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.addFirst(DefaultChannelPipeline.java:187)^
^at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.addFirst(DefaultChannelPipeline.java:380)^
^at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.addFirst(DefaultChannelPipeline.java:359)^
^at org.apache.hadoop.hbase.ipc.NettyRpcConnection.saslNegotiate(NettyRpcConnection.java:200)^
^... 18 more^
^Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)^
^at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)^
^at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)^
^at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)^
^at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224)^
^at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)^
^at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)^
^at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192)^
^... 30 more^
The following are my submission parameters:
^spark-submit^
^--master yarn^
^--name PHOENIX_SPARK_PLUGIN^
^--deploy-mode cluster^
^--keytab /tesdmp/keytabs/nnjKorRc37PPPjLf/dmp/testdmp.keytab^
^--principal dmp@TESTDIP.ORG^
^--jars /usr/hdp/3.0.0.0-1634/phoenix/phoenix-5.0.0.3.0.0.0-1634-client.jar^
^--conf spark.yarn.maxAppAttempts=1^
^/opt/workspace/plugin/phoenix-spark-plugin-example-1.10.0-SNAPSHOT-jar-with-dependencies.jar "DMP_CONF=\{\"spark\":{\"sparkMaster\":\"yarn\"},\"zkUrl\":\"jdbc:phoenix:test-dmp5.fengdai.org,test-dmp3.fengdai.org,test-dmp4.fengdai.org:2181/hbase-secure\",\"tableName\":\"DMP_TEST\" ,\"isDS\":true}"^
This program runs on YARN. Do I need to do something extra? Can anyone give me some advice?
> GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
> -----------------------------------------------------------------------------------------------
>
> Key: PHOENIX-5145
> URL: https://issues.apache.org/jira/browse/PHOENIX-5145
> Project: Phoenix
> Issue Type: Bug
> Affects Versions: 5.0.0
> Environment: >HDP 3.0.0
> >Phoenix 5.0.0
> >HBase 2.0.0
> >Spark 2.3.1
> >Hadoop 3.0.1
> Reporter: MariaCarrie
> Priority: Major
> Original Estimate: 72h
> Remaining Estimate: 72h
>
> I can successfully read the data using the local mode. Here is my code:
> ^val sqlContext: SQLContext = missionSession.app.ss.sqlContext^
> ^System.setProperty("sun.security.krb5.debug", "true")^
> ^System.setProperty("sun.security.spnego.debug", "true")^
> ^UserGroupInformation.loginUserFromKeytab("dmp@DEVDIP.ORG", "devdmp.keytab")^
> ^// Load as a DataFrame directly using a Configuration object^
> ^val df: DataFrame = sqlContext.phoenixTableAsDataFrame(missionSession.config.tableName, Seq("ID"), zkUrl = Some(missionSession.config.zkUrl))^
> ^df.show(5)^
> But when I submit this to YARN for execution, an exception will be thrown. The following is the exception information:
> ^Tue Feb 19 13:07:53 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: java.io.IOException: Can not send request because relogin is in progress.^
> ^Tue Feb 19 13:07:53 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: java.io.IOException: Can not send request because relogin is in progress.^
> ^Tue Feb 19 13:07:53 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: java.io.IOException: Can not send request because relogin is in progress.^
> ^Tue Feb 19 13:07:54 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: java.io.IOException: Can not send request because relogin is in progress.^
> ^Tue Feb 19 13:07:54 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: java.io.IOException: Can not send request because relogin is in progress.^
> ^Tue Feb 19 13:07:55 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: java.io.IOException: Can not send request because relogin is in progress.^
> ^Tue Feb 19 13:07:57 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: java.io.IOException: Can not send request because relogin is in progress.^
> ^Tue Feb 19 13:08:01 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
> ^Tue Feb 19 13:08:11 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
> ^Tue Feb 19 13:08:21 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
> ^Tue Feb 19 13:08:31 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
> ^Tue Feb 19 13:08:41 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
> ^Tue Feb 19 13:09:01 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
> ^Tue Feb 19 13:09:22 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
> ^Tue Feb 19 13:09:42 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
> ^Tue Feb 19 13:10:02 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
> ^Tue Feb 19 13:10:22 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
> ^Tue Feb 19 13:10:42 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
> ^Tue Feb 19 13:11:02 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
> ^Tue Feb 19 13:11:22 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
> ^Tue Feb 19 13:11:42 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
> ^Tue Feb 19 13:12:02 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
> ^Tue Feb 19 13:12:23 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
> ^Tue Feb 19 13:12:43 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
> ^Tue Feb 19 13:13:03 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
> ^Tue Feb 19 13:13:23 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
> ^Tue Feb 19 13:13:43 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
> ^Tue Feb 19 13:14:03 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
> ^Tue Feb 19 13:14:23 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
> ^Tue Feb 19 13:14:43 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
> ^Tue Feb 19 13:15:03 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
> ^Tue Feb 19 13:15:23 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
> ^Tue Feb 19 13:15:44 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
> ^Tue Feb 19 13:16:04 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
> ^Tue Feb 19 13:16:24 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
> ^Tue Feb 19 13:16:44 CST 2019, RpcRetryingCaller\{globalStartTime=1550552873361, pause=100, maxAttempts=36}, java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
> ^at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:145)^
> ^at org.apache.hadoop.hbase.client.HTable.get(HTable.java:386)^
> ^at org.apache.hadoop.hbase.client.HTable.get(HTable.java:360)^
> ^at org.apache.hadoop.hbase.MetaTableAccessor.getTableState(MetaTableAccessor.java:1076)^
> ^at org.apache.hadoop.hbase.MetaTableAccessor.tableExists(MetaTableAccessor.java:399)^
> ^at org.apache.hadoop.hbase.client.HBaseAdmin$6.rpcCall(HBaseAdmin.java:437)^
> ^at org.apache.hadoop.hbase.client.HBaseAdmin$6.rpcCall(HBaseAdmin.java:434)^
> ^at org.apache.hadoop.hbase.client.RpcRetryingCallable.call(RpcRetryingCallable.java:58)^
> ^at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:107)^
> ^... 55 more^
> ^Caused by: java.io.IOException: Call to test-dmp5.fengdai.org/10.200.162.26:16020 failed on local exception: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
> ^at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:180)^
> ^at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:390)^
> ^at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:95)^
> ^at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410)^
> ^at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:406)^
> ^at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103)^
> ^at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118)^
> ^at org.apache.hadoop.hbase.ipc.BufferCallBeforeInitHandler.userEventTriggered(BufferCallBeforeInitHandler.java:92)^
> ^at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:329)^
> ^at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:315)^
> ^at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:307)^
> ^at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:108)^
> ^at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:329)^
> ^at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:315)^
> ^at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:307)^
> ^at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:108)^
> ^at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:353)^
> ^at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:329)^
> ^at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:315)^
> ^at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:307)^
> ^at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.userEventTriggered(DefaultChannelPipeline.java:1377)^
> ^at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:329)^
> ^at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:315)^
> ^at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireUserEventTriggered(DefaultChannelPipeline.java:929)^
> ^at org.apache.hadoop.hbase.ipc.NettyRpcConnection.failInit(NettyRpcConnection.java:179)^
> ^at org.apache.hadoop.hbase.ipc.NettyRpcConnection.access$500(NettyRpcConnection.java:71)^
> ^at org.apache.hadoop.hbase.ipc.NettyRpcConnection$2.operationComplete(NettyRpcConnection.java:247)^
> ^at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)^
> ^at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:481)^
> ^at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420)^
> ^at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:163)^
> ^at org.apache.hadoop.hbase.ipc.NettyRpcConnection.saslNegotiate(NettyRpcConnection.java:201)^
> ^at org.apache.hadoop.hbase.ipc.NettyRpcConnection.access$800(NettyRpcConnection.java:71)^
> ^at org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:273)^
> ^at org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:261)^
> ^at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)^
> ^at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:500)^
> ^at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:479)^
> ^at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420)^
> ^at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104)^
> ^at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82)^
> ^at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:306)^
> ^at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:341)^
> ^at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:633)^
> ^at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)^
> ^at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)^
> ^at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)^
> ^at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)^
> ^at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)^
> ^... 1 more^
> ^Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]^
> ^at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)^
> ^at org.apache.hadoop.hbase.security.AbstractHBaseSaslRpcClient.getInitialResponse(AbstractHBaseSaslRpcClient.java:131)^
> ^at org.apache.hadoop.hbase.security.NettyHBaseSaslRpcClientHandler$1.run(NettyHBaseSaslRpcClientHandler.java:110)^
> ^at org.apache.hadoop.hbase.security.NettyHBaseSaslRpcClientHandler$1.run(NettyHBaseSaslRpcClientHandler.java:106)^
> ^at java.security.AccessController.doPrivileged(Native Method)^
> ^at javax.security.auth.Subject.doAs(Subject.java:422)^
> ^at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1688)^
> ^at org.apache.hadoop.hbase.security.NettyHBaseSaslRpcClientHandler.handlerAdded(NettyHBaseSaslRpcClientHandler.java:106)^
> ^at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.callHandlerAdded0(DefaultChannelPipeline.java:606)^
> ^at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.addFirst(DefaultChannelPipeline.java:187)^
> ^at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.addFirst(DefaultChannelPipeline.java:380)^
> ^at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.addFirst(DefaultChannelPipeline.java:359)^
> ^at org.apache.hadoop.hbase.ipc.NettyRpcConnection.saslNegotiate(NettyRpcConnection.java:200)^
> ^... 18 more^
> ^Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)^
> ^at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)^
> ^at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)^
> ^at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)^
> ^at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224)^
> ^at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)^
> ^at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)^
> ^at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192)^
> ^... 30 more^
> The following are my submission parameters:
> ^spark-submit^
> ^--master yarn^
> ^--name PHOENIX_SPARK_PLUGIN^
> ^--deploy-mode cluster^
> ^--keytab /tesdmp/keytabs/nnjKorRc37PPPjLf/dmp/testdmp.keytab^
> ^--principal dmp@TESTDIP.ORG^
> ^--jars /usr/hdp/3.0.0.0-1634/phoenix/phoenix-5.0.0.3.0.0.0-1634-client.jar^
> ^--conf spark.yarn.maxAppAttempts=1^
> ^/path/phoenix-spark-plugin.jar^
> This program runs on YARN. Do I need to do something extra? Can anyone give me some advice?
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)