You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zeppelin.apache.org by "Greg Senia (JIRA)" <ji...@apache.org> on 2018/03/21 16:23:00 UTC
[jira] [Created] (ZEPPELIN-3356) Zeppelin FileSystemStorage
reloginfrom keytab needed
Greg Senia created ZEPPELIN-3356:
------------------------------------
Summary: Zeppelin FileSystemStorage reloginfrom keytab needed
Key: ZEPPELIN-3356
URL: https://issues.apache.org/jira/browse/ZEPPELIN-3356
Project: Zeppelin
Issue Type: Bug
Affects Versions: 0.8.0, 0.9.0
Reporter: Greg Senia
Assignee: Greg Senia
Fix For: 0.8.0
After running the current Zeppelin snapshot. Users were complaining that they could no longer make notebook changes as we are storing the notebook configuration and settings in HDFS. When I went to look at the Zeppelin logs I saw the following below. I propose some slight changes as similar code is used in KMSClientProvider in core Hadoop and I have used similar code in homegrown HDFS management software. This error occurs exactly 7days after the initial keytab is logged in at startup. Unfortunately getCurrentUser doesn't seem to relogin a keytab once the ticket reaches max expiration in our case 7days.
INFO [2018-03-16 08:50:01,275] (\{qtp2081303229-1297} RetryInvocationHandler.java[invoke]:144) - Exception while invoking create of class ClientNamenodeProtocolTranslatorPB over ha20p01nn.pro
d.hdp.example.com/10.70.1.6:8020. Trying to fail over immediately.
java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism l
evel: Failed to find any Kerberos tgt)]; Host Details : local host is: "ha21p05en.prod.hdp.example.com/10.70.8.211"; destination host is: "ha20p01nn.prod.hdp.example.com":8020;
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:776)
at org.apache.hadoop.ipc.Client.call(Client.java:1479)
at org.apache.hadoop.ipc.Client.call(Client.java:1412)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy24.create(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:296)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy25.create(Unknown Source)
at org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1648)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1689)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1624)
at org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:448)
at org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:444)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:459)
at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:387)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:789)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:778)
at org.apache.zeppelin.notebook.FileSystemStorage$6.call(FileSystemStorage.java:144)
at org.apache.zeppelin.notebook.FileSystemStorage$6.call(FileSystemStorage.java:138)
at org.apache.zeppelin.notebook.FileSystemStorage$7.run(FileSystemStorage.java:162)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
at org.apache.zeppelin.notebook.FileSystemStorage.callHdfsOperation(FileSystemStorage.java:159)
at org.apache.zeppelin.notebook.FileSystemStorage.writeFile(FileSystemStorage.java:138)
at org.apache.zeppelin.storage.FileSystemConfigStorage.save(FileSystemConfigStorage.java:66)
at org.apache.zeppelin.interpreter.InterpreterSettingManager.saveToFile(InterpreterSettingManager.java:285)
at org.apache.zeppelin.interpreter.InterpreterSettingManager.setInterpreterBinding(InterpreterSettingManager.java:705)
at org.apache.zeppelin.notebook.Notebook.bindInterpretersToNote(Notebook.java:275)
at org.apache.zeppelin.notebook.Notebook.createNote(Notebook.java:166)
at org.apache.zeppelin.socket.NotebookServer.createNote(NotebookServer.java:1037)
at org.apache.zeppelin.socket.NotebookServer.onMessage(NotebookServer.java:226)
at org.apache.zeppelin.socket.NotebookSocket.onWebSocketText(NotebookSocket.java:59)
at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextMessage(JettyListenerEventDriver.java:128)
at org.eclipse.jetty.websocket.common.message.SimpleTextMessage.messageComplete(SimpleTextMessage.java:69)
at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.appendMessage(AbstractEventDriver.java:65)
at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextFrame(JettyListenerEventDriver.java:122)
at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.incomingFrame(AbstractEventDriver.java:161)
at org.eclipse.jetty.websocket.common.WebSocketSession.incomingFrame(WebSocketSession.java:309)
at org.eclipse.jetty.websocket.common.extensions.ExtensionStack.incomingFrame(ExtensionStack.java:214)
at org.eclipse.jetty.websocket.common.Parser.notifyFrame(Parser.java:220)
at org.eclipse.jetty.websocket.common.Parser.parse(Parser.java:258)
at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.readParse(AbstractWebSocketConnection.java:632)
at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:480)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:687)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)