Zeppelin stops working after a few days of uptime with RemoteEndpoint unavailable, outgoing connection not open

[Clemens Valiente <Cl...@trivago.com>]

Hi,

I am running a zeppelin 0.9.0-snapshot instance and it works fine most of the time, but after running it for a while we eventually encounter issues.

The notebooks suddenly show up empty (are not loading) and the logfile (apache-zeppelin-dev.log) shows the following error:

 INFO [2018-07-13 09:57:38,484] ({qtp2091156596-404} NotebookServer.java[sendNote]:865) - New operation from 172.21.95.21 : 62462 : johndoe : GET_NOTE : 2DJK3XG1R
ERROR [2018-07-13 09:57:38,484] ({qtp2091156596-404} NotebookServer.java[onMessage]:395) - Can't handle message: {"op":"GET_NOTE","data":{"id":"2DJK3XG1R"},"principal":"johndoe","ticket":"52ceedde-5654-49b7-88c1-bcd95ec9ae8c","roles":"[]"}
org.eclipse.jetty.websocket.api.WebSocketException: RemoteEndpoint unavailable, outgoing connection not open
at org.eclipse.jetty.websocket.common.WebSocketSession.getRemote(WebSocketSession.java:252)
at org.apache.zeppelin.socket.NotebookSocket.send(NotebookSocket.java:70)
at org.apache.zeppelin.socket.NotebookServer.broadcast(NotebookServer.java:553)
at org.apache.zeppelin.socket.NotebookServer.checkCollaborativeStatus(NotebookServer.java:496)
at org.apache.zeppelin.socket.NotebookServer.removeConnectionFromNote(NotebookServer.java:457)
at org.apache.zeppelin.socket.NotebookServer.removeConnectionFromAllNote(NotebookServer.java:471)
at org.apache.zeppelin.socket.NotebookServer.addConnectionToNote(NotebookServer.java:437)
at org.apache.zeppelin.socket.NotebookServer.sendNote(NotebookServer.java:884)
at org.apache.zeppelin.socket.NotebookServer.onMessage(NotebookServer.java:249)
at org.apache.zeppelin.socket.NotebookSocket.onWebSocketText(NotebookSocket.java:58)
at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextMessage(JettyListenerEventDriver.java:128)
at org.eclipse.jetty.websocket.common.message.SimpleTextMessage.messageComplete(SimpleTextMessage.java:69)
at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.appendMessage(AbstractEventDriver.java:65)
at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextFrame(JettyListenerEventDriver.java:122)
at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.incomingFrame(AbstractEventDriver.java:161)
at org.eclipse.jetty.websocket.common.WebSocketSession.incomingFrame(WebSocketSession.java:309)
at org.eclipse.jetty.websocket.common.extensions.ExtensionStack.incomingFrame(ExtensionStack.java:214)
at org.eclipse.jetty.websocket.common.Parser.notifyFrame(Parser.java:220)
at org.eclipse.jetty.websocket.common.Parser.parse(Parser.java:258)
at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.readParse(AbstractWebSocketConnection.java:632)
at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:480)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:748)



after restarting the server with zeppelin-daemon.sh restart, everything works again (for a while).

this happens for all users on different systems, after logging in/logging out, so it doesn't look like a client related problem.

The little websocket indicator next to the username in the webUI shows up green, so I don't know which websocket this error is referring to.

I didn't find this error in any tasks or reported issue. Did anyone else encounter this problem?

Any hints on how I can further debug this?


Cheers

Clemens

Re: Zeppelin stops working after a few days of uptime with RemoteEndpoint unavailable, outgoing connection not open

[Jeff Zhang <zj...@gmail.com>]

Could you create a ticket for this issue ?


Clemens Valiente <Cl...@trivago.com>于2018年7月13日周五 下午6:14写道：

> Hi,
>
> I am running a zeppelin 0.9.0-snapshot instance and it works fine most of
> the time, but after running it for a while we eventually encounter issues.
>
> The notebooks suddenly show up empty (are not loading) and the logfile
> (apache-zeppelin-dev.log) shows the following error:
>
>  INFO [2018-07-13 09:57:38,484] ({qtp2091156596-404}
> NotebookServer.java[sendNote]:865) - New operation from 172.21.95.21 :
> 62462 : johndoe : GET_NOTE : 2DJK3XG1R
> ERROR [2018-07-13 09:57:38,484] ({qtp2091156596-404}
> NotebookServer.java[onMessage]:395) - Can't handle message:
> {"op":"GET_NOTE","data":{"id":"2DJK3XG1R"},"principal":"johndoe
> ","ticket":"52ceedde-5654-49b7-88c1-bcd95ec9ae8c","roles":"[]"}
> org.eclipse.jetty.websocket.api.WebSocketException: RemoteEndpoint
> unavailable, outgoing connection not open
> at
> org.eclipse.jetty.websocket.common.WebSocketSession.getRemote(WebSocketSession.java:252)
> at org.apache.zeppelin.socket.NotebookSocket.send(NotebookSocket.java:70)
> at
> org.apache.zeppelin.socket.NotebookServer.broadcast(NotebookServer.java:553)
> at
> org.apache.zeppelin.socket.NotebookServer.checkCollaborativeStatus(NotebookServer.java:496)
> at
> org.apache.zeppelin.socket.NotebookServer.removeConnectionFromNote(NotebookServer.java:457)
> at
> org.apache.zeppelin.socket.NotebookServer.removeConnectionFromAllNote(NotebookServer.java:471)
> at
> org.apache.zeppelin.socket.NotebookServer.addConnectionToNote(NotebookServer.java:437)
> at
> org.apache.zeppelin.socket.NotebookServer.sendNote(NotebookServer.java:884)
> at
> org.apache.zeppelin.socket.NotebookServer.onMessage(NotebookServer.java:249)
> at
> org.apache.zeppelin.socket.NotebookSocket.onWebSocketText(NotebookSocket.java:58)
> at
> org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextMessage(JettyListenerEventDriver.java:128)
> at
> org.eclipse.jetty.websocket.common.message.SimpleTextMessage.messageComplete(SimpleTextMessage.java:69)
> at
> org.eclipse.jetty.websocket.common.events.AbstractEventDriver.appendMessage(AbstractEventDriver.java:65)
> at
> org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextFrame(JettyListenerEventDriver.java:122)
> at
> org.eclipse.jetty.websocket.common.events.AbstractEventDriver.incomingFrame(AbstractEventDriver.java:161)
> at
> org.eclipse.jetty.websocket.common.WebSocketSession.incomingFrame(WebSocketSession.java:309)
> at
> org.eclipse.jetty.websocket.common.extensions.ExtensionStack.incomingFrame(ExtensionStack.java:214)
> at org.eclipse.jetty.websocket.common.Parser.notifyFrame(Parser.java:220)
> at org.eclipse.jetty.websocket.common.Parser.parse(Parser.java:258)
> at
> org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.readParse(AbstractWebSocketConnection.java:632)
> at
> org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:480)
> at
> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
> at java.lang.Thread.run(Thread.java:748)
>
>
> after restarting the server with zeppelin-daemon.sh restart, everything
> works again (for a while).
>
> this happens for all users on different systems, after logging in/logging
> out, so it doesn't look like a client related problem.
>
> The little websocket indicator next to the username in the webUI shows up
> green, so I don't know which websocket this error is referring to.
>
> I didn't find this error in any tasks or reported issue. Did anyone else
> encounter this problem?
>
> Any hints on how I can further debug this?
>
>
> Cheers
>
> Clemens
>