You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@livy.apache.org by "Jeff Zhang (JIRA)" <ji...@apache.org> on 2018/01/26 09:49:00 UTC
[jira] [Commented] (LIVY-436) Client RPC channel closed
unexpectedly
[ https://issues.apache.org/jira/browse/LIVY-436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16340810#comment-16340810 ]
Jeff Zhang commented on LIVY-436:
---------------------------------
Could you check the yarn app log ? Usually yarn app log contains the root cause.
> Client RPC channel closed unexpectedly
> --------------------------------------
>
> Key: LIVY-436
> URL: https://issues.apache.org/jira/browse/LIVY-436
> Project: Livy
> Issue Type: Bug
> Affects Versions: 0.4.0
> Environment: Cloudera CDH 5.14
> Spark 2.2 / Scala
> Apache Livy 4.0
> Apache Zeppelin 0.7.3
> Ubuntu 16.04 LTS
> Reporter: Maziyar PANAHI
> Priority: Major
>
> Hi,
> We are getting this error time to time and we have to re-create the session and start the execution of our codes from the beginning. However, it is really hard to find out why the error happens since it's not always the same code nor the Spark commands. The session is suddenly closed without any WARN or ERROR in different time with different code.
> I checked the logs from YARN and Spark and there is nothing that indicates the session was closed because of any error or warning before hand.
> I have Livy server and Apache Zeppelin on the same machine so the IP address is the same for both of them.
> I am just wondering, what could cause this error in a multi-user environment. (number of connections, open files, heap size, etc.). Is it the connection unstable or is there some configs that could help preventing this sudden death of sessions.
>
> {code:java}
> 18/01/26 10:08:59 WARN RSCClient: Client RPC channel closed unexpectedly.
> 18/01/26 10:08:59 WARN Rpc: [id: 0x94c07360, /Livy-IP-Address:57846 :> executor-hostname/executor-IP-Address:10000] CLOSE()
> 18/01/26 10:08:59 WARN RSCClient: Error stopping RPC.
> io.netty.util.concurrent.BlockingOperationException: DefaultChannelPromise@1062b60(uncancellable)
> at io.netty.util.concurrent.DefaultPromise.checkDeadLock(DefaultPromise.java:390)
> at io.netty.channel.DefaultChannelPromise.checkDeadLock(DefaultChannelPromise.java:157)
> at io.netty.util.concurrent.DefaultPromise.await(DefaultPromise.java:251)
> at io.netty.channel.DefaultChannelPromise.await(DefaultChannelPromise.java:129)
> at io.netty.channel.DefaultChannelPromise.await(DefaultChannelPromise.java:28)
> at io.netty.util.concurrent.DefaultPromise.sync(DefaultPromise.java:218)
> at io.netty.channel.DefaultChannelPromise.sync(DefaultChannelPromise.java:117)
> at io.netty.channel.DefaultChannelPromise.sync(DefaultChannelPromise.java:28)
> at org.apache.livy.rsc.rpc.Rpc.close(Rpc.java:307)
> at org.apache.livy.rsc.RSCClient.stop(RSCClient.java:232)
> at org.apache.livy.rsc.RSCClient$2$1.onSuccess(RSCClient.java:129)
> at org.apache.livy.rsc.RSCClient$2$1.onSuccess(RSCClient.java:123)
> at org.apache.livy.rsc.Utils$2.operationComplete(Utils.java:108)
> at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)
> at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:567)
> at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:406)
> at io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82)
> at io.netty.channel.AbstractChannel$CloseFuture.setClosed(AbstractChannel.java:956)
> at io.netty.channel.AbstractChannel$AbstractUnsafe.doClose0(AbstractChannel.java:608)
> at io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:586)
> at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.closeOnRead(AbstractNioByteChannel.java:71)
> at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:158)
> at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
> at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
> at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
> at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
> at java.lang.Thread.run(Thread.java:748)
> 18/01/26 10:08:59 WARN Rpc: [id: 0x94c07360,/Livy-IP-Address:57846 :> executor-hostname/executor-IP-Address:10000] INACTIVE
> 18/01/26 10:08:59 WARN Rpc: [id: 0x94c07360,/Livy-IP-Address:57846 :> executor-hostname/executor-IP-Address:10000] UNREGISTERED
> 18/01/26 10:09:09 ERROR SessionServlet$: internal error
> java.lang.IllegalStateException: Session is in state dead
> at org.apache.livy.server.interactive.InteractiveSession.ensureRunning(InteractiveSession.scala:568)
> at org.apache.livy.server.interactive.InteractiveSession.executeStatement(InteractiveSession.scala:490)
> at org.apache.livy.server.interactive.InteractiveSessionServlet$$anonfun$11$$anonfun$apply$6.apply(InteractiveSessionServlet.scala:124)
> at org.apache.livy.server.interactive.InteractiveSessionServlet$$anonfun$11$$anonfun$apply$6.apply(InteractiveSessionServlet.scala:123)
> at org.apache.livy.server.interactive.SessionHeartbeatNotifier$$anonfun$withModifyAccessSession$1.apply(SessionHeartbeat.scala:76)
> at org.apache.livy.server.interactive.SessionHeartbeatNotifier$$anonfun$withModifyAccessSession$1.apply(SessionHeartbeat.scala:74)
> at org.apache.livy.server.SessionServlet.doWithSession(SessionServlet.scala:221)
> at org.apache.livy.server.SessionServlet.withModifyAccessSession(SessionServlet.scala:212)
> at org.apache.livy.server.interactive.InteractiveSessionServlet.org$apache$livy$server$interactive$SessionHeartbeatNotifier$$super$withModifyAccessSession(InteractiveSessionServlet.scala:40)
> at org.apache.livy.server.interactive.SessionHeartbeatNotifier$class.withModifyAccessSession(SessionHeartbeat.scala:74)
> at org.apache.livy.server.interactive.InteractiveSessionServlet.withModifyAccessSession(InteractiveSessionServlet.scala:40)
> at org.apache.livy.server.interactive.InteractiveSessionServlet$$anonfun$11.apply(InteractiveSessionServlet.scala:123)
> at org.apache.livy.server.interactive.InteractiveSessionServlet$$anonfun$11.apply(InteractiveSessionServlet.scala:122)
> at org.apache.livy.server.JsonServlet.org$apache$livy$server$JsonServlet$$doAction(JsonServlet.scala:113)
> at org.apache.livy.server.JsonServlet$$anonfun$jpost$1.apply(JsonServlet.scala:75)
> at org.scalatra.ScalatraBase$class.org$scalatra$ScalatraBase$$liftAction(ScalatraBase.scala:270)
> at org.scalatra.ScalatraBase$$anonfun$invoke$1.apply(ScalatraBase.scala:265)
> at org.scalatra.ScalatraBase$$anonfun$invoke$1.apply(ScalatraBase.scala:265)
> at org.scalatra.ApiFormats$class.withRouteMultiParams(ApiFormats.scala:178)
> at org.apache.livy.server.JsonServlet.withRouteMultiParams(JsonServlet.scala:39)
> at org.scalatra.ScalatraBase$class.invoke(ScalatraBase.scala:264)
> at org.scalatra.ScalatraServlet.invoke(ScalatraServlet.scala:49)
> at org.scalatra.ScalatraBase$$anonfun$runRoutes$1$$anonfun$apply$8.apply(ScalatraBase.scala:240)
> at org.scalatra.ScalatraBase$$anonfun$runRoutes$1$$anonfun$apply$8.apply(ScalatraBase.scala:238)
> at scala.Option.flatMap(Option.scala:170)
> at org.scalatra.ScalatraBase$$anonfun$runRoutes$1.apply(ScalatraBase.scala:238)
> at org.scalatra.ScalatraBase$$anonfun$runRoutes$1.apply(ScalatraBase.scala:237)
> at scala.collection.immutable.Stream.flatMap(Stream.scala:446)
> at org.scalatra.ScalatraBase$class.runRoutes(ScalatraBase.scala:237)
> at org.scalatra.ScalatraServlet.runRoutes(ScalatraServlet.scala:49)
> at org.scalatra.ScalatraBase$class.runActions$1(ScalatraBase.scala:163)
> at org.scalatra.ScalatraBase$$anonfun$executeRoutes$1.apply$mcV$sp(ScalatraBase.scala:175)
> at org.scalatra.ScalatraBase$$anonfun$executeRoutes$1.apply(ScalatraBase.scala:175)
> at org.scalatra.ScalatraBase$$anonfun$executeRoutes$1.apply(ScalatraBase.scala:175)
> at org.scalatra.ScalatraBase$class.org$scalatra$ScalatraBase$$cradleHalt(ScalatraBase.scala:193)
> at org.scalatra.ScalatraBase$class.executeRoutes(ScalatraBase.scala:175)
> at org.scalatra.ScalatraServlet.executeRoutes(ScalatraServlet.scala:49)
> at org.scalatra.ScalatraBase$$anonfun$handle$1.apply$mcV$sp(ScalatraBase.scala:113)
> at org.scalatra.ScalatraBase$$anonfun$handle$1.apply(ScalatraBase.scala:113)
> at org.scalatra.ScalatraBase$$anonfun$handle$1.apply(ScalatraBase.scala:113)
> at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
> at org.scalatra.DynamicScope$class.withResponse(DynamicScope.scala:80)
> at org.scalatra.ScalatraServlet.withResponse(ScalatraServlet.scala:49)
> at org.scalatra.DynamicScope$$anonfun$withRequestResponse$1.apply(DynamicScope.scala:60)
> at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
> at org.scalatra.DynamicScope$class.withRequest(DynamicScope.scala:71)
> at org.scalatra.ScalatraServlet.withRequest(ScalatraServlet.scala:49)
> at org.scalatra.DynamicScope$class.withRequestResponse(DynamicScope.scala:59)
> at org.scalatra.ScalatraServlet.withRequestResponse(ScalatraServlet.scala:49)
> at org.scalatra.ScalatraBase$class.handle(ScalatraBase.scala:111)
> at org.scalatra.ScalatraServlet.org$scalatra$servlet$ServletBase$$super$handle(ScalatraServlet.scala:49)
> at org.scalatra.servlet.ServletBase$class.handle(ServletBase.scala:43)
> at org.apache.livy.server.SessionServlet.org$scalatra$MethodOverride$$super$handle(SessionServlet.scala:39)
> at org.scalatra.MethodOverride$class.handle(MethodOverride.scala:28)
> at org.apache.livy.server.SessionServlet.org$scalatra$GZipSupport$$super$handle(SessionServlet.scala:39)
> at org.scalatra.GZipSupport$$anonfun$handle$1.apply$mcV$sp(GZipSupport.scala:36)
> at org.scalatra.GZipSupport$$anonfun$handle$1.apply(GZipSupport.scala:19)
> at org.scalatra.GZipSupport$$anonfun$handle$1.apply(GZipSupport.scala:19)
> at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
> at org.scalatra.DynamicScope$class.withResponse(DynamicScope.scala:80)
> at org.scalatra.ScalatraServlet.withResponse(ScalatraServlet.scala:49)
> at org.scalatra.DynamicScope$$anonfun$withRequestResponse$1.apply(DynamicScope.scala:60)
> at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
> at org.scalatra.DynamicScope$class.withRequest(DynamicScope.scala:71)
> at org.scalatra.ScalatraServlet.withRequest(ScalatraServlet.scala:49)
> at org.scalatra.DynamicScope$class.withRequestResponse(DynamicScope.scala:59)
> at org.scalatra.ScalatraServlet.withRequestResponse(ScalatraServlet.scala:49) [7557/9847]
> at org.scalatra.GZipSupport$class.handle(GZipSupport.scala:18)
> at org.apache.livy.server.interactive.InteractiveSessionServlet.org$scalatra$servlet$FileUploadSupport$$super$handle(InteractiveSessionServlet.scala:40)
> at org.scalatra.servlet.FileUploadSupport$class.handle(FileUploadSupport.scala:93)
> at org.apache.livy.server.interactive.InteractiveSessionServlet.handle(InteractiveSessionServlet.scala:40)
> at org.scalatra.ScalatraServlet.service(ScalatraServlet.scala:54)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
> at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:587)
> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
> at org.eclipse.jetty.server.Server.handle(Server.java:499)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
> at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
> at java.lang.Thread.run(Thread.java:748)
> 18/01/26 10:09:11 ERROR SessionServlet$: internal error
> java.lang.IllegalStateException: Session is in state dead
> at org.apache.livy.server.interactive.InteractiveSession.ensureRunning(InteractiveSession.scala:568)
> at org.apache.livy.server.interactive.InteractiveSession.executeStatement(InteractiveSession.scala:490)
> at org.apache.livy.server.interactive.InteractiveSessionServlet$$anonfun$11$$anonfun$apply$6.apply(InteractiveSessionServlet.scala:124)
> at org.apache.livy.server.interactive.InteractiveSessionServlet$$anonfun$11$$anonfun$apply$6.apply(InteractiveSessionServlet.scala:123)
> at org.apache.livy.server.interactive.SessionHeartbeatNotifier$$anonfun$withModifyAccessSession$1.apply(SessionHeartbeat.scala:76)
> at org.apache.livy.server.interactive.SessionHeartbeatNotifier$$anonfun$withModifyAccessSession$1.apply(SessionHeartbeat.scala:74)
> at org.apache.livy.server.SessionServlet.doWithSession(SessionServlet.scala:221)
> at org.apache.livy.server.SessionServlet.withModifyAccessSession(SessionServlet.scala:212)
> at org.apache.livy.server.interactive.InteractiveSessionServlet.org$apache$livy$server$interactive$SessionHeartbeatNotifier$$super$withModifyAccessSession(InteractiveSessionServlet.scala:40)
> at org.apache.livy.server.interactive.SessionHeartbeatNotifier$class.withModifyAccessSession(SessionHeartbeat.scala:74)
> at org.apache.livy.server.interactive.InteractiveSessionServlet.withModifyAccessSession(InteractiveSessionServlet.scala:40)
> at org.apache.livy.server.interactive.InteractiveSessionServlet$$anonfun$11.apply(InteractiveSessionServlet.scala:123)
> at org.apache.livy.server.interactive.InteractiveSessionServlet$$anonfun$11.apply(InteractiveSessionServlet.scala:122)
> at org.apache.livy.server.JsonServlet.org$apache$livy$server$JsonServlet$$doAction(JsonServlet.scala:113)
> at org.apache.livy.server.JsonServlet$$anonfun$jpost$1.apply(JsonServlet.scala:75)
> at org.scalatra.ScalatraBase$class.org$scalatra$ScalatraBase$$liftAction(ScalatraBase.scala:270)
> at org.scalatra.ScalatraBase$$anonfun$invoke$1.apply(ScalatraBase.scala:265)
> at org.scalatra.ScalatraBase$$anonfun$invoke$1.apply(ScalatraBase.scala:265)
> at org.scalatra.ApiFormats$class.withRouteMultiParams(ApiFormats.scala:178)
> at org.apache.livy.server.JsonServlet.withRouteMultiParams(JsonServlet.scala:39)
> at org.scalatra.ScalatraBase$class.invoke(ScalatraBase.scala:264)
> at org.scalatra.ScalatraServlet.invoke(ScalatraServlet.scala:49)
> at org.scalatra.ScalatraBase$$anonfun$runRoutes$1$$anonfun$apply$8.apply(ScalatraBase.scala:240)
> at org.scalatra.ScalatraBase$$anonfun$runRoutes$1$$anonfun$apply$8.apply(ScalatraBase.scala:238)
> at scala.Option.flatMap(Option.scala:170)
> at org.scalatra.ScalatraBase$$anonfun$runRoutes$1.apply(ScalatraBase.scala:238)
> at org.scalatra.ScalatraBase$$anonfun$runRoutes$1.apply(ScalatraBase.scala:237)
> at scala.collection.immutable.Stream.flatMap(Stream.scala:446)
> at org.scalatra.ScalatraBase$class.runRoutes(ScalatraBase.scala:237)
> at org.scalatra.ScalatraServlet.runRoutes(ScalatraServlet.scala:49)
> at org.scalatra.ScalatraBase$class.runActions$1(ScalatraBase.scala:163)
> at org.scalatra.ScalatraBase$$anonfun$executeRoutes$1.apply$mcV$sp(ScalatraBase.scala:175)
> at org.scalatra.ScalatraBase$$anonfun$executeRoutes$1.apply(ScalatraBase.scala:175)
> at org.scalatra.ScalatraBase$$anonfun$executeRoutes$1.apply(ScalatraBase.scala:175)
> at org.scalatra.ScalatraBase$class.org$scalatra$ScalatraBase$$cradleHalt(ScalatraBase.scala:193)
> at org.scalatra.ScalatraBase$class.executeRoutes(ScalatraBase.scala:175)
> at org.scalatra.ScalatraServlet.executeRoutes(ScalatraServlet.scala:49)
> at org.scalatra.ScalatraBase$$anonfun$handle$1.apply$mcV$sp(ScalatraBase.scala:113)
> at org.scalatra.ScalatraBase$$anonfun$handle$1.apply(ScalatraBase.scala:113)
> at org.scalatra.ScalatraBase$$anonfun$handle$1.apply(ScalatraBase.scala:113)
> at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
> at org.scalatra.DynamicScope$class.withResponse(DynamicScope.scala:80)
> at org.scalatra.ScalatraServlet.withResponse(ScalatraServlet.scala:49)
> at org.scalatra.DynamicScope$$anonfun$withRequestResponse$1.apply(DynamicScope.scala:60)
> at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
> at org.scalatra.DynamicScope$class.withRequest(DynamicScope.scala:71)
> at org.scalatra.ScalatraServlet.withRequest(ScalatraServlet.scala:49)
> at org.scalatra.DynamicScope$class.withRequestResponse(DynamicScope.scala:59)
> at org.scalatra.ScalatraServlet.withRequestResponse(ScalatraServlet.scala:49)
> at org.scalatra.ScalatraBase$class.handle(ScalatraBase.scala:111)
> at org.scalatra.ScalatraServlet.org$scalatra$servlet$ServletBase$$super$handle(ScalatraServlet.scala:49)
> at org.scalatra.servlet.ServletBase$class.handle(ServletBase.scala:43)
> at org.apache.livy.server.SessionServlet.org$scalatra$MethodOverride$$super$handle(SessionServlet.scala:39)
> at org.scalatra.MethodOverride$class.handle(MethodOverride.scala:28)
> at org.apache.livy.server.SessionServlet.org$scalatra$GZipSupport$$super$handle(SessionServlet.scala:39)
> at org.scalatra.GZipSupport$$anonfun$handle$1.apply$mcV$sp(GZipSupport.scala:36)
> at org.scalatra.GZipSupport$$anonfun$handle$1.apply(GZipSupport.scala:19)
> at org.scalatra.GZipSupport$$anonfun$handle$1.apply(GZipSupport.scala:19)
> at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
> at org.scalatra.DynamicScope$class.withResponse(DynamicScope.scala:80)
> at org.scalatra.ScalatraServlet.withResponse(ScalatraServlet.scala:49)
> at org.scalatra.DynamicScope$$anonfun$withRequestResponse$1.apply(DynamicScope.scala:60)
> at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
> at org.scalatra.DynamicScope$class.withRequest(DynamicScope.scala:71)
> at org.scalatra.ScalatraServlet.withRequest(ScalatraServlet.scala:49)
> at org.scalatra.DynamicScope$class.withRequestResponse(DynamicScope.scala:59)
> at org.scalatra.ScalatraServlet.withRequestResponse(ScalatraServlet.scala:49)
> at org.scalatra.GZipSupport$class.handle(GZipSupport.scala:18)
> at org.apache.livy.server.interactive.InteractiveSessionServlet.org$scalatra$servlet$FileUploadSupport$$super$handle(InteractiveSessionServlet.scala:40)
> at org.scalatra.servlet.FileUploadSupport$class.handle(FileUploadSupport.scala:93)
> at org.apache.livy.server.interactive.InteractiveSessionServlet.handle(InteractiveSessionServlet.scala:40)
> at org.scalatra.ScalatraServlet.service(ScalatraServlet.scala:54)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
> at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:587)
> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
> at org.eclipse.jetty.server.Server.handle(Server.java:499)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
> at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
> at java.lang.Thread.run(Thread.java:748)
> {code}
>
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)