keep or remove sc.stop() coz of RpcEnv already stopped error

[nancy henry <na...@gmail.com>]

 Hi Team,


getting this error if we put sc.stop() in application..



can we remove it from application but i read if yu dont explicitly stop
using sc.stop the yarn application will not get registered in history
service.. SO what to do ?


 WARN Dispatcher: Message RemoteProcessDisconnected dropped.

java.lang.IllegalStateException: RpcEnv already stopped.

        at
org.apache.spark.rpc.netty.Dispatcher.postMessage(Dispatcher.scala:159)

        at
org.apache.spark.rpc.netty.Dispatcher.postToAll(Dispatcher.scala:109)

        at
org.apache.spark.rpc.netty.NettyRpcHandler.connectionTerminated(NettyRpcEnv.scala:630)

        at
org.apache.spark.network.server.TransportRequestHandler.channelUnregistered(TransportRequestHandler.java:94)

Re: keep or remove sc.stop() coz of RpcEnv already stopped error

[Alex <si...@gmail.com>]

Hi ,
I am using spark-1.6 how to ignore this warning because of this Illegal
state exception my production jobs which are scheduld are showing completed
abnormally... I cant even handle exception as after sc.stop if i try to
execute any code again this exception will come from catch block.. so i
removed sc.stop completely and let it exit on its own is it okay?

On Mon, Mar 13, 2017 at 7:01 PM, Yong Zhang <ja...@hotmail.com> wrote:

> What version of Spark you are using?
>
>
> Based on Spark-12967, it is fixed on Spark 2.0 and later. If you are using
> Spark 1.x, you can ignore this Warning. It shouldn't affect any functions.
>
>
> Yong
>
> ------------------------------
> *From:* nancy henry <na...@gmail.com>
> *Sent:* Monday, March 13, 2017 7:08 AM
> *To:* user @spark
> *Subject:* keep or remove sc.stop() coz of RpcEnv already stopped error
>
>
>  Hi Team,
>
>
> getting this error if we put sc.stop() in application..
>
>
>
> can we remove it from application but i read if yu dont explicitly stop
> using sc.stop the yarn application will not get registered in history
> service.. SO what to do ?
>
>
>  WARN Dispatcher: Message RemoteProcessDisconnected dropped.
>
> java.lang.IllegalStateException: RpcEnv already stopped.
>
>         at org.apache.spark.rpc.netty.Dispatcher.postMessage(
> Dispatcher.scala:159)
>
>         at org.apache.spark.rpc.netty.Dispatcher.postToAll(
> Dispatcher.scala:109)
>
>         at org.apache.spark.rpc.netty.NettyRpcHandler.
> connectionTerminated(NettyRpcEnv.scala:630)
>
>         at org.apache.spark.network.server.TransportRequestHandler.
> channelUnregistered(TransportRequestHandler.java:94)
>
>
>
>
>

Re: keep or remove sc.stop() coz of RpcEnv already stopped error

[Yong Zhang <ja...@hotmail.com>]

What version of Spark you are using?


Based on Spark-12967, it is fixed on Spark 2.0 and later. If you are using Spark 1.x, you can ignore this Warning. It shouldn't affect any functions.


Yong

________________________________
From: nancy henry <na...@gmail.com>
Sent: Monday, March 13, 2017 7:08 AM
To: user @spark
Subject: keep or remove sc.stop() coz of RpcEnv already stopped error

 Hi Team,

getting this error if we put sc.stop() in application..

can we remove it from application but i read if yu dont explicitly stop using sc.stop the yarn application will not get registered in history service.. SO what to do ?

 WARN Dispatcher: Message RemoteProcessDisconnected dropped.
java.lang.IllegalStateException: RpcEnv already stopped.
        at org.apache.spark.rpc.netty.Dispatcher.postMessage(Dispatcher.scala:159)
        at org.apache.spark.rpc.netty.Dispatcher.postToAll(Dispatcher.scala:109)
        at org.apache.spark.rpc.netty.NettyRpcHandler.connectionTerminated(NettyRpcEnv.scala:630)
        at org.apache.spark.network.server.TransportRequestHandler.channelUnregistered(TransportRequestHandler.java:94)