You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Jacek Laskowski <ja...@japila.pl> on 2015/10/03 20:27:58 UTC
WARN NettyRpcEndpointRef: Error sending message [message = Heartbeat(driver,...
Hi,
The following WARN happens in Spark built from today's sources. There
were some changes about RPC so it may be related. Should I report an
issue in JIRA?
I use a sbt project and `sbt console` to play with Spark.
```
scala> import org.apache.spark.SparkConf
import org.apache.spark.SparkConf
scala> val conf = new
SparkConf().setMaster("local[*]").setAppName("Hello world!")
conf: org.apache.spark.SparkConf = org.apache.spark.SparkConf@7a8f69d6
scala> import org.apache.spark.SparkContext
import org.apache.spark.SparkContext
scala> val sc = new SparkContext(conf)
```
So far so good. After a while I can see my CPU going up and the REPL
shows up with the WARN:
```
15/10/03 14:26:03 WARN NettyRpcEndpointRef: Error sending message
[message = Heartbeat(driver,[Lscala.Tuple2;@711362a6,BlockManagerId(driver,
localhost, 50804))] in 1 attempts
org.apache.spark.rpc.RpcTimeoutException: Futures timed out after [120
seconds]. This timeout is controlled by spark.rpc.askTimeout
at org.apache.spark.rpc.RpcTimeout.org$apache$spark$rpc$RpcTimeout$$createRpcTimeoutException(RpcEnv.scala:215)
at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcEnv.scala:230)
at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcEnv.scala:226)
at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcEnv.scala:243)
at org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:101)
at org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:77)
at org.apache.spark.executor.Executor.org$apache$spark$executor$Executor$$reportHeartBeat(Executor.scala:452)
at org.apache.spark.executor.Executor$$anon$1$$anonfun$run$1.apply$mcV$sp(Executor.scala:472)
at org.apache.spark.executor.Executor$$anon$1$$anonfun$run$1.apply(Executor.scala:472)
at org.apache.spark.executor.Executor$$anon$1$$anonfun$run$1.apply(Executor.scala:472)
at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1697)
at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:472)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.TimeoutException: Futures timed out
after [120 seconds]
at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)
at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)
at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:190)
at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
at scala.concurrent.Await$.result(package.scala:190)
at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcEnv.scala:242)
... 15 more
```
Pozdrawiam,
Jacek
--
Jacek Laskowski | http://blog.japila.pl | http://blog.jaceklaskowski.pl
Follow me at https://twitter.com/jaceklaskowski
Upvote at http://stackoverflow.com/users/1305344/jacek-laskowski
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: WARN NettyRpcEndpointRef: Error sending message [message = Heartbeat(driver,...
Posted by Jacek Laskowski <ja...@japila.pl>.
Hi Ted,
You're absolutely right - twice! It was initially indeed in
spark-shell and there can only be only active SparkContext. I did
`sc.stop`, but it didn't help much.
That's why I switched to a Scala/sbt/Spark project and run it all from
scratch (as shown). The stacktrace was showing up regularly. Give it a
shot yourself as it's easy to reproduce - build Spark from the sources
and have a project with libraryDependencies set with Spark core
1.6.0-SNAPSHOT.
Pozdrawiam,
Jacek
--
Jacek Laskowski | http://blog.japila.pl | http://blog.jaceklaskowski.pl
Follow me at https://twitter.com/jaceklaskowski
Upvote at http://stackoverflow.com/users/1305344/jacek-laskowski
On Sat, Oct 3, 2015 at 7:15 PM, Ted Yu <yu...@gmail.com> wrote:
> Did you use spark-shell ?
>
> In spark-shell, there can only be one running SparkContext which is created
> automatically.
>
> Cheers
>
> On Sat, Oct 3, 2015 at 11:27 AM, Jacek Laskowski <ja...@japila.pl> wrote:
>>
>> Hi,
>>
>> The following WARN happens in Spark built from today's sources. There
>> were some changes about RPC so it may be related. Should I report an
>> issue in JIRA?
>>
>> I use a sbt project and `sbt console` to play with Spark.
>>
>> ```
>> scala> import org.apache.spark.SparkConf
>> import org.apache.spark.SparkConf
>>
>> scala> val conf = new
>> SparkConf().setMaster("local[*]").setAppName("Hello world!")
>> conf: org.apache.spark.SparkConf = org.apache.spark.SparkConf@7a8f69d6
>>
>> scala> import org.apache.spark.SparkContext
>> import org.apache.spark.SparkContext
>>
>> scala> val sc = new SparkContext(conf)
>> ```
>>
>> So far so good. After a while I can see my CPU going up and the REPL
>> shows up with the WARN:
>>
>> ```
>> 15/10/03 14:26:03 WARN NettyRpcEndpointRef: Error sending message
>> [message =
>> Heartbeat(driver,[Lscala.Tuple2;@711362a6,BlockManagerId(driver,
>> localhost, 50804))] in 1 attempts
>> org.apache.spark.rpc.RpcTimeoutException: Futures timed out after [120
>> seconds]. This timeout is controlled by spark.rpc.askTimeout
>> at
>> org.apache.spark.rpc.RpcTimeout.org$apache$spark$rpc$RpcTimeout$$createRpcTimeoutException(RpcEnv.scala:215)
>> at
>> org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcEnv.scala:230)
>> at
>> org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcEnv.scala:226)
>> at
>> scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
>> at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcEnv.scala:243)
>> at
>> org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:101)
>> at
>> org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:77)
>> at
>> org.apache.spark.executor.Executor.org$apache$spark$executor$Executor$$reportHeartBeat(Executor.scala:452)
>> at
>> org.apache.spark.executor.Executor$$anon$1$$anonfun$run$1.apply$mcV$sp(Executor.scala:472)
>> at
>> org.apache.spark.executor.Executor$$anon$1$$anonfun$run$1.apply(Executor.scala:472)
>> at
>> org.apache.spark.executor.Executor$$anon$1$$anonfun$run$1.apply(Executor.scala:472)
>> at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1697)
>> at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:472)
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>> Caused by: java.util.concurrent.TimeoutException: Futures timed out
>> after [120 seconds]
>> at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)
>> at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)
>> at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:190)
>> at
>> scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
>> at scala.concurrent.Await$.result(package.scala:190)
>> at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcEnv.scala:242)
>> ... 15 more
>> ```
>>
>> Pozdrawiam,
>> Jacek
>>
>> --
>> Jacek Laskowski | http://blog.japila.pl | http://blog.jaceklaskowski.pl
>> Follow me at https://twitter.com/jaceklaskowski
>> Upvote at http://stackoverflow.com/users/1305344/jacek-laskowski
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: WARN NettyRpcEndpointRef: Error sending message [message = Heartbeat(driver,...
Posted by Ted Yu <yu...@gmail.com>.
Did you use spark-shell ?
In spark-shell, there can only be one running SparkContext which is created
automatically.
Cheers
On Sat, Oct 3, 2015 at 11:27 AM, Jacek Laskowski <ja...@japila.pl> wrote:
> Hi,
>
> The following WARN happens in Spark built from today's sources. There
> were some changes about RPC so it may be related. Should I report an
> issue in JIRA?
>
> I use a sbt project and `sbt console` to play with Spark.
>
> ```
> scala> import org.apache.spark.SparkConf
> import org.apache.spark.SparkConf
>
> scala> val conf = new
> SparkConf().setMaster("local[*]").setAppName("Hello world!")
> conf: org.apache.spark.SparkConf = org.apache.spark.SparkConf@7a8f69d6
>
> scala> import org.apache.spark.SparkContext
> import org.apache.spark.SparkContext
>
> scala> val sc = new SparkContext(conf)
> ```
>
> So far so good. After a while I can see my CPU going up and the REPL
> shows up with the WARN:
>
> ```
> 15/10/03 14:26:03 WARN NettyRpcEndpointRef: Error sending message
> [message = Heartbeat(driver,[Lscala.Tuple2;@711362a6
> ,BlockManagerId(driver,
> localhost, 50804))] in 1 attempts
> org.apache.spark.rpc.RpcTimeoutException: Futures timed out after [120
> seconds]. This timeout is controlled by spark.rpc.askTimeout
> at org.apache.spark.rpc.RpcTimeout.org
> $apache$spark$rpc$RpcTimeout$$createRpcTimeoutException(RpcEnv.scala:215)
> at
> org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcEnv.scala:230)
> at
> org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcEnv.scala:226)
> at
> scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
> at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcEnv.scala:243)
> at
> org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:101)
> at
> org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:77)
> at org.apache.spark.executor.Executor.org
> $apache$spark$executor$Executor$$reportHeartBeat(Executor.scala:452)
> at
> org.apache.spark.executor.Executor$$anon$1$$anonfun$run$1.apply$mcV$sp(Executor.scala:472)
> at
> org.apache.spark.executor.Executor$$anon$1$$anonfun$run$1.apply(Executor.scala:472)
> at
> org.apache.spark.executor.Executor$$anon$1$$anonfun$run$1.apply(Executor.scala:472)
> at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1697)
> at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:472)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.util.concurrent.TimeoutException: Futures timed out
> after [120 seconds]
> at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)
> at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)
> at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:190)
> at
> scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
> at scala.concurrent.Await$.result(package.scala:190)
> at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcEnv.scala:242)
> ... 15 more
> ```
>
> Pozdrawiam,
> Jacek
>
> --
> Jacek Laskowski | http://blog.japila.pl | http://blog.jaceklaskowski.pl
> Follow me at https://twitter.com/jaceklaskowski
> Upvote at http://stackoverflow.com/users/1305344/jacek-laskowski
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>