Cause of akka.pattern.AskTimeoutException

[Rachana Srivastava <Ra...@markmonitor.com>]

Hello all,

I have a spark process that is running successfully for few days then suddenly the process stop with following error.  Do we know the cause of this error and how to resolve it.

WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@1c50c2a8,BlockManagerId(<driver>, localhost, 42507))] in 3 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)

Caused by: akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                ... 1 more
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@357039d9,BlockManagerId(<driver>, localhost, 42507))] in 1 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@357039d9,BlockManagerId(<driver>, localhost, 42507))] in 2 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@357039d9,BlockManagerId(<driver>, localhost, 42507))] in 3 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.executor.Executor - Issue communicating with driver in heartbeater
org.apache.spark.SparkException: Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@357039d9,BlockManagerId(<driver>, localhost, 42507))]
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:209)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
Caused by: akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                ... 1 more
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@a45e872,BlockManagerId(<driver>, localhost, 42507))] in 1 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@a45e872,BlockManagerId(<driver>, localhost, 42507))] in 2 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@a45e872,BlockManagerId(<driver>, localhost, 42507))] in 3 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.executor.Executor - Issue communicating with driver in heartbeater
org.apache.spark.SparkException: Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@a45e872,BlockManagerId(<driver>, localhost, 42507))]
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:209)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
Caused by: akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                ... 1 more
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@5d0f05af,BlockManagerId(<driver>, localhost, 42507))] in 1 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@5d0f05af,BlockManagerId(<driver>, localhost, 42507))] in 2 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@5d0f05af,BlockManagerId(<driver>, localhost, 42507))] in 3 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.executor.Executor - Issue communicating with driver in heartbeater
org.apache.spark.SparkException: Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@5d0f05af,BlockManagerId(<driver>, localhost, 42507))]
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:209)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
Caused by: akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                ... 1 more
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@6728837f,BlockManagerId(<driver>, localhost, 42507))] in 1 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@6728837f,BlockManagerId(<driver>, localhost, 42507))] in 2 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@6728837f,BlockManagerId(<driver>, localhost, 42507))] in 3 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.executor.Executor - Issue communicating with driver in heartbeater
org.apache.spark.SparkException: Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@6728837f,BlockManagerId(<driver>, localhost, 42507))]
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:209)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
Caused by: akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                ... 1 more
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@27314434,BlockManagerId(<driver>, localhost, 42507))] in 1 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@27314434,BlockManagerId(<driver>, localhost, 42507))] in 2 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@27314434,BlockManagerId(<driver>, localhost, 42507))] in 3 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.executor.Executor - Issue communicating with driver in heartbeater
org.apache.spark.SparkException: Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@27314434,BlockManagerId(<driver>, localhost, 42507))]
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:209)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
Caused by: akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                ... 1 more
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@42b7b5f3,BlockManagerId(<driver>, localhost, 42507))] in 1 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@42b7b5f3,BlockManagerId(<driver>, localhost, 42507))] in 2 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@42b7b5f3,BlockManagerId(<driver>, localhost, 42507))] in 3 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.executor.Executor - Issue communicating with driver in heartbeater
org.apache.spark.SparkException: Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@42b7b5f3,BlockManagerId(<driver>, localhost, 42507))]
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:209)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
Caused by: akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                ... 1 more
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@4dc790e1,BlockManagerId(<driver>, localhost, 42507))] in 1 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@4dc790e1,BlockManagerId(<driver>, localhost, 42507))] in 2 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@4dc790e1,BlockManagerId(<driver>, localhost, 42507))] in 3 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.executor.Executor - Issue communicating with driver in heartbeater
org.apache.spark.SparkException: Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@4dc790e1,BlockManagerId(<driver>, localhost, 42507))]
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:209)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
Caused by: akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                ... 1 more
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@47969bc5,BlockManagerId(<driver>, localhost, 42507))] in 1 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@47969bc5,BlockManagerId(<driver>, localhost, 42507))] in 2 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@47969bc5,BlockManagerId(<driver>, localhost, 42507))] in 3 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.executor.Executor - Issue communicating with driver in heartbeater
org.apache.spark.SparkException: Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@47969bc5,BlockManagerId(<driver>, localhost, 42507))]
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:209)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
Caused by: akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                ... 1 more
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@57cca1d5,BlockManagerId(<driver>, localhost, 42507))] in 1 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@57cca1d5,BlockManagerId(<driver>, localhost, 42507))] in 2 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@57cca1d5,BlockManagerId(<driver>, localhost, 42507))] in 3 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.executor.Executor - Issue communicating with driver in heartbeater
org.apache.spark.SparkException: Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@57cca1d5,BlockManagerId(<driver>, localhost, 42507))]
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:209)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
Caused by: akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                ... 1 more
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@46ae2f22,BlockManagerId(<driver>, localhost, 42507))] in 1 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@46ae2f22,BlockManagerId(<driver>, localhost, 42507))] in 2 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@46ae2f22,BlockManagerId(<driver>, localhost, 42507))] in 3 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.executor.Executor - Issue communicating with driver in heartbeater
org.apache.spark.SparkException: Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@46ae2f22,BlockManagerId(<driver>, localhost, 42507))]
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:209)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
Caused by: akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                ... 1 more
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@6e7da9d8,BlockManagerId(<driver>, localhost, 42507))] in 1 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@6e7da9d8,BlockManagerId(<driver>, localhost, 42507))] in 2 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@6e7da9d8,BlockManagerId(<driver>, localhost, 42507))] in 3 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.executor.Executor - Issue communicating with driver in heartbeater
org.apache.spark.SparkException: Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@6e7da9d8,BlockManagerId(<driver>, localhost, 42507))]
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:209)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
Caused by: akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                ... 1 more
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@cd4d4ac,BlockManagerId(<driver>, localhost, 42507))] in 1 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@cd4d4ac,BlockManagerId(<driver>, localhost, 42507))] in 2 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@cd4d4ac,BlockManagerId(<driver>, localhost, 42507))] in 3 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.executor.Executor - Issue communicating with driver in heartbeater
org.apache.spark.SparkException: Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@cd4d4ac,BlockManagerId(<driver>, localhost, 42507))]
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:209)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
Caused by: akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                ... 1 more
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@1237dcce,BlockManagerId(<driver>, localhost, 42507))] in 1 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@1237dcce,BlockManagerId(<driver>, localhost, 42507))] in 2 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@1237dcce,BlockManagerId(<driver>, localhost, 42507))] in 3 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.executor.Executor - Issue communicating with driver in heartbeater
org.apache.spark.SparkException: Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@1237dcce,BlockManagerId(<driver>, localhost, 42507))]
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:209)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
Caused by: akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                ... 1 more
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@45e5527e,BlockManagerId(<driver>, localhost, 42507))] in 1 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@45e5527e,BlockManagerId(<driver>, localhost, 42507))] in 2 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@45e5527e,BlockManagerId(<driver>, localhost, 42507))] in 3 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.executor.Executor - Issue communicating with driver in heartbeater
org.apache.spark.SparkException: Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@45e5527e,BlockManagerId(<driver>, localhost, 42507))]
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:209)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
Caused by: akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                ... 1 more
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@737a14e,BlockManagerId(<driver>, localhost, 42507))] in 1 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@737a14e,BlockManagerId(<driver>, localhost, 42507))] in 2 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@737a14e,BlockManagerId(<driver>, localhost, 42507))] in 3 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.executor.Executor - Issue communicating with driver in heartbeater
org.apache.spark.SparkException: Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@737a14e,BlockManagerId(<driver>, localhost, 42507))]
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:209)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
Caused by: akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                ... 1 more
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@543aff2a,BlockManagerId(<driver>, localhost, 42507))] in 1 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@543aff2a,BlockManagerId(<driver>, localhost, 42507))] in 2 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@543aff2a,BlockManagerId(<driver>, localhost, 42507))] in 3 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.executor.Executor - Issue communicating with driver in heartbeater
org.apache.spark.SparkException: Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@543aff2a,BlockManagerId(<driver>, localhost, 42507))]
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:209)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
Caused by: akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                ... 1 more
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@d55fd64,BlockManagerId(<driver>, localhost, 42507))] in 1 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@d55fd64,BlockManagerId(<driver>, localhost, 42507))] in 2 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@d55fd64,BlockManagerId(<driver>, localhost, 42507))] in 3 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.executor.Executor - Issue communicating with driver in heartbeater
org.apache.spark.SparkException: Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@d55fd64,BlockManagerId(<driver>, localhost, 42507))]
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:209)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
Caused by: akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                ... 1 more
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@3759cf46,BlockManagerId(<driver>, localhost, 42507))] in 1 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@3759cf46,BlockManagerId(<driver>, localhost, 42507))] in 2 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@3759cf46,BlockManagerId(<driver>, localhost, 42507))] in 3 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.executor.Executor - Issue communicating with driver in heartbeater
org.apache.spark.SparkException: Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@3759cf46,BlockManagerId(<driver>, localhost, 42507))]
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:209)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
Caused by: akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                ... 1 more
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@629a9984,BlockManagerId(<driver>, localhost, 42507))] in 1 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@629a9984,BlockManagerId(<driver>, localhost, 42507))] in 2 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@629a9984,BlockManagerId(<driver>, localhost, 42507))] in 3 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.executor.Executor - Issue communicating with driver in heartbeater
org.apache.spark.SparkException: Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@629a9984,BlockManagerId(<driver>, localhost, 42507))]
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:209)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
Caused by: akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                ... 1 more
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@3ecaf226,BlockManagerId(<driver>, localhost, 42507))] in 1 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@3ecaf226,BlockManagerId(<driver>, localhost, 42507))] in 2 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@3ecaf226,BlockManagerId(<driver>, localhost, 42507))] in 3 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.executor.Executor - Issue communicating with driver in heartbeater
org.apache.spark.SparkException: Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@3ecaf226,BlockManagerId(<driver>, localhost, 42507))]
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:209)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
Caused by: akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                ... 1 more
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@3db42d26,BlockManagerId(<driver>, localhost, 42507))] in 1 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@3db42d26,BlockManagerId(<driver>, localhost, 42507))] in 2 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@3db42d26,BlockManagerId(<driver>, localhost, 42507))] in 3 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.executor.Executor - Issue communicating with driver in heartbeater
org.apache.spark.SparkException: Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@3db42d26,BlockManagerId(<driver>, localhost, 42507))]
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:209)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
Caused by: akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                ... 1 more
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@679c5979,BlockManagerId(<driver>, localhost, 42507))] in 1 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@679c5979,BlockManagerId(<driver>, localhost, 42507))] in 2 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@679c5979,BlockManagerId(<driver>, localhost, 42507))] in 3 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.executor.Executor - Issue communicating with driver in heartbeater
org.apache.spark.SparkException: Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@679c5979,BlockManagerId(<driver>, localhost, 42507))]
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:209)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
Caused by: akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                ... 1 more
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@5f328085,BlockManagerId(<driver>, localhost, 42507))] in 1 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@5f328085,BlockManagerId(<driver>, localhost, 42507))] in 2 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@5f328085,BlockManagerId(<driver>, localhost, 42507))] in 3 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.executor.Executor - Issue communicating with driver in heartbeater
org.apache.spark.SparkException: Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@5f328085,BlockManagerId(<driver>, localhost, 42507))]
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:209)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
Caused by: akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                ... 1 more
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@4dbcf1a,BlockManagerId(<driver>, localhost, 42507))] in 1 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@4dbcf1a,BlockManagerId(<driver>, localhost, 42507))] in 2 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@4dbcf1a,BlockManagerId(<driver>, localhost, 42507))] in 3 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.executor.Executor - Issue communicating with driver in heartbeater
org.apache.spark.SparkException: Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@4dbcf1a,BlockManagerId(<driver>, localhost, 42507))]
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:209)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
Caused by: akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                ... 1 more
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@784bd62b,BlockManagerId(<driver>, localhost, 42507))] in 1 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@784bd62b,BlockManagerId(<driver>, localhost, 42507))] in 2 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@784bd62b,BlockManagerId(<driver>, localhost, 42507))] in 3 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.executor.Executor - Issue communicating with driver in heartbeater
org.apache.spark.SparkException: Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@784bd62b,BlockManagerId(<driver>, localhost, 42507))]
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:209)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
Caused by: akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                ... 1 more
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@1d76884a,BlockManagerId(<driver>, localhost, 42507))] in 1 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@1d76884a,BlockManagerId(<driver>, localhost, 42507))] in 2 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@1d76884a,BlockManagerId(<driver>, localhost, 42507))] in 3 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.executor.Executor - Issue communicating with driver in heartbeater
org.apache.spark.SparkException: Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@1d76884a,BlockManagerId(<driver>, localhost, 42507))]
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:209)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
Caused by: akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                ... 1 more
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@270e056,BlockManagerId(<driver>, localhost, 42507))] in 1 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@270e056,BlockManagerId(<driver>, localhost, 42507))] in 2 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@270e056,BlockManagerId(<driver>, localhost, 42507))] in 3 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.executor.Executor - Issue communicating with driver in heartbeater
org.apache.spark.SparkException: Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@270e056,BlockManagerId(<driver>, localhost, 42507))]
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:209)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
Caused by: akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                ... 1 more
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@37aab060,BlockManagerId(<driver>, localhost, 42507))] in 1 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@37aab060,BlockManagerId(<driver>, localhost, 42507))] in 2 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.util.AkkaUtils - Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@37aab060,BlockManagerId(<driver>, localhost, 42507))] in 3 attempts
akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
WARN : org.apache.spark.executor.Executor - Issue communicating with driver in heartbeater
org.apache.spark.SparkException: Error sending message [message = Heartbeat(<driver>,[Lscala.Tuple2;@37aab060,BlockManagerId(<driver>, localhost, 42507))]
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:209)
                at org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
Caused by: akka.pattern.AskTimeoutException: Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had already been terminated.
                at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
                at org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
                ... 1 more

Thanks,

Rachana

Re: Cause of akka.pattern.AskTimeoutException

[Akhil Das <ak...@sigmoidanalytics.com>]

You need to look for the root cause in the logs to see what went wrong.
There is a high chance of your process getting OOM before the throwing up
the error that you pasted.

Thanks
Best Regards

On Thu, Dec 10, 2015 at 3:15 AM, Rachana Srivastava <
Rachana.Srivastava@markmonitor.com> wrote:

> Hello all,
>
>
>
> I have a spark process that is running successfully for few days then
> suddenly the process stop with following error.  Do we know the cause of
> this error and how to resolve it.
>
>
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@1c50c2a8,BlockManagerId(<driver>,
> localhost, 42507))] in 3 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
>
>
> Caused by: akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 ... 1 more
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@357039d9,BlockManagerId(<driver>,
> localhost, 42507))] in 1 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@357039d9,BlockManagerId(<driver>,
> localhost, 42507))] in 2 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@357039d9,BlockManagerId(<driver>,
> localhost, 42507))] in 3 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.executor.Executor - Issue communicating with
> driver in heartbeater
>
> org.apache.spark.SparkException: Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@357039d9,BlockManagerId(<driver>,
> localhost, 42507))]
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:209)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> Caused by: akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 ... 1 more
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@a45e872,BlockManagerId(<driver>,
> localhost, 42507))] in 1 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@a45e872,BlockManagerId(<driver>,
> localhost, 42507))] in 2 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@a45e872,BlockManagerId(<driver>,
> localhost, 42507))] in 3 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.executor.Executor - Issue communicating with
> driver in heartbeater
>
> org.apache.spark.SparkException: Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@a45e872,BlockManagerId(<driver>,
> localhost, 42507))]
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:209)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> Caused by: akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 ... 1 more
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@5d0f05af,BlockManagerId(<driver>,
> localhost, 42507))] in 1 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@5d0f05af,BlockManagerId(<driver>,
> localhost, 42507))] in 2 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@5d0f05af,BlockManagerId(<driver>,
> localhost, 42507))] in 3 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.executor.Executor - Issue communicating with
> driver in heartbeater
>
> org.apache.spark.SparkException: Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@5d0f05af,BlockManagerId(<driver>,
> localhost, 42507))]
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:209)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> Caused by: akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 ... 1 more
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@6728837f,BlockManagerId(<driver>,
> localhost, 42507))] in 1 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@6728837f,BlockManagerId(<driver>,
> localhost, 42507))] in 2 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@6728837f,BlockManagerId(<driver>,
> localhost, 42507))] in 3 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.executor.Executor - Issue communicating with
> driver in heartbeater
>
> org.apache.spark.SparkException: Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@6728837f,BlockManagerId(<driver>,
> localhost, 42507))]
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:209)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> Caused by: akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 ... 1 more
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@27314434,BlockManagerId(<driver>,
> localhost, 42507))] in 1 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@27314434,BlockManagerId(<driver>,
> localhost, 42507))] in 2 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@27314434,BlockManagerId(<driver>,
> localhost, 42507))] in 3 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.executor.Executor - Issue communicating with
> driver in heartbeater
>
> org.apache.spark.SparkException: Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@27314434,BlockManagerId(<driver>,
> localhost, 42507))]
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:209)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> Caused by: akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 ... 1 more
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@42b7b5f3,BlockManagerId(<driver>,
> localhost, 42507))] in 1 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@42b7b5f3,BlockManagerId(<driver>,
> localhost, 42507))] in 2 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@42b7b5f3,BlockManagerId(<driver>,
> localhost, 42507))] in 3 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.executor.Executor - Issue communicating with
> driver in heartbeater
>
> org.apache.spark.SparkException: Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@42b7b5f3,BlockManagerId(<driver>,
> localhost, 42507))]
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:209)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> Caused by: akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 ... 1 more
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@4dc790e1,BlockManagerId(<driver>,
> localhost, 42507))] in 1 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@4dc790e1,BlockManagerId(<driver>,
> localhost, 42507))] in 2 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@4dc790e1,BlockManagerId(<driver>,
> localhost, 42507))] in 3 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.executor.Executor - Issue communicating with
> driver in heartbeater
>
> org.apache.spark.SparkException: Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@4dc790e1,BlockManagerId(<driver>,
> localhost, 42507))]
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:209)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> Caused by: akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 ... 1 more
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@47969bc5,BlockManagerId(<driver>,
> localhost, 42507))] in 1 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@47969bc5,BlockManagerId(<driver>,
> localhost, 42507))] in 2 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@47969bc5,BlockManagerId(<driver>,
> localhost, 42507))] in 3 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.executor.Executor - Issue communicating with
> driver in heartbeater
>
> org.apache.spark.SparkException: Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@47969bc5,BlockManagerId(<driver>,
> localhost, 42507))]
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:209)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> Caused by: akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 ... 1 more
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@57cca1d5,BlockManagerId(<driver>,
> localhost, 42507))] in 1 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@57cca1d5,BlockManagerId(<driver>,
> localhost, 42507))] in 2 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@57cca1d5,BlockManagerId(<driver>,
> localhost, 42507))] in 3 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.executor.Executor - Issue communicating with
> driver in heartbeater
>
> org.apache.spark.SparkException: Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@57cca1d5,BlockManagerId(<driver>,
> localhost, 42507))]
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:209)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> Caused by: akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 ... 1 more
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@46ae2f22,BlockManagerId(<driver>,
> localhost, 42507))] in 1 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@46ae2f22,BlockManagerId(<driver>,
> localhost, 42507))] in 2 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@46ae2f22,BlockManagerId(<driver>,
> localhost, 42507))] in 3 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.executor.Executor - Issue communicating with
> driver in heartbeater
>
> org.apache.spark.SparkException: Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@46ae2f22,BlockManagerId(<driver>,
> localhost, 42507))]
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:209)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> Caused by: akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 ... 1 more
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@6e7da9d8,BlockManagerId(<driver>,
> localhost, 42507))] in 1 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@6e7da9d8,BlockManagerId(<driver>,
> localhost, 42507))] in 2 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@6e7da9d8,BlockManagerId(<driver>,
> localhost, 42507))] in 3 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.executor.Executor - Issue communicating with
> driver in heartbeater
>
> org.apache.spark.SparkException: Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@6e7da9d8,BlockManagerId(<driver>,
> localhost, 42507))]
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:209)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> Caused by: akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 ... 1 more
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@cd4d4ac,BlockManagerId(<driver>,
> localhost, 42507))] in 1 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@cd4d4ac,BlockManagerId(<driver>,
> localhost, 42507))] in 2 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@cd4d4ac,BlockManagerId(<driver>,
> localhost, 42507))] in 3 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.executor.Executor - Issue communicating with
> driver in heartbeater
>
> org.apache.spark.SparkException: Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@cd4d4ac,BlockManagerId(<driver>,
> localhost, 42507))]
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:209)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> Caused by: akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 ... 1 more
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@1237dcce,BlockManagerId(<driver>,
> localhost, 42507))] in 1 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@1237dcce,BlockManagerId(<driver>,
> localhost, 42507))] in 2 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@1237dcce,BlockManagerId(<driver>,
> localhost, 42507))] in 3 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.executor.Executor - Issue communicating with
> driver in heartbeater
>
> org.apache.spark.SparkException: Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@1237dcce,BlockManagerId(<driver>,
> localhost, 42507))]
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:209)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> Caused by: akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 ... 1 more
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@45e5527e,BlockManagerId(<driver>,
> localhost, 42507))] in 1 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@45e5527e,BlockManagerId(<driver>,
> localhost, 42507))] in 2 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@45e5527e,BlockManagerId(<driver>,
> localhost, 42507))] in 3 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.executor.Executor - Issue communicating with
> driver in heartbeater
>
> org.apache.spark.SparkException: Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@45e5527e,BlockManagerId(<driver>,
> localhost, 42507))]
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:209)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> Caused by: akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 ... 1 more
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@737a14e,BlockManagerId(<driver>,
> localhost, 42507))] in 1 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@737a14e,BlockManagerId(<driver>,
> localhost, 42507))] in 2 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@737a14e,BlockManagerId(<driver>,
> localhost, 42507))] in 3 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.executor.Executor - Issue communicating with
> driver in heartbeater
>
> org.apache.spark.SparkException: Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@737a14e,BlockManagerId(<driver>,
> localhost, 42507))]
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:209)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> Caused by: akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 ... 1 more
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@543aff2a,BlockManagerId(<driver>,
> localhost, 42507))] in 1 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@543aff2a,BlockManagerId(<driver>,
> localhost, 42507))] in 2 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@543aff2a,BlockManagerId(<driver>,
> localhost, 42507))] in 3 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.executor.Executor - Issue communicating with
> driver in heartbeater
>
> org.apache.spark.SparkException: Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@543aff2a,BlockManagerId(<driver>,
> localhost, 42507))]
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:209)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> Caused by: akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 ... 1 more
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@d55fd64,BlockManagerId(<driver>,
> localhost, 42507))] in 1 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@d55fd64,BlockManagerId(<driver>,
> localhost, 42507))] in 2 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@d55fd64,BlockManagerId(<driver>,
> localhost, 42507))] in 3 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.executor.Executor - Issue communicating with
> driver in heartbeater
>
> org.apache.spark.SparkException: Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@d55fd64,BlockManagerId(<driver>,
> localhost, 42507))]
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:209)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> Caused by: akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 ... 1 more
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@3759cf46,BlockManagerId(<driver>,
> localhost, 42507))] in 1 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@3759cf46,BlockManagerId(<driver>,
> localhost, 42507))] in 2 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@3759cf46,BlockManagerId(<driver>,
> localhost, 42507))] in 3 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.executor.Executor - Issue communicating with
> driver in heartbeater
>
> org.apache.spark.SparkException: Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@3759cf46,BlockManagerId(<driver>,
> localhost, 42507))]
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:209)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> Caused by: akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 ... 1 more
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@629a9984,BlockManagerId(<driver>,
> localhost, 42507))] in 1 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@629a9984,BlockManagerId(<driver>,
> localhost, 42507))] in 2 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@629a9984,BlockManagerId(<driver>,
> localhost, 42507))] in 3 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.executor.Executor - Issue communicating with
> driver in heartbeater
>
> org.apache.spark.SparkException: Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@629a9984,BlockManagerId(<driver>,
> localhost, 42507))]
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:209)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> Caused by: akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 ... 1 more
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@3ecaf226,BlockManagerId(<driver>,
> localhost, 42507))] in 1 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@3ecaf226,BlockManagerId(<driver>,
> localhost, 42507))] in 2 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@3ecaf226,BlockManagerId(<driver>,
> localhost, 42507))] in 3 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.executor.Executor - Issue communicating with
> driver in heartbeater
>
> org.apache.spark.SparkException: Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@3ecaf226,BlockManagerId(<driver>,
> localhost, 42507))]
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:209)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> Caused by: akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 ... 1 more
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@3db42d26,BlockManagerId(<driver>,
> localhost, 42507))] in 1 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@3db42d26,BlockManagerId(<driver>,
> localhost, 42507))] in 2 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@3db42d26,BlockManagerId(<driver>,
> localhost, 42507))] in 3 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.executor.Executor - Issue communicating with
> driver in heartbeater
>
> org.apache.spark.SparkException: Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@3db42d26,BlockManagerId(<driver>,
> localhost, 42507))]
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:209)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> Caused by: akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 ... 1 more
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@679c5979,BlockManagerId(<driver>,
> localhost, 42507))] in 1 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@679c5979,BlockManagerId(<driver>,
> localhost, 42507))] in 2 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@679c5979,BlockManagerId(<driver>,
> localhost, 42507))] in 3 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.executor.Executor - Issue communicating with
> driver in heartbeater
>
> org.apache.spark.SparkException: Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@679c5979,BlockManagerId(<driver>,
> localhost, 42507))]
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:209)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> Caused by: akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 ... 1 more
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@5f328085,BlockManagerId(<driver>,
> localhost, 42507))] in 1 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@5f328085,BlockManagerId(<driver>,
> localhost, 42507))] in 2 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@5f328085,BlockManagerId(<driver>,
> localhost, 42507))] in 3 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.executor.Executor - Issue communicating with
> driver in heartbeater
>
> org.apache.spark.SparkException: Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@5f328085,BlockManagerId(<driver>,
> localhost, 42507))]
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:209)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> Caused by: akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 ... 1 more
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@4dbcf1a,BlockManagerId(<driver>,
> localhost, 42507))] in 1 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@4dbcf1a,BlockManagerId(<driver>,
> localhost, 42507))] in 2 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@4dbcf1a,BlockManagerId(<driver>,
> localhost, 42507))] in 3 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.executor.Executor - Issue communicating with
> driver in heartbeater
>
> org.apache.spark.SparkException: Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@4dbcf1a,BlockManagerId(<driver>,
> localhost, 42507))]
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:209)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> Caused by: akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 ... 1 more
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@784bd62b,BlockManagerId(<driver>,
> localhost, 42507))] in 1 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@784bd62b,BlockManagerId(<driver>,
> localhost, 42507))] in 2 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@784bd62b,BlockManagerId(<driver>,
> localhost, 42507))] in 3 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.executor.Executor - Issue communicating with
> driver in heartbeater
>
> org.apache.spark.SparkException: Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@784bd62b,BlockManagerId(<driver>,
> localhost, 42507))]
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:209)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> Caused by: akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 ... 1 more
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@1d76884a,BlockManagerId(<driver>,
> localhost, 42507))] in 1 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@1d76884a,BlockManagerId(<driver>,
> localhost, 42507))] in 2 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@1d76884a,BlockManagerId(<driver>,
> localhost, 42507))] in 3 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.executor.Executor - Issue communicating with
> driver in heartbeater
>
> org.apache.spark.SparkException: Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@1d76884a,BlockManagerId(<driver>,
> localhost, 42507))]
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:209)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> Caused by: akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 ... 1 more
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@270e056,BlockManagerId(<driver>,
> localhost, 42507))] in 1 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@270e056,BlockManagerId(<driver>,
> localhost, 42507))] in 2 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@270e056,BlockManagerId(<driver>,
> localhost, 42507))] in 3 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.executor.Executor - Issue communicating with
> driver in heartbeater
>
> org.apache.spark.SparkException: Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@270e056,BlockManagerId(<driver>,
> localhost, 42507))]
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:209)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> Caused by: akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 ... 1 more
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@37aab060,BlockManagerId(<driver>,
> localhost, 42507))] in 1 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@37aab060,BlockManagerId(<driver>,
> localhost, 42507))] in 2 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.util.AkkaUtils - Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@37aab060,BlockManagerId(<driver>,
> localhost, 42507))] in 3 attempts
>
> akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> WARN : org.apache.spark.executor.Executor - Issue communicating with
> driver in heartbeater
>
> org.apache.spark.SparkException: Error sending message [message =
> Heartbeat(<driver>,[Lscala.Tuple2;@37aab060,BlockManagerId(<driver>,
> localhost, 42507))]
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:209)
>
>                 at
> org.apache.spark.executor.Executor$$anon$1.run(Executor.scala:427)
>
> Caused by: akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://sparkDriver/user/HeartbeatReceiver#122822265]] had
> already been terminated.
>
>                 at
> akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>
>                 at
> org.apache.spark.util.AkkaUtils$.askWithReply(AkkaUtils.scala:194)
>
>                 ... 1 more
>
>
>
> Thanks,
>
>
>
> Rachana
>