You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by ja...@centrum.cz on 2014/11/08 10:28:58 UTC
Spark on YARN, ExecutorLostFailure for long running computations in map
Hi,
I am getting ExecutorLostFailure when I run spark on YARN and in map I perform very long tasks (couple of hours). Error Log is below.
Do you know if it is possible to set something to make it possible for Spark to perform these very long running jobs in map?
Thank you very much for any advice.
Best regards,
Jan
Spark log:
4533,931: [GC 394578K->20882K(1472000K), 0,0226470 secs]
Traceback (most recent call last):
File "/home/hadoop/spark_stuff/spark_lda.py", line 112, in <module>
models.saveAsTextFile(sys.argv[1])
File "/home/hadoop/spark/python/pyspark/rdd.py", line 1324, in saveAsTextFile
keyed._jrdd.map(self.ctx._jvm.BytesToString()).saveAsTextFile(path)
File "/home/hadoop/spark/python/lib/py4j-0.8.2.1-src.zip/py4j/java_gateway.py", line 538, in __call__
File "/home/hadoop/spark/python/lib/py4j-0.8.2.1-src.zip/py4j/protocol.py", line 300, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o36.saveAsTextFile.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 28 in stage 0.0 failed 4 times, most recent failure: Lost task 28.3 in stage 0.0 (TID 41, ip-172-16-1-90.us-west-2.compute.internal): ExecutorLostFailure (executor lost)
Driver stacktrace:
at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1185)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1174)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1173)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1173)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:688)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:688)
at scala.Option.foreach(Option.scala:236)
at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:688)
at org.apache.spark.scheduler.DAGSchedulerEventProcessActor$$anonfun$receive$2.applyOrElse(DAGScheduler.scala:1391)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)
at akka.actor.ActorCell.invoke(ActorCell.scala:456)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)
at akka.dispatch.Mailbox.run(Mailbox.scala:219)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Yarn log:
14/11/08 08:20:34 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on ip-172-16-1-152.us-west-2.compute.internal:41091 (size: 596.9 KB, free: 775.7 MB)
14/11/08 08:20:34 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on ip-172-16-1-152.us-west-2.compute.internal:39160 (size: 596.9 KB, free: 775.7 MB)
14/11/08 08:20:34 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on ip-172-16-1-152.us-west-2.compute.internal:45058 (size: 596.9 KB, free: 775.7 MB)
14/11/08 08:20:34 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on ip-172-16-1-241.us-west-2.compute.internal:54111 (size: 596.9 KB, free: 775.7 MB)
14/11/08 08:20:34 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on ip-172-16-1-238.us-west-2.compute.internal:45772 (size: 596.9 KB, free: 775.7 MB)
14/11/08 08:20:34 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on ip-172-16-1-241.us-west-2.compute.internal:59509 (size: 596.9 KB, free: 775.7 MB)
14/11/08 08:20:34 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on ip-172-16-1-238.us-west-2.compute.internal:35720 (size: 596.9 KB, free: 775.7 MB)
14/11/08 08:21:11 INFO network.ConnectionManager: Removing SendingConnection to ConnectionManagerId(ip-172-16-1-241.us-west-2.compute.internal,59509)
14/11/08 08:21:11 INFO network.ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(ip-172-16-1-241.us-west-2.compute.internal,59509)
14/11/08 08:21:11 ERROR network.ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(ip-172-16-1-241.us-west-2.compute.internal,59509) not found
14/11/08 08:21:11 INFO cluster.YarnClientSchedulerBackend: Executor 10 disconnected, so removing it
14/11/08 08:21:11 ERROR cluster.YarnClientClusterScheduler: Lost executor 10 on ip-172-16-1-241.us-west-2.compute.internal: remote Akka client disassociated
14/11/08 08:21:11 INFO scheduler.TaskSetManager: Re-queueing tasks for 10 from TaskSet 0.0
14/11/08 08:21:11 WARN scheduler.TaskSetManager: Lost task 28.0 in stage 0.0 (TID 28, ip-172-16-1-241.us-west-2.compute.internal): ExecutorLostFailure (executor lost)
14/11/08 08:21:11 INFO scheduler.DAGScheduler: Executor lost: 10 (epoch 0)
14/11/08 08:21:11 INFO storage.BlockManagerMasterActor: Trying to remove executor 10 from BlockManagerMaster.
14/11/08 08:21:11 INFO storage.BlockManagerMaster: Removed 10 successfully in removeExecutor
14/11/08 08:21:20 INFO network.ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(ip-172-16-1-194.us-west-2.compute.internal,45823)
14/11/08 08:21:20 INFO network.ConnectionManager: Removing SendingConnection to ConnectionManagerId(ip-172-16-1-194.us-west-2.compute.internal,45823)
14/11/08 08:21:20 INFO network.ConnectionManager: Removing SendingConnection to ConnectionManagerId(ip-172-16-1-194.us-west-2.compute.internal,45823)
14/11/08 08:21:20 INFO cluster.YarnClientSchedulerBackend: Executor 5 disconnected, so removing it
14/11/08 08:21:20 ERROR cluster.YarnClientClusterScheduler: Lost executor 5 on ip-172-16-1-194.us-west-2.compute.internal: remote Akka client disassociated
14/11/08 08:21:20 INFO scheduler.TaskSetManager: Re-queueing tasks for 5 from TaskSet 0.0
14/11/08 08:21:20 WARN scheduler.TaskSetManager: Lost task 21.0 in stage 0.0 (TID 21, ip-172-16-1-194.us-west-2.compute.internal): ExecutorLostFailure (executor lost)
14/11/08 08:21:20 INFO scheduler.DAGScheduler: Executor lost: 5 (epoch 1)
14/11/08 08:21:20 INFO network.ConnectionManager: key already cancelled ? sun.nio.ch.SelectionKeyImpl@3bb633cd
java.nio.channels.CancelledKeyException
at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77)
at org.apache.spark.network.ConnectionManager.run(ConnectionManager.scala:289)
at org.apache.spark.network.ConnectionManager$$anon$4.run(ConnectionManager.scala:139)
14/11/08 08:21:20 INFO storage.BlockManagerMasterActor: Trying to remove executor 5 from BlockManagerMaster.
14/11/08 08:21:20 INFO storage.BlockManagerMaster: Removed 5 successfully in removeExecutor
14/11/08 08:21:21 INFO network.ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(ip-172-16-1-92.us-west-2.compute.internal,50928)
14/11/08 08:21:21 INFO network.ConnectionManager: Removing SendingConnection to ConnectionManagerId(ip-172-16-1-92.us-west-2.compute.internal,50928)
14/11/08 08:21:21 INFO network.ConnectionManager: Removing SendingConnection to ConnectionManagerId(ip-172-16-1-92.us-west-2.compute.internal,50928)
14/11/08 08:21:21 INFO cluster.YarnClientSchedulerBackend: Executor 27 disconnected, so removing it
14/11/08 08:21:21 ERROR cluster.YarnClientClusterScheduler: Lost executor 27 on ip-172-16-1-92.us-west-2.compute.internal: remote Akka client disassociated
14/11/08 08:21:21 INFO scheduler.TaskSetManager: Re-queueing tasks for 27 from TaskSet 0.0
14/11/08 08:21:21 WARN scheduler.TaskSetManager: Lost task 27.0 in stage 0.0 (TID 27, ip-172-16-1-92.us-west-2.compute.internal): ExecutorLostFailure (executor lost)
14/11/08 08:21:21 INFO scheduler.DAGScheduler: Executor lost: 27 (epoch 2)
14/11/08 08:21:21 INFO storage.BlockManagerMasterActor: Trying to remove executor 27 from BlockManagerMaster.
14/11/08 08:21:21 INFO storage.BlockManagerMaster: Removed 27 successfully in removeExecutor
14/11/08 08:21:21 INFO network.ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(ip-172-16-1-152.us-west-2.compute.internal,41091)
14/11/08 08:21:21 INFO network.ConnectionManager: Removing SendingConnection to ConnectionManagerId(ip-172-16-1-152.us-west-2.compute.internal,41091)
14/11/08 08:21:21 INFO network.ConnectionManager: Removing SendingConnection to ConnectionManagerId(ip-172-16-1-152.us-west-2.compute.internal,41091)
14/11/08 08:21:21 INFO cluster.YarnClientSchedulerBackend: Executor 20 disconnected, so removing it
14/11/08 08:21:21 ERROR cluster.YarnClientClusterScheduler: Lost executor 20 on ip-172-16-1-152.us-west-2.compute.internal: remote Akka client disassociated
14/11/08 08:21:21 INFO scheduler.TaskSetManager: Re-queueing tasks for 20 from TaskSet 0.0
14/11/08 08:21:21 WARN scheduler.TaskSetManager: Lost task 29.0 in stage 0.0 (TID 29, ip-172-16-1-152.us-west-2.compute.internal): ExecutorLostFailure (executor lost)
14/11/08 08:21:21 INFO scheduler.DAGScheduler: Executor lost: 20 (epoch 3)
14/11/08 08:21:21 INFO storage.BlockManagerMasterActor: Trying to remove executor 20 from BlockManagerMaster.
14/11/08 08:21:21 INFO storage.BlockManagerMaster: Removed 20 successfully in removeExecutor
14/11/08 08:21:26 INFO network.ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(ip-172-16-1-23.us-west-2.compute.internal,51269)
14/11/08 08:21:26 INFO network.ConnectionManager: Removing SendingConnection to ConnectionManagerId(ip-172-16-1-23.us-west-2.compute.internal,51269)
14/11/08 08:21:26 INFO network.ConnectionManager: Removing SendingConnection to ConnectionManagerId(ip-172-16-1-23.us-west-2.compute.internal,51269)
14/11/08 08:21:26 INFO cluster.YarnClientSchedulerBackend: Executor 6 disconnected, so removing it
14/11/08 08:21:26 ERROR cluster.YarnClientClusterScheduler: Lost executor 6 on ip-172-16-1-23.us-west-2.compute.internal: remote Akka client disassociated
14/11/08 08:21:26 INFO scheduler.TaskSetManager: Re-queueing tasks for 6 from TaskSet 0.0
14/11/08 08:21:26 WARN scheduler.TaskSetManager: Lost task 24.0 in stage 0.0 (TID 24, ip-172-16-1-23.us-west-2.compute.internal): ExecutorLostFailure (executor lost)
14/11/08 08:21:26 INFO scheduler.DAGScheduler: Executor lost: 6 (epoch 4)
14/11/08 08:21:26 INFO storage.BlockManagerMasterActor: Trying to remove executor 6 from BlockManagerMaster.
14/11/08 08:21:26 INFO storage.BlockManagerMaster: Removed 6 successfully in removeExecutor
14/11/08 08:21:26 INFO network.ConnectionManager: Removing SendingConnection to ConnectionManagerId(ip-172-16-1-90.us-west-2.compute.internal,46792)
14/11/08 08:21:26 INFO network.ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(ip-172-16-1-90.us-west-2.compute.internal,46792)
14/11/08 08:21:26 ERROR network.ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(ip-172-16-1-90.us-west-2.compute.internal,46792) not found
14/11/08 08:21:26 INFO cluster.YarnClientSchedulerBackend: Executor 21 disconnected, so removing it
14/11/08 08:21:26 ERROR cluster.YarnClientClusterScheduler: Lost executor 21 on ip-172-16-1-90.us-west-2.compute.internal: remote Akka client disassociated
14/11/08 08:21:26 INFO scheduler.TaskSetManager: Re-queueing tasks for 21 from TaskSet 0.0
14/11/08 08:21:26 WARN scheduler.TaskSetManager: Lost task 25.0 in stage 0.0 (TID 25, ip-172-16-1-90.us-west-2.compute.internal): ExecutorLostFailure (executor lost)
14/11/08 08:21:26 INFO scheduler.DAGScheduler: Executor lost: 21 (epoch 5)
14/11/08 08:21:26 INFO storage.BlockManagerMasterActor: Trying to remove executor 21 from BlockManagerMaster.
14/11/08 08:21:26 INFO storage.BlockManagerMaster: Removed 21 successfully in removeExecutor
14/11/08 08:21:29 INFO cluster.YarnClientSchedulerBackend: Executor 18 disconnected, so removing it
14/11/08 08:21:29 INFO network.ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(ip-172-16-1-222.us-west-2.compute.internal,43883)
14/11/08 08:21:29 INFO network.ConnectionManager: Removing SendingConnection to ConnectionManagerId(ip-172-16-1-222.us-west-2.compute.internal,43883)
14/11/08 08:21:29 INFO network.ConnectionManager: Removing SendingConnection to ConnectionManagerId(ip-172-16-1-222.us-west-2.compute.internal,43883)
14/11/08 08:21:29 ERROR cluster.YarnClientClusterScheduler: Lost executor 18 on ip-172-16-1-222.us-west-2.compute.internal: remote Akka client disassociated
14/11/08 08:21:29 INFO scheduler.TaskSetManager: Re-queueing tasks for 18 from TaskSet 0.0
14/11/08 08:21:29 WARN scheduler.TaskSetManager: Lost task 26.0 in stage 0.0 (TID 26, ip-172-16-1-222.us-west-2.compute.internal): ExecutorLostFailure (executor lost)
14/11/08 08:21:29 INFO scheduler.DAGScheduler: Executor lost: 18 (epoch 6)
14/11/08 08:21:29 INFO storage.BlockManagerMasterActor: Trying to remove executor 18 from BlockManagerMaster.
14/11/08 08:21:29 INFO storage.BlockManagerMaster: Removed 18 successfully in removeExecutor
14/11/08 08:21:30 INFO cluster.YarnClientSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@ip-172-16-1-194.us-west-2.compute.internal:50858/user/Executor#935992941] with ID 31
14/11/08 08:21:30 INFO scheduler.TaskSetManager: Starting task 26.1 in stage 0.0 (TID 30, ip-172-16-1-194.us-west-2.compute.internal, PROCESS_LOCAL, 1122 bytes)
14/11/08 08:21:30 INFO storage.BlockManagerMasterActor: Registering block manager ip-172-16-1-194.us-west-2.compute.internal:44263 with 776.3 MB RAM
14/11/08 08:21:31 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on ip-172-16-1-194.us-west-2.compute.internal:44263 (size: 596.9 KB, free: 775.7 MB)
14/11/08 08:21:33 INFO network.ConnectionManager: Removing SendingConnection to ConnectionManagerId(ip-172-16-1-222.us-west-2.compute.internal,40102)
14/11/08 08:21:33 INFO network.ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(ip-172-16-1-222.us-west-2.compute.internal,40102)
14/11/08 08:21:33 ERROR network.ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(ip-172-16-1-222.us-west-2.compute.internal,40102) not found
14/11/08 08:21:33 INFO cluster.YarnClientSchedulerBackend: Executor 26 disconnected, so removing it
14/11/08 08:21:33 ERROR cluster.YarnClientClusterScheduler: Lost executor 26 on ip-172-16-1-222.us-west-2.compute.internal: remote Akka client disassociated
14/11/08 08:21:33 INFO scheduler.TaskSetManager: Re-queueing tasks for 26 from TaskSet 0.0
14/11/08 08:21:33 WARN scheduler.TaskSetManager: Lost task 23.0 in stage 0.0 (TID 23, ip-172-16-1-222.us-west-2.compute.internal): ExecutorLostFailure (executor lost)
14/11/08 08:21:33 INFO scheduler.DAGScheduler: Executor lost: 26 (epoch 7)
14/11/08 08:21:33 INFO storage.BlockManagerMasterActor: Trying to remove executor 26 from BlockManagerMaster.
14/11/08 08:21:33 INFO storage.BlockManagerMaster: Removed 26 successfully in removeExecutor
14/11/08 08:21:36 INFO network.ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(ip-172-16-1-241.us-west-2.compute.internal,43310)
14/11/08 08:21:36 INFO network.ConnectionManager: Removing SendingConnection to ConnectionManagerId(ip-172-16-1-241.us-west-2.compute.internal,43310)
14/11/08 08:21:36 INFO cluster.YarnClientSchedulerBackend: Executor 1 disconnected, so removing it
14/11/08 08:21:36 ERROR cluster.YarnClientClusterScheduler: Lost executor 1 on ip-172-16-1-241.us-west-2.compute.internal: remote Akka client disassociated
14/11/08 08:21:36 INFO scheduler.TaskSetManager: Re-queueing tasks for 1 from TaskSet 0.0
14/11/08 08:21:36 WARN scheduler.TaskSetManager: Lost task 22.0 in stage 0.0 (TID 22, ip-172-16-1-241.us-west-2.compute.internal): ExecutorLostFailure (executor lost)
14/11/08 08:21:36 ERROR network.SendingConnection: Exception while reading SendingConnection to ConnectionManagerId(ip-172-16-1-241.us-west-2.compute.internal,43310)
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
at org.apache.spark.network.SendingConnection.read(Connection.scala:390)
at org.apache.spark.network.ConnectionManager$$anon$7.run(ConnectionManager.scala:199)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
14/11/08 08:21:36 INFO scheduler.DAGScheduler: Executor lost: 1 (epoch 8)
14/11/08 08:21:36 INFO storage.BlockManagerMasterActor: Trying to remove executor 1 from BlockManagerMaster.
14/11/08 08:21:36 INFO storage.BlockManagerMaster: Removed 1 successfully in removeExecutor
14/11/08 08:21:36 INFO network.ConnectionManager: Handling connection error on connection to ConnectionManagerId(ip-172-16-1-241.us-west-2.compute.internal,43310)
14/11/08 08:21:36 INFO network.ConnectionManager: Removing SendingConnection to ConnectionManagerId(ip-172-16-1-241.us-west-2.compute.internal,43310)
14/11/08 08:21:36 INFO network.ConnectionManager: Removing SendingConnection to ConnectionManagerId(ip-172-16-1-241.us-west-2.compute.internal,43310)
14/11/08 08:21:40 INFO cluster.YarnClientSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@ip-172-16-1-194.us-west-2.compute.internal:58099/user/Executor#-112835629] with ID 34
14/11/08 08:21:40 INFO scheduler.TaskSetManager: Starting task 22.1 in stage 0.0 (TID 31, ip-172-16-1-194.us-west-2.compute.internal, PROCESS_LOCAL, 1122 bytes)
14/11/08 08:21:41 INFO storage.BlockManagerMasterActor: Registering block manager ip-172-16-1-194.us-west-2.compute.internal:41093 with 776.3 MB RAM
14/11/08 08:21:41 INFO cluster.YarnClientSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@ip-172-16-1-228.us-west-2.compute.internal:36136/user/Executor#318736262] with ID 32
14/11/08 08:21:41 INFO scheduler.TaskSetManager: Starting task 23.1 in stage 0.0 (TID 32, ip-172-16-1-228.us-west-2.compute.internal, PROCESS_LOCAL, 1122 bytes)
14/11/08 08:21:41 INFO cluster.YarnClientSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@ip-172-16-1-90.us-west-2.compute.internal:33130/user/Executor#1744030597] with ID 33
14/11/08 08:21:41 INFO scheduler.TaskSetManager: Starting task 25.1 in stage 0.0 (TID 33, ip-172-16-1-90.us-west-2.compute.internal, PROCESS_LOCAL, 1122 bytes)
14/11/08 08:21:41 INFO cluster.YarnClientSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@ip-172-16-1-92.us-west-2.compute.internal:55503/user/Executor#574084779] with ID 35
14/11/08 08:21:41 INFO scheduler.TaskSetManager: Starting task 24.1 in stage 0.0 (TID 34, ip-172-16-1-92.us-west-2.compute.internal, PROCESS_LOCAL, 1122 bytes)
14/11/08 08:21:42 INFO storage.BlockManagerMasterActor: Registering block manager ip-172-16-1-228.us-west-2.compute.internal:40128 with 776.3 MB RAM
14/11/08 08:21:42 INFO storage.BlockManagerMasterActor: Registering block manager ip-172-16-1-90.us-west-2.compute.internal:32839 with 776.3 MB RAM
14/11/08 08:21:42 INFO storage.BlockManagerMasterActor: Registering block manager ip-172-16-1-92.us-west-2.compute.internal:58081 with 776.3 MB RAM
14/11/08 08:21:42 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on ip-172-16-1-194.us-west-2.compute.internal:41093 (size: 596.9 KB, free: 775.7 MB)
14/11/08 08:21:43 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on ip-172-16-1-228.us-west-2.compute.internal:40128 (size: 596.9 KB, free: 775.7 MB)
14/11/08 08:21:43 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on ip-172-16-1-92.us-west-2.compute.internal:58081 (size: 596.9 KB, free: 775.7 MB)
14/11/08 08:21:43 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on ip-172-16-1-90.us-west-2.compute.internal:32839 (size: 596.9 KB, free: 775.7 MB)
14/11/08 08:21:43 INFO cluster.YarnClientSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@ip-172-16-1-152.us-west-2.compute.internal:34268/user/Executor#-937582169] with ID 36
14/11/08 08:21:43 INFO scheduler.TaskSetManager: Starting task 29.1 in stage 0.0 (TID 35, ip-172-16-1-152.us-west-2.compute.internal, PROCESS_LOCAL, 1122 bytes)
14/11/08 08:21:44 INFO storage.BlockManagerMasterActor: Registering block manager ip-172-16-1-152.us-west-2.compute.internal:52550 with 776.3 MB RAM
14/11/08 08:21:45 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on ip-172-16-1-152.us-west-2.compute.internal:52550 (size: 596.9 KB, free: 775.7 MB)
14/11/08 08:21:46 INFO cluster.YarnClientSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@ip-172-16-1-90.us-west-2.compute.internal:34555/user/Executor#-94727554] with ID 37
14/11/08 08:21:46 INFO scheduler.TaskSetManager: Starting task 27.1 in stage 0.0 (TID 36, ip-172-16-1-90.us-west-2.compute.internal, PROCESS_LOCAL, 1122 bytes)
14/11/08 08:21:46 INFO cluster.YarnClientSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@ip-172-16-1-228.us-west-2.compute.internal:34471/user/Executor#1412546630] with ID 38
14/11/08 08:21:46 INFO scheduler.TaskSetManager: Starting task 21.1 in stage 0.0 (TID 37, ip-172-16-1-228.us-west-2.compute.internal, PROCESS_LOCAL, 1122 bytes)
14/11/08 08:21:47 INFO storage.BlockManagerMasterActor: Registering block manager ip-172-16-1-90.us-west-2.compute.internal:46194 with 776.3 MB RAM
14/11/08 08:21:47 INFO storage.BlockManagerMasterActor: Registering block manager ip-172-16-1-228.us-west-2.compute.internal:42275 with 776.3 MB RAM
14/11/08 08:21:48 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on ip-172-16-1-90.us-west-2.compute.internal:46194 (size: 596.9 KB, free: 775.7 MB)
14/11/08 08:21:48 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on ip-172-16-1-228.us-west-2.compute.internal:42275 (size: 596.9 KB, free: 775.7 MB)
14/11/08 08:21:50 INFO cluster.YarnClientSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@ip-172-16-1-23.us-west-2.compute.internal:37122/user/Executor#1404320204] with ID 39
14/11/08 08:21:51 INFO scheduler.TaskSetManager: Starting task 28.1 in stage 0.0 (TID 38, ip-172-16-1-23.us-west-2.compute.internal, PROCESS_LOCAL, 1122 bytes)
14/11/08 08:21:51 INFO storage.BlockManagerMasterActor: Registering block manager ip-172-16-1-23.us-west-2.compute.internal:33106 with 776.3 MB RAM
14/11/08 08:21:52 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on ip-172-16-1-23.us-west-2.compute.internal:33106 (size: 596.9 KB, free: 775.7 MB)
14/11/08 08:22:36 INFO cluster.YarnClientSchedulerBackend: Executor 39 disconnected, so removing it
14/11/08 08:22:36 ERROR cluster.YarnClientClusterScheduler: Lost executor 39 on ip-172-16-1-23.us-west-2.compute.internal: remote Akka client disassociated
14/11/08 08:22:36 INFO scheduler.TaskSetManager: Re-queueing tasks for 39 from TaskSet 0.0
14/11/08 08:22:36 WARN scheduler.TaskSetManager: Lost task 28.1 in stage 0.0 (TID 38, ip-172-16-1-23.us-west-2.compute.internal): ExecutorLostFailure (executor lost)
14/11/08 08:22:36 INFO scheduler.DAGScheduler: Executor lost: 39 (epoch 9)
14/11/08 08:22:36 INFO storage.BlockManagerMasterActor: Trying to remove executor 39 from BlockManagerMaster.
14/11/08 08:22:36 INFO storage.BlockManagerMaster: Removed 39 successfully in removeExecutor
14/11/08 08:22:57 INFO cluster.YarnClientSchedulerBackend: Executor 36 disconnected, so removing it
14/11/08 08:22:57 ERROR cluster.YarnClientClusterScheduler: Lost executor 36 on ip-172-16-1-152.us-west-2.compute.internal: remote Akka client disassociated
14/11/08 08:22:57 INFO scheduler.TaskSetManager: Re-queueing tasks for 36 from TaskSet 0.0
14/11/08 08:22:57 WARN scheduler.TaskSetManager: Lost task 29.1 in stage 0.0 (TID 35, ip-172-16-1-152.us-west-2.compute.internal): ExecutorLostFailure (executor lost)
14/11/08 08:22:57 INFO scheduler.DAGScheduler: Executor lost: 36 (epoch 10)
14/11/08 08:22:57 INFO storage.BlockManagerMasterActor: Trying to remove executor 36 from BlockManagerMaster.
14/11/08 08:22:57 INFO storage.BlockManagerMaster: Removed 36 successfully in removeExecutor
14/11/08 08:23:00 INFO cluster.YarnClientSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@ip-172-16-1-90.us-west-2.compute.internal:48033/user/Executor#-1088273404] with ID 40
14/11/08 08:23:00 INFO scheduler.TaskSetManager: Starting task 29.2 in stage 0.0 (TID 39, ip-172-16-1-90.us-west-2.compute.internal, PROCESS_LOCAL, 1122 bytes)
14/11/08 08:23:01 INFO storage.BlockManagerMasterActor: Registering block manager ip-172-16-1-90.us-west-2.compute.internal:39067 with 776.3 MB RAM
14/11/08 08:23:03 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on ip-172-16-1-90.us-west-2.compute.internal:39067 (size: 596.9 KB, free: 775.7 MB)
14/11/08 08:23:15 INFO cluster.YarnClientSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@ip-172-16-1-23.us-west-2.compute.internal:48860/user/Executor#-369895446] with ID 41
14/11/08 08:23:15 INFO scheduler.TaskSetManager: Starting task 28.2 in stage 0.0 (TID 40, ip-172-16-1-23.us-west-2.compute.internal, PROCESS_LOCAL, 1122 bytes)
14/11/08 08:23:16 INFO storage.BlockManagerMasterActor: Registering block manager ip-172-16-1-23.us-west-2.compute.internal:38093 with 776.3 MB RAM
14/11/08 08:23:17 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on ip-172-16-1-23.us-west-2.compute.internal:38093 (size: 596.9 KB, free: 775.7 MB)
14/11/08 08:23:32 INFO cluster.YarnClientSchedulerBackend: Executor 34 disconnected, so removing it
14/11/08 08:23:32 ERROR cluster.YarnClientClusterScheduler: Lost executor 34 on ip-172-16-1-194.us-west-2.compute.internal: remote Akka client disassociated
14/11/08 08:23:32 INFO scheduler.TaskSetManager: Re-queueing tasks for 34 from TaskSet 0.0
14/11/08 08:23:32 WARN scheduler.TaskSetManager: Lost task 22.1 in stage 0.0 (TID 31, ip-172-16-1-194.us-west-2.compute.internal): ExecutorLostFailure (executor lost)
14/11/08 08:23:32 INFO scheduler.DAGScheduler: Executor lost: 34 (epoch 11)
14/11/08 08:23:32 INFO storage.BlockManagerMasterActor: Trying to remove executor 34 from BlockManagerMaster.
14/11/08 08:23:32 INFO storage.BlockManagerMaster: Removed 34 successfully in removeExecutor
14/11/08 08:23:53 INFO cluster.YarnClientSchedulerBackend: Executor 41 disconnected, so removing it
14/11/08 08:23:53 ERROR cluster.YarnClientClusterScheduler: Lost executor 41 on ip-172-16-1-23.us-west-2.compute.internal: remote Akka client disassociated
14/11/08 08:23:53 INFO scheduler.TaskSetManager: Re-queueing tasks for 41 from TaskSet 0.0
14/11/08 08:23:53 WARN scheduler.TaskSetManager: Lost task 28.2 in stage 0.0 (TID 40, ip-172-16-1-23.us-west-2.compute.internal): ExecutorLostFailure (executor lost)
14/11/08 08:23:53 INFO scheduler.DAGScheduler: Executor lost: 41 (epoch 12)
14/11/08 08:23:53 INFO storage.BlockManagerMasterActor: Trying to remove executor 41 from BlockManagerMaster.
14/11/08 08:23:53 INFO storage.BlockManagerMaster: Removed 41 successfully in removeExecutor
14/11/08 08:23:57 INFO cluster.YarnClientSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@ip-172-16-1-90.us-west-2.compute.internal:58017/user/Executor#2094507560] with ID 42
14/11/08 08:23:57 INFO scheduler.TaskSetManager: Starting task 28.3 in stage 0.0 (TID 41, ip-172-16-1-90.us-west-2.compute.internal, PROCESS_LOCAL, 1122 bytes)
14/11/08 08:23:58 INFO storage.BlockManagerMasterActor: Registering block manager ip-172-16-1-90.us-west-2.compute.internal:41182 with 776.3 MB RAM
14/11/08 08:24:00 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on ip-172-16-1-90.us-west-2.compute.internal:41182 (size: 596.9 KB, free: 775.7 MB)
14/11/08 08:24:04 INFO cluster.YarnClientSchedulerBackend: Executor 35 disconnected, so removing it
14/11/08 08:24:04 ERROR cluster.YarnClientClusterScheduler: Lost executor 35 on ip-172-16-1-92.us-west-2.compute.internal: remote Akka client disassociated
14/11/08 08:24:04 INFO scheduler.TaskSetManager: Re-queueing tasks for 35 from TaskSet 0.0
14/11/08 08:24:04 WARN scheduler.TaskSetManager: Lost task 24.1 in stage 0.0 (TID 34, ip-172-16-1-92.us-west-2.compute.internal): ExecutorLostFailure (executor lost)
14/11/08 08:24:04 INFO scheduler.DAGScheduler: Executor lost: 35 (epoch 13)
14/11/08 08:24:04 INFO storage.BlockManagerMasterActor: Trying to remove executor 35 from BlockManagerMaster.
14/11/08 08:24:04 INFO storage.BlockManagerMaster: Removed 35 successfully in removeExecutor
14/11/08 08:24:17 INFO cluster.YarnClientSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@ip-172-16-1-90.us-west-2.compute.internal:36395/user/Executor#-1907878650] with ID 43
14/11/08 08:24:17 INFO scheduler.TaskSetManager: Starting task 24.2 in stage 0.0 (TID 42, ip-172-16-1-90.us-west-2.compute.internal, PROCESS_LOCAL, 1122 bytes)
14/11/08 08:24:18 INFO storage.BlockManagerMasterActor: Registering block manager ip-172-16-1-90.us-west-2.compute.internal:46948 with 776.3 MB RAM
14/11/08 08:24:20 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on ip-172-16-1-90.us-west-2.compute.internal:46948 (size: 596.9 KB, free: 775.7 MB)
14/11/08 08:24:21 INFO cluster.YarnClientSchedulerBackend: Executor 40 disconnected, so removing it
14/11/08 08:24:21 ERROR cluster.YarnClientClusterScheduler: Lost executor 40 on ip-172-16-1-90.us-west-2.compute.internal: remote Akka client disassociated
14/11/08 08:24:21 INFO scheduler.TaskSetManager: Re-queueing tasks for 40 from TaskSet 0.0
14/11/08 08:24:21 WARN scheduler.TaskSetManager: Lost task 29.2 in stage 0.0 (TID 39, ip-172-16-1-90.us-west-2.compute.internal): ExecutorLostFailure (executor lost)
14/11/08 08:24:21 INFO scheduler.DAGScheduler: Executor lost: 40 (epoch 14)
14/11/08 08:24:21 INFO storage.BlockManagerMasterActor: Trying to remove executor 40 from BlockManagerMaster.
14/11/08 08:24:21 INFO storage.BlockManagerMaster: Removed 40 successfully in removeExecutor
14/11/08 08:24:31 INFO cluster.YarnClientSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@ip-172-16-1-90.us-west-2.compute.internal:34467/user/Executor#-1100688472] with ID 44
14/11/08 08:24:31 INFO scheduler.TaskSetManager: Starting task 29.3 in stage 0.0 (TID 43, ip-172-16-1-90.us-west-2.compute.internal, PROCESS_LOCAL, 1122 bytes)
14/11/08 08:24:32 INFO storage.BlockManagerMasterActor: Registering block manager ip-172-16-1-90.us-west-2.compute.internal:40126 with 776.3 MB RAM
14/11/08 08:24:34 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on ip-172-16-1-90.us-west-2.compute.internal:40126 (size: 596.9 KB, free: 775.7 MB)
14/11/08 08:24:48 INFO cluster.YarnClientSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@ip-172-16-1-90.us-west-2.compute.internal:53257/user/Executor#-745380917] with ID 45
14/11/08 08:24:48 INFO scheduler.TaskSetManager: Starting task 22.2 in stage 0.0 (TID 44, ip-172-16-1-90.us-west-2.compute.internal, PROCESS_LOCAL, 1122 bytes)
14/11/08 08:24:49 INFO storage.BlockManagerMasterActor: Registering block manager ip-172-16-1-90.us-west-2.compute.internal:46252 with 776.3 MB RAM
14/11/08 08:24:52 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on ip-172-16-1-90.us-west-2.compute.internal:46252 (size: 596.9 KB, free: 775.7 MB)
14/11/08 08:25:16 INFO cluster.YarnClientSchedulerBackend: Executor 38 disconnected, so removing it
14/11/08 08:25:16 ERROR cluster.YarnClientClusterScheduler: Lost executor 38 on ip-172-16-1-228.us-west-2.compute.internal: remote Akka client disassociated
14/11/08 08:25:16 INFO scheduler.TaskSetManager: Re-queueing tasks for 38 from TaskSet 0.0
14/11/08 08:25:16 WARN scheduler.TaskSetManager: Lost task 21.1 in stage 0.0 (TID 37, ip-172-16-1-228.us-west-2.compute.internal): ExecutorLostFailure (executor lost)
14/11/08 08:25:16 INFO scheduler.DAGScheduler: Executor lost: 38 (epoch 15)
14/11/08 08:25:16 INFO storage.BlockManagerMasterActor: Trying to remove executor 38 from BlockManagerMaster.
14/11/08 08:25:16 INFO storage.BlockManagerMaster: Removed 38 successfully in removeExecutor
14/11/08 08:25:37 INFO cluster.YarnClientSchedulerBackend: Executor 42 disconnected, so removing it
14/11/08 08:25:37 ERROR cluster.YarnClientClusterScheduler: Lost executor 42 on ip-172-16-1-90.us-west-2.compute.internal: remote Akka client disassociated
14/11/08 08:25:37 INFO scheduler.TaskSetManager: Re-queueing tasks for 42 from TaskSet 0.0
14/11/08 08:25:37 WARN scheduler.TaskSetManager: Lost task 28.3 in stage 0.0 (TID 41, ip-172-16-1-90.us-west-2.compute.internal): ExecutorLostFailure (executor lost)
14/11/08 08:25:37 ERROR scheduler.TaskSetManager: Task 28 in stage 0.0 failed 4 times; aborting job
14/11/08 08:25:37 INFO cluster.YarnClientClusterScheduler: Cancelling stage 0
14/11/08 08:25:37 INFO cluster.YarnClientClusterScheduler: Stage 0 was cancelled
14/11/08 08:25:37 INFO scheduler.DAGScheduler: Failed to run saveAsTextFile at NativeMethodAccessorImpl.java:-2
14/11/08 08:25:37 INFO scheduler.DAGScheduler: Executor lost: 42 (epoch 16)
14/11/08 08:25:37 INFO storage.BlockManagerMasterActor: Trying to remove executor 42 from BlockManagerMaster.
14/11/08 08:25:37 INFO storage.BlockManagerMaster: Removed 42 successfully in removeExecutor
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Spark on YARN, ExecutorLostFailure for long running computations in map
Posted by ja...@centrum.cz.
So it seems that this problem was related to http://apache-spark-developers-list.1001551.n3.nabble.com/Lost-executor-on-YARN-ALS-iterations-td7916.html and increasing the executor memory worked for me.
______________________________________________________________
Hi,
I am getting ExecutorLostFailure when I run spark on YARN and in map I perform very long tasks (couple of hours). Error Log is below.
Do you know if it is possible to set something to make it possible for Spark to perform these very long running jobs in map?
Thank you very much for any advice.
Best regards,
Jan
Spark log:
4533,931: [GC 394578K->20882K(1472000K), 0,0226470 secs]
Traceback (most recent call last):
File "/home/hadoop/spark_stuff/spark_lda.py", line 112, in <module>
models.saveAsTextFile(sys.argv[1])
File "/home/hadoop/spark/python/pyspark/rdd.py", line 1324, in saveAsTextFile
keyed._jrdd.map(self.ctx._jvm.BytesToString()).saveAsTextFile(path)
File "/home/hadoop/spark/python/lib/py4j-0.8.2.1-src.zip/py4j/java_gateway.py", line 538, in __call__
File "/home/hadoop/spark/python/lib/py4j-0.8.2.1-src.zip/py4j/protocol.py", line 300, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o36.saveAsTextFile.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 28 in stage 0.0 failed 4 times, most recent failure: Lost task 28.3 in stage 0.0 (TID 41, ip-172-16-1-90.us-west-2.compute.internal): ExecutorLostFailure (executor lost)
Driver stacktrace:
at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1185)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1174)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1173)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1173)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:688)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:688)
at scala.Option.foreach(Option.scala:236)
at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:688)
at org.apache.spark.scheduler.DAGSchedulerEventProcessActor$$anonfun$receive$2.applyOrElse(DAGScheduler.scala:1391)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)
at akka.actor.ActorCell.invoke(ActorCell.scala:456)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)
at akka.dispatch.Mailbox.run(Mailbox.scala:219)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Yarn log:
14/11/08 08:20:34 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on ip-172-16-1-152.us-west-2.compute.internal:41091 (size: 596.9 KB, free: 775.7 MB)
14/11/08 08:20:34 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on ip-172-16-1-152.us-west-2.compute.internal:39160 (size: 596.9 KB, free: 775.7 MB)
14/11/08 08:20:34 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on ip-172-16-1-152.us-west-2.compute.internal:45058 (size: 596.9 KB, free: 775.7 MB)
14/11/08 08:20:34 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on ip-172-16-1-241.us-west-2.compute.internal:54111 (size: 596.9 KB, free: 775.7 MB)
14/11/08 08:20:34 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on ip-172-16-1-238.us-west-2.compute.internal:45772 (size: 596.9 KB, free: 775.7 MB)
14/11/08 08:20:34 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on ip-172-16-1-241.us-west-2.compute.internal:59509 (size: 596.9 KB, free: 775.7 MB)
14/11/08 08:20:34 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on ip-172-16-1-238.us-west-2.compute.internal:35720 (size: 596.9 KB, free: 775.7 MB)
14/11/08 08:21:11 INFO network.ConnectionManager: Removing SendingConnection to ConnectionManagerId(ip-172-16-1-241.us-west-2.compute.internal,59509)
14/11/08 08:21:11 INFO network.ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(ip-172-16-1-241.us-west-2.compute.internal,59509)
14/11/08 08:21:11 ERROR network.ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(ip-172-16-1-241.us-west-2.compute.internal,59509) not found
14/11/08 08:21:11 INFO cluster.YarnClientSchedulerBackend: Executor 10 disconnected, so removing it
14/11/08 08:21:11 ERROR cluster.YarnClientClusterScheduler: Lost executor 10 on ip-172-16-1-241.us-west-2.compute.internal: remote Akka client disassociated
14/11/08 08:21:11 INFO scheduler.TaskSetManager: Re-queueing tasks for 10 from TaskSet 0.0
14/11/08 08:21:11 WARN scheduler.TaskSetManager: Lost task 28.0 in stage 0.0 (TID 28, ip-172-16-1-241.us-west-2.compute.internal): ExecutorLostFailure (executor lost)
14/11/08 08:21:11 INFO scheduler.DAGScheduler: Executor lost: 10 (epoch 0)
14/11/08 08:21:11 INFO storage.BlockManagerMasterActor: Trying to remove executor 10 from BlockManagerMaster.
14/11/08 08:21:11 INFO storage.BlockManagerMaster: Removed 10 successfully in removeExecutor
14/11/08 08:21:20 INFO network.ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(ip-172-16-1-194.us-west-2.compute.internal,45823)
14/11/08 08:21:20 INFO network.ConnectionManager: Removing SendingConnection to ConnectionManagerId(ip-172-16-1-194.us-west-2.compute.internal,45823)
14/11/08 08:21:20 INFO network.ConnectionManager: Removing SendingConnection to ConnectionManagerId(ip-172-16-1-194.us-west-2.compute.internal,45823)
14/11/08 08:21:20 INFO cluster.YarnClientSchedulerBackend: Executor 5 disconnected, so removing it
14/11/08 08:21:20 ERROR cluster.YarnClientClusterScheduler: Lost executor 5 on ip-172-16-1-194.us-west-2.compute.internal: remote Akka client disassociated
14/11/08 08:21:20 INFO scheduler.TaskSetManager: Re-queueing tasks for 5 from TaskSet 0.0
14/11/08 08:21:20 WARN scheduler.TaskSetManager: Lost task 21.0 in stage 0.0 (TID 21, ip-172-16-1-194.us-west-2.compute.internal): ExecutorLostFailure (executor lost)
14/11/08 08:21:20 INFO scheduler.DAGScheduler: Executor lost: 5 (epoch 1)
14/11/08 08:21:20 INFO network.ConnectionManager: key already cancelled ? sun.nio.ch.SelectionKeyImpl@3bb633cd
java.nio.channels.CancelledKeyException
at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77)
at org.apache.spark.network.ConnectionManager.run(ConnectionManager.scala:289)
at org.apache.spark.network.ConnectionManager$$anon$4.run(ConnectionManager.scala:139)
14/11/08 08:21:20 INFO storage.BlockManagerMasterActor: Trying to remove executor 5 from BlockManagerMaster.
14/11/08 08:21:20 INFO storage.BlockManagerMaster: Removed 5 successfully in removeExecutor
14/11/08 08:21:21 INFO network.ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(ip-172-16-1-92.us-west-2.compute.internal,50928)
14/11/08 08:21:21 INFO network.ConnectionManager: Removing SendingConnection to ConnectionManagerId(ip-172-16-1-92.us-west-2.compute.internal,50928)
14/11/08 08:21:21 INFO network.ConnectionManager: Removing SendingConnection to ConnectionManagerId(ip-172-16-1-92.us-west-2.compute.internal,50928)
14/11/08 08:21:21 INFO cluster.YarnClientSchedulerBackend: Executor 27 disconnected, so removing it
14/11/08 08:21:21 ERROR cluster.YarnClientClusterScheduler: Lost executor 27 on ip-172-16-1-92.us-west-2.compute.internal: remote Akka client disassociated
14/11/08 08:21:21 INFO scheduler.TaskSetManager: Re-queueing tasks for 27 from TaskSet 0.0
14/11/08 08:21:21 WARN scheduler.TaskSetManager: Lost task 27.0 in stage 0.0 (TID 27, ip-172-16-1-92.us-west-2.compute.internal): ExecutorLostFailure (executor lost)
14/11/08 08:21:21 INFO scheduler.DAGScheduler: Executor lost: 27 (epoch 2)
14/11/08 08:21:21 INFO storage.BlockManagerMasterActor: Trying to remove executor 27 from BlockManagerMaster.
14/11/08 08:21:21 INFO storage.BlockManagerMaster: Removed 27 successfully in removeExecutor
14/11/08 08:21:21 INFO network.ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(ip-172-16-1-152.us-west-2.compute.internal,41091)
14/11/08 08:21:21 INFO network.ConnectionManager: Removing SendingConnection to ConnectionManagerId(ip-172-16-1-152.us-west-2.compute.internal,41091)
14/11/08 08:21:21 INFO network.ConnectionManager: Removing SendingConnection to ConnectionManagerId(ip-172-16-1-152.us-west-2.compute.internal,41091)
14/11/08 08:21:21 INFO cluster.YarnClientSchedulerBackend: Executor 20 disconnected, so removing it
14/11/08 08:21:21 ERROR cluster.YarnClientClusterScheduler: Lost executor 20 on ip-172-16-1-152.us-west-2.compute.internal: remote Akka client disassociated
14/11/08 08:21:21 INFO scheduler.TaskSetManager: Re-queueing tasks for 20 from TaskSet 0.0
14/11/08 08:21:21 WARN scheduler.TaskSetManager: Lost task 29.0 in stage 0.0 (TID 29, ip-172-16-1-152.us-west-2.compute.internal): ExecutorLostFailure (executor lost)
14/11/08 08:21:21 INFO scheduler.DAGScheduler: Executor lost: 20 (epoch 3)
14/11/08 08:21:21 INFO storage.BlockManagerMasterActor: Trying to remove executor 20 from BlockManagerMaster.
14/11/08 08:21:21 INFO storage.BlockManagerMaster: Removed 20 successfully in removeExecutor
14/11/08 08:21:26 INFO network.ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(ip-172-16-1-23.us-west-2.compute.internal,51269)
14/11/08 08:21:26 INFO network.ConnectionManager: Removing SendingConnection to ConnectionManagerId(ip-172-16-1-23.us-west-2.compute.internal,51269)
14/11/08 08:21:26 INFO network.ConnectionManager: Removing SendingConnection to ConnectionManagerId(ip-172-16-1-23.us-west-2.compute.internal,51269)
14/11/08 08:21:26 INFO cluster.YarnClientSchedulerBackend: Executor 6 disconnected, so removing it
14/11/08 08:21:26 ERROR cluster.YarnClientClusterScheduler: Lost executor 6 on ip-172-16-1-23.us-west-2.compute.internal: remote Akka client disassociated
14/11/08 08:21:26 INFO scheduler.TaskSetManager: Re-queueing tasks for 6 from TaskSet 0.0
14/11/08 08:21:26 WARN scheduler.TaskSetManager: Lost task 24.0 in stage 0.0 (TID 24, ip-172-16-1-23.us-west-2.compute.internal): ExecutorLostFailure (executor lost)
14/11/08 08:21:26 INFO scheduler.DAGScheduler: Executor lost: 6 (epoch 4)
14/11/08 08:21:26 INFO storage.BlockManagerMasterActor: Trying to remove executor 6 from BlockManagerMaster.
14/11/08 08:21:26 INFO storage.BlockManagerMaster: Removed 6 successfully in removeExecutor
14/11/08 08:21:26 INFO network.ConnectionManager: Removing SendingConnection to ConnectionManagerId(ip-172-16-1-90.us-west-2.compute.internal,46792)
14/11/08 08:21:26 INFO network.ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(ip-172-16-1-90.us-west-2.compute.internal,46792)
14/11/08 08:21:26 ERROR network.ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(ip-172-16-1-90.us-west-2.compute.internal,46792) not found
14/11/08 08:21:26 INFO cluster.YarnClientSchedulerBackend: Executor 21 disconnected, so removing it
14/11/08 08:21:26 ERROR cluster.YarnClientClusterScheduler: Lost executor 21 on ip-172-16-1-90.us-west-2.compute.internal: remote Akka client disassociated
14/11/08 08:21:26 INFO scheduler.TaskSetManager: Re-queueing tasks for 21 from TaskSet 0.0
14/11/08 08:21:26 WARN scheduler.TaskSetManager: Lost task 25.0 in stage 0.0 (TID 25, ip-172-16-1-90.us-west-2.compute.internal): ExecutorLostFailure (executor lost)
14/11/08 08:21:26 INFO scheduler.DAGScheduler: Executor lost: 21 (epoch 5)
14/11/08 08:21:26 INFO storage.BlockManagerMasterActor: Trying to remove executor 21 from BlockManagerMaster.
14/11/08 08:21:26 INFO storage.BlockManagerMaster: Removed 21 successfully in removeExecutor
14/11/08 08:21:29 INFO cluster.YarnClientSchedulerBackend: Executor 18 disconnected, so removing it
14/11/08 08:21:29 INFO network.ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(ip-172-16-1-222.us-west-2.compute.internal,43883)
14/11/08 08:21:29 INFO network.ConnectionManager: Removing SendingConnection to ConnectionManagerId(ip-172-16-1-222.us-west-2.compute.internal,43883)
14/11/08 08:21:29 INFO network.ConnectionManager: Removing SendingConnection to ConnectionManagerId(ip-172-16-1-222.us-west-2.compute.internal,43883)
14/11/08 08:21:29 ERROR cluster.YarnClientClusterScheduler: Lost executor 18 on ip-172-16-1-222.us-west-2.compute.internal: remote Akka client disassociated
14/11/08 08:21:29 INFO scheduler.TaskSetManager: Re-queueing tasks for 18 from TaskSet 0.0
14/11/08 08:21:29 WARN scheduler.TaskSetManager: Lost task 26.0 in stage 0.0 (TID 26, ip-172-16-1-222.us-west-2.compute.internal): ExecutorLostFailure (executor lost)
14/11/08 08:21:29 INFO scheduler.DAGScheduler: Executor lost: 18 (epoch 6)
14/11/08 08:21:29 INFO storage.BlockManagerMasterActor: Trying to remove executor 18 from BlockManagerMaster.
14/11/08 08:21:29 INFO storage.BlockManagerMaster: Removed 18 successfully in removeExecutor
14/11/08 08:21:30 INFO cluster.YarnClientSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@ip-172-16-1-194.us-west-2.compute.internal:50858/user/Executor#935992941] with ID 31
14/11/08 08:21:30 INFO scheduler.TaskSetManager: Starting task 26.1 in stage 0.0 (TID 30, ip-172-16-1-194.us-west-2.compute.internal, PROCESS_LOCAL, 1122 bytes)
14/11/08 08:21:30 INFO storage.BlockManagerMasterActor: Registering block manager ip-172-16-1-194.us-west-2.compute.internal:44263 with 776.3 MB RAM
14/11/08 08:21:31 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on ip-172-16-1-194.us-west-2.compute.internal:44263 (size: 596.9 KB, free: 775.7 MB)
14/11/08 08:21:33 INFO network.ConnectionManager: Removing SendingConnection to ConnectionManagerId(ip-172-16-1-222.us-west-2.compute.internal,40102)
14/11/08 08:21:33 INFO network.ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(ip-172-16-1-222.us-west-2.compute.internal,40102)
14/11/08 08:21:33 ERROR network.ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(ip-172-16-1-222.us-west-2.compute.internal,40102) not found
14/11/08 08:21:33 INFO cluster.YarnClientSchedulerBackend: Executor 26 disconnected, so removing it
14/11/08 08:21:33 ERROR cluster.YarnClientClusterScheduler: Lost executor 26 on ip-172-16-1-222.us-west-2.compute.internal: remote Akka client disassociated
14/11/08 08:21:33 INFO scheduler.TaskSetManager: Re-queueing tasks for 26 from TaskSet 0.0
14/11/08 08:21:33 WARN scheduler.TaskSetManager: Lost task 23.0 in stage 0.0 (TID 23, ip-172-16-1-222.us-west-2.compute.internal): ExecutorLostFailure (executor lost)
14/11/08 08:21:33 INFO scheduler.DAGScheduler: Executor lost: 26 (epoch 7)
14/11/08 08:21:33 INFO storage.BlockManagerMasterActor: Trying to remove executor 26 from BlockManagerMaster.
14/11/08 08:21:33 INFO storage.BlockManagerMaster: Removed 26 successfully in removeExecutor
14/11/08 08:21:36 INFO network.ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(ip-172-16-1-241.us-west-2.compute.internal,43310)
14/11/08 08:21:36 INFO network.ConnectionManager: Removing SendingConnection to ConnectionManagerId(ip-172-16-1-241.us-west-2.compute.internal,43310)
14/11/08 08:21:36 INFO cluster.YarnClientSchedulerBackend: Executor 1 disconnected, so removing it
14/11/08 08:21:36 ERROR cluster.YarnClientClusterScheduler: Lost executor 1 on ip-172-16-1-241.us-west-2.compute.internal: remote Akka client disassociated
14/11/08 08:21:36 INFO scheduler.TaskSetManager: Re-queueing tasks for 1 from TaskSet 0.0
14/11/08 08:21:36 WARN scheduler.TaskSetManager: Lost task 22.0 in stage 0.0 (TID 22, ip-172-16-1-241.us-west-2.compute.internal): ExecutorLostFailure (executor lost)
14/11/08 08:21:36 ERROR network.SendingConnection: Exception while reading SendingConnection to ConnectionManagerId(ip-172-16-1-241.us-west-2.compute.internal,43310)
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
at org.apache.spark.network.SendingConnection.read(Connection.scala:390)
at org.apache.spark.network.ConnectionManager$$anon$7.run(ConnectionManager.scala:199)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
14/11/08 08:21:36 INFO scheduler.DAGScheduler: Executor lost: 1 (epoch 8)
14/11/08 08:21:36 INFO storage.BlockManagerMasterActor: Trying to remove executor 1 from BlockManagerMaster.
14/11/08 08:21:36 INFO storage.BlockManagerMaster: Removed 1 successfully in removeExecutor
14/11/08 08:21:36 INFO network.ConnectionManager: Handling connection error on connection to ConnectionManagerId(ip-172-16-1-241.us-west-2.compute.internal,43310)
14/11/08 08:21:36 INFO network.ConnectionManager: Removing SendingConnection to ConnectionManagerId(ip-172-16-1-241.us-west-2.compute.internal,43310)
14/11/08 08:21:36 INFO network.ConnectionManager: Removing SendingConnection to ConnectionManagerId(ip-172-16-1-241.us-west-2.compute.internal,43310)
14/11/08 08:21:40 INFO cluster.YarnClientSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@ip-172-16-1-194.us-west-2.compute.internal:58099/user/Executor#-112835629] with ID 34
14/11/08 08:21:40 INFO scheduler.TaskSetManager: Starting task 22.1 in stage 0.0 (TID 31, ip-172-16-1-194.us-west-2.compute.internal, PROCESS_LOCAL, 1122 bytes)
14/11/08 08:21:41 INFO storage.BlockManagerMasterActor: Registering block manager ip-172-16-1-194.us-west-2.compute.internal:41093 with 776.3 MB RAM
14/11/08 08:21:41 INFO cluster.YarnClientSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@ip-172-16-1-228.us-west-2.compute.internal:36136/user/Executor#318736262] with ID 32
14/11/08 08:21:41 INFO scheduler.TaskSetManager: Starting task 23.1 in stage 0.0 (TID 32, ip-172-16-1-228.us-west-2.compute.internal, PROCESS_LOCAL, 1122 bytes)
14/11/08 08:21:41 INFO cluster.YarnClientSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@ip-172-16-1-90.us-west-2.compute.internal:33130/user/Executor#1744030597] with ID 33
14/11/08 08:21:41 INFO scheduler.TaskSetManager: Starting task 25.1 in stage 0.0 (TID 33, ip-172-16-1-90.us-west-2.compute.internal, PROCESS_LOCAL, 1122 bytes)
14/11/08 08:21:41 INFO cluster.YarnClientSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@ip-172-16-1-92.us-west-2.compute.internal:55503/user/Executor#574084779] with ID 35
14/11/08 08:21:41 INFO scheduler.TaskSetManager: Starting task 24.1 in stage 0.0 (TID 34, ip-172-16-1-92.us-west-2.compute.internal, PROCESS_LOCAL, 1122 bytes)
14/11/08 08:21:42 INFO storage.BlockManagerMasterActor: Registering block manager ip-172-16-1-228.us-west-2.compute.internal:40128 with 776.3 MB RAM
14/11/08 08:21:42 INFO storage.BlockManagerMasterActor: Registering block manager ip-172-16-1-90.us-west-2.compute.internal:32839 with 776.3 MB RAM
14/11/08 08:21:42 INFO storage.BlockManagerMasterActor: Registering block manager ip-172-16-1-92.us-west-2.compute.internal:58081 with 776.3 MB RAM
14/11/08 08:21:42 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on ip-172-16-1-194.us-west-2.compute.internal:41093 (size: 596.9 KB, free: 775.7 MB)
14/11/08 08:21:43 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on ip-172-16-1-228.us-west-2.compute.internal:40128 (size: 596.9 KB, free: 775.7 MB)
14/11/08 08:21:43 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on ip-172-16-1-92.us-west-2.compute.internal:58081 (size: 596.9 KB, free: 775.7 MB)
14/11/08 08:21:43 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on ip-172-16-1-90.us-west-2.compute.internal:32839 (size: 596.9 KB, free: 775.7 MB)
14/11/08 08:21:43 INFO cluster.YarnClientSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@ip-172-16-1-152.us-west-2.compute.internal:34268/user/Executor#-937582169] with ID 36
14/11/08 08:21:43 INFO scheduler.TaskSetManager: Starting task 29.1 in stage 0.0 (TID 35, ip-172-16-1-152.us-west-2.compute.internal, PROCESS_LOCAL, 1122 bytes)
14/11/08 08:21:44 INFO storage.BlockManagerMasterActor: Registering block manager ip-172-16-1-152.us-west-2.compute.internal:52550 with 776.3 MB RAM
14/11/08 08:21:45 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on ip-172-16-1-152.us-west-2.compute.internal:52550 (size: 596.9 KB, free: 775.7 MB)
14/11/08 08:21:46 INFO cluster.YarnClientSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@ip-172-16-1-90.us-west-2.compute.internal:34555/user/Executor#-94727554] with ID 37
14/11/08 08:21:46 INFO scheduler.TaskSetManager: Starting task 27.1 in stage 0.0 (TID 36, ip-172-16-1-90.us-west-2.compute.internal, PROCESS_LOCAL, 1122 bytes)
14/11/08 08:21:46 INFO cluster.YarnClientSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@ip-172-16-1-228.us-west-2.compute.internal:34471/user/Executor#1412546630] with ID 38
14/11/08 08:21:46 INFO scheduler.TaskSetManager: Starting task 21.1 in stage 0.0 (TID 37, ip-172-16-1-228.us-west-2.compute.internal, PROCESS_LOCAL, 1122 bytes)
14/11/08 08:21:47 INFO storage.BlockManagerMasterActor: Registering block manager ip-172-16-1-90.us-west-2.compute.internal:46194 with 776.3 MB RAM
14/11/08 08:21:47 INFO storage.BlockManagerMasterActor: Registering block manager ip-172-16-1-228.us-west-2.compute.internal:42275 with 776.3 MB RAM
14/11/08 08:21:48 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on ip-172-16-1-90.us-west-2.compute.internal:46194 (size: 596.9 KB, free: 775.7 MB)
14/11/08 08:21:48 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on ip-172-16-1-228.us-west-2.compute.internal:42275 (size: 596.9 KB, free: 775.7 MB)
14/11/08 08:21:50 INFO cluster.YarnClientSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@ip-172-16-1-23.us-west-2.compute.internal:37122/user/Executor#1404320204] with ID 39
14/11/08 08:21:51 INFO scheduler.TaskSetManager: Starting task 28.1 in stage 0.0 (TID 38, ip-172-16-1-23.us-west-2.compute.internal, PROCESS_LOCAL, 1122 bytes)
14/11/08 08:21:51 INFO storage.BlockManagerMasterActor: Registering block manager ip-172-16-1-23.us-west-2.compute.internal:33106 with 776.3 MB RAM
14/11/08 08:21:52 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on ip-172-16-1-23.us-west-2.compute.internal:33106 (size: 596.9 KB, free: 775.7 MB)
14/11/08 08:22:36 INFO cluster.YarnClientSchedulerBackend: Executor 39 disconnected, so removing it
14/11/08 08:22:36 ERROR cluster.YarnClientClusterScheduler: Lost executor 39 on ip-172-16-1-23.us-west-2.compute.internal: remote Akka client disassociated
14/11/08 08:22:36 INFO scheduler.TaskSetManager: Re-queueing tasks for 39 from TaskSet 0.0
14/11/08 08:22:36 WARN scheduler.TaskSetManager: Lost task 28.1 in stage 0.0 (TID 38, ip-172-16-1-23.us-west-2.compute.internal): ExecutorLostFailure (executor lost)
14/11/08 08:22:36 INFO scheduler.DAGScheduler: Executor lost: 39 (epoch 9)
14/11/08 08:22:36 INFO storage.BlockManagerMasterActor: Trying to remove executor 39 from BlockManagerMaster.
14/11/08 08:22:36 INFO storage.BlockManagerMaster: Removed 39 successfully in removeExecutor
14/11/08 08:22:57 INFO cluster.YarnClientSchedulerBackend: Executor 36 disconnected, so removing it
14/11/08 08:22:57 ERROR cluster.YarnClientClusterScheduler: Lost executor 36 on ip-172-16-1-152.us-west-2.compute.internal: remote Akka client disassociated
14/11/08 08:22:57 INFO scheduler.TaskSetManager: Re-queueing tasks for 36 from TaskSet 0.0
14/11/08 08:22:57 WARN scheduler.TaskSetManager: Lost task 29.1 in stage 0.0 (TID 35, ip-172-16-1-152.us-west-2.compute.internal): ExecutorLostFailure (executor lost)
14/11/08 08:22:57 INFO scheduler.DAGScheduler: Executor lost: 36 (epoch 10)
14/11/08 08:22:57 INFO storage.BlockManagerMasterActor: Trying to remove executor 36 from BlockManagerMaster.
14/11/08 08:22:57 INFO storage.BlockManagerMaster: Removed 36 successfully in removeExecutor
14/11/08 08:23:00 INFO cluster.YarnClientSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@ip-172-16-1-90.us-west-2.compute.internal:48033/user/Executor#-1088273404] with ID 40
14/11/08 08:23:00 INFO scheduler.TaskSetManager: Starting task 29.2 in stage 0.0 (TID 39, ip-172-16-1-90.us-west-2.compute.internal, PROCESS_LOCAL, 1122 bytes)
14/11/08 08:23:01 INFO storage.BlockManagerMasterActor: Registering block manager ip-172-16-1-90.us-west-2.compute.internal:39067 with 776.3 MB RAM
14/11/08 08:23:03 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on ip-172-16-1-90.us-west-2.compute.internal:39067 (size: 596.9 KB, free: 775.7 MB)
14/11/08 08:23:15 INFO cluster.YarnClientSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@ip-172-16-1-23.us-west-2.compute.internal:48860/user/Executor#-369895446] with ID 41
14/11/08 08:23:15 INFO scheduler.TaskSetManager: Starting task 28.2 in stage 0.0 (TID 40, ip-172-16-1-23.us-west-2.compute.internal, PROCESS_LOCAL, 1122 bytes)
14/11/08 08:23:16 INFO storage.BlockManagerMasterActor: Registering block manager ip-172-16-1-23.us-west-2.compute.internal:38093 with 776.3 MB RAM
14/11/08 08:23:17 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on ip-172-16-1-23.us-west-2.compute.internal:38093 (size: 596.9 KB, free: 775.7 MB)
14/11/08 08:23:32 INFO cluster.YarnClientSchedulerBackend: Executor 34 disconnected, so removing it
14/11/08 08:23:32 ERROR cluster.YarnClientClusterScheduler: Lost executor 34 on ip-172-16-1-194.us-west-2.compute.internal: remote Akka client disassociated
14/11/08 08:23:32 INFO scheduler.TaskSetManager: Re-queueing tasks for 34 from TaskSet 0.0
14/11/08 08:23:32 WARN scheduler.TaskSetManager: Lost task 22.1 in stage 0.0 (TID 31, ip-172-16-1-194.us-west-2.compute.internal): ExecutorLostFailure (executor lost)
14/11/08 08:23:32 INFO scheduler.DAGScheduler: Executor lost: 34 (epoch 11)
14/11/08 08:23:32 INFO storage.BlockManagerMasterActor: Trying to remove executor 34 from BlockManagerMaster.
14/11/08 08:23:32 INFO storage.BlockManagerMaster: Removed 34 successfully in removeExecutor
14/11/08 08:23:53 INFO cluster.YarnClientSchedulerBackend: Executor 41 disconnected, so removing it
14/11/08 08:23:53 ERROR cluster.YarnClientClusterScheduler: Lost executor 41 on ip-172-16-1-23.us-west-2.compute.internal: remote Akka client disassociated
14/11/08 08:23:53 INFO scheduler.TaskSetManager: Re-queueing tasks for 41 from TaskSet 0.0
14/11/08 08:23:53 WARN scheduler.TaskSetManager: Lost task 28.2 in stage 0.0 (TID 40, ip-172-16-1-23.us-west-2.compute.internal): ExecutorLostFailure (executor lost)
14/11/08 08:23:53 INFO scheduler.DAGScheduler: Executor lost: 41 (epoch 12)
14/11/08 08:23:53 INFO storage.BlockManagerMasterActor: Trying to remove executor 41 from BlockManagerMaster.
14/11/08 08:23:53 INFO storage.BlockManagerMaster: Removed 41 successfully in removeExecutor
14/11/08 08:23:57 INFO cluster.YarnClientSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@ip-172-16-1-90.us-west-2.compute.internal:58017/user/Executor#2094507560] with ID 42
14/11/08 08:23:57 INFO scheduler.TaskSetManager: Starting task 28.3 in stage 0.0 (TID 41, ip-172-16-1-90.us-west-2.compute.internal, PROCESS_LOCAL, 1122 bytes)
14/11/08 08:23:58 INFO storage.BlockManagerMasterActor: Registering block manager ip-172-16-1-90.us-west-2.compute.internal:41182 with 776.3 MB RAM
14/11/08 08:24:00 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on ip-172-16-1-90.us-west-2.compute.internal:41182 (size: 596.9 KB, free: 775.7 MB)
14/11/08 08:24:04 INFO cluster.YarnClientSchedulerBackend: Executor 35 disconnected, so removing it
14/11/08 08:24:04 ERROR cluster.YarnClientClusterScheduler: Lost executor 35 on ip-172-16-1-92.us-west-2.compute.internal: remote Akka client disassociated
14/11/08 08:24:04 INFO scheduler.TaskSetManager: Re-queueing tasks for 35 from TaskSet 0.0
14/11/08 08:24:04 WARN scheduler.TaskSetManager: Lost task 24.1 in stage 0.0 (TID 34, ip-172-16-1-92.us-west-2.compute.internal): ExecutorLostFailure (executor lost)
14/11/08 08:24:04 INFO scheduler.DAGScheduler: Executor lost: 35 (epoch 13)
14/11/08 08:24:04 INFO storage.BlockManagerMasterActor: Trying to remove executor 35 from BlockManagerMaster.
14/11/08 08:24:04 INFO storage.BlockManagerMaster: Removed 35 successfully in removeExecutor
14/11/08 08:24:17 INFO cluster.YarnClientSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@ip-172-16-1-90.us-west-2.compute.internal:36395/user/Executor#-1907878650] with ID 43
14/11/08 08:24:17 INFO scheduler.TaskSetManager: Starting task 24.2 in stage 0.0 (TID 42, ip-172-16-1-90.us-west-2.compute.internal, PROCESS_LOCAL, 1122 bytes)
14/11/08 08:24:18 INFO storage.BlockManagerMasterActor: Registering block manager ip-172-16-1-90.us-west-2.compute.internal:46948 with 776.3 MB RAM
14/11/08 08:24:20 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on ip-172-16-1-90.us-west-2.compute.internal:46948 (size: 596.9 KB, free: 775.7 MB)
14/11/08 08:24:21 INFO cluster.YarnClientSchedulerBackend: Executor 40 disconnected, so removing it
14/11/08 08:24:21 ERROR cluster.YarnClientClusterScheduler: Lost executor 40 on ip-172-16-1-90.us-west-2.compute.internal: remote Akka client disassociated
14/11/08 08:24:21 INFO scheduler.TaskSetManager: Re-queueing tasks for 40 from TaskSet 0.0
14/11/08 08:24:21 WARN scheduler.TaskSetManager: Lost task 29.2 in stage 0.0 (TID 39, ip-172-16-1-90.us-west-2.compute.internal): ExecutorLostFailure (executor lost)
14/11/08 08:24:21 INFO scheduler.DAGScheduler: Executor lost: 40 (epoch 14)
14/11/08 08:24:21 INFO storage.BlockManagerMasterActor: Trying to remove executor 40 from BlockManagerMaster.
14/11/08 08:24:21 INFO storage.BlockManagerMaster: Removed 40 successfully in removeExecutor
14/11/08 08:24:31 INFO cluster.YarnClientSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@ip-172-16-1-90.us-west-2.compute.internal:34467/user/Executor#-1100688472] with ID 44
14/11/08 08:24:31 INFO scheduler.TaskSetManager: Starting task 29.3 in stage 0.0 (TID 43, ip-172-16-1-90.us-west-2.compute.internal, PROCESS_LOCAL, 1122 bytes)
14/11/08 08:24:32 INFO storage.BlockManagerMasterActor: Registering block manager ip-172-16-1-90.us-west-2.compute.internal:40126 with 776.3 MB RAM
14/11/08 08:24:34 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on ip-172-16-1-90.us-west-2.compute.internal:40126 (size: 596.9 KB, free: 775.7 MB)
14/11/08 08:24:48 INFO cluster.YarnClientSchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@ip-172-16-1-90.us-west-2.compute.internal:53257/user/Executor#-745380917] with ID 45
14/11/08 08:24:48 INFO scheduler.TaskSetManager: Starting task 22.2 in stage 0.0 (TID 44, ip-172-16-1-90.us-west-2.compute.internal, PROCESS_LOCAL, 1122 bytes)
14/11/08 08:24:49 INFO storage.BlockManagerMasterActor: Registering block manager ip-172-16-1-90.us-west-2.compute.internal:46252 with 776.3 MB RAM
14/11/08 08:24:52 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on ip-172-16-1-90.us-west-2.compute.internal:46252 (size: 596.9 KB, free: 775.7 MB)
14/11/08 08:25:16 INFO cluster.YarnClientSchedulerBackend: Executor 38 disconnected, so removing it
14/11/08 08:25:16 ERROR cluster.YarnClientClusterScheduler: Lost executor 38 on ip-172-16-1-228.us-west-2.compute.internal: remote Akka client disassociated
14/11/08 08:25:16 INFO scheduler.TaskSetManager: Re-queueing tasks for 38 from TaskSet 0.0
14/11/08 08:25:16 WARN scheduler.TaskSetManager: Lost task 21.1 in stage 0.0 (TID 37, ip-172-16-1-228.us-west-2.compute.internal): ExecutorLostFailure (executor lost)
14/11/08 08:25:16 INFO scheduler.DAGScheduler: Executor lost: 38 (epoch 15)
14/11/08 08:25:16 INFO storage.BlockManagerMasterActor: Trying to remove executor 38 from BlockManagerMaster.
14/11/08 08:25:16 INFO storage.BlockManagerMaster: Removed 38 successfully in removeExecutor
14/11/08 08:25:37 INFO cluster.YarnClientSchedulerBackend: Executor 42 disconnected, so removing it
14/11/08 08:25:37 ERROR cluster.YarnClientClusterScheduler: Lost executor 42 on ip-172-16-1-90.us-west-2.compute.internal: remote Akka client disassociated
14/11/08 08:25:37 INFO scheduler.TaskSetManager: Re-queueing tasks for 42 from TaskSet 0.0
14/11/08 08:25:37 WARN scheduler.TaskSetManager: Lost task 28.3 in stage 0.0 (TID 41, ip-172-16-1-90.us-west-2.compute.internal): ExecutorLostFailure (executor lost)
14/11/08 08:25:37 ERROR scheduler.TaskSetManager: Task 28 in stage 0.0 failed 4 times; aborting job
14/11/08 08:25:37 INFO cluster.YarnClientClusterScheduler: Cancelling stage 0
14/11/08 08:25:37 INFO cluster.YarnClientClusterScheduler: Stage 0 was cancelled
14/11/08 08:25:37 INFO scheduler.DAGScheduler: Failed to run saveAsTextFile at NativeMethodAccessorImpl.java:-2
14/11/08 08:25:37 INFO scheduler.DAGScheduler: Executor lost: 42 (epoch 16)
14/11/08 08:25:37 INFO storage.BlockManagerMasterActor: Trying to remove executor 42 from BlockManagerMaster.
14/11/08 08:25:37 INFO storage.BlockManagerMaster: Removed 42 successfully in removeExecutor
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Spark on YARN, ExecutorLostFailure for long running computations
in map
Posted by Akhil Das <ak...@sigmoidanalytics.com>.
Used to hit this issue, but setting the following confs while creating the
sparkcontext seems working.
.set("spark.rdd.compress","true")
.set("spark.storage.memoryFraction","1")
.set("spark.core.connection.ack.wait.timeout","6000")
.set("spark.akka.frameSize","50")
Most likely, one of the executor is stuck on a GC Pause and meanwhile
master thinks its dead and throws timeout/cancel key exception.
Thanks
Best Regards
On Sat, Nov 8, 2014 at 2:58 PM, <ja...@centrum.cz> wrote:
> Hi,
>
> I am getting ExecutorLostFailure when I run spark on YARN and in map I
> perform very long tasks (couple of hours). Error Log is below.
>
> Do you know if it is possible to set something to make it possible for
> Spark to perform these very long running jobs in map?
>
> Thank you very much for any advice.
>
> Best regards,
> Jan
>
> Spark log:
> 4533,931: [GC 394578K->20882K(1472000K), 0,0226470 secs]
> Traceback (most recent call last):
> File "/home/hadoop/spark_stuff/spark_lda.py", line 112, in <module>
> models.saveAsTextFile(sys.argv[1])
> File "/home/hadoop/spark/python/pyspark/rdd.py", line 1324, in
> saveAsTextFile
> keyed._jrdd.map(self.ctx._jvm.BytesToString()).saveAsTextFile(path)
> File
> "/home/hadoop/spark/python/lib/py4j-0.8.2.1-src.zip/py4j/java_gateway.py",
> line 538, in __call__
> File
> "/home/hadoop/spark/python/lib/py4j-0.8.2.1-src.zip/py4j/protocol.py", line
> 300, in get_return_value
> py4j.protocol.Py4JJavaError: An error occurred while calling
> o36.saveAsTextFile.
> : org.apache.spark.SparkException: Job aborted due to stage failure: Task
> 28 in stage 0.0 failed 4 times, most recent failure: Lost task 28.3 in
> stage 0.0 (TID 41, ip-172-16-1-90.us-west-2.compute.internal):
> ExecutorLostFailure (executor lost)
> Driver stacktrace:
> at org.apache.spark.scheduler.DAGScheduler.org
> $apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1185)
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1174)
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1173)
> at
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> at
> scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
> at
> org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1173)
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:688)
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:688)
> at scala.Option.foreach(Option.scala:236)
> at
> org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:688)
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessActor$$anonfun$receive$2.applyOrElse(DAGScheduler.scala:1391)
> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)
> at akka.actor.ActorCell.invoke(ActorCell.scala:456)
> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)
> at akka.dispatch.Mailbox.run(Mailbox.scala:219)
> at
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
> at
> scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> at
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> at
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
>
> Yarn log:
> 14/11/08 08:20:34 INFO storage.BlockManagerInfo: Added broadcast_0_piece0
> in memory on ip-172-16-1-152.us-west-2.compute.internal:41091 (size: 596.9
> KB, free: 775.7 MB)
> 14/11/08 08:20:34 INFO storage.BlockManagerInfo: Added broadcast_0_piece0
> in memory on ip-172-16-1-152.us-west-2.compute.internal:39160 (size: 596.9
> KB, free: 775.7 MB)
> 14/11/08 08:20:34 INFO storage.BlockManagerInfo: Added broadcast_0_piece0
> in memory on ip-172-16-1-152.us-west-2.compute.internal:45058 (size: 596.9
> KB, free: 775.7 MB)
> 14/11/08 08:20:34 INFO storage.BlockManagerInfo: Added broadcast_0_piece0
> in memory on ip-172-16-1-241.us-west-2.compute.internal:54111 (size: 596.9
> KB, free: 775.7 MB)
> 14/11/08 08:20:34 INFO storage.BlockManagerInfo: Added broadcast_0_piece0
> in memory on ip-172-16-1-238.us-west-2.compute.internal:45772 (size: 596.9
> KB, free: 775.7 MB)
> 14/11/08 08:20:34 INFO storage.BlockManagerInfo: Added broadcast_0_piece0
> in memory on ip-172-16-1-241.us-west-2.compute.internal:59509 (size: 596.9
> KB, free: 775.7 MB)
> 14/11/08 08:20:34 INFO storage.BlockManagerInfo: Added broadcast_0_piece0
> in memory on ip-172-16-1-238.us-west-2.compute.internal:35720 (size: 596.9
> KB, free: 775.7 MB)
> 14/11/08 08:21:11 INFO network.ConnectionManager: Removing
> SendingConnection to
> ConnectionManagerId(ip-172-16-1-241.us-west-2.compute.internal,59509)
> 14/11/08 08:21:11 INFO network.ConnectionManager: Removing
> ReceivingConnection to
> ConnectionManagerId(ip-172-16-1-241.us-west-2.compute.internal,59509)
> 14/11/08 08:21:11 ERROR network.ConnectionManager: Corresponding
> SendingConnection to
> ConnectionManagerId(ip-172-16-1-241.us-west-2.compute.internal,59509) not
> found
> 14/11/08 08:21:11 INFO cluster.YarnClientSchedulerBackend: Executor 10
> disconnected, so removing it
> 14/11/08 08:21:11 ERROR cluster.YarnClientClusterScheduler: Lost executor
> 10 on ip-172-16-1-241.us-west-2.compute.internal: remote Akka client
> disassociated
> 14/11/08 08:21:11 INFO scheduler.TaskSetManager: Re-queueing tasks for 10
> from TaskSet 0.0
> 14/11/08 08:21:11 WARN scheduler.TaskSetManager: Lost task 28.0 in stage
> 0.0 (TID 28, ip-172-16-1-241.us-west-2.compute.internal):
> ExecutorLostFailure (executor lost)
> 14/11/08 08:21:11 INFO scheduler.DAGScheduler: Executor lost: 10 (epoch 0)
> 14/11/08 08:21:11 INFO storage.BlockManagerMasterActor: Trying to remove
> executor 10 from BlockManagerMaster.
> 14/11/08 08:21:11 INFO storage.BlockManagerMaster: Removed 10 successfully
> in removeExecutor
> 14/11/08 08:21:20 INFO network.ConnectionManager: Removing
> ReceivingConnection to
> ConnectionManagerId(ip-172-16-1-194.us-west-2.compute.internal,45823)
> 14/11/08 08:21:20 INFO network.ConnectionManager: Removing
> SendingConnection to
> ConnectionManagerId(ip-172-16-1-194.us-west-2.compute.internal,45823)
> 14/11/08 08:21:20 INFO network.ConnectionManager: Removing
> SendingConnection to
> ConnectionManagerId(ip-172-16-1-194.us-west-2.compute.internal,45823)
> 14/11/08 08:21:20 INFO cluster.YarnClientSchedulerBackend: Executor 5
> disconnected, so removing it
> 14/11/08 08:21:20 ERROR cluster.YarnClientClusterScheduler: Lost executor
> 5 on ip-172-16-1-194.us-west-2.compute.internal: remote Akka client
> disassociated
> 14/11/08 08:21:20 INFO scheduler.TaskSetManager: Re-queueing tasks for 5
> from TaskSet 0.0
> 14/11/08 08:21:20 WARN scheduler.TaskSetManager: Lost task 21.0 in stage
> 0.0 (TID 21, ip-172-16-1-194.us-west-2.compute.internal):
> ExecutorLostFailure (executor lost)
> 14/11/08 08:21:20 INFO scheduler.DAGScheduler: Executor lost: 5 (epoch 1)
> 14/11/08 08:21:20 INFO network.ConnectionManager: key already cancelled ?
> sun.nio.ch.SelectionKeyImpl@3bb633cd
> java.nio.channels.CancelledKeyException
> at
> sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
> at
> sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77)
> at
> org.apache.spark.network.ConnectionManager.run(ConnectionManager.scala:289)
> at
> org.apache.spark.network.ConnectionManager$$anon$4.run(ConnectionManager.scala:139)
> 14/11/08 08:21:20 INFO storage.BlockManagerMasterActor: Trying to remove
> executor 5 from BlockManagerMaster.
> 14/11/08 08:21:20 INFO storage.BlockManagerMaster: Removed 5 successfully
> in removeExecutor
> 14/11/08 08:21:21 INFO network.ConnectionManager: Removing
> ReceivingConnection to
> ConnectionManagerId(ip-172-16-1-92.us-west-2.compute.internal,50928)
> 14/11/08 08:21:21 INFO network.ConnectionManager: Removing
> SendingConnection to
> ConnectionManagerId(ip-172-16-1-92.us-west-2.compute.internal,50928)
> 14/11/08 08:21:21 INFO network.ConnectionManager: Removing
> SendingConnection to
> ConnectionManagerId(ip-172-16-1-92.us-west-2.compute.internal,50928)
> 14/11/08 08:21:21 INFO cluster.YarnClientSchedulerBackend: Executor 27
> disconnected, so removing it
> 14/11/08 08:21:21 ERROR cluster.YarnClientClusterScheduler: Lost executor
> 27 on ip-172-16-1-92.us-west-2.compute.internal: remote Akka client
> disassociated
> 14/11/08 08:21:21 INFO scheduler.TaskSetManager: Re-queueing tasks for 27
> from TaskSet 0.0
> 14/11/08 08:21:21 WARN scheduler.TaskSetManager: Lost task 27.0 in stage
> 0.0 (TID 27, ip-172-16-1-92.us-west-2.compute.internal):
> ExecutorLostFailure (executor lost)
> 14/11/08 08:21:21 INFO scheduler.DAGScheduler: Executor lost: 27 (epoch 2)
> 14/11/08 08:21:21 INFO storage.BlockManagerMasterActor: Trying to remove
> executor 27 from BlockManagerMaster.
> 14/11/08 08:21:21 INFO storage.BlockManagerMaster: Removed 27 successfully
> in removeExecutor
> 14/11/08 08:21:21 INFO network.ConnectionManager: Removing
> ReceivingConnection to
> ConnectionManagerId(ip-172-16-1-152.us-west-2.compute.internal,41091)
> 14/11/08 08:21:21 INFO network.ConnectionManager: Removing
> SendingConnection to
> ConnectionManagerId(ip-172-16-1-152.us-west-2.compute.internal,41091)
> 14/11/08 08:21:21 INFO network.ConnectionManager: Removing
> SendingConnection to
> ConnectionManagerId(ip-172-16-1-152.us-west-2.compute.internal,41091)
> 14/11/08 08:21:21 INFO cluster.YarnClientSchedulerBackend: Executor 20
> disconnected, so removing it
> 14/11/08 08:21:21 ERROR cluster.YarnClientClusterScheduler: Lost executor
> 20 on ip-172-16-1-152.us-west-2.compute.internal: remote Akka client
> disassociated
> 14/11/08 08:21:21 INFO scheduler.TaskSetManager: Re-queueing tasks for 20
> from TaskSet 0.0
> 14/11/08 08:21:21 WARN scheduler.TaskSetManager: Lost task 29.0 in stage
> 0.0 (TID 29, ip-172-16-1-152.us-west-2.compute.internal):
> ExecutorLostFailure (executor lost)
> 14/11/08 08:21:21 INFO scheduler.DAGScheduler: Executor lost: 20 (epoch 3)
> 14/11/08 08:21:21 INFO storage.BlockManagerMasterActor: Trying to remove
> executor 20 from BlockManagerMaster.
> 14/11/08 08:21:21 INFO storage.BlockManagerMaster: Removed 20 successfully
> in removeExecutor
> 14/11/08 08:21:26 INFO network.ConnectionManager: Removing
> ReceivingConnection to
> ConnectionManagerId(ip-172-16-1-23.us-west-2.compute.internal,51269)
> 14/11/08 08:21:26 INFO network.ConnectionManager: Removing
> SendingConnection to
> ConnectionManagerId(ip-172-16-1-23.us-west-2.compute.internal,51269)
> 14/11/08 08:21:26 INFO network.ConnectionManager: Removing
> SendingConnection to
> ConnectionManagerId(ip-172-16-1-23.us-west-2.compute.internal,51269)
> 14/11/08 08:21:26 INFO cluster.YarnClientSchedulerBackend: Executor 6
> disconnected, so removing it
> 14/11/08 08:21:26 ERROR cluster.YarnClientClusterScheduler: Lost executor
> 6 on ip-172-16-1-23.us-west-2.compute.internal: remote Akka client
> disassociated
> 14/11/08 08:21:26 INFO scheduler.TaskSetManager: Re-queueing tasks for 6
> from TaskSet 0.0
> 14/11/08 08:21:26 WARN scheduler.TaskSetManager: Lost task 24.0 in stage
> 0.0 (TID 24, ip-172-16-1-23.us-west-2.compute.internal):
> ExecutorLostFailure (executor lost)
> 14/11/08 08:21:26 INFO scheduler.DAGScheduler: Executor lost: 6 (epoch 4)
> 14/11/08 08:21:26 INFO storage.BlockManagerMasterActor: Trying to remove
> executor 6 from BlockManagerMaster.
> 14/11/08 08:21:26 INFO storage.BlockManagerMaster: Removed 6 successfully
> in removeExecutor
> 14/11/08 08:21:26 INFO network.ConnectionManager: Removing
> SendingConnection to
> ConnectionManagerId(ip-172-16-1-90.us-west-2.compute.internal,46792)
> 14/11/08 08:21:26 INFO network.ConnectionManager: Removing
> ReceivingConnection to
> ConnectionManagerId(ip-172-16-1-90.us-west-2.compute.internal,46792)
> 14/11/08 08:21:26 ERROR network.ConnectionManager: Corresponding
> SendingConnection to
> ConnectionManagerId(ip-172-16-1-90.us-west-2.compute.internal,46792) not
> found
> 14/11/08 08:21:26 INFO cluster.YarnClientSchedulerBackend: Executor 21
> disconnected, so removing it
> 14/11/08 08:21:26 ERROR cluster.YarnClientClusterScheduler: Lost executor
> 21 on ip-172-16-1-90.us-west-2.compute.internal: remote Akka client
> disassociated
> 14/11/08 08:21:26 INFO scheduler.TaskSetManager: Re-queueing tasks for 21
> from TaskSet 0.0
> 14/11/08 08:21:26 WARN scheduler.TaskSetManager: Lost task 25.0 in stage
> 0.0 (TID 25, ip-172-16-1-90.us-west-2.compute.internal):
> ExecutorLostFailure (executor lost)
> 14/11/08 08:21:26 INFO scheduler.DAGScheduler: Executor lost: 21 (epoch 5)
> 14/11/08 08:21:26 INFO storage.BlockManagerMasterActor: Trying to remove
> executor 21 from BlockManagerMaster.
> 14/11/08 08:21:26 INFO storage.BlockManagerMaster: Removed 21 successfully
> in removeExecutor
> 14/11/08 08:21:29 INFO cluster.YarnClientSchedulerBackend: Executor 18
> disconnected, so removing it
> 14/11/08 08:21:29 INFO network.ConnectionManager: Removing
> ReceivingConnection to
> ConnectionManagerId(ip-172-16-1-222.us-west-2.compute.internal,43883)
> 14/11/08 08:21:29 INFO network.ConnectionManager: Removing
> SendingConnection to
> ConnectionManagerId(ip-172-16-1-222.us-west-2.compute.internal,43883)
> 14/11/08 08:21:29 INFO network.ConnectionManager: Removing
> SendingConnection to
> ConnectionManagerId(ip-172-16-1-222.us-west-2.compute.internal,43883)
> 14/11/08 08:21:29 ERROR cluster.YarnClientClusterScheduler: Lost executor
> 18 on ip-172-16-1-222.us-west-2.compute.internal: remote Akka client
> disassociated
> 14/11/08 08:21:29 INFO scheduler.TaskSetManager: Re-queueing tasks for 18
> from TaskSet 0.0
> 14/11/08 08:21:29 WARN scheduler.TaskSetManager: Lost task 26.0 in stage
> 0.0 (TID 26, ip-172-16-1-222.us-west-2.compute.internal):
> ExecutorLostFailure (executor lost)
> 14/11/08 08:21:29 INFO scheduler.DAGScheduler: Executor lost: 18 (epoch 6)
> 14/11/08 08:21:29 INFO storage.BlockManagerMasterActor: Trying to remove
> executor 18 from BlockManagerMaster.
> 14/11/08 08:21:29 INFO storage.BlockManagerMaster: Removed 18 successfully
> in removeExecutor
> 14/11/08 08:21:30 INFO cluster.YarnClientSchedulerBackend: Registered
> executor:
> Actor[akka.tcp://sparkExecutor@ip-172-16-1-194.us-west-2.compute.internal:50858/user/Executor#935992941]
> with ID 31
> 14/11/08 08:21:30 INFO scheduler.TaskSetManager: Starting task 26.1 in
> stage 0.0 (TID 30, ip-172-16-1-194.us-west-2.compute.internal,
> PROCESS_LOCAL, 1122 bytes)
> 14/11/08 08:21:30 INFO storage.BlockManagerMasterActor: Registering block
> manager ip-172-16-1-194.us-west-2.compute.internal:44263 with 776.3 MB RAM
> 14/11/08 08:21:31 INFO storage.BlockManagerInfo: Added broadcast_0_piece0
> in memory on ip-172-16-1-194.us-west-2.compute.internal:44263 (size: 596.9
> KB, free: 775.7 MB)
> 14/11/08 08:21:33 INFO network.ConnectionManager: Removing
> SendingConnection to
> ConnectionManagerId(ip-172-16-1-222.us-west-2.compute.internal,40102)
> 14/11/08 08:21:33 INFO network.ConnectionManager: Removing
> ReceivingConnection to
> ConnectionManagerId(ip-172-16-1-222.us-west-2.compute.internal,40102)
> 14/11/08 08:21:33 ERROR network.ConnectionManager: Corresponding
> SendingConnection to
> ConnectionManagerId(ip-172-16-1-222.us-west-2.compute.internal,40102) not
> found
> 14/11/08 08:21:33 INFO cluster.YarnClientSchedulerBackend: Executor 26
> disconnected, so removing it
> 14/11/08 08:21:33 ERROR cluster.YarnClientClusterScheduler: Lost executor
> 26 on ip-172-16-1-222.us-west-2.compute.internal: remote Akka client
> disassociated
> 14/11/08 08:21:33 INFO scheduler.TaskSetManager: Re-queueing tasks for 26
> from TaskSet 0.0
> 14/11/08 08:21:33 WARN scheduler.TaskSetManager: Lost task 23.0 in stage
> 0.0 (TID 23, ip-172-16-1-222.us-west-2.compute.internal):
> ExecutorLostFailure (executor lost)
> 14/11/08 08:21:33 INFO scheduler.DAGScheduler: Executor lost: 26 (epoch 7)
> 14/11/08 08:21:33 INFO storage.BlockManagerMasterActor: Trying to remove
> executor 26 from BlockManagerMaster.
> 14/11/08 08:21:33 INFO storage.BlockManagerMaster: Removed 26 successfully
> in removeExecutor
> 14/11/08 08:21:36 INFO network.ConnectionManager: Removing
> ReceivingConnection to
> ConnectionManagerId(ip-172-16-1-241.us-west-2.compute.internal,43310)
> 14/11/08 08:21:36 INFO network.ConnectionManager: Removing
> SendingConnection to
> ConnectionManagerId(ip-172-16-1-241.us-west-2.compute.internal,43310)
> 14/11/08 08:21:36 INFO cluster.YarnClientSchedulerBackend: Executor 1
> disconnected, so removing it
> 14/11/08 08:21:36 ERROR cluster.YarnClientClusterScheduler: Lost executor
> 1 on ip-172-16-1-241.us-west-2.compute.internal: remote Akka client
> disassociated
> 14/11/08 08:21:36 INFO scheduler.TaskSetManager: Re-queueing tasks for 1
> from TaskSet 0.0
> 14/11/08 08:21:36 WARN scheduler.TaskSetManager: Lost task 22.0 in stage
> 0.0 (TID 22, ip-172-16-1-241.us-west-2.compute.internal):
> ExecutorLostFailure (executor lost)
> 14/11/08 08:21:36 ERROR network.SendingConnection: Exception while reading
> SendingConnection to
> ConnectionManagerId(ip-172-16-1-241.us-west-2.compute.internal,43310)
> java.nio.channels.ClosedChannelException
> at
> sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
> at
> org.apache.spark.network.SendingConnection.read(Connection.scala:390)
> at
> org.apache.spark.network.ConnectionManager$$anon$7.run(ConnectionManager.scala:199)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> 14/11/08 08:21:36 INFO scheduler.DAGScheduler: Executor lost: 1 (epoch 8)
> 14/11/08 08:21:36 INFO storage.BlockManagerMasterActor: Trying to remove
> executor 1 from BlockManagerMaster.
> 14/11/08 08:21:36 INFO storage.BlockManagerMaster: Removed 1 successfully
> in removeExecutor
> 14/11/08 08:21:36 INFO network.ConnectionManager: Handling connection
> error on connection to
> ConnectionManagerId(ip-172-16-1-241.us-west-2.compute.internal,43310)
> 14/11/08 08:21:36 INFO network.ConnectionManager: Removing
> SendingConnection to
> ConnectionManagerId(ip-172-16-1-241.us-west-2.compute.internal,43310)
> 14/11/08 08:21:36 INFO network.ConnectionManager: Removing
> SendingConnection to
> ConnectionManagerId(ip-172-16-1-241.us-west-2.compute.internal,43310)
> 14/11/08 08:21:40 INFO cluster.YarnClientSchedulerBackend: Registered
> executor:
> Actor[akka.tcp://sparkExecutor@ip-172-16-1-194.us-west-2.compute.internal:58099/user/Executor#-112835629]
> with ID 34
> 14/11/08 08:21:40 INFO scheduler.TaskSetManager: Starting task 22.1 in
> stage 0.0 (TID 31, ip-172-16-1-194.us-west-2.compute.internal,
> PROCESS_LOCAL, 1122 bytes)
> 14/11/08 08:21:41 INFO storage.BlockManagerMasterActor: Registering block
> manager ip-172-16-1-194.us-west-2.compute.internal:41093 with 776.3 MB RAM
> 14/11/08 08:21:41 INFO cluster.YarnClientSchedulerBackend: Registered
> executor:
> Actor[akka.tcp://sparkExecutor@ip-172-16-1-228.us-west-2.compute.internal:36136/user/Executor#318736262]
> with ID 32
> 14/11/08 08:21:41 INFO scheduler.TaskSetManager: Starting task 23.1 in
> stage 0.0 (TID 32, ip-172-16-1-228.us-west-2.compute.internal,
> PROCESS_LOCAL, 1122 bytes)
> 14/11/08 08:21:41 INFO cluster.YarnClientSchedulerBackend: Registered
> executor:
> Actor[akka.tcp://sparkExecutor@ip-172-16-1-90.us-west-2.compute.internal:33130/user/Executor#1744030597]
> with ID 33
> 14/11/08 08:21:41 INFO scheduler.TaskSetManager: Starting task 25.1 in
> stage 0.0 (TID 33, ip-172-16-1-90.us-west-2.compute.internal,
> PROCESS_LOCAL, 1122 bytes)
> 14/11/08 08:21:41 INFO cluster.YarnClientSchedulerBackend: Registered
> executor:
> Actor[akka.tcp://sparkExecutor@ip-172-16-1-92.us-west-2.compute.internal:55503/user/Executor#574084779]
> with ID 35
> 14/11/08 08:21:41 INFO scheduler.TaskSetManager: Starting task 24.1 in
> stage 0.0 (TID 34, ip-172-16-1-92.us-west-2.compute.internal,
> PROCESS_LOCAL, 1122 bytes)
> 14/11/08 08:21:42 INFO storage.BlockManagerMasterActor: Registering block
> manager ip-172-16-1-228.us-west-2.compute.internal:40128 with 776.3 MB RAM
> 14/11/08 08:21:42 INFO storage.BlockManagerMasterActor: Registering block
> manager ip-172-16-1-90.us-west-2.compute.internal:32839 with 776.3 MB RAM
> 14/11/08 08:21:42 INFO storage.BlockManagerMasterActor: Registering block
> manager ip-172-16-1-92.us-west-2.compute.internal:58081 with 776.3 MB RAM
> 14/11/08 08:21:42 INFO storage.BlockManagerInfo: Added broadcast_0_piece0
> in memory on ip-172-16-1-194.us-west-2.compute.internal:41093 (size: 596.9
> KB, free: 775.7 MB)
> 14/11/08 08:21:43 INFO storage.BlockManagerInfo: Added broadcast_0_piece0
> in memory on ip-172-16-1-228.us-west-2.compute.internal:40128 (size: 596.9
> KB, free: 775.7 MB)
> 14/11/08 08:21:43 INFO storage.BlockManagerInfo: Added broadcast_0_piece0
> in memory on ip-172-16-1-92.us-west-2.compute.internal:58081 (size: 596.9
> KB, free: 775.7 MB)
> 14/11/08 08:21:43 INFO storage.BlockManagerInfo: Added broadcast_0_piece0
> in memory on ip-172-16-1-90.us-west-2.compute.internal:32839 (size: 596.9
> KB, free: 775.7 MB)
> 14/11/08 08:21:43 INFO cluster.YarnClientSchedulerBackend: Registered
> executor:
> Actor[akka.tcp://sparkExecutor@ip-172-16-1-152.us-west-2.compute.internal:34268/user/Executor#-937582169]
> with ID 36
> 14/11/08 08:21:43 INFO scheduler.TaskSetManager: Starting task 29.1 in
> stage 0.0 (TID 35, ip-172-16-1-152.us-west-2.compute.internal,
> PROCESS_LOCAL, 1122 bytes)
> 14/11/08 08:21:44 INFO storage.BlockManagerMasterActor: Registering block
> manager ip-172-16-1-152.us-west-2.compute.internal:52550 with 776.3 MB RAM
> 14/11/08 08:21:45 INFO storage.BlockManagerInfo: Added broadcast_0_piece0
> in memory on ip-172-16-1-152.us-west-2.compute.internal:52550 (size: 596.9
> KB, free: 775.7 MB)
> 14/11/08 08:21:46 INFO cluster.YarnClientSchedulerBackend: Registered
> executor:
> Actor[akka.tcp://sparkExecutor@ip-172-16-1-90.us-west-2.compute.internal:34555/user/Executor#-94727554]
> with ID 37
> 14/11/08 08:21:46 INFO scheduler.TaskSetManager: Starting task 27.1 in
> stage 0.0 (TID 36, ip-172-16-1-90.us-west-2.compute.internal,
> PROCESS_LOCAL, 1122 bytes)
> 14/11/08 08:21:46 INFO cluster.YarnClientSchedulerBackend: Registered
> executor:
> Actor[akka.tcp://sparkExecutor@ip-172-16-1-228.us-west-2.compute.internal:34471/user/Executor#1412546630]
> with ID 38
> 14/11/08 08:21:46 INFO scheduler.TaskSetManager: Starting task 21.1 in
> stage 0.0 (TID 37, ip-172-16-1-228.us-west-2.compute.internal,
> PROCESS_LOCAL, 1122 bytes)
> 14/11/08 08:21:47 INFO storage.BlockManagerMasterActor: Registering block
> manager ip-172-16-1-90.us-west-2.compute.internal:46194 with 776.3 MB RAM
> 14/11/08 08:21:47 INFO storage.BlockManagerMasterActor: Registering block
> manager ip-172-16-1-228.us-west-2.compute.internal:42275 with 776.3 MB RAM
> 14/11/08 08:21:48 INFO storage.BlockManagerInfo: Added broadcast_0_piece0
> in memory on ip-172-16-1-90.us-west-2.compute.internal:46194 (size: 596.9
> KB, free: 775.7 MB)
> 14/11/08 08:21:48 INFO storage.BlockManagerInfo: Added broadcast_0_piece0
> in memory on ip-172-16-1-228.us-west-2.compute.internal:42275 (size: 596.9
> KB, free: 775.7 MB)
> 14/11/08 08:21:50 INFO cluster.YarnClientSchedulerBackend: Registered
> executor:
> Actor[akka.tcp://sparkExecutor@ip-172-16-1-23.us-west-2.compute.internal:37122/user/Executor#1404320204]
> with ID 39
> 14/11/08 08:21:51 INFO scheduler.TaskSetManager: Starting task 28.1 in
> stage 0.0 (TID 38, ip-172-16-1-23.us-west-2.compute.internal,
> PROCESS_LOCAL, 1122 bytes)
> 14/11/08 08:21:51 INFO storage.BlockManagerMasterActor: Registering block
> manager ip-172-16-1-23.us-west-2.compute.internal:33106 with 776.3 MB RAM
> 14/11/08 08:21:52 INFO storage.BlockManagerInfo: Added broadcast_0_piece0
> in memory on ip-172-16-1-23.us-west-2.compute.internal:33106 (size: 596.9
> KB, free: 775.7 MB)
> 14/11/08 08:22:36 INFO cluster.YarnClientSchedulerBackend: Executor 39
> disconnected, so removing it
> 14/11/08 08:22:36 ERROR cluster.YarnClientClusterScheduler: Lost executor
> 39 on ip-172-16-1-23.us-west-2.compute.internal: remote Akka client
> disassociated
> 14/11/08 08:22:36 INFO scheduler.TaskSetManager: Re-queueing tasks for 39
> from TaskSet 0.0
> 14/11/08 08:22:36 WARN scheduler.TaskSetManager: Lost task 28.1 in stage
> 0.0 (TID 38, ip-172-16-1-23.us-west-2.compute.internal):
> ExecutorLostFailure (executor lost)
> 14/11/08 08:22:36 INFO scheduler.DAGScheduler: Executor lost: 39 (epoch 9)
> 14/11/08 08:22:36 INFO storage.BlockManagerMasterActor: Trying to remove
> executor 39 from BlockManagerMaster.
> 14/11/08 08:22:36 INFO storage.BlockManagerMaster: Removed 39 successfully
> in removeExecutor
> 14/11/08 08:22:57 INFO cluster.YarnClientSchedulerBackend: Executor 36
> disconnected, so removing it
> 14/11/08 08:22:57 ERROR cluster.YarnClientClusterScheduler: Lost executor
> 36 on ip-172-16-1-152.us-west-2.compute.internal: remote Akka client
> disassociated
> 14/11/08 08:22:57 INFO scheduler.TaskSetManager: Re-queueing tasks for 36
> from TaskSet 0.0
> 14/11/08 08:22:57 WARN scheduler.TaskSetManager: Lost task 29.1 in stage
> 0.0 (TID 35, ip-172-16-1-152.us-west-2.compute.internal):
> ExecutorLostFailure (executor lost)
> 14/11/08 08:22:57 INFO scheduler.DAGScheduler: Executor lost: 36 (epoch 10)
> 14/11/08 08:22:57 INFO storage.BlockManagerMasterActor: Trying to remove
> executor 36 from BlockManagerMaster.
> 14/11/08 08:22:57 INFO storage.BlockManagerMaster: Removed 36 successfully
> in removeExecutor
> 14/11/08 08:23:00 INFO cluster.YarnClientSchedulerBackend: Registered
> executor:
> Actor[akka.tcp://sparkExecutor@ip-172-16-1-90.us-west-2.compute.internal:48033/user/Executor#-1088273404]
> with ID 40
> 14/11/08 08:23:00 INFO scheduler.TaskSetManager: Starting task 29.2 in
> stage 0.0 (TID 39, ip-172-16-1-90.us-west-2.compute.internal,
> PROCESS_LOCAL, 1122 bytes)
> 14/11/08 08:23:01 INFO storage.BlockManagerMasterActor: Registering block
> manager ip-172-16-1-90.us-west-2.compute.internal:39067 with 776.3 MB RAM
> 14/11/08 08:23:03 INFO storage.BlockManagerInfo: Added broadcast_0_piece0
> in memory on ip-172-16-1-90.us-west-2.compute.internal:39067 (size: 596.9
> KB, free: 775.7 MB)
> 14/11/08 08:23:15 INFO cluster.YarnClientSchedulerBackend: Registered
> executor:
> Actor[akka.tcp://sparkExecutor@ip-172-16-1-23.us-west-2.compute.internal:48860/user/Executor#-369895446]
> with ID 41
> 14/11/08 08:23:15 INFO scheduler.TaskSetManager: Starting task 28.2 in
> stage 0.0 (TID 40, ip-172-16-1-23.us-west-2.compute.internal,
> PROCESS_LOCAL, 1122 bytes)
> 14/11/08 08:23:16 INFO storage.BlockManagerMasterActor: Registering block
> manager ip-172-16-1-23.us-west-2.compute.internal:38093 with 776.3 MB RAM
> 14/11/08 08:23:17 INFO storage.BlockManagerInfo: Added broadcast_0_piece0
> in memory on ip-172-16-1-23.us-west-2.compute.internal:38093 (size: 596.9
> KB, free: 775.7 MB)
> 14/11/08 08:23:32 INFO cluster.YarnClientSchedulerBackend: Executor 34
> disconnected, so removing it
> 14/11/08 08:23:32 ERROR cluster.YarnClientClusterScheduler: Lost executor
> 34 on ip-172-16-1-194.us-west-2.compute.internal: remote Akka client
> disassociated
> 14/11/08 08:23:32 INFO scheduler.TaskSetManager: Re-queueing tasks for 34
> from TaskSet 0.0
> 14/11/08 08:23:32 WARN scheduler.TaskSetManager: Lost task 22.1 in stage
> 0.0 (TID 31, ip-172-16-1-194.us-west-2.compute.internal):
> ExecutorLostFailure (executor lost)
> 14/11/08 08:23:32 INFO scheduler.DAGScheduler: Executor lost: 34 (epoch 11)
> 14/11/08 08:23:32 INFO storage.BlockManagerMasterActor: Trying to remove
> executor 34 from BlockManagerMaster.
> 14/11/08 08:23:32 INFO storage.BlockManagerMaster: Removed 34 successfully
> in removeExecutor
> 14/11/08 08:23:53 INFO cluster.YarnClientSchedulerBackend: Executor 41
> disconnected, so removing it
> 14/11/08 08:23:53 ERROR cluster.YarnClientClusterScheduler: Lost executor
> 41 on ip-172-16-1-23.us-west-2.compute.internal: remote Akka client
> disassociated
> 14/11/08 08:23:53 INFO scheduler.TaskSetManager: Re-queueing tasks for 41
> from TaskSet 0.0
> 14/11/08 08:23:53 WARN scheduler.TaskSetManager: Lost task 28.2 in stage
> 0.0 (TID 40, ip-172-16-1-23.us-west-2.compute.internal):
> ExecutorLostFailure (executor lost)
> 14/11/08 08:23:53 INFO scheduler.DAGScheduler: Executor lost: 41 (epoch 12)
> 14/11/08 08:23:53 INFO storage.BlockManagerMasterActor: Trying to remove
> executor 41 from BlockManagerMaster.
> 14/11/08 08:23:53 INFO storage.BlockManagerMaster: Removed 41 successfully
> in removeExecutor
> 14/11/08 08:23:57 INFO cluster.YarnClientSchedulerBackend: Registered
> executor:
> Actor[akka.tcp://sparkExecutor@ip-172-16-1-90.us-west-2.compute.internal:58017/user/Executor#2094507560]
> with ID 42
> 14/11/08 08:23:57 INFO scheduler.TaskSetManager: Starting task 28.3 in
> stage 0.0 (TID 41, ip-172-16-1-90.us-west-2.compute.internal,
> PROCESS_LOCAL, 1122 bytes)
> 14/11/08 08:23:58 INFO storage.BlockManagerMasterActor: Registering block
> manager ip-172-16-1-90.us-west-2.compute.internal:41182 with 776.3 MB RAM
> 14/11/08 08:24:00 INFO storage.BlockManagerInfo: Added broadcast_0_piece0
> in memory on ip-172-16-1-90.us-west-2.compute.internal:41182 (size: 596.9
> KB, free: 775.7 MB)
> 14/11/08 08:24:04 INFO cluster.YarnClientSchedulerBackend: Executor 35
> disconnected, so removing it
> 14/11/08 08:24:04 ERROR cluster.YarnClientClusterScheduler: Lost executor
> 35 on ip-172-16-1-92.us-west-2.compute.internal: remote Akka client
> disassociated
> 14/11/08 08:24:04 INFO scheduler.TaskSetManager: Re-queueing tasks for 35
> from TaskSet 0.0
> 14/11/08 08:24:04 WARN scheduler.TaskSetManager: Lost task 24.1 in stage
> 0.0 (TID 34, ip-172-16-1-92.us-west-2.compute.internal):
> ExecutorLostFailure (executor lost)
> 14/11/08 08:24:04 INFO scheduler.DAGScheduler: Executor lost: 35 (epoch 13)
> 14/11/08 08:24:04 INFO storage.BlockManagerMasterActor: Trying to remove
> executor 35 from BlockManagerMaster.
> 14/11/08 08:24:04 INFO storage.BlockManagerMaster: Removed 35 successfully
> in removeExecutor
> 14/11/08 08:24:17 INFO cluster.YarnClientSchedulerBackend: Registered
> executor:
> Actor[akka.tcp://sparkExecutor@ip-172-16-1-90.us-west-2.compute.internal:36395/user/Executor#-1907878650]
> with ID 43
> 14/11/08 08:24:17 INFO scheduler.TaskSetManager: Starting task 24.2 in
> stage 0.0 (TID 42, ip-172-16-1-90.us-west-2.compute.internal,
> PROCESS_LOCAL, 1122 bytes)
> 14/11/08 08:24:18 INFO storage.BlockManagerMasterActor: Registering block
> manager ip-172-16-1-90.us-west-2.compute.internal:46948 with 776.3 MB RAM
> 14/11/08 08:24:20 INFO storage.BlockManagerInfo: Added broadcast_0_piece0
> in memory on ip-172-16-1-90.us-west-2.compute.internal:46948 (size: 596.9
> KB, free: 775.7 MB)
> 14/11/08 08:24:21 INFO cluster.YarnClientSchedulerBackend: Executor 40
> disconnected, so removing it
> 14/11/08 08:24:21 ERROR cluster.YarnClientClusterScheduler: Lost executor
> 40 on ip-172-16-1-90.us-west-2.compute.internal: remote Akka client
> disassociated
> 14/11/08 08:24:21 INFO scheduler.TaskSetManager: Re-queueing tasks for 40
> from TaskSet 0.0
> 14/11/08 08:24:21 WARN scheduler.TaskSetManager: Lost task 29.2 in stage
> 0.0 (TID 39, ip-172-16-1-90.us-west-2.compute.internal):
> ExecutorLostFailure (executor lost)
> 14/11/08 08:24:21 INFO scheduler.DAGScheduler: Executor lost: 40 (epoch 14)
> 14/11/08 08:24:21 INFO storage.BlockManagerMasterActor: Trying to remove
> executor 40 from BlockManagerMaster.
> 14/11/08 08:24:21 INFO storage.BlockManagerMaster: Removed 40 successfully
> in removeExecutor
> 14/11/08 08:24:31 INFO cluster.YarnClientSchedulerBackend: Registered
> executor:
> Actor[akka.tcp://sparkExecutor@ip-172-16-1-90.us-west-2.compute.internal:34467/user/Executor#-1100688472]
> with ID 44
> 14/11/08 08:24:31 INFO scheduler.TaskSetManager: Starting task 29.3 in
> stage 0.0 (TID 43, ip-172-16-1-90.us-west-2.compute.internal,
> PROCESS_LOCAL, 1122 bytes)
> 14/11/08 08:24:32 INFO storage.BlockManagerMasterActor: Registering block
> manager ip-172-16-1-90.us-west-2.compute.internal:40126 with 776.3 MB RAM
> 14/11/08 08:24:34 INFO storage.BlockManagerInfo: Added broadcast_0_piece0
> in memory on ip-172-16-1-90.us-west-2.compute.internal:40126 (size: 596.9
> KB, free: 775.7 MB)
> 14/11/08 08:24:48 INFO cluster.YarnClientSchedulerBackend: Registered
> executor:
> Actor[akka.tcp://sparkExecutor@ip-172-16-1-90.us-west-2.compute.internal:53257/user/Executor#-745380917]
> with ID 45
> 14/11/08 08:24:48 INFO scheduler.TaskSetManager: Starting task 22.2 in
> stage 0.0 (TID 44, ip-172-16-1-90.us-west-2.compute.internal,
> PROCESS_LOCAL, 1122 bytes)
> 14/11/08 08:24:49 INFO storage.BlockManagerMasterActor: Registering block
> manager ip-172-16-1-90.us-west-2.compute.internal:46252 with 776.3 MB RAM
> 14/11/08 08:24:52 INFO storage.BlockManagerInfo: Added broadcast_0_piece0
> in memory on ip-172-16-1-90.us-west-2.compute.internal:46252 (size: 596.9
> KB, free: 775.7 MB)
> 14/11/08 08:25:16 INFO cluster.YarnClientSchedulerBackend: Executor 38
> disconnected, so removing it
> 14/11/08 08:25:16 ERROR cluster.YarnClientClusterScheduler: Lost executor
> 38 on ip-172-16-1-228.us-west-2.compute.internal: remote Akka client
> disassociated
> 14/11/08 08:25:16 INFO scheduler.TaskSetManager: Re-queueing tasks for 38
> from TaskSet 0.0
> 14/11/08 08:25:16 WARN scheduler.TaskSetManager: Lost task 21.1 in stage
> 0.0 (TID 37, ip-172-16-1-228.us-west-2.compute.internal):
> ExecutorLostFailure (executor lost)
> 14/11/08 08:25:16 INFO scheduler.DAGScheduler: Executor lost: 38 (epoch 15)
> 14/11/08 08:25:16 INFO storage.BlockManagerMasterActor: Trying to remove
> executor 38 from BlockManagerMaster.
> 14/11/08 08:25:16 INFO storage.BlockManagerMaster: Removed 38 successfully
> in removeExecutor
> 14/11/08 08:25:37 INFO cluster.YarnClientSchedulerBackend: Executor 42
> disconnected, so removing it
> 14/11/08 08:25:37 ERROR cluster.YarnClientClusterScheduler: Lost executor
> 42 on ip-172-16-1-90.us-west-2.compute.internal: remote Akka client
> disassociated
> 14/11/08 08:25:37 INFO scheduler.TaskSetManager: Re-queueing tasks for 42
> from TaskSet 0.0
> 14/11/08 08:25:37 WARN scheduler.TaskSetManager: Lost task 28.3 in stage
> 0.0 (TID 41, ip-172-16-1-90.us-west-2.compute.internal):
> ExecutorLostFailure (executor lost)
> 14/11/08 08:25:37 ERROR scheduler.TaskSetManager: Task 28 in stage 0.0
> failed 4 times; aborting job
> 14/11/08 08:25:37 INFO cluster.YarnClientClusterScheduler: Cancelling
> stage 0
> 14/11/08 08:25:37 INFO cluster.YarnClientClusterScheduler: Stage 0 was
> cancelled
> 14/11/08 08:25:37 INFO scheduler.DAGScheduler: Failed to run
> saveAsTextFile at NativeMethodAccessorImpl.java:-2
> 14/11/08 08:25:37 INFO scheduler.DAGScheduler: Executor lost: 42 (epoch 16)
> 14/11/08 08:25:37 INFO storage.BlockManagerMasterActor: Trying to remove
> executor 42 from BlockManagerMaster.
> 14/11/08 08:25:37 INFO storage.BlockManagerMaster: Removed 42 successfully
> in removeExecutor
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>