You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Muhammad Ahsan <mu...@gmail.com> on 2014/12/11 13:42:09 UTC

ERROR YarnClientClusterScheduler: Lost executor Akka client disassociated

Hi Everyone !

I am trying to execute some simple spark jobs with large data of volume of
almost 2TB. The jobs finish successfully on small dataset on local machine
but when I use the same code for large dataset of size 2TB, I get some
errors of remote Akka client disassociated. 

Please help thanks

I am using the following configurations over YARN cluster

./bin/spark-shell --master yarn-client \
    --num-executors 400 \
    --driver-memory 7g \
    --executor-memory 7g \
    --executor-cores 1 \


Spark Command: /usr/java/jdk1.7.0_45-cloudera/bin/java -cp
::/home/ahsan/spark-deploy/conf:/home/ahsan/spark-deploy/assembly/target/scala-2.10/spark-assembly-1.2.0-SNAPSHOT-hadoop2.3.0-cdh5.1.3.jar:/etc/hadoop/conf.cloudera.yarn:/etc/hadoop/conf.cloudera.yarn
-XX:MaxPermSize=512m -XX:+UseConcMarkSweepGC -Xms7g -Xmx7g
-XX:MaxPermSize=256m -Xms8G -Xmx8G org.apache.spark.deploy.SparkSubmit
--class org.apache.spark.repl.Main --master yarn-client --num-executors 400
--driver-memory 7g --executor-memory 7g --executor-cores 1 spark-shell
========================================

Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 1.2.0-SNAPSHOT
      /_/

Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java
1.7.0_45)
Type in expressions to have them evaluated.
Type :help for more information.
14/12/11 11:12:12 WARN NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
Spark context available as sc.


---------------------------
ERROR AND WARNINGS
---------------------------


14/12/11 11:24:07 WARN TaskSetManager: Lost task 190.0 in stage 0.0 (TID
240, ams05-005.ff.company.com): java.net.ConnectException: Connection timed
out
        sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
       
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735)
       
org.apache.spark.network.nio.SendingConnection.finishConnect(Connection.scala:342)
       
org.apache.spark.network.nio.ConnectionManager$$anon$11.run(ConnectionManager.scala:271)
       
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
       
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        java.lang.Thread.run(Thread.java:744)
14/12/11 11:28:28 ERROR YarnClientClusterScheduler: Lost executor 374 on
ams03-006.ff.company.com: remote Akka client disassociated
14/12/11 11:28:28 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams03-006.ff.company.com:46972] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:28:28 WARN TaskSetManager: Lost task 3149.0 in stage 0.0 (TID
2965, ams03-006.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:28:28 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 374
14/12/11 11:29:09 ERROR YarnClientClusterScheduler: Lost executor 37 on
ams02-010.ff.company.com: remote Akka client disassociated
14/12/11 11:29:09 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams02-010.ff.company.com:45892] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:29:09 ERROR ConnectionManager: Corresponding SendingConnection
to ConnectionManagerId(ams02-010.ff.company.com,39424) not found
14/12/11 11:29:09 WARN TaskSetManager: Lost task 3519.0 in stage 0.0 (TID
3402, ams02-010.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:29:09 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 37
14/12/11 11:29:10 ERROR YarnClientClusterScheduler: Lost executor 100 on
ams06-005.ff.company.com: remote Akka client disassociated
14/12/11 11:29:10 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams06-005.ff.company.com:42345] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:29:10 WARN TaskSetManager: Lost task 3215.0 in stage 0.0 (TID
3433, ams06-005.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:29:10 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 100
14/12/11 11:29:16 ERROR YarnClientClusterScheduler: Lost executor 17 on
ams04-005.ff.company.com: remote Akka client disassociated
14/12/11 11:29:16 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams04-005.ff.company.com:36822] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:29:16 WARN TaskSetManager: Lost task 4518.0 in stage 0.0 (TID
3481, ams04-005.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:29:16 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 17
14/12/11 11:29:34 ERROR YarnClientClusterScheduler: Lost executor 133 on
ams03-016.ff.company.com: remote Akka client disassociated
14/12/11 11:29:34 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams03-016.ff.company.com:46446] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:29:34 WARN TaskSetManager: Lost task 3634.0 in stage 0.0 (TID
3680, ams03-016.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:29:34 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 133
14/12/11 11:29:41 ERROR ConnectionManager: Corresponding SendingConnection
to ConnectionManagerId(ams02-010.ff.company.com,40878) not found
14/12/11 11:29:41 ERROR YarnClientClusterScheduler: Lost executor 241 on
ams02-010.ff.company.com: remote Akka client disassociated
14/12/11 11:29:41 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams02-010.ff.company.com:46617] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:29:41 WARN TaskSetManager: Lost task 3519.1 in stage 0.0 (TID
3783, ams02-010.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:29:41 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 241
14/12/11 11:30:07 ERROR YarnClientClusterScheduler: Lost executor 283 on
ams05-007.ff.company.com: remote Akka client disassociated
14/12/11 11:30:07 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams05-007.ff.company.com:33156] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:30:07 WARN TaskSetManager: Lost task 4283.0 in stage 0.0 (TID
4109, ams05-007.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:30:07 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 283
14/12/11 11:30:14 ERROR YarnClientClusterScheduler: Lost executor 202 on
ams06-005.ff.company.com: remote Akka client disassociated
14/12/11 11:30:14 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams06-005.ff.company.com:46207] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:30:14 ERROR SendingConnection: Exception while reading
SendingConnection to ConnectionManagerId(ams06-005.ff.company.com,40220)
java.nio.channels.ClosedChannelException
	at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
	at
org.apache.spark.network.nio.SendingConnection.read(Connection.scala:419)
	at
org.apache.spark.network.nio.ConnectionManager$$anon$10.run(ConnectionManager.scala:238)
	at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
14/12/11 11:30:14 ERROR ConnectionManager: Exception while sending message.
java.nio.channels.ClosedChannelException
	at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
	at
org.apache.spark.network.nio.SendingConnection.read(Connection.scala:419)
	at
org.apache.spark.network.nio.ConnectionManager$$anon$10.run(ConnectionManager.scala:238)
	at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
14/12/11 11:30:14 ERROR ConnectionManager: no messageStatus for failed
message id: 177
14/12/11 11:30:14 WARN TaskSetManager: Lost task 3709.0 in stage 0.0 (TID
4040, ams06-005.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:30:14 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 202
14/12/11 11:31:15 ERROR ConnectionManager: Corresponding SendingConnection
to ConnectionManagerId(ams07-005.ff.company.com,35510) not found
14/12/11 11:31:15 ERROR YarnClientClusterScheduler: Lost executor 274 on
ams07-005.ff.company.com: remote Akka client disassociated
14/12/11 11:31:15 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams07-005.ff.company.com:47730] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:31:15 WARN TaskSetManager: Lost task 4775.0 in stage 0.0 (TID
4467, ams07-005.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:31:15 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 274
14/12/11 11:31:53 ERROR YarnClientClusterScheduler: Lost executor 376 on
ams07-005.ff.company.com: remote Akka client disassociated
14/12/11 11:31:53 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams07-005.ff.company.com:48502] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:31:53 WARN TaskSetManager: Lost task 4775.1 in stage 0.0 (TID
4733, ams07-005.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:31:53 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 376
14/12/11 11:38:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(284, ams02-004.ff.company.com, 33103) with no recent heart
beats: 53763ms exceeds 45000ms
14/12/11 11:38:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(117, ams03-017.ff.company.com, 47312) with no recent heart
beats: 75361ms exceeds 45000ms
14/12/11 11:38:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(319, ams04-013.ff.company.com, 41162) with no recent heart
beats: 54240ms exceeds 45000ms
14/12/11 11:38:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(214, ams03-012.ff.company.com, 41824) with no recent heart
beats: 73400ms exceeds 45000ms
14/12/11 11:38:28 WARN TaskSetManager: Lost task 4166.0 in stage 0.0 (TID
4436, ams03-011.ff.company.com): java.lang.OutOfMemoryError: GC overhead
limit exceeded
        java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:322)
       
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1133)
       
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
       
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
       
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
       
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
       
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
       
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
       
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
       
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
        java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347)
       
org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:42)
       
org.apache.spark.storage.DiskBlockObjectWriter.write(BlockObjectWriter.scala:195)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4$$anonfun$apply$2.apply(ExternalSorter.scala:744)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4$$anonfun$apply$2.apply(ExternalSorter.scala:743)
        scala.collection.Iterator$class.foreach(Iterator.scala:727)
       
org.apache.spark.util.collection.ExternalSorter$IteratorForPartition.foreach(ExternalSorter.scala:801)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4.apply(ExternalSorter.scala:743)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4.apply(ExternalSorter.scala:739)
        scala.collection.Iterator$class.foreach(Iterator.scala:727)
        scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
       
org.apache.spark.util.collection.ExternalSorter.writePartitionedFile(ExternalSorter.scala:739)
       
org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:70)
       
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)
       
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
        org.apache.spark.scheduler.Task.run(Task.scala:56)
       
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:186)
       
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
       
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        java.lang.Thread.run(Thread.java:744)
14/12/11 11:38:30 ERROR YarnClientClusterScheduler: Lost executor 303 on
ams03-011.ff.company.com: remote Akka client disassociated
14/12/11 11:38:30 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams03-011.ff.company.com:45320] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:38:30 WARN TaskSetManager: Lost task 1409.1 in stage 0.0 (TID
5535, ams03-011.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:38:30 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 303
14/12/11 11:39:00 ERROR YarnClientClusterScheduler: Lost executor 406 on
ams02-011.ff.company.com: remote Akka client disassociated
14/12/11 11:39:00 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams02-011.ff.company.com:49627] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:39:00 WARN TaskSetManager: Lost task 6407.0 in stage 0.0 (TID
5524, ams02-011.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:39:00 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 406
14/12/11 11:39:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(90, ams05-010.ff.company.com, 46830) with no recent heart
beats: 77684ms exceeds 45000ms
14/12/11 11:39:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(264, ams06-004.ff.company.com, 46639) with no recent heart
beats: 62653ms exceeds 45000ms
14/12/11 11:39:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(366, ams06-004.ff.company.com, 46181) with no recent heart
beats: 59485ms exceeds 45000ms
14/12/11 11:39:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(105, ams05-015.ff.company.com, 48759) with no recent heart
beats: 73984ms exceeds 45000ms
14/12/11 11:39:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(25, ams05-018.ff.company.com, 36831) with no recent heart
beats: 65835ms exceeds 45000ms
14/12/11 11:39:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(47, ams07-010.ff.company.com, 46783) with no recent heart
beats: 85222ms exceeds 45000ms
14/12/11 11:39:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(153, ams07-006.ff.company.com, 35692) with no recent heart
beats: 103407ms exceeds 45000ms
14/12/11 11:39:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(269, ams06-013.ff.company.com, 40855) with no recent heart
beats: 63718ms exceeds 45000ms
14/12/11 11:39:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(319, ams04-013.ff.company.com, 41162) with no recent heart
beats: 51806ms exceeds 45000ms
14/12/11 11:39:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(122, ams04-009.ff.company.com, 35108) with no recent heart
beats: 46651ms exceeds 45000ms
14/12/11 11:39:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(213, ams07-012.ff.company.com, 37239) with no recent heart
beats: 63734ms exceeds 45000ms
14/12/11 11:39:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(110, ams02-008.ff.company.com, 34143) with no recent heart
beats: 47385ms exceeds 45000ms
14/12/11 11:39:47 WARN TaskSetManager: Lost task 4546.0 in stage 0.0 (TID
5238, ams02-010.ff.company.com): java.lang.OutOfMemoryError: GC overhead
limit exceeded
       
java.io.ObjectOutputStream$HandleTable.growEntries(ObjectOutputStream.java:2350)
       
java.io.ObjectOutputStream$HandleTable.assign(ObjectOutputStream.java:2275)
        java.io.ObjectOutputStream.writeString(ObjectOutputStream.java:1301)
       
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1171)
       
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
       
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
       
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
       
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
       
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
       
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
       
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
       
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
        java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347)
       
org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:42)
       
org.apache.spark.storage.DiskBlockObjectWriter.write(BlockObjectWriter.scala:195)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4$$anonfun$apply$2.apply(ExternalSorter.scala:744)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4$$anonfun$apply$2.apply(ExternalSorter.scala:743)
        scala.collection.Iterator$class.foreach(Iterator.scala:727)
       
org.apache.spark.util.collection.ExternalSorter$IteratorForPartition.foreach(ExternalSorter.scala:801)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4.apply(ExternalSorter.scala:743)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4.apply(ExternalSorter.scala:739)
        scala.collection.Iterator$class.foreach(Iterator.scala:727)
        scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
       
org.apache.spark.util.collection.ExternalSorter.writePartitionedFile(ExternalSorter.scala:739)
       
org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:70)
       
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)
       
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
        org.apache.spark.scheduler.Task.run(Task.scala:56)
       
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:186)
       
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
       
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        java.lang.Thread.run(Thread.java:744)
14/12/11 11:39:50 ERROR YarnClientClusterScheduler: Lost executor 139 on
ams02-010.ff.company.com: remote Akka client disassociated
14/12/11 11:39:50 WARN TaskSetManager: Lost task 5653.0 in stage 0.0 (TID
5573, ams02-010.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:39:50 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams02-010.ff.company.com:37094] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:39:50 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 139
14/12/11 11:40:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(49, ams05-017.ff.company.com, 47581) with no recent heart
beats: 66780ms exceeds 45000ms
14/12/11 11:40:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(243, ams02-001.ff.company.com, 47519) with no recent heart
beats: 62633ms exceeds 45000ms
14/12/11 11:40:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(228, ams07-011.ff.company.com, 49867) with no recent heart
beats: 77027ms exceeds 45000ms
14/12/11 11:40:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(272, ams03-006.ff.company.com, 39870) with no recent heart
beats: 45215ms exceeds 45000ms
14/12/11 11:40:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(94, ams05-002.ff.company.com, 40006) with no recent heart
beats: 45691ms exceeds 45000ms
14/12/11 11:40:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(22, ams05-014.ff.company.com, 44348) with no recent heart
beats: 46928ms exceeds 45000ms
14/12/11 11:40:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(357, ams07-006.ff.company.com, 42543) with no recent heart
beats: 85413ms exceeds 45000ms
14/12/11 11:40:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(122, ams04-009.ff.company.com, 35108) with no recent heart
beats: 49511ms exceeds 45000ms
14/12/11 11:40:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(227, ams03-003.ff.company.com, 49795) with no recent heart
beats: 76471ms exceeds 45000ms
14/12/11 11:40:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(160, ams04-007.ff.company.com, 33817) with no recent heart
beats: 88614ms exceeds 45000ms
14/12/11 11:40:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(354, ams06-014.ff.company.com, 48023) with no recent heart
beats: 74273ms exceeds 45000ms
14/12/11 11:40:19 WARN TaskSetManager: Lost task 5317.0 in stage 0.0 (TID
5562, ams02-010.ff.company.com): java.io.IOException: sendMessageReliably
failed because ack was not received within 60 sec
       
org.apache.spark.network.nio.ConnectionManager$$anon$8$$anonfun$run$19.apply(ConnectionManager.scala:903)
       
org.apache.spark.network.nio.ConnectionManager$$anon$8$$anonfun$run$19.apply(ConnectionManager.scala:902)
        scala.Option.foreach(Option.scala:236)
       
org.apache.spark.network.nio.ConnectionManager$$anon$8.run(ConnectionManager.scala:902)
        java.util.TimerThread.mainLoop(Timer.java:555)
        java.util.TimerThread.run(Timer.java:505)
14/12/11 11:41:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(278, ams04-015.ff.company.com, 44067) with no recent heart
beats: 49379ms exceeds 45000ms
14/12/11 11:41:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(243, ams02-001.ff.company.com, 47519) with no recent heart
beats: 45512ms exceeds 45000ms
14/12/11 11:41:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(154, ams07-014.ff.company.com, 47197) with no recent heart
beats: 85398ms exceeds 45000ms
14/12/11 11:41:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(391, ams07-004.ff.company.com, 45193) with no recent heart
beats: 86808ms exceeds 45000ms
14/12/11 11:41:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(63, ams02-013.ff.company.com, 35729) with no recent heart
beats: 64054ms exceeds 45000ms
14/12/11 11:41:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(225, ams06-010.ff.company.com, 39763) with no recent heart
beats: 80734ms exceeds 45000ms
14/12/11 11:41:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(70, ams07-005.ff.company.com, 43480) with no recent heart
beats: 68173ms exceeds 45000ms
14/12/11 11:41:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(350, ams03-010.ff.company.com, 34850) with no recent heart
beats: 63529ms exceeds 45000ms
14/12/11 11:41:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(132, ams04-012.ff.company.com, 36525) with no recent heart
beats: 84641ms exceeds 45000ms
14/12/11 11:41:19 ERROR YarnClientClusterScheduler: Lost executor 401 on
ams06-011.ff.company.com: remote Akka client disassociated
14/12/11 11:41:19 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams06-011.ff.company.com:35623] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:41:19 WARN TaskSetManager: Lost task 6046.0 in stage 0.0 (TID
5583, ams06-011.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:41:19 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 401
14/12/11 11:41:19 WARN TaskSetManager: Lost task 900.1 in stage 0.0 (TID
5587, ams02-010.ff.company.com): java.io.IOException: sendMessageReliably
failed because ack was not received within 60 sec
       
org.apache.spark.network.nio.ConnectionManager$$anon$8$$anonfun$run$19.apply(ConnectionManager.scala:903)
       
org.apache.spark.network.nio.ConnectionManager$$anon$8$$anonfun$run$19.apply(ConnectionManager.scala:902)
        scala.Option.foreach(Option.scala:236)
       
org.apache.spark.network.nio.ConnectionManager$$anon$8.run(ConnectionManager.scala:902)
        java.util.TimerThread.mainLoop(Timer.java:555)
        java.util.TimerThread.run(Timer.java:505)
14/12/11 11:41:42 WARN TaskSetManager: Lost task 5280.0 in stage 0.0 (TID
4557, ams04-013.ff.company.com): java.lang.OutOfMemoryError: GC overhead
limit exceeded
       
org.xerial.snappy.buffer.CachedBufferAllocator.allocate(CachedBufferAllocator.java:48)
       
org.xerial.snappy.SnappyOutputStream.<init>(SnappyOutputStream.java:96)
       
org.xerial.snappy.SnappyOutputStream.<init>(SnappyOutputStream.java:83)
       
org.apache.spark.io.SnappyCompressionCodec.compressedOutputStream(CompressionCodec.scala:125)
       
org.apache.spark.storage.BlockManager.wrapForCompression(BlockManager.scala:1084)
       
org.apache.spark.storage.BlockManager$$anonfun$7.apply(BlockManager.scala:579)
       
org.apache.spark.storage.BlockManager$$anonfun$7.apply(BlockManager.scala:579)
       
org.apache.spark.storage.DiskBlockObjectWriter.open(BlockObjectWriter.scala:126)
       
org.apache.spark.storage.DiskBlockObjectWriter.write(BlockObjectWriter.scala:192)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4$$anonfun$apply$2.apply(ExternalSorter.scala:744)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4$$anonfun$apply$2.apply(ExternalSorter.scala:743)
        scala.collection.Iterator$class.foreach(Iterator.scala:727)
       
org.apache.spark.util.collection.ExternalSorter$IteratorForPartition.foreach(ExternalSorter.scala:801)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4.apply(ExternalSorter.scala:743)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4.apply(ExternalSorter.scala:739)
        scala.collection.Iterator$class.foreach(Iterator.scala:727)
        scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
       
org.apache.spark.util.collection.ExternalSorter.writePartitionedFile(ExternalSorter.scala:739)
       
org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:70)
       
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)
       
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
        org.apache.spark.scheduler.Task.run(Task.scala:56)
       
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:186)
       
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
       
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        java.lang.Thread.run(Thread.java:744)
14/12/11 11:41:44 ERROR YarnClientClusterScheduler: Lost executor 319 on
ams04-013.ff.company.com: remote Akka client disassociated
14/12/11 11:41:44 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams04-013.ff.company.com:43247] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:41:44 WARN TaskSetManager: Lost task 6530.0 in stage 0.0 (TID
5622, ams04-013.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:41:44 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 319
14/12/11 11:41:48 ERROR YarnClientClusterScheduler: Lost executor 389 on
ams03-005.ff.company.com: remote Akka client disassociated
14/12/11 11:41:48 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams03-005.ff.company.com:37995] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:41:48 WARN TaskSetManager: Lost task 6036.1 in stage 0.0 (TID
5625, ams03-005.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:41:48 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 389
14/12/11 11:41:57 WARN TaskSetManager: Lost task 5018.0 in stage 0.0 (TID
5172, ams03-004.ff.company.com): java.lang.OutOfMemoryError: GC overhead
limit exceeded
        java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:322)
       
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1133)
       
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
       
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
       
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
       
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
        java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347)
       
org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:42)
       
org.apache.spark.storage.DiskBlockObjectWriter.write(BlockObjectWriter.scala:195)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4$$anonfun$apply$2.apply(ExternalSorter.scala:744)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4$$anonfun$apply$2.apply(ExternalSorter.scala:743)
        scala.collection.Iterator$class.foreach(Iterator.scala:727)
       
org.apache.spark.util.collection.ExternalSorter$IteratorForPartition.foreach(ExternalSorter.scala:801)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4.apply(ExternalSorter.scala:743)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4.apply(ExternalSorter.scala:739)
        scala.collection.Iterator$class.foreach(Iterator.scala:727)
        scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
       
org.apache.spark.util.collection.ExternalSorter.writePartitionedFile(ExternalSorter.scala:739)
       
org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:70)
       
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)
       
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
        org.apache.spark.scheduler.Task.run(Task.scala:56)
       
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:186)
       
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
       
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        java.lang.Thread.run(Thread.java:744)
14/12/11 11:41:59 ERROR YarnClientClusterScheduler: Lost executor 96 on
ams03-004.ff.company.com: remote Akka client disassociated
14/12/11 11:41:59 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams03-004.ff.company.com:48601] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:41:59 WARN TaskSetManager: Lost task 5302.0 in stage 0.0 (TID
5636, ams03-004.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:41:59 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 96
14/12/11 11:42:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(284, ams02-004.ff.company.com, 33103) with no recent heart
beats: 96959ms exceeds 45000ms
14/12/11 11:42:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(53, ams04-011.ff.company.com, 37314) with no recent heart
beats: 51536ms exceeds 45000ms
14/12/11 11:42:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(299, ams06-015.ff.company.com, 34264) with no recent heart
beats: 75423ms exceeds 45000ms
14/12/11 11:42:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(27, ams07-002.ff.company.com, 35873) with no recent heart
beats: 69518ms exceeds 45000ms
14/12/11 11:42:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(101, ams04-016.ff.company.com, 48586) with no recent heart
beats: 57084ms exceeds 45000ms
14/12/11 11:42:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(124, ams05-014.ff.company.com, 43779) with no recent heart
beats: 61928ms exceeds 45000ms
14/12/11 11:42:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(64, ams02-005.ff.company.com, 37446) with no recent heart
beats: 48791ms exceeds 45000ms
14/12/11 11:42:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(247, ams02-006.ff.company.com, 34425) with no recent heart
beats: 82464ms exceeds 45000ms
14/12/11 11:42:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(28, ams02-015.ff.company.com, 46323) with no recent heart
beats: 62659ms exceeds 45000ms
14/12/11 11:42:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(341, ams06-001.ff.company.com, 38687) with no recent heart
beats: 46911ms exceeds 45000ms
14/12/11 11:42:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(71, ams02-018.ff.company.com, 49975) with no recent heart
beats: 65944ms exceeds 45000ms
14/12/11 11:42:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(107, ams02-011.ff.company.com, 36163) with no recent heart
beats: 48259ms exceeds 45000ms
14/12/11 11:42:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(360, ams04-003.ff.company.com, 48594) with no recent heart
beats: 60558ms exceeds 45000ms
14/12/11 11:42:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(30, ams04-012.ff.company.com, 37996) with no recent heart
beats: 50411ms exceeds 45000ms
14/12/11 11:42:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(69, ams07-013.ff.company.com, 34344) with no recent heart
beats: 47756ms exceeds 45000ms
14/12/11 11:42:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(44, ams03-010.ff.company.com, 46150) with no recent heart
beats: 47539ms exceeds 45000ms
14/12/11 11:42:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(265, ams05-008.ff.company.com, 33695) with no recent heart
beats: 52546ms exceeds 45000ms
14/12/11 11:42:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(378, ams04-010.ff.company.com, 46141) with no recent heart
beats: 47098ms exceeds 45000ms
14/12/11 11:42:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(255, ams07-006.ff.company.com, 34848) with no recent heart
beats: 98145ms exceeds 45000ms
14/12/11 11:42:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(151, ams05-017.ff.company.com, 35734) with no recent heart
beats: 52862ms exceeds 45000ms
14/12/11 11:42:19 WARN TaskSetManager: Lost task 5095.0 in stage 0.0 (TID
5225, ams05-005.ff.company.com): java.lang.OutOfMemoryError: GC overhead
limit exceeded
       
java.io.ObjectOutputStream$HandleTable.growEntries(ObjectOutputStream.java:2350)
       
java.io.ObjectOutputStream$HandleTable.assign(ObjectOutputStream.java:2275)
        java.io.ObjectOutputStream.writeString(ObjectOutputStream.java:1301)
       
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1171)
       
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
       
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
       
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
       
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
       
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
       
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
       
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
       
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
        java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347)
       
org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:42)
       
org.apache.spark.storage.DiskBlockObjectWriter.write(BlockObjectWriter.scala:195)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4$$anonfun$apply$2.apply(ExternalSorter.scala:744)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4$$anonfun$apply$2.apply(ExternalSorter.scala:743)
        scala.collection.Iterator$class.foreach(Iterator.scala:727)
       
org.apache.spark.util.collection.ExternalSorter$IteratorForPartition.foreach(ExternalSorter.scala:801)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4.apply(ExternalSorter.scala:743)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4.apply(ExternalSorter.scala:739)
        scala.collection.Iterator$class.foreach(Iterator.scala:727)
        scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
       
org.apache.spark.util.collection.ExternalSorter.writePartitionedFile(ExternalSorter.scala:739)
       
org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:70)
       
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)
       
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
        org.apache.spark.scheduler.Task.run(Task.scala:56)
       
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:186)
       
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
       
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        java.lang.Thread.run(Thread.java:744)
14/12/11 11:42:20 WARN TaskSetManager: Lost task 5317.1 in stage 0.0 (TID
5611, ams02-010.ff.company.com): java.io.IOException: sendMessageReliably
failed because ack was not received within 60 sec
       
org.apache.spark.network.nio.ConnectionManager$$anon$8$$anonfun$run$19.apply(ConnectionManager.scala:903)
       
org.apache.spark.network.nio.ConnectionManager$$anon$8$$anonfun$run$19.apply(ConnectionManager.scala:902)
        scala.Option.foreach(Option.scala:236)
       
org.apache.spark.network.nio.ConnectionManager$$anon$8.run(ConnectionManager.scala:902)
        java.util.TimerThread.mainLoop(Timer.java:555)
        java.util.TimerThread.run(Timer.java:505)
14/12/11 11:42:22 ERROR YarnClientClusterScheduler: Lost executor 98 on
ams05-005.ff.company.com: remote Akka client disassociated
14/12/11 11:42:22 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams05-005.ff.company.com:37375] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:42:22 WARN TaskSetManager: Lost task 6043.1 in stage 0.0 (TID
5645, ams05-005.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:42:22 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 98
14/12/11 11:42:31 ERROR YarnClientClusterScheduler: Lost executor 300 on
ams03-004.ff.company.com: remote Akka client disassociated
14/12/11 11:42:31 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams03-004.ff.company.com:32903] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:42:31 ERROR ConnectionManager: Exception while sending message.
java.nio.channels.CancelledKeyException
	at
org.apache.spark.network.nio.ConnectionManager.run(ConnectionManager.scala:437)
	at
org.apache.spark.network.nio.ConnectionManager$$anon$7.run(ConnectionManager.scala:170)
14/12/11 11:42:31 ERROR ConnectionManager: no messageStatus for failed
message id: 455
14/12/11 11:42:31 WARN TaskSetManager: Lost task 571.1 in stage 0.0 (TID
5651, ams03-004.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:42:31 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 300
14/12/11 11:42:33 WARN TaskSetManager: Lost task 5002.0 in stage 0.0 (TID
4984, ams07-008.ff.company.com): java.lang.OutOfMemoryError: GC overhead
limit exceeded
        java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:322)
       
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1133)
       
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
       
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
       
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
       
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
       
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
       
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
       
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
       
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
        java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347)
       
org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:42)
       
org.apache.spark.storage.DiskBlockObjectWriter.write(BlockObjectWriter.scala:195)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4$$anonfun$apply$2.apply(ExternalSorter.scala:744)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4$$anonfun$apply$2.apply(ExternalSorter.scala:743)
        scala.collection.Iterator$class.foreach(Iterator.scala:727)
       
org.apache.spark.util.collection.ExternalSorter$IteratorForPartition.foreach(ExternalSorter.scala:801)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4.apply(ExternalSorter.scala:743)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4.apply(ExternalSorter.scala:739)
        scala.collection.Iterator$class.foreach(Iterator.scala:727)
        scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
       
org.apache.spark.util.collection.ExternalSorter.writePartitionedFile(ExternalSorter.scala:739)
       
org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:70)
       
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)
       
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
        org.apache.spark.scheduler.Task.run(Task.scala:56)
       
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:186)
       
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
       
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        java.lang.Thread.run(Thread.java:744)
14/12/11 11:42:33 WARN TaskSetManager: Lost task 6046.1 in stage 0.0 (TID
5618, ams04-005.ff.company.com): java.io.IOException: sendMessageReliably
failed because ack was not received within 60 sec
       
org.apache.spark.network.nio.ConnectionManager$$anon$8$$anonfun$run$19.apply(ConnectionManager.scala:903)
       
org.apache.spark.network.nio.ConnectionManager$$anon$8$$anonfun$run$19.apply(ConnectionManager.scala:902)
        scala.Option.foreach(Option.scala:236)
       
org.apache.spark.network.nio.ConnectionManager$$anon$8.run(ConnectionManager.scala:902)
        java.util.TimerThread.mainLoop(Timer.java:555)
        java.util.TimerThread.run(Timer.java:505)
14/12/11 11:42:36 ERROR ConnectionManager: Corresponding SendingConnection
to ConnectionManagerId(ams07-008.ff.company.com,48165) not found
14/12/11 11:42:36 ERROR YarnClientClusterScheduler: Lost executor 16 on
ams07-008.ff.company.com: remote Akka client disassociated
14/12/11 11:42:36 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams07-008.ff.company.com:42535] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:42:36 WARN TaskSetManager: Lost task 755.1 in stage 0.0 (TID
5654, ams07-008.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:42:36 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 16
14/12/11 11:42:36 ERROR YarnClientClusterScheduler: Lost executor 203 on
ams04-016.ff.company.com: remote Akka client disassociated
14/12/11 11:42:36 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams04-016.ff.company.com:44487] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:42:36 WARN TaskSetManager: Lost task 678.1 in stage 0.0 (TID
5658, ams04-016.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:42:36 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 203
14/12/11 11:42:41 ERROR YarnClientClusterScheduler: Lost executor 198 on
ams03-004.ff.company.com: remote Akka client disassociated
14/12/11 11:42:41 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams03-004.ff.company.com:46196] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:42:41 WARN TaskSetManager: Lost task 638.1 in stage 0.0 (TID
5666, ams03-004.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:42:41 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 198
14/12/11 11:42:41 ERROR ConnectionManager: Corresponding SendingConnection
to ConnectionManagerId(ams05-012.ff.company.com,43431) not found
14/12/11 11:42:41 ERROR YarnClientClusterScheduler: Lost executor 57 on
ams05-012.ff.company.com: remote Akka client disassociated
14/12/11 11:42:41 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams05-012.ff.company.com:36387] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:42:41 WARN TaskSetManager: Lost task 399.1 in stage 0.0 (TID
5664, ams05-012.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:42:41 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 57
14/12/11 11:42:43 ERROR SendingConnection: Exception while reading
SendingConnection to ConnectionManagerId(ams06-007.ff.company.com,45398)
java.nio.channels.ClosedChannelException
	at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
	at
org.apache.spark.network.nio.SendingConnection.read(Connection.scala:419)
	at
org.apache.spark.network.nio.ConnectionManager$$anon$10.run(ConnectionManager.scala:238)
	at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
14/12/11 11:42:43 ERROR ConnectionManager: Exception while sending message.
java.nio.channels.ClosedChannelException
	at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
	at
org.apache.spark.network.nio.SendingConnection.read(Connection.scala:419)
	at
org.apache.spark.network.nio.ConnectionManager$$anon$10.run(ConnectionManager.scala:238)
	at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
14/12/11 11:42:43 ERROR YarnClientClusterScheduler: Lost executor 301 on
ams06-007.ff.company.com: remote Akka client disassociated
14/12/11 11:42:43 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams06-007.ff.company.com:40332] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:42:43 ERROR ConnectionManager: no messageStatus for failed
message id: 291
14/12/11 11:42:43 WARN TaskSetManager: Lost task 5604.0 in stage 0.0 (TID
5668, ams06-007.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:42:43 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 301
14/12/11 11:42:52 WARN TaskSetManager: Lost task 5098.0 in stage 0.0 (TID
5224, ams05-018.ff.company.com): java.lang.OutOfMemoryError: GC overhead
limit exceeded
        java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:322)
       
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1133)
       
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
       
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
       
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
       
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
       
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
       
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
       
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
       
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
        java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347)
       
org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:42)
       
org.apache.spark.storage.DiskBlockObjectWriter.write(BlockObjectWriter.scala:195)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4$$anonfun$apply$2.apply(ExternalSorter.scala:744)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4$$anonfun$apply$2.apply(ExternalSorter.scala:743)
        scala.collection.Iterator$class.foreach(Iterator.scala:727)
       
org.apache.spark.util.collection.ExternalSorter$IteratorForPartition.foreach(ExternalSorter.scala:801)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4.apply(ExternalSorter.scala:743)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4.apply(ExternalSorter.scala:739)
        scala.collection.Iterator$class.foreach(Iterator.scala:727)
        scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
       
org.apache.spark.util.collection.ExternalSorter.writePartitionedFile(ExternalSorter.scala:739)
       
org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:70)
       
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)
       
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
        org.apache.spark.scheduler.Task.run(Task.scala:56)
       
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:186)
       
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
       
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        java.lang.Thread.run(Thread.java:744)
14/12/11 11:42:54 ERROR YarnClientClusterScheduler: Lost executor 229 on
ams05-018.ff.company.com: remote Akka client disassociated
14/12/11 11:42:54 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams05-018.ff.company.com:47265] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:42:54 WARN TaskSetManager: Lost task 5662.0 in stage 0.0 (TID
5676, ams05-018.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:42:54 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 229
14/12/11 11:42:58 ERROR ConnectionManager: Exception while sending message.
java.nio.channels.CancelledKeyException
	at
org.apache.spark.network.nio.ConnectionManager.run(ConnectionManager.scala:437)
	at
org.apache.spark.network.nio.ConnectionManager$$anon$7.run(ConnectionManager.scala:170)
14/12/11 11:42:58 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams02-015.ff.company.com:46871] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:42:58 ERROR YarnClientClusterScheduler: Lost executor 232 on
ams02-015.ff.company.com: remote Akka client disassociated
14/12/11 11:42:58 ERROR ConnectionManager: no messageStatus for failed
message id: 465
14/12/11 11:42:58 ERROR ConnectionManager: Exception while sending message.
java.nio.channels.CancelledKeyException
	at
org.apache.spark.network.nio.ConnectionManager.run(ConnectionManager.scala:437)
	at
org.apache.spark.network.nio.ConnectionManager$$anon$7.run(ConnectionManager.scala:170)
14/12/11 11:42:58 ERROR ConnectionManager: no messageStatus for failed
message id: 465
14/12/11 11:42:58 ERROR ConnectionManager: Exception while sending message.
java.nio.channels.CancelledKeyException
	at
org.apache.spark.network.nio.ConnectionManager.run(ConnectionManager.scala:437)
	at
org.apache.spark.network.nio.ConnectionManager$$anon$7.run(ConnectionManager.scala:170)
14/12/11 11:42:58 ERROR ConnectionManager: no messageStatus for failed
message id: 465
14/12/11 11:42:58 ERROR ConnectionManager: Exception while sending message.
java.nio.channels.CancelledKeyException
	at
org.apache.spark.network.nio.ConnectionManager.run(ConnectionManager.scala:437)
	at
org.apache.spark.network.nio.ConnectionManager$$anon$7.run(ConnectionManager.scala:170)
14/12/11 11:42:58 ERROR ConnectionManager: no messageStatus for failed
message id: 465
14/12/11 11:42:58 ERROR ConnectionManager: Exception while sending message.
java.nio.channels.CancelledKeyException
	at
org.apache.spark.network.nio.ConnectionManager.run(ConnectionManager.scala:437)
	at
org.apache.spark.network.nio.ConnectionManager$$anon$7.run(ConnectionManager.scala:170)
14/12/11 11:42:58 ERROR ConnectionManager: no messageStatus for failed
message id: 465
14/12/11 11:42:58 ERROR ConnectionManager: Exception while sending message.
java.nio.channels.CancelledKeyException
	at
org.apache.spark.network.nio.ConnectionManager.run(ConnectionManager.scala:437)
	at
org.apache.spark.network.nio.ConnectionManager$$anon$7.run(ConnectionManager.scala:170)
14/12/11 11:42:58 ERROR ConnectionManager: no messageStatus for failed
message id: 465
14/12/11 11:42:58 ERROR ConnectionManager: Exception while sending message.
java.nio.channels.CancelledKeyException
	at
org.apache.spark.network.nio.ConnectionManager.run(ConnectionManager.scala:437)
	at
org.apache.spark.network.nio.ConnectionManager$$anon$7.run(ConnectionManager.scala:170)
14/12/11 11:42:58 ERROR ConnectionManager: no messageStatus for failed
message id: 465
14/12/11 11:42:58 ERROR ConnectionManager: Exception while sending message.
java.nio.channels.CancelledKeyException
	at
org.apache.spark.network.nio.ConnectionManager.run(ConnectionManager.scala:437)
	at
org.apache.spark.network.nio.ConnectionManager$$anon$7.run(ConnectionManager.scala:170)
14/12/11 11:42:58 ERROR ConnectionManager: no messageStatus for failed
message id: 465
14/12/11 11:42:58 ERROR ConnectionManager: Exception while sending message.
java.nio.channels.CancelledKeyException
	at
org.apache.spark.network.nio.ConnectionManager.run(ConnectionManager.scala:437)
	at
org.apache.spark.network.nio.ConnectionManager$$anon$7.run(ConnectionManager.scala:170)
14/12/11 11:42:58 ERROR ConnectionManager: no messageStatus for failed
message id: 465
14/12/11 11:42:58 ERROR ConnectionManager: Exception while sending message.
java.nio.channels.CancelledKeyException
	at
org.apache.spark.network.nio.ConnectionManager.run(ConnectionManager.scala:437)
	at
org.apache.spark.network.nio.ConnectionManager$$anon$7.run(ConnectionManager.scala:170)
14/12/11 11:42:58 ERROR ConnectionManager: no messageStatus for failed
message id: 465
14/12/11 11:42:58 ERROR ConnectionManager: Exception while sending message.
java.nio.channels.CancelledKeyException
	at
org.apache.spark.network.nio.ConnectionManager.run(ConnectionManager.scala:437)
	at
org.apache.spark.network.nio.ConnectionManager$$anon$7.run(ConnectionManager.scala:170)
14/12/11 11:42:58 ERROR ConnectionManager: no messageStatus for failed
message id: 465
14/12/11 11:42:58 WARN TaskSetManager: Lost task 1170.1 in stage 0.0 (TID
5680, ams02-015.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:42:58 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 232
14/12/11 11:43:03 WARN TaskSetManager: Lost task 5009.0 in stage 0.0 (TID
5453, ams04-005.ff.company.com): java.lang.OutOfMemoryError: GC overhead
limit exceeded
        java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:322)
       
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1133)
       
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
       
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
       
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
       
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
       
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
       
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
       
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
       
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
        java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347)
       
org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:42)
       
org.apache.spark.storage.DiskBlockObjectWriter.write(BlockObjectWriter.scala:195)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4$$anonfun$apply$2.apply(ExternalSorter.scala:744)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4$$anonfun$apply$2.apply(ExternalSorter.scala:743)
        scala.collection.Iterator$class.foreach(Iterator.scala:727)
       
org.apache.spark.util.collection.ExternalSorter$IteratorForPartition.foreach(ExternalSorter.scala:801)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4.apply(ExternalSorter.scala:743)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4.apply(ExternalSorter.scala:739)
        scala.collection.Iterator$class.foreach(Iterator.scala:727)
        scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
       
org.apache.spark.util.collection.ExternalSorter.writePartitionedFile(ExternalSorter.scala:739)
       
org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:70)
       
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)
       
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
        org.apache.spark.scheduler.Task.run(Task.scala:56)
       
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:186)
       
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
       
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        java.lang.Thread.run(Thread.java:744)
14/12/11 11:43:04 ERROR YarnClientClusterScheduler: Lost executor 323 on
ams04-005.ff.company.com: remote Akka client disassociated
14/12/11 11:43:04 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams04-005.ff.company.com:41439] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:43:04 WARN TaskSetManager: Lost task 217.1 in stage 0.0 (TID
5688, ams04-005.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:43:04 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 323
14/12/11 11:43:07 WARN TaskSetManager: Lost task 6530.1 in stage 0.0 (TID
5640, ams04-013.ff.company.com): java.io.IOException: sendMessageReliably
failed because ack was not received within 60 sec
       
org.apache.spark.network.nio.ConnectionManager$$anon$8$$anonfun$run$19.apply(ConnectionManager.scala:903)
       
org.apache.spark.network.nio.ConnectionManager$$anon$8$$anonfun$run$19.apply(ConnectionManager.scala:902)
        scala.Option.foreach(Option.scala:236)
       
org.apache.spark.network.nio.ConnectionManager$$anon$8.run(ConnectionManager.scala:902)
        java.util.TimerThread.mainLoop(Timer.java:555)
        java.util.TimerThread.run(Timer.java:505)
14/12/11 11:43:11 ERROR SendingConnection: Exception while reading
SendingConnection to ConnectionManagerId(ams03-009.ff.company.com,42604)
java.nio.channels.ClosedChannelException
	at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
	at
org.apache.spark.network.nio.SendingConnection.read(Connection.scala:419)
	at
org.apache.spark.network.nio.ConnectionManager$$anon$10.run(ConnectionManager.scala:238)
	at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
14/12/11 11:43:11 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams03-009.ff.company.com:34157] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:43:11 ERROR ConnectionManager: Exception while sending message.
java.nio.channels.ClosedChannelException
	at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
	at
org.apache.spark.network.nio.SendingConnection.read(Connection.scala:419)
	at
org.apache.spark.network.nio.ConnectionManager$$anon$10.run(ConnectionManager.scala:238)
	at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
14/12/11 11:43:11 ERROR YarnClientClusterScheduler: Lost executor 91 on
ams03-009.ff.company.com: remote Akka client disassociated
14/12/11 11:43:11 ERROR ConnectionManager: no messageStatus for failed
message id: 18
14/12/11 11:43:11 WARN TaskSetManager: Lost task 3521.1 in stage 0.0 (TID
5693, ams03-009.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:43:11 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 91
14/12/11 11:43:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(248, ams03-010.ff.company.com, 48854) with no recent heart
beats: 47156ms exceeds 45000ms
14/12/11 11:43:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(256, ams07-014.ff.company.com, 36630) with no recent heart
beats: 50977ms exceeds 45000ms
14/12/11 11:43:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(61, ams05-008.ff.company.com, 39788) with no recent heart
beats: 65135ms exceeds 45000ms
14/12/11 11:43:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(72, ams04-010.ff.company.com, 49712) with no recent heart
beats: 47890ms exceeds 45000ms
14/12/11 11:43:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(68, ams03-006.ff.company.com, 49574) with no recent heart
beats: 96309ms exceeds 45000ms
14/12/11 11:43:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(218, ams03-008.ff.company.com, 42311) with no recent heart
beats: 72219ms exceeds 45000ms
14/12/11 11:43:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(200, ams05-005.ff.company.com, 36768) with no recent heart
beats: 94365ms exceeds 45000ms
14/12/11 11:43:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(393, ams06-008.ff.company.com, 46167) with no recent heart
beats: 86182ms exceeds 45000ms
14/12/11 11:43:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(212, ams02-008.ff.company.com, 34642) with no recent heart
beats: 54050ms exceeds 45000ms
14/12/11 11:43:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(145, ams02-006.ff.company.com, 48475) with no recent heart
beats: 94362ms exceeds 45000ms
14/12/11 11:43:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(102, ams03-018.ff.company.com, 34979) with no recent heart
beats: 48007ms exceeds 45000ms
14/12/11 11:43:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(142, ams04-017.ff.company.com, 33319) with no recent heart
beats: 46638ms exceeds 45000ms
14/12/11 11:43:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(313, ams02-003.ff.company.com, 46140) with no recent heart
beats: 78537ms exceeds 45000ms
14/12/11 11:43:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(7, ams02-003.ff.company.com, 33131) with no recent heart
beats: 69484ms exceeds 45000ms
14/12/11 11:43:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(196, ams05-002.ff.company.com, 34810) with no recent heart
beats: 76019ms exceeds 45000ms
14/12/11 11:43:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(271, ams04-002.ff.company.com, 47740) with no recent heart
beats: 61019ms exceeds 45000ms
14/12/11 11:43:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(181, ams05-007.ff.company.com, 47937) with no recent heart
beats: 91201ms exceeds 45000ms
14/12/11 11:43:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(254, ams03-002.ff.company.com, 49530) with no recent heart
beats: 49478ms exceeds 45000ms
14/12/11 11:43:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(1, ams06-011.ff.company.com, 48132) with no recent heart
beats: 78312ms exceeds 45000ms
14/12/11 11:43:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(266, ams05-016.ff.company.com, 38478) with no recent heart
beats: 58402ms exceeds 45000ms
14/12/11 11:43:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(80, ams02-004.ff.company.com, 47966) with no recent heart
beats: 45508ms exceeds 45000ms
14/12/11 11:43:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(94, ams05-002.ff.company.com, 40006) with no recent heart
beats: 57376ms exceeds 45000ms
14/12/11 11:43:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(77, ams05-003.ff.company.com, 49588) with no recent heart
beats: 57328ms exceeds 45000ms
14/12/11 11:43:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(296, ams04-014.ff.company.com, 47157) with no recent heart
beats: 48908ms exceeds 45000ms
14/12/11 11:43:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(304, ams06-005.ff.company.com, 45450) with no recent heart
beats: 82040ms exceeds 45000ms
14/12/11 11:43:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(56, ams03-015.ff.company.com, 43420) with no recent heart
beats: 52124ms exceeds 45000ms
14/12/11 11:43:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(211, ams02-003.ff.company.com, 34326) with no recent heart
beats: 54123ms exceeds 45000ms
14/12/11 11:43:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(138, ams05-013.ff.company.com, 43105) with no recent heart
beats: 76849ms exceeds 45000ms
14/12/11 11:43:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(316, ams03-012.ff.company.com, 37243) with no recent heart
beats: 52970ms exceeds 45000ms
14/12/11 11:43:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(188, ams04-001.ff.company.com, 38742) with no recent heart
beats: 98684ms exceeds 45000ms
14/12/11 11:43:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(92, ams04-014.ff.company.com, 41186) with no recent heart
beats: 60376ms exceeds 45000ms
14/12/11 11:43:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(104, ams06-002.ff.company.com, 35894) with no recent heart
beats: 46553ms exceeds 45000ms
14/12/11 11:43:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(123, ams06-010.ff.company.com, 49691) with no recent heart
beats: 46000ms exceeds 45000ms
14/12/11 11:43:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(78, ams06-012.ff.company.com, 49643) with no recent heart
beats: 102292ms exceeds 45000ms
14/12/11 11:43:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(325, ams02-002.ff.company.com, 40604) with no recent heart
beats: 80448ms exceeds 45000ms
14/12/11 11:43:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(125, ams03-003.ff.company.com, 40805) with no recent heart
beats: 57280ms exceeds 45000ms
14/12/11 11:43:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(277, ams06-009.ff.company.com, 36795) with no recent heart
beats: 96741ms exceeds 45000ms
14/12/11 11:43:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(82, ams06-003.ff.company.com, 40527) with no recent heart
beats: 83847ms exceeds 45000ms
14/12/11 11:43:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(281, ams05-003.ff.company.com, 38694) with no recent heart
beats: 56295ms exceeds 45000ms
14/12/11 11:43:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(40, ams04-017.ff.company.com, 39384) with no recent heart
beats: 47674ms exceeds 45000ms
14/12/11 11:43:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(237, ams07-007.ff.company.com, 42002) with no recent heart
beats: 79072ms exceeds 45000ms
14/12/11 11:43:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(120, ams05-001.ff.company.com, 33460) with no recent heart
beats: 89915ms exceeds 45000ms
14/12/11 11:43:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(187, ams07-004.ff.company.com, 44596) with no recent heart
beats: 102905ms exceeds 45000ms
14/12/11 11:43:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(306, ams03-018.ff.company.com, 46013) with no recent heart
beats: 92941ms exceeds 45000ms
14/12/11 11:43:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(110, ams02-008.ff.company.com, 34143) with no recent heart
beats: 53007ms exceeds 45000ms
14/12/11 11:43:13 WARN TaskSetManager: Lost task 5050.0 in stage 0.0 (TID
5035, ams06-013.ff.company.com): java.lang.OutOfMemoryError: GC overhead
limit exceeded
       
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1536)
       
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
       
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
       
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
        java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347)
       
org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:42)
       
org.apache.spark.storage.DiskBlockObjectWriter.write(BlockObjectWriter.scala:195)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4$$anonfun$apply$2.apply(ExternalSorter.scala:744)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4$$anonfun$apply$2.apply(ExternalSorter.scala:743)
        scala.collection.Iterator$class.foreach(Iterator.scala:727)
       
org.apache.spark.util.collection.ExternalSorter$IteratorForPartition.foreach(ExternalSorter.scala:801)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4.apply(ExternalSorter.scala:743)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4.apply(ExternalSorter.scala:739)
        scala.collection.Iterator$class.foreach(Iterator.scala:727)
        scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
       
org.apache.spark.util.collection.ExternalSorter.writePartitionedFile(ExternalSorter.scala:739)
       
org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:70)
       
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)
       
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
        org.apache.spark.scheduler.Task.run(Task.scala:56)
       
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:186)
       
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
       
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        java.lang.Thread.run(Thread.java:744)
14/12/11 11:43:16 ERROR SendingConnection: Exception while reading
SendingConnection to ConnectionManagerId(ams06-013.ff.company.com,43920)
java.nio.channels.ClosedChannelException
	at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
	at
org.apache.spark.network.nio.SendingConnection.read(Connection.scala:419)
	at
org.apache.spark.network.nio.ConnectionManager$$anon$10.run(ConnectionManager.scala:238)
	at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
14/12/11 11:43:16 ERROR ConnectionManager: Exception while sending message.
java.nio.channels.ClosedChannelException
	at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
	at
org.apache.spark.network.nio.SendingConnection.read(Connection.scala:419)
	at
org.apache.spark.network.nio.ConnectionManager$$anon$10.run(ConnectionManager.scala:238)
	at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
14/12/11 11:43:16 ERROR YarnClientClusterScheduler: Lost executor 167 on
ams06-013.ff.company.com: remote Akka client disassociated
14/12/11 11:43:16 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams06-013.ff.company.com:33556] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:43:16 ERROR ConnectionManager: no messageStatus for failed
message id: 802
14/12/11 11:43:16 WARN TaskSetManager: Lost task 5608.0 in stage 0.0 (TID
5698, ams06-013.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:43:16 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 167
14/12/11 11:43:16 ERROR YarnClientClusterScheduler: Lost executor 182 on
ams02-004.ff.company.com: remote Akka client disassociated
14/12/11 11:43:16 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams02-004.ff.company.com:43208] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:43:16 ERROR SendingConnection: Exception while reading
SendingConnection to ConnectionManagerId(ams02-004.ff.company.com,34770)
java.nio.channels.ClosedChannelException
	at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
	at
org.apache.spark.network.nio.SendingConnection.read(Connection.scala:419)
	at
org.apache.spark.network.nio.ConnectionManager$$anon$10.run(ConnectionManager.scala:238)
	at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
14/12/11 11:43:16 ERROR ConnectionManager: Exception while sending message.
java.nio.channels.ClosedChannelException
	at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
	at
org.apache.spark.network.nio.SendingConnection.read(Connection.scala:419)
	at
org.apache.spark.network.nio.ConnectionManager$$anon$10.run(ConnectionManager.scala:238)
	at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
14/12/11 11:43:16 ERROR ConnectionManager: no messageStatus for failed
message id: 397
14/12/11 11:43:16 WARN TaskSetManager: Lost task 949.1 in stage 0.0 (TID
5699, ams02-004.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:43:16 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 182
14/12/11 11:43:20 ERROR YarnClientClusterScheduler: Lost executor 30 on
ams04-012.ff.company.com: remote Akka client disassociated
14/12/11 11:43:20 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams04-012.ff.company.com:41589] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:43:20 WARN TaskSetManager: Lost task 1199.1 in stage 0.0 (TID
5702, ams04-012.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:43:20 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 30
14/12/11 11:43:22 ERROR YarnClientClusterScheduler: Lost executor 147 on
ams07-001.ff.company.com: remote Akka client disassociated
14/12/11 11:43:22 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams07-001.ff.company.com:48343] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:43:22 WARN TaskSetManager: Lost task 1244.0 in stage 1.0 (TID
5705, ams07-001.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:43:22 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 147
14/12/11 11:43:28 WARN TaskSetManager: Lost task 4503.0 in stage 0.0 (TID
5168, ams06-009.ff.company.com): java.lang.OutOfMemoryError: GC overhead
limit exceeded
        java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:322)
       
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1133)
       
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
       
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
       
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
       
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
       
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
       
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
       
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
       
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
        java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347)
       
org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:42)
       
org.apache.spark.storage.DiskBlockObjectWriter.write(BlockObjectWriter.scala:195)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4$$anonfun$apply$2.apply(ExternalSorter.scala:744)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4$$anonfun$apply$2.apply(ExternalSorter.scala:743)
        scala.collection.Iterator$class.foreach(Iterator.scala:727)
       
org.apache.spark.util.collection.ExternalSorter$IteratorForPartition.foreach(ExternalSorter.scala:801)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4.apply(ExternalSorter.scala:743)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4.apply(ExternalSorter.scala:739)
        scala.collection.Iterator$class.foreach(Iterator.scala:727)
        scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
       
org.apache.spark.util.collection.ExternalSorter.writePartitionedFile(ExternalSorter.scala:739)
       
org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:70)
       
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)
       
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
        org.apache.spark.scheduler.Task.run(Task.scala:56)
       
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:186)
       
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
       
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        java.lang.Thread.run(Thread.java:744)
14/12/11 11:43:31 ERROR YarnClientClusterScheduler: Lost executor 73 on
ams06-009.ff.company.com: remote Akka client disassociated
14/12/11 11:43:31 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams06-009.ff.company.com:33747] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:43:31 WARN TaskSetManager: Lost task 4648.1 in stage 0.0 (TID
5712, ams06-009.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:43:31 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 73
14/12/11 11:43:31 ERROR YarnClientClusterScheduler: Lost executor 181 on
ams05-007.ff.company.com: remote Akka client disassociated
14/12/11 11:43:31 WARN TaskSetManager: Lost task 5091.0 in stage 0.0 (TID
5143, ams05-007.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:43:39 WARN TaskSetManager: Lost task 571.2 in stage 0.0 (TID
5663, ams03-004.ff.company.com): java.io.IOException: sendMessageReliably
failed because ack was not received within 60 sec
       
org.apache.spark.network.nio.ConnectionManager$$anon$8$$anonfun$run$19.apply(ConnectionManager.scala:903)
       
org.apache.spark.network.nio.ConnectionManager$$anon$8$$anonfun$run$19.apply(ConnectionManager.scala:902)
        scala.Option.foreach(Option.scala:236)
       
org.apache.spark.network.nio.ConnectionManager$$anon$8.run(ConnectionManager.scala:902)
        java.util.TimerThread.mainLoop(Timer.java:555)
        java.util.TimerThread.run(Timer.java:505)
14/12/11 11:43:45 WARN TaskSetManager: Lost task 4628.0 in stage 0.0 (TID
5118, ams02-008.ff.company.com): java.lang.OutOfMemoryError: GC overhead
limit exceeded
       
java.io.ObjectOutputStream$HandleTable.growEntries(ObjectOutputStream.java:2346)
       
java.io.ObjectOutputStream$HandleTable.assign(ObjectOutputStream.java:2275)
        java.io.ObjectOutputStream.writeString(ObjectOutputStream.java:1301)
       
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1171)
       
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
       
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
       
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
       
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
       
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
       
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
       
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
       
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
        java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347)
       
org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:42)
       
org.apache.spark.storage.DiskBlockObjectWriter.write(BlockObjectWriter.scala:195)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4$$anonfun$apply$2.apply(ExternalSorter.scala:744)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4$$anonfun$apply$2.apply(ExternalSorter.scala:743)
        scala.collection.Iterator$class.foreach(Iterator.scala:727)
       
org.apache.spark.util.collection.ExternalSorter$IteratorForPartition.foreach(ExternalSorter.scala:801)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4.apply(ExternalSorter.scala:743)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4.apply(ExternalSorter.scala:739)
        scala.collection.Iterator$class.foreach(Iterator.scala:727)
        scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
       
org.apache.spark.util.collection.ExternalSorter.writePartitionedFile(ExternalSorter.scala:739)
       
org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:70)
       
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)
       
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
        org.apache.spark.scheduler.Task.run(Task.scala:56)
       
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:186)
       
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
       
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        java.lang.Thread.run(Thread.java:744)
14/12/11 11:43:49 ERROR YarnClientClusterScheduler: Lost executor 314 on
ams02-008.ff.company.com: remote Akka client disassociated
14/12/11 11:43:49 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams02-008.ff.company.com:42911] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:43:49 WARN TaskSetManager: Lost task 1758.1 in stage 0.0 (TID
5729, ams02-008.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:43:49 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 314
14/12/11 11:43:49 WARN TaskSetManager: Lost task 5604.1 in stage 0.0 (TID
5675, ams06-007.ff.company.com): java.io.IOException: sendMessageReliably
failed because ack was not received within 60 sec
       
org.apache.spark.network.nio.ConnectionManager$$anon$8$$anonfun$run$19.apply(ConnectionManager.scala:903)
       
org.apache.spark.network.nio.ConnectionManager$$anon$8$$anonfun$run$19.apply(ConnectionManager.scala:902)
        scala.Option.foreach(Option.scala:236)
       
org.apache.spark.network.nio.ConnectionManager$$anon$8.run(ConnectionManager.scala:902)
        java.util.TimerThread.mainLoop(Timer.java:555)
        java.util.TimerThread.run(Timer.java:505)
14/12/11 11:43:50 ERROR YarnClientClusterScheduler: Lost executor 120 on
ams05-001.ff.company.com: remote Akka client disassociated
14/12/11 11:43:50 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams05-001.ff.company.com:43760] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:43:50 WARN TaskSetManager: Lost task 636.1 in stage 0.0 (TID
5732, ams05-001.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:43:50 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 120
14/12/11 11:43:57 WARN TaskSetManager: Lost task 5146.0 in stage 0.0 (TID
5448, ams03-016.ff.company.com): java.lang.OutOfMemoryError: GC overhead
limit exceeded
        java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:322)
       
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1133)
       
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
       
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
       
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
       
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
       
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
       
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
       
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
       
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
        java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347)
       
org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:42)
       
org.apache.spark.storage.DiskBlockObjectWriter.write(BlockObjectWriter.scala:195)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4$$anonfun$apply$2.apply(ExternalSorter.scala:744)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4$$anonfun$apply$2.apply(ExternalSorter.scala:743)
        scala.collection.Iterator$class.foreach(Iterator.scala:727)
       
org.apache.spark.util.collection.ExternalSorter$IteratorForPartition.foreach(ExternalSorter.scala:801)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4.apply(ExternalSorter.scala:743)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4.apply(ExternalSorter.scala:739)
        scala.collection.Iterator$class.foreach(Iterator.scala:727)
        scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
       
org.apache.spark.util.collection.ExternalSorter.writePartitionedFile(ExternalSorter.scala:739)
       
org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:70)
       
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)
       
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
        org.apache.spark.scheduler.Task.run(Task.scala:56)
       
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:186)
       
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
       
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        java.lang.Thread.run(Thread.java:744)
14/12/11 11:43:59 ERROR YarnClientClusterScheduler: Lost executor 235 on
ams03-016.ff.company.com: remote Akka client disassociated
14/12/11 11:43:59 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams03-016.ff.company.com:43499] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:43:59 ERROR SendingConnection: Exception while reading
SendingConnection to ConnectionManagerId(ams03-016.ff.company.com,46291)
java.nio.channels.ClosedChannelException
	at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
	at
org.apache.spark.network.nio.SendingConnection.read(Connection.scala:419)
	at
org.apache.spark.network.nio.ConnectionManager$$anon$10.run(ConnectionManager.scala:238)
	at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
14/12/11 11:43:59 ERROR ConnectionManager: Exception while sending message.
java.nio.channels.ClosedChannelException
	at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
	at
org.apache.spark.network.nio.SendingConnection.read(Connection.scala:419)
	at
org.apache.spark.network.nio.ConnectionManager$$anon$10.run(ConnectionManager.scala:238)
	at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
14/12/11 11:43:59 ERROR ConnectionManager: no messageStatus for failed
message id: 625
14/12/11 11:43:59 WARN TaskSetManager: Lost task 5342.0 in stage 0.0 (TID
5746, ams03-016.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:43:59 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 235
14/12/11 11:44:01 WARN TaskSetManager: Lost task 2544.1 in stage 0.0 (TID
5686, ams02-003.ff.company.com): java.io.IOException: sendMessageReliably
failed because ack was not received within 60 sec
       
org.apache.spark.network.nio.ConnectionManager$$anon$8$$anonfun$run$19.apply(ConnectionManager.scala:903)
       
org.apache.spark.network.nio.ConnectionManager$$anon$8$$anonfun$run$19.apply(ConnectionManager.scala:902)
        scala.Option.foreach(Option.scala:236)
       
org.apache.spark.network.nio.ConnectionManager$$anon$8.run(ConnectionManager.scala:902)
        java.util.TimerThread.mainLoop(Timer.java:555)
        java.util.TimerThread.run(Timer.java:505)
14/12/11 11:44:08 WARN TaskSetManager: Lost task 6132.0 in stage 0.0 (TID
5250, ams04-018.ff.company.com): java.lang.OutOfMemoryError: GC overhead
limit exceeded
       
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1536)
       
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
       
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
       
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
       
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
       
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
       
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
       
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
        java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347)
       
org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:42)
       
org.apache.spark.storage.DiskBlockObjectWriter.write(BlockObjectWriter.scala:195)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4$$anonfun$apply$2.apply(ExternalSorter.scala:744)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4$$anonfun$apply$2.apply(ExternalSorter.scala:743)
        scala.collection.Iterator$class.foreach(Iterator.scala:727)
       
org.apache.spark.util.collection.ExternalSorter$IteratorForPartition.foreach(ExternalSorter.scala:801)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4.apply(ExternalSorter.scala:743)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4.apply(ExternalSorter.scala:739)
        scala.collection.Iterator$class.foreach(Iterator.scala:727)
        scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
       
org.apache.spark.util.collection.ExternalSorter.writePartitionedFile(ExternalSorter.scala:739)
       
org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:70)
       
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)
       
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
        org.apache.spark.scheduler.Task.run(Task.scala:56)
       
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:186)
       
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
       
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        java.lang.Thread.run(Thread.java:744)
14/12/11 11:44:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(381, ams04-006.ff.company.com, 35778) with no recent heart
beats: 48856ms exceeds 45000ms
14/12/11 11:44:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(49, ams05-017.ff.company.com, 47581) with no recent heart
beats: 53411ms exceeds 45000ms
14/12/11 11:44:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(169, ams04-002.ff.company.com, 42471) with no recent heart
beats: 63929ms exceeds 45000ms
14/12/11 11:44:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(258, ams04-003.ff.company.com, 44220) with no recent heart
beats: 64668ms exceeds 45000ms
14/12/11 11:44:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(209, ams02-011.ff.company.com, 39392) with no recent heart
beats: 62865ms exceeds 45000ms
14/12/11 11:44:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(108, ams05-006.ff.company.com, 49510) with no recent heart
beats: 99733ms exceeds 45000ms
14/12/11 11:44:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(218, ams03-008.ff.company.com, 42311) with no recent heart
beats: 46832ms exceeds 45000ms
14/12/11 11:44:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(208, ams04-018.ff.company.com, 47488) with no recent heart
beats: 61436ms exceeds 45000ms
14/12/11 11:44:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(278, ams04-015.ff.company.com, 44067) with no recent heart
beats: 45403ms exceeds 45000ms
14/12/11 11:44:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(14, ams03-008.ff.company.com, 43712) with no recent heart
beats: 75902ms exceeds 45000ms
14/12/11 11:44:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(41, ams05-004.ff.company.com, 49152) with no recent heart
beats: 82702ms exceeds 45000ms
14/12/11 11:44:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(293, ams03-013.ff.company.com, 38503) with no recent heart
beats: 100042ms exceeds 45000ms
14/12/11 11:44:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(164, ams05-016.ff.company.com, 47609) with no recent heart
beats: 59440ms exceeds 45000ms
14/12/11 11:44:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(191, ams03-013.ff.company.com, 35569) with no recent heart
beats: 64633ms exceeds 45000ms
14/12/11 11:44:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(348, ams05-009.ff.company.com, 41425) with no recent heart
beats: 71650ms exceeds 45000ms
14/12/11 11:44:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(103, ams06-011.ff.company.com, 37755) with no recent heart
beats: 90077ms exceeds 45000ms
14/12/11 11:44:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(183, ams02-012.ff.company.com, 49263) with no recent heart
beats: 50809ms exceeds 45000ms
14/12/11 11:44:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(36, ams05-013.ff.company.com, 38275) with no recent heart
beats: 60037ms exceeds 45000ms
14/12/11 11:44:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(298, ams05-002.ff.company.com, 46919) with no recent heart
beats: 50778ms exceeds 45000ms
14/12/11 11:44:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(152, ams03-002.ff.company.com, 44072) with no recent heart
beats: 48934ms exceeds 45000ms
14/12/11 11:44:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(355, ams05-017.ff.company.com, 45132) with no recent heart
beats: 72840ms exceeds 45000ms
14/12/11 11:44:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(324, ams05-001.ff.company.com, 37708) with no recent heart
beats: 49763ms exceeds 45000ms
14/12/11 11:44:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(344, ams04-008.ff.company.com, 45719) with no recent heart
beats: 81961ms exceeds 45000ms
14/12/11 11:44:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(197, ams06-015.ff.company.com, 37364) with no recent heart
beats: 71387ms exceeds 45000ms
14/12/11 11:44:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(99, ams03-011.ff.company.com, 35480) with no recent heart
beats: 61395ms exceeds 45000ms
14/12/11 11:44:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(201, ams03-011.ff.company.com, 38902) with no recent heart
beats: 75001ms exceeds 45000ms
14/12/11 11:44:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(230, ams06-006.ff.company.com, 36865) with no recent heart
beats: 70927ms exceeds 45000ms
14/12/11 11:44:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(195, ams07-009.ff.company.com, 33955) with no recent heart
beats: 49157ms exceeds 45000ms
14/12/11 11:44:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(11, ams07-003.ff.company.com, 43976) with no recent heart
beats: 66686ms exceeds 45000ms
14/12/11 11:44:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(216, ams02-016.ff.company.com, 35582) with no recent heart
beats: 46091ms exceeds 45000ms
14/12/11 11:44:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(360, ams04-003.ff.company.com, 48594) with no recent heart
beats: 85294ms exceeds 45000ms
14/12/11 11:44:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(383, ams05-003.ff.company.com, 33745) with no recent heart
beats: 62415ms exceeds 45000ms
14/12/11 11:44:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(122, ams04-009.ff.company.com, 35108) with no recent heart
beats: 67061ms exceeds 45000ms
14/12/11 11:44:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(361, ams03-007.ff.company.com, 43900) with no recent heart
beats: 69293ms exceeds 45000ms
14/12/11 11:44:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(239, ams06-001.ff.company.com, 35139) with no recent heart
beats: 99192ms exceeds 45000ms
14/12/11 11:44:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(287, ams03-005.ff.company.com, 41789) with no recent heart
beats: 48916ms exceeds 45000ms
14/12/11 11:44:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(13, ams04-013.ff.company.com, 45353) with no recent heart
beats: 68212ms exceeds 45000ms
14/12/11 11:44:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(6, ams05-006.ff.company.com, 34890) with no recent heart
beats: 88887ms exceeds 45000ms
14/12/11 11:44:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(163, ams05-008.ff.company.com, 43196) with no recent heart
beats: 59039ms exceeds 45000ms
14/12/11 11:44:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(388, ams06-003.ff.company.com, 48077) with no recent heart
beats: 104260ms exceeds 45000ms
14/12/11 11:44:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(160, ams04-007.ff.company.com, 33817) with no recent heart
beats: 82608ms exceeds 45000ms
14/12/11 11:44:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(46, ams02-014.ff.company.com, 44772) with no recent heart
beats: 46641ms exceeds 45000ms
14/12/11 11:44:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(156, ams04-003.ff.company.com, 47986) with no recent heart
beats: 73961ms exceeds 45000ms
14/12/11 11:44:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(339, ams07-007.ff.company.com, 45695) with no recent heart
beats: 64116ms exceeds 45000ms
14/12/11 11:44:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(292, ams02-017.ff.company.com, 34648) with no recent heart
beats: 67764ms exceeds 45000ms
14/12/11 11:44:12 ERROR ConnectionManager: Corresponding SendingConnection
to ConnectionManagerId(ams04-018.ff.company.com,39795) not found
14/12/11 11:44:12 ERROR YarnClientClusterScheduler: Lost executor 4 on
ams04-018.ff.company.com: remote Akka client disassociated
14/12/11 11:44:12 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams04-018.ff.company.com:39915] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:44:12 WARN TaskSetManager: Lost task 2908.1 in stage 0.0 (TID
5757, ams04-018.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:44:12 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 4
14/12/11 11:44:12 ERROR YarnClientClusterScheduler: Lost executor 359 on
ams04-011.ff.company.com: remote Akka client disassociated
14/12/11 11:44:12 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams04-011.ff.company.com:45273] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:44:12 WARN TaskSetManager: Lost task 2296.1 in stage 0.0 (TID
5758, ams04-011.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:44:12 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 359
14/12/11 11:44:18 ERROR YarnClientClusterScheduler: Lost executor 276 on
ams04-010.ff.company.com: remote Akka client disassociated
14/12/11 11:44:18 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams04-010.ff.company.com:49225] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:44:18 WARN TaskSetManager: Lost task 5233.0 in stage 0.0 (TID
5765, ams04-010.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:44:18 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 276
14/12/11 11:44:20 WARN TaskSetManager: Lost task 742.1 in stage 0.0 (TID
5704, ams07-008.ff.company.com): java.io.IOException: sendMessageReliably
failed because ack was not received within 60 sec
       
org.apache.spark.network.nio.ConnectionManager$$anon$8$$anonfun$run$19.apply(ConnectionManager.scala:903)
       
org.apache.spark.network.nio.ConnectionManager$$anon$8$$anonfun$run$19.apply(ConnectionManager.scala:902)
        scala.Option.foreach(Option.scala:236)
       
org.apache.spark.network.nio.ConnectionManager$$anon$8.run(ConnectionManager.scala:902)
        java.util.TimerThread.mainLoop(Timer.java:555)
        java.util.TimerThread.run(Timer.java:505)
14/12/11 11:44:24 WARN TaskSetManager: Lost task 5526.0 in stage 0.0 (TID
5059, ams04-008.ff.company.com): java.lang.OutOfMemoryError: GC overhead
limit exceeded
        java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:322)
       
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1133)
       
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
       
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
       
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
       
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
       
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
       
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
       
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
       
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
        java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347)
       
org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:42)
       
org.apache.spark.storage.DiskBlockObjectWriter.write(BlockObjectWriter.scala:195)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4$$anonfun$apply$2.apply(ExternalSorter.scala:744)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4$$anonfun$apply$2.apply(ExternalSorter.scala:743)
        scala.collection.Iterator$class.foreach(Iterator.scala:727)
       
org.apache.spark.util.collection.ExternalSorter$IteratorForPartition.foreach(ExternalSorter.scala:801)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4.apply(ExternalSorter.scala:743)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4.apply(ExternalSorter.scala:739)
        scala.collection.Iterator$class.foreach(Iterator.scala:727)
        scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
       
org.apache.spark.util.collection.ExternalSorter.writePartitionedFile(ExternalSorter.scala:739)
       
org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:70)
       
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)
       
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
        org.apache.spark.scheduler.Task.run(Task.scala:56)
       
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:186)
       
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
       
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        java.lang.Thread.run(Thread.java:744)
14/12/11 11:44:27 ERROR YarnClientClusterScheduler: Lost executor 38 on
ams04-008.ff.company.com: remote Akka client disassociated
14/12/11 11:44:27 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams04-008.ff.company.com:41575] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:44:27 WARN TaskSetManager: Lost task 2315.1 in stage 0.0 (TID
5772, ams04-008.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:44:27 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 38
14/12/11 11:44:31 WARN TaskSetManager: Lost task 4634.1 in stage 0.0 (TID
5714, ams04-005.ff.company.com): java.io.IOException: sendMessageReliably
failed because ack was not received within 60 sec
       
org.apache.spark.network.nio.ConnectionManager$$anon$8$$anonfun$run$19.apply(ConnectionManager.scala:903)
       
org.apache.spark.network.nio.ConnectionManager$$anon$8$$anonfun$run$19.apply(ConnectionManager.scala:902)
        scala.Option.foreach(Option.scala:236)
       
org.apache.spark.network.nio.ConnectionManager$$anon$8.run(ConnectionManager.scala:902)
        java.util.TimerThread.mainLoop(Timer.java:555)
        java.util.TimerThread.run(Timer.java:505)
14/12/11 11:44:32 ERROR YarnClientClusterScheduler: Lost executor 74 on
ams04-015.ff.company.com: remote Akka client disassociated
14/12/11 11:44:32 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams04-015.ff.company.com:37517] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:44:32 WARN TaskSetManager: Lost task 2936.1 in stage 0.0 (TID
5777, ams04-015.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:44:32 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 74
14/12/11 11:44:35 ERROR SendingConnection: Exception while reading
SendingConnection to ConnectionManagerId(ams03-007.ff.company.com,48637)
java.nio.channels.ClosedChannelException
	at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
	at
org.apache.spark.network.nio.SendingConnection.read(Connection.scala:419)
	at
org.apache.spark.network.nio.ConnectionManager$$anon$10.run(ConnectionManager.scala:238)
	at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
14/12/11 11:44:35 ERROR ConnectionManager: Exception while sending message.
java.nio.channels.ClosedChannelException
	at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
	at
org.apache.spark.network.nio.SendingConnection.read(Connection.scala:419)
	at
org.apache.spark.network.nio.ConnectionManager$$anon$10.run(ConnectionManager.scala:238)
	at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
14/12/11 11:44:35 ERROR YarnClientClusterScheduler: Lost executor 55 on
ams03-007.ff.company.com: remote Akka client disassociated
14/12/11 11:44:35 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams03-007.ff.company.com:32989] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:44:35 WARN TaskSetManager: Lost task 4066.1 in stage 0.0 (TID
5783, ams03-007.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:44:35 ERROR ConnectionManager: no messageStatus for failed
message id: 541
14/12/11 11:44:35 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 55
14/12/11 11:44:41 WARN TaskSetManager: Lost task 3312.1 in stage 0.0 (TID
5725, ams03-012.ff.company.com): java.io.IOException: sendMessageReliably
failed because ack was not received within 60 sec
       
org.apache.spark.network.nio.ConnectionManager$$anon$8$$anonfun$run$19.apply(ConnectionManager.scala:903)
       
org.apache.spark.network.nio.ConnectionManager$$anon$8$$anonfun$run$19.apply(ConnectionManager.scala:902)
        scala.Option.foreach(Option.scala:236)
       
org.apache.spark.network.nio.ConnectionManager$$anon$8.run(ConnectionManager.scala:902)
        java.util.TimerThread.mainLoop(Timer.java:555)
        java.util.TimerThread.run(Timer.java:505)
14/12/11 11:44:47 WARN TaskSetManager: Lost task 5195.0 in stage 0.0 (TID
5372, ams05-018.ff.company.com): java.lang.OutOfMemoryError: GC overhead
limit exceeded
       
java.io.ObjectOutputStream$HandleTable.growSpine(ObjectOutputStream.java:2333)
       
java.io.ObjectOutputStream$HandleTable.assign(ObjectOutputStream.java:2278)
        java.io.ObjectOutputStream.writeString(ObjectOutputStream.java:1301)
       
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1171)
       
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
       
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
       
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
       
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
       
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
       
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
       
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
       
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
        java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347)
       
org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:42)
       
org.apache.spark.storage.DiskBlockObjectWriter.write(BlockObjectWriter.scala:195)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4$$anonfun$apply$2.apply(ExternalSorter.scala:744)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4$$anonfun$apply$2.apply(ExternalSorter.scala:743)
        scala.collection.Iterator$class.foreach(Iterator.scala:727)
       
org.apache.spark.util.collection.ExternalSorter$IteratorForPartition.foreach(ExternalSorter.scala:801)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4.apply(ExternalSorter.scala:743)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4.apply(ExternalSorter.scala:739)
        scala.collection.Iterator$class.foreach(Iterator.scala:727)
        scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
       
org.apache.spark.util.collection.ExternalSorter.writePartitionedFile(ExternalSorter.scala:739)
       
org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:70)
       
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)
       
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
        org.apache.spark.scheduler.Task.run(Task.scala:56)
       
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:186)
       
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
       
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        java.lang.Thread.run(Thread.java:744)
14/12/11 11:44:50 ERROR YarnClientClusterScheduler: Lost executor 331 on
ams05-018.ff.company.com: remote Akka client disassociated
14/12/11 11:44:50 ERROR ConnectionManager: Corresponding SendingConnection
to ConnectionManagerId(ams05-018.ff.company.com,37419) not found
14/12/11 11:44:50 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams05-018.ff.company.com:46094] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:44:50 WARN TaskSetManager: Lost task 5662.2 in stage 0.0 (TID
5802, ams05-018.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:44:50 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 331
14/12/11 11:44:54 ERROR YarnClientClusterScheduler: Lost executor 142 on
ams04-017.ff.company.com: remote Akka client disassociated
14/12/11 11:44:54 WARN TaskSetManager: Lost task 391.1 in stage 0.0 (TID
4835, ams04-017.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:44:55 WARN TaskSetManager: Lost task 1438.1 in stage 0.0 (TID
5738, ams02-002.ff.company.com): java.io.IOException: sendMessageReliably
failed because ack was not received within 60 sec
       
org.apache.spark.network.nio.ConnectionManager$$anon$8$$anonfun$run$19.apply(ConnectionManager.scala:903)
       
org.apache.spark.network.nio.ConnectionManager$$anon$8$$anonfun$run$19.apply(ConnectionManager.scala:902)
        scala.Option.foreach(Option.scala:236)
       
org.apache.spark.network.nio.ConnectionManager$$anon$8.run(ConnectionManager.scala:902)
        java.util.TimerThread.mainLoop(Timer.java:555)
        java.util.TimerThread.run(Timer.java:505)
14/12/11 11:44:56 ERROR YarnClientClusterScheduler: Lost executor 185 on
ams03-005.ff.company.com: remote Akka client disassociated
14/12/11 11:44:56 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams03-005.ff.company.com:48788] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:44:56 WARN TaskSetManager: Lost task 4218.1 in stage 0.0 (TID
5811, ams03-005.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:44:56 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 185
14/12/11 11:44:56 ERROR SendingConnection: Exception while reading
SendingConnection to ConnectionManagerId(ams04-017.ff.company.com,33319)
java.nio.channels.ClosedChannelException
	at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
	at
org.apache.spark.network.nio.SendingConnection.read(Connection.scala:419)
	at
org.apache.spark.network.nio.ConnectionManager$$anon$10.run(ConnectionManager.scala:238)
	at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
14/12/11 11:44:56 ERROR ConnectionManager: Exception while sending message.
java.nio.channels.ClosedChannelException
	at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
	at
org.apache.spark.network.nio.SendingConnection.read(Connection.scala:419)
	at
org.apache.spark.network.nio.ConnectionManager$$anon$10.run(ConnectionManager.scala:238)
	at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
14/12/11 11:44:56 ERROR ConnectionManager: no messageStatus for failed
message id: 437
14/12/11 11:45:00 WARN TaskSetManager: Lost task 4866.0 in stage 0.0 (TID
5223, ams06-010.ff.company.com): java.lang.OutOfMemoryError: GC overhead
limit exceeded
       
org.apache.spark.util.collection.ExternalSorter$IteratorForPartition.next(ExternalSorter.scala:811)
       
org.apache.spark.util.collection.ExternalSorter$IteratorForPartition.next(ExternalSorter.scala:801)
        scala.collection.Iterator$class.foreach(Iterator.scala:727)
       
org.apache.spark.util.collection.ExternalSorter$IteratorForPartition.foreach(ExternalSorter.scala:801)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4.apply(ExternalSorter.scala:743)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4.apply(ExternalSorter.scala:739)
        scala.collection.Iterator$class.foreach(Iterator.scala:727)
        scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
       
org.apache.spark.util.collection.ExternalSorter.writePartitionedFile(ExternalSorter.scala:739)
       
org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:70)
       
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)
       
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
        org.apache.spark.scheduler.Task.run(Task.scala:56)
       
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:186)
       
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
       
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        java.lang.Thread.run(Thread.java:744)
14/12/11 11:45:03 ERROR YarnClientClusterScheduler: Lost executor 327 on
ams06-010.ff.company.com: remote Akka client disassociated
14/12/11 11:45:03 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams06-010.ff.company.com:45244] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:45:03 ERROR SendingConnection: Exception while reading
SendingConnection to ConnectionManagerId(ams06-010.ff.company.com,44945)
java.nio.channels.ClosedChannelException
	at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
	at
org.apache.spark.network.nio.SendingConnection.read(Connection.scala:419)
	at
org.apache.spark.network.nio.ConnectionManager$$anon$10.run(ConnectionManager.scala:238)
	at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
14/12/11 11:45:03 ERROR ConnectionManager: Exception while sending message.
java.nio.channels.ClosedChannelException
	at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
	at
org.apache.spark.network.nio.SendingConnection.read(Connection.scala:419)
	at
org.apache.spark.network.nio.ConnectionManager$$anon$10.run(ConnectionManager.scala:238)
	at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
14/12/11 11:45:03 ERROR ConnectionManager: no messageStatus for failed
message id: 745
14/12/11 11:45:03 WARN TaskSetManager: Lost task 1360.1 in stage 0.0 (TID
5820, ams06-010.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:45:03 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 327
14/12/11 11:45:08 WARN TaskSetManager: Lost task 5342.1 in stage 0.0 (TID
5756, ams03-016.ff.company.com): java.io.IOException: sendMessageReliably
failed because ack was not received within 60 sec
       
org.apache.spark.network.nio.ConnectionManager$$anon$8$$anonfun$run$19.apply(ConnectionManager.scala:903)
       
org.apache.spark.network.nio.ConnectionManager$$anon$8$$anonfun$run$19.apply(ConnectionManager.scala:902)
        scala.Option.foreach(Option.scala:236)
       
org.apache.spark.network.nio.ConnectionManager$$anon$8.run(ConnectionManager.scala:902)
        java.util.TimerThread.mainLoop(Timer.java:555)
        java.util.TimerThread.run(Timer.java:505)
14/12/11 11:45:08 ERROR YarnClientClusterScheduler: Lost executor 125 on
ams03-003.ff.company.com: remote Akka client disassociated
14/12/11 11:45:08 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams03-003.ff.company.com:36163] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:45:08 ERROR SendingConnection: Exception while reading
SendingConnection to ConnectionManagerId(ams03-003.ff.company.com,40805)
java.nio.channels.ClosedChannelException
	at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
	at
org.apache.spark.network.nio.SendingConnection.read(Connection.scala:419)
	at
org.apache.spark.network.nio.ConnectionManager$$anon$10.run(ConnectionManager.scala:238)
	at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
14/12/11 11:45:08 ERROR ConnectionManager: Exception while sending message.
java.nio.channels.ClosedChannelException
	at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
	at
org.apache.spark.network.nio.SendingConnection.read(Connection.scala:419)
	at
org.apache.spark.network.nio.ConnectionManager$$anon$10.run(ConnectionManager.scala:238)
	at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
14/12/11 11:45:08 ERROR ConnectionManager: no messageStatus for failed
message id: 67
14/12/11 11:45:08 WARN TaskSetManager: Lost task 816.1 in stage 0.0 (TID
5828, ams03-003.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:45:08 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 125
14/12/11 11:45:09 ERROR YarnClientClusterScheduler: Lost executor 11 on
ams07-003.ff.company.com: remote Akka client disassociated
14/12/11 11:45:09 WARN TaskSetManager: Lost task 4878.0 in stage 0.0 (TID
5048, ams07-003.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:45:10 ERROR SendingConnection: Exception while reading
SendingConnection to ConnectionManagerId(ams03-009.ff.company.com,40343)
java.nio.channels.ClosedChannelException
	at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
	at
org.apache.spark.network.nio.SendingConnection.read(Connection.scala:419)
	at
org.apache.spark.network.nio.ConnectionManager$$anon$10.run(ConnectionManager.scala:238)
	at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
14/12/11 11:45:10 ERROR YarnClientClusterScheduler: Lost executor 193 on
ams03-009.ff.company.com: remote Akka client disassociated
14/12/11 11:45:10 ERROR ConnectionManager: Exception while sending message.
java.nio.channels.ClosedChannelException
	at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
	at
org.apache.spark.network.nio.SendingConnection.read(Connection.scala:419)
	at
org.apache.spark.network.nio.ConnectionManager$$anon$10.run(ConnectionManager.scala:238)
	at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
14/12/11 11:45:10 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams03-009.ff.company.com:40822] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:45:10 ERROR ConnectionManager: no messageStatus for failed
message id: 729
14/12/11 11:45:10 WARN TaskSetManager: Lost task 3514.1 in stage 0.0 (TID
5833, ams03-009.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:45:10 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 193
14/12/11 11:45:11 ERROR ConnectionManager: Corresponding SendingConnection
to ConnectionManagerId(ams06-013.ff.company.com,35317) not found
14/12/11 11:45:11 ERROR YarnClientClusterScheduler: Lost executor 371 on
ams06-013.ff.company.com: remote Akka client disassociated
14/12/11 11:45:11 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams06-013.ff.company.com:44289] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:45:11 WARN TaskSetManager: Lost task 2315.2 in stage 0.0 (TID
5834, ams06-013.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:45:11 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 371
14/12/11 11:45:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(248, ams03-010.ff.company.com, 48854) with no recent heart
beats: 52009ms exceeds 45000ms
14/12/11 11:45:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(39, ams02-001.ff.company.com, 41513) with no recent heart
beats: 83083ms exceeds 45000ms
14/12/11 11:45:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(342, ams05-013.ff.company.com, 46759) with no recent heart
beats: 53917ms exceeds 45000ms
14/12/11 11:45:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(257, ams04-011.ff.company.com, 48683) with no recent heart
beats: 63547ms exceeds 45000ms
14/12/11 11:45:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(329, ams03-003.ff.company.com, 32768) with no recent heart
beats: 61499ms exceeds 45000ms
14/12/11 11:45:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(243, ams02-001.ff.company.com, 47519) with no recent heart
beats: 58035ms exceeds 45000ms
14/12/11 11:45:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(19, ams02-002.ff.company.com, 38804) with no recent heart
beats: 64592ms exceeds 45000ms
14/12/11 11:45:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(260, ams03-015.ff.company.com, 38515) with no recent heart
beats: 73552ms exceeds 45000ms
14/12/11 11:45:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(43, ams02-006.ff.company.com, 36105) with no recent heart
beats: 77821ms exceeds 45000ms
14/12/11 11:45:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(399, ams07-009.ff.company.com, 36803) with no recent heart
beats: 81176ms exceeds 45000ms
14/12/11 11:45:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(162, ams06-004.ff.company.com, 33267) with no recent heart
beats: 49065ms exceeds 45000ms
14/12/11 11:45:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(170, ams03-006.ff.company.com, 45693) with no recent heart
beats: 66715ms exceeds 45000ms
14/12/11 11:45:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(262, ams04-007.ff.company.com, 33851) with no recent heart
beats: 77720ms exceeds 45000ms
14/12/11 11:45:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(26, ams06-006.ff.company.com, 45208) with no recent heart
beats: 75549ms exceeds 45000ms
14/12/11 11:45:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(154, ams07-014.ff.company.com, 47197) with no recent heart
beats: 71885ms exceeds 45000ms
14/12/11 11:45:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(9, ams07-012.ff.company.com, 38113) with no recent heart
beats: 55843ms exceeds 45000ms
14/12/11 11:45:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(95, ams06-015.ff.company.com, 34770) with no recent heart
beats: 82751ms exceeds 45000ms
14/12/11 11:45:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(309, ams05-015.ff.company.com, 46483) with no recent heart
beats: 65788ms exceeds 45000ms
14/12/11 11:45:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(330, ams07-011.ff.company.com, 35252) with no recent heart
beats: 48259ms exceeds 45000ms
14/12/11 11:45:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(42, ams05-009.ff.company.com, 46950) with no recent heart
beats: 90022ms exceeds 45000ms
14/12/11 11:45:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(148, ams02-014.ff.company.com, 35698) with no recent heart
beats: 95874ms exceeds 45000ms
14/12/11 11:45:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(118, ams07-008.ff.company.com, 34613) with no recent heart
beats: 97277ms exceeds 45000ms
14/12/11 11:45:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(346, ams04-017.ff.company.com, 38528) with no recent heart
beats: 55961ms exceeds 45000ms
14/12/11 11:45:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(305, ams04-016.ff.company.com, 46821) with no recent heart
beats: 56851ms exceeds 45000ms
14/12/11 11:45:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(84, ams02-009.ff.company.com, 35103) with no recent heart
beats: 53127ms exceeds 45000ms
14/12/11 11:45:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(32, ams07-015.ff.company.com, 41299) with no recent heart
beats: 65952ms exceeds 45000ms
14/12/11 11:45:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(282, ams06-012.ff.company.com, 44208) with no recent heart
beats: 74070ms exceeds 45000ms
14/12/11 11:45:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(87, ams06-008.ff.company.com, 37070) with no recent heart
beats: 45694ms exceeds 45000ms
14/12/11 11:45:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(382, ams05-011.ff.company.com, 38205) with no recent heart
beats: 51319ms exceeds 45000ms
14/12/11 11:45:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(375, ams07-013.ff.company.com, 48121) with no recent heart
beats: 45341ms exceeds 45000ms
14/12/11 11:45:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(345, ams02-001.ff.company.com, 42078) with no recent heart
beats: 89922ms exceeds 45000ms
14/12/11 11:45:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(392, ams04-001.ff.company.com, 45923) with no recent heart
beats: 49656ms exceeds 45000ms
14/12/11 11:45:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(127, ams05-018.ff.company.com, 49580) with no recent heart
beats: 46871ms exceeds 45000ms
14/12/11 11:45:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(275, ams02-018.ff.company.com, 46695) with no recent heart
beats: 45623ms exceeds 45000ms
14/12/11 11:45:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(44, ams03-010.ff.company.com, 46150) with no recent heart
beats: 74156ms exceeds 45000ms
14/12/11 11:45:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(123, ams06-010.ff.company.com, 49691) with no recent heart
beats: 98141ms exceeds 45000ms
14/12/11 11:45:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(140, ams04-008.ff.company.com, 36396) with no recent heart
beats: 92354ms exceeds 45000ms
14/12/11 11:45:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(325, ams02-002.ff.company.com, 40604) with no recent heart
beats: 91079ms exceeds 45000ms
14/12/11 11:45:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(66, ams03-014.ff.company.com, 49157) with no recent heart
beats: 49750ms exceeds 45000ms
14/12/11 11:45:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(409, ams06-005.ff.company.com, 48628) with no recent heart
beats: 88496ms exceeds 45000ms
14/12/11 11:45:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(149, ams07-010.ff.company.com, 49636) with no recent heart
beats: 47206ms exceeds 45000ms
14/12/11 11:45:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(204, ams03-018.ff.company.com, 33554) with no recent heart
beats: 84894ms exceeds 45000ms
14/12/11 11:45:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(295, ams03-009.ff.company.com, 35589) with no recent heart
beats: 63907ms exceeds 45000ms
14/12/11 11:45:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(237, ams07-007.ff.company.com, 42002) with no recent heart
beats: 93861ms exceeds 45000ms
14/12/11 11:45:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(143, ams05-004.ff.company.com, 34168) with no recent heart
beats: 74124ms exceeds 45000ms
14/12/11 11:45:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(62, ams05-016.ff.company.com, 48895) with no recent heart
beats: 79005ms exceeds 45000ms
14/12/11 11:45:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(364, ams04-007.ff.company.com, 49087) with no recent heart
beats: 99473ms exceeds 45000ms
14/12/11 11:45:12 ERROR SendingConnection: Exception while reading
SendingConnection to ConnectionManagerId(ams03-015.ff.company.com,48986)
java.nio.channels.ClosedChannelException
	at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
	at
org.apache.spark.network.nio.SendingConnection.read(Connection.scala:419)
	at
org.apache.spark.network.nio.ConnectionManager$$anon$10.run(ConnectionManager.scala:238)
	at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
14/12/11 11:45:12 ERROR YarnClientClusterScheduler: Lost executor 362 on
ams03-015.ff.company.com: remote Akka client disassociated
14/12/11 11:45:12 ERROR ConnectionManager: Exception while sending message.
java.nio.channels.ClosedChannelException
	at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
	at
org.apache.spark.network.nio.SendingConnection.read(Connection.scala:419)
	at
org.apache.spark.network.nio.ConnectionManager$$anon$10.run(ConnectionManager.scala:238)
	at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
14/12/11 11:45:12 ERROR ConnectionManager: no messageStatus for failed
message id: 287
14/12/11 11:45:12 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams03-015.ff.company.com:42765] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:45:12 WARN TaskSetManager: Lost task 1156.2 in stage 0.0 (TID
5836, ams03-015.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:45:12 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 362
14/12/11 11:45:12 ERROR SendingConnection: Exception while reading
SendingConnection to ConnectionManagerId(ams07-003.ff.company.com,43976)
java.nio.channels.ClosedChannelException
	at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
	at
org.apache.spark.network.nio.SendingConnection.read(Connection.scala:419)
	at
org.apache.spark.network.nio.ConnectionManager$$anon$10.run(ConnectionManager.scala:238)
	at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
14/12/11 11:45:12 ERROR ConnectionManager: Exception while sending message.
java.nio.channels.ClosedChannelException
	at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
	at
org.apache.spark.network.nio.SendingConnection.read(Connection.scala:419)
	at
org.apache.spark.network.nio.ConnectionManager$$anon$10.run(ConnectionManager.scala:238)
	at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
14/12/11 11:45:12 ERROR ConnectionManager: no messageStatus for failed
message id: 220
14/12/11 11:45:20 WARN TaskSetManager: Lost task 755.3 in stage 0.0 (TID
5768, ams07-008.ff.company.com): java.io.IOException: sendMessageReliably
failed because ack was not received within 60 sec
       
org.apache.spark.network.nio.ConnectionManager$$anon$8$$anonfun$run$19.apply(ConnectionManager.scala:903)
       
org.apache.spark.network.nio.ConnectionManager$$anon$8$$anonfun$run$19.apply(ConnectionManager.scala:902)
        scala.Option.foreach(Option.scala:236)
       
org.apache.spark.network.nio.ConnectionManager$$anon$8.run(ConnectionManager.scala:902)
        java.util.TimerThread.mainLoop(Timer.java:555)
        java.util.TimerThread.run(Timer.java:505)
14/12/11 11:45:26 WARN TaskSetManager: Lost task 4670.0 in stage 0.0 (TID
5103, ams03-017.ff.company.com): java.lang.OutOfMemoryError: GC overhead
limit exceeded
        java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:322)
       
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1133)
       
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
       
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
       
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
       
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
       
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
       
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
       
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
       
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
        java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347)
       
org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:42)
       
org.apache.spark.storage.DiskBlockObjectWriter.write(BlockObjectWriter.scala:195)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4$$anonfun$apply$2.apply(ExternalSorter.scala:744)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4$$anonfun$apply$2.apply(ExternalSorter.scala:743)
        scala.collection.Iterator$class.foreach(Iterator.scala:727)
       
org.apache.spark.util.collection.ExternalSorter$IteratorForPartition.foreach(ExternalSorter.scala:801)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4.apply(ExternalSorter.scala:743)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4.apply(ExternalSorter.scala:739)
        scala.collection.Iterator$class.foreach(Iterator.scala:727)
        scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
       
org.apache.spark.util.collection.ExternalSorter.writePartitionedFile(ExternalSorter.scala:739)
       
org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:70)
       
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)
       
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
        org.apache.spark.scheduler.Task.run(Task.scala:56)
       
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:186)
       
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
       
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        java.lang.Thread.run(Thread.java:744)
14/12/11 11:45:28 ERROR YarnClientClusterScheduler: Lost executor 219 on
ams03-017.ff.company.com: remote Akka client disassociated
14/12/11 11:45:28 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams03-017.ff.company.com:33828] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:45:28 WARN TaskSetManager: Lost task 6150.2 in stage 0.0 (TID
5856, ams03-017.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:45:28 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 219
14/12/11 11:45:30 ERROR ConnectionManager: Corresponding SendingConnection
to ConnectionManagerId(ams03-014.ff.company.com,42029) not found
14/12/11 11:45:30 ERROR YarnClientClusterScheduler: Lost executor 168 on
ams03-014.ff.company.com: remote Akka client disassociated
14/12/11 11:45:30 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams03-014.ff.company.com:34313] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:45:30 WARN TaskSetManager: Lost task 3114.1 in stage 0.0 (TID
5862, ams03-014.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:45:30 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 168
14/12/11 11:45:31 WARN TaskSetManager: Lost task 4658.1 in stage 0.0 (TID
5780, ams04-005.ff.company.com): java.io.IOException: sendMessageReliably
failed because ack was not received within 60 sec
       
org.apache.spark.network.nio.ConnectionManager$$anon$8$$anonfun$run$19.apply(ConnectionManager.scala:903)
       
org.apache.spark.network.nio.ConnectionManager$$anon$8$$anonfun$run$19.apply(ConnectionManager.scala:902)
        scala.Option.foreach(Option.scala:236)
       
org.apache.spark.network.nio.ConnectionManager$$anon$8.run(ConnectionManager.scala:902)
        java.util.TimerThread.mainLoop(Timer.java:555)
        java.util.TimerThread.run(Timer.java:505)
14/12/11 11:45:31 ERROR YarnClientClusterScheduler: Lost executor 356 on
ams03-002.ff.company.com: remote Akka client disassociated
14/12/11 11:45:31 ERROR ConnectionManager: Corresponding SendingConnection
to ConnectionManagerId(ams03-002.ff.company.com,36528) not found
14/12/11 11:45:31 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams03-002.ff.company.com:47919] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:45:31 WARN TaskSetManager: Lost task 2404.1 in stage 0.0 (TID
5864, ams03-002.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:45:31 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 356
14/12/11 11:45:36 ERROR SendingConnection: Exception while reading
SendingConnection to ConnectionManagerId(ams04-016.ff.company.com,46821)
java.nio.channels.ClosedChannelException
	at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
	at
org.apache.spark.network.nio.SendingConnection.read(Connection.scala:419)
	at
org.apache.spark.network.nio.ConnectionManager$$anon$10.run(ConnectionManager.scala:238)
	at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
14/12/11 11:45:36 ERROR YarnClientClusterScheduler: Lost executor 305 on
ams04-016.ff.company.com: remote Akka client disassociated
14/12/11 11:45:36 ERROR ConnectionManager: Exception while sending message.
java.nio.channels.ClosedChannelException
	at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
	at
org.apache.spark.network.nio.SendingConnection.read(Connection.scala:419)
	at
org.apache.spark.network.nio.ConnectionManager$$anon$10.run(ConnectionManager.scala:238)
	at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
14/12/11 11:45:36 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams04-016.ff.company.com:47395] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:45:36 ERROR ConnectionManager: no messageStatus for failed
message id: 807
14/12/11 11:45:36 WARN TaskSetManager: Lost task 1703.1 in stage 0.0 (TID
5874, ams04-016.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:45:36 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 305
14/12/11 11:45:37 WARN TaskSetManager: Lost task 5156.0 in stage 0.0 (TID
5291, ams07-008.ff.company.com): java.lang.OutOfMemoryError: GC overhead
limit exceeded
       
java.io.ObjectOutputStream$HandleTable.growEntries(ObjectOutputStream.java:2350)
       
java.io.ObjectOutputStream$HandleTable.assign(ObjectOutputStream.java:2275)
        java.io.ObjectOutputStream.writeString(ObjectOutputStream.java:1301)
       
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1171)
       
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
       
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
       
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
       
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
       
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
       
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
       
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
       
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
        java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347)
       
org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:42)
       
org.apache.spark.storage.DiskBlockObjectWriter.write(BlockObjectWriter.scala:195)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4$$anonfun$apply$2.apply(ExternalSorter.scala:744)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4$$anonfun$apply$2.apply(ExternalSorter.scala:743)
        scala.collection.Iterator$class.foreach(Iterator.scala:727)
       
org.apache.spark.util.collection.ExternalSorter$IteratorForPartition.foreach(ExternalSorter.scala:801)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4.apply(ExternalSorter.scala:743)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4.apply(ExternalSorter.scala:739)
        scala.collection.Iterator$class.foreach(Iterator.scala:727)
        scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
       
org.apache.spark.util.collection.ExternalSorter.writePartitionedFile(ExternalSorter.scala:739)
       
org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:70)
       
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)
       
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
        org.apache.spark.scheduler.Task.run(Task.scala:56)
       
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:186)
       
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
       
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        java.lang.Thread.run(Thread.java:744)
14/12/11 11:45:40 ERROR YarnClientClusterScheduler: Lost executor 322 on
ams07-008.ff.company.com: remote Akka client disassociated
14/12/11 11:45:40 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams07-008.ff.company.com:42539] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:45:40 WARN TaskSetManager: Lost task 742.3 in stage 0.0 (TID
5878, ams07-008.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:45:40 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 322
14/12/11 11:45:41 WARN TaskSetManager: Lost task 3299.1 in stage 0.0 (TID
5793, ams03-012.ff.company.com): java.io.IOException: sendMessageReliably
failed because ack was not received within 60 sec
       
org.apache.spark.network.nio.ConnectionManager$$anon$8$$anonfun$run$19.apply(ConnectionManager.scala:903)
       
org.apache.spark.network.nio.ConnectionManager$$anon$8$$anonfun$run$19.apply(ConnectionManager.scala:902)
        scala.Option.foreach(Option.scala:236)
       
org.apache.spark.network.nio.ConnectionManager$$anon$8.run(ConnectionManager.scala:902)
        java.util.TimerThread.mainLoop(Timer.java:555)
        java.util.TimerThread.run(Timer.java:505)
14/12/11 11:45:45 ERROR SendingConnection: Exception while reading
SendingConnection to ConnectionManagerId(ams04-004.ff.company.com,46375)
java.nio.channels.ClosedChannelException
	at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
	at
org.apache.spark.network.nio.SendingConnection.read(Connection.scala:419)
	at
org.apache.spark.network.nio.ConnectionManager$$anon$10.run(ConnectionManager.scala:238)
	at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
14/12/11 11:45:45 ERROR ConnectionManager: Exception while sending message.
java.nio.channels.ClosedChannelException
	at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
	at
org.apache.spark.network.nio.SendingConnection.read(Connection.scala:419)
	at
org.apache.spark.network.nio.ConnectionManager$$anon$10.run(ConnectionManager.scala:238)
	at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
14/12/11 11:45:45 ERROR YarnClientClusterScheduler: Lost executor 340 on
ams04-004.ff.company.com: remote Akka client disassociated
14/12/11 11:45:45 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams04-004.ff.company.com:36967] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:45:45 ERROR ConnectionManager: no messageStatus for failed
message id: 356
14/12/11 11:45:45 WARN TaskSetManager: Lost task 374.1 in stage 0.0 (TID
5884, ams04-004.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:45:45 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 340
14/12/11 11:45:46 ERROR SendingConnection: Exception while reading
SendingConnection to ConnectionManagerId(ams02-011.ff.company.com,39392)
java.nio.channels.ClosedChannelException
	at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
	at
org.apache.spark.network.nio.SendingConnection.read(Connection.scala:419)
	at
org.apache.spark.network.nio.ConnectionManager$$anon$10.run(ConnectionManager.scala:238)
	at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
14/12/11 11:45:46 ERROR YarnClientClusterScheduler: Lost executor 209 on
ams02-011.ff.company.com: remote Akka client disassociated
14/12/11 11:45:46 ERROR ConnectionManager: Exception while sending message.
java.nio.channels.ClosedChannelException
	at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
	at
org.apache.spark.network.nio.SendingConnection.read(Connection.scala:419)
	at
org.apache.spark.network.nio.ConnectionManager$$anon$10.run(ConnectionManager.scala:238)
	at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
14/12/11 11:45:46 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams02-011.ff.company.com:36974] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:45:46 ERROR ConnectionManager: no messageStatus for failed
message id: 143
14/12/11 11:45:46 WARN TaskSetManager: Lost task 661.1 in stage 0.0 (TID
5885, ams02-011.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:45:46 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 209
14/12/11 11:45:51 WARN TaskSetManager: Lost task 5031.0 in stage 0.0 (TID
5218, ams07-014.ff.company.com): java.lang.OutOfMemoryError: GC overhead
limit exceeded
       
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1536)
       
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
       
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
       
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
        java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347)
       
org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:42)
       
org.apache.spark.storage.DiskBlockObjectWriter.write(BlockObjectWriter.scala:195)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4$$anonfun$apply$2.apply(ExternalSorter.scala:744)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4$$anonfun$apply$2.apply(ExternalSorter.scala:743)
        scala.collection.Iterator$class.foreach(Iterator.scala:727)
       
org.apache.spark.util.collection.ExternalSorter$IteratorForPartition.foreach(ExternalSorter.scala:801)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4.apply(ExternalSorter.scala:743)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4.apply(ExternalSorter.scala:739)
        scala.collection.Iterator$class.foreach(Iterator.scala:727)
        scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
       
org.apache.spark.util.collection.ExternalSorter.writePartitionedFile(ExternalSorter.scala:739)
       
org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:70)
       
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)
       
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
        org.apache.spark.scheduler.Task.run(Task.scala:56)
       
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:186)
       
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
       
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        java.lang.Thread.run(Thread.java:744)
14/12/11 11:45:54 ERROR SendingConnection: Exception while reading
SendingConnection to ConnectionManagerId(ams02-007.ff.company.com,39870)
java.nio.channels.ClosedChannelException
	at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
	at
org.apache.spark.network.nio.SendingConnection.read(Connection.scala:419)
	at
org.apache.spark.network.nio.ConnectionManager$$anon$10.run(ConnectionManager.scala:238)
	at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
14/12/11 11:45:54 ERROR ConnectionManager: Exception while sending message.
java.nio.channels.ClosedChannelException
	at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
	at
org.apache.spark.network.nio.SendingConnection.read(Connection.scala:419)
	at
org.apache.spark.network.nio.ConnectionManager$$anon$10.run(ConnectionManager.scala:238)
	at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
14/12/11 11:45:54 ERROR YarnClientClusterScheduler: Lost executor 131 on
ams02-007.ff.company.com: remote Akka client disassociated
14/12/11 11:45:54 ERROR ConnectionManager: no messageStatus for failed
message id: 818
14/12/11 11:45:54 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams02-007.ff.company.com:43427] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:45:54 WARN TaskSetManager: Lost task 5765.0 in stage 0.0 (TID
5899, ams02-007.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:45:54 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 131
14/12/11 11:45:54 ERROR SendingConnection: Exception while reading
SendingConnection to ConnectionManagerId(ams07-014.ff.company.com,36630)
java.nio.channels.ClosedChannelException
	at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
	at
org.apache.spark.network.nio.SendingConnection.read(Connection.scala:419)
	at
org.apache.spark.network.nio.ConnectionManager$$anon$10.run(ConnectionManager.scala:238)
	at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
14/12/11 11:45:54 ERROR YarnClientClusterScheduler: Lost executor 256 on
ams07-014.ff.company.com: remote Akka client disassociated
14/12/11 11:45:54 ERROR ConnectionManager: Exception while sending message.
java.nio.channels.ClosedChannelException
	at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
	at
org.apache.spark.network.nio.SendingConnection.read(Connection.scala:419)
	at
org.apache.spark.network.nio.ConnectionManager$$anon$10.run(ConnectionManager.scala:238)
	at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
14/12/11 11:45:54 ERROR ConnectionManager: no messageStatus for failed
message id: 723
14/12/11 11:45:54 WARN TaskSetManager: Lost task 1790.1 in stage 0.0 (TID
5896, ams07-014.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:45:54 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams07-014.ff.company.com:49670] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:45:54 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 256
14/12/11 11:45:54 WARN TaskSetManager: Lost task 4067.1 in stage 0.0 (TID
5810, ams04-018.ff.company.com): java.io.IOException: sendMessageReliably
failed because ack was not received within 60 sec
       
org.apache.spark.network.nio.ConnectionManager$$anon$8$$anonfun$run$19.apply(ConnectionManager.scala:903)
       
org.apache.spark.network.nio.ConnectionManager$$anon$8$$anonfun$run$19.apply(ConnectionManager.scala:902)
        scala.Option.foreach(Option.scala:236)
       
org.apache.spark.network.nio.ConnectionManager$$anon$8.run(ConnectionManager.scala:902)
        java.util.TimerThread.mainLoop(Timer.java:555)
        java.util.TimerThread.run(Timer.java:505)
14/12/11 11:45:55 ERROR SendingConnection: Exception while reading
SendingConnection to ConnectionManagerId(ams02-002.ff.company.com,43949)
java.nio.channels.ClosedChannelException
	at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
	at
org.apache.spark.network.nio.SendingConnection.read(Connection.scala:419)
	at
org.apache.spark.network.nio.ConnectionManager$$anon$10.run(ConnectionManager.scala:238)
	at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
14/12/11 11:45:55 ERROR YarnClientClusterScheduler: Lost executor 223 on
ams02-002.ff.company.com: remote Akka client disassociated
14/12/11 11:45:55 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams02-002.ff.company.com:33909] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:45:55 ERROR ConnectionManager: Exception while sending message.
java.nio.channels.ClosedChannelException
	at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
	at
org.apache.spark.network.nio.SendingConnection.read(Connection.scala:419)
	at
org.apache.spark.network.nio.ConnectionManager$$anon$10.run(ConnectionManager.scala:238)
	at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
14/12/11 11:45:55 ERROR ConnectionManager: no messageStatus for failed
message id: 803
14/12/11 11:45:55 WARN TaskSetManager: Lost task 1430.1 in stage 0.0 (TID
5902, ams02-002.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:45:55 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 223
14/12/11 11:45:57 ERROR YarnClientClusterScheduler: Lost executor 58 on
ams04-007.ff.company.com: remote Akka client disassociated
14/12/11 11:45:57 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams04-007.ff.company.com:48221] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:45:57 WARN TaskSetManager: Lost task 5137.1 in stage 0.0 (TID
5905, ams04-007.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:45:57 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 58
14/12/11 11:46:01 ERROR YarnClientClusterScheduler: Lost executor 115 on
ams04-013.ff.company.com: remote Akka client disassociated
14/12/11 11:46:01 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams04-013.ff.company.com:44645] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:46:01 WARN TaskSetManager: Lost task 1676.2 in stage 0.0 (TID
5920, ams04-013.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:46:01 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 115
14/12/11 11:46:03 WARN TaskSetManager: Lost task 4837.0 in stage 0.0 (TID
5217, ams06-006.ff.company.com): java.lang.OutOfMemoryError: GC overhead
limit exceeded
        java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:322)
       
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1133)
       
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
       
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
       
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
       
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
       
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
       
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
       
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
       
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
        java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347)
       
org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:42)
       
org.apache.spark.storage.DiskBlockObjectWriter.write(BlockObjectWriter.scala:195)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4$$anonfun$apply$2.apply(ExternalSorter.scala:744)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4$$anonfun$apply$2.apply(ExternalSorter.scala:743)
        scala.collection.Iterator$class.foreach(Iterator.scala:727)
       
org.apache.spark.util.collection.ExternalSorter$IteratorForPartition.foreach(ExternalSorter.scala:801)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4.apply(ExternalSorter.scala:743)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4.apply(ExternalSorter.scala:739)
        scala.collection.Iterator$class.foreach(Iterator.scala:727)
        scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
       
org.apache.spark.util.collection.ExternalSorter.writePartitionedFile(ExternalSorter.scala:739)
       
org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:70)
       
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)
       
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
        org.apache.spark.scheduler.Task.run(Task.scala:56)
       
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:186)
       
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
       
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        java.lang.Thread.run(Thread.java:744)
14/12/11 11:46:05 WARN TaskSetManager: Lost task 1616.1 in stage 0.0 (TID
5824, ams02-010.ff.company.com): java.io.IOException: sendMessageReliably
failed because ack was not received within 60 sec
       
org.apache.spark.network.nio.ConnectionManager$$anon$8$$anonfun$run$19.apply(ConnectionManager.scala:903)
       
org.apache.spark.network.nio.ConnectionManager$$anon$8$$anonfun$run$19.apply(ConnectionManager.scala:902)
        scala.Option.foreach(Option.scala:236)
       
org.apache.spark.network.nio.ConnectionManager$$anon$8.run(ConnectionManager.scala:902)
        java.util.TimerThread.mainLoop(Timer.java:555)
        java.util.TimerThread.run(Timer.java:505)
14/12/11 11:46:06 ERROR YarnClientClusterScheduler: Lost executor 271 on
ams04-002.ff.company.com: remote Akka client disassociated
14/12/11 11:46:06 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams04-002.ff.company.com:35273] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:46:06 WARN TaskSetManager: Lost task 3369.1 in stage 0.0 (TID
5929, ams04-002.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:46:06 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 271
14/12/11 11:46:06 ERROR SendingConnection: Exception while reading
SendingConnection to ConnectionManagerId(ams06-006.ff.company.com,36865)
java.nio.channels.ClosedChannelException
	at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
	at
org.apache.spark.network.nio.SendingConnection.read(Connection.scala:419)
	at
org.apache.spark.network.nio.ConnectionManager$$anon$10.run(ConnectionManager.scala:238)
	at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
14/12/11 11:46:06 ERROR YarnClientClusterScheduler: Lost executor 230 on
ams06-006.ff.company.com: remote Akka client disassociated
14/12/11 11:46:06 ERROR ConnectionManager: Exception while sending message.
java.nio.channels.ClosedChannelException
	at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
	at
org.apache.spark.network.nio.SendingConnection.read(Connection.scala:419)
	at
org.apache.spark.network.nio.ConnectionManager$$anon$10.run(ConnectionManager.scala:238)
	at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
14/12/11 11:46:06 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams06-006.ff.company.com:34794] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:46:06 ERROR ConnectionManager: no messageStatus for failed
message id: 407
14/12/11 11:46:06 WARN TaskSetManager: Lost task 713.1 in stage 0.0 (TID
5928, ams06-006.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:46:06 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 230
14/12/11 11:46:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(31, ams03-016.ff.company.com, 35679) with no recent heart
beats: 67298ms exceeds 45000ms
14/12/11 11:46:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(111, ams07-012.ff.company.com, 33898) with no recent heart
beats: 69496ms exceeds 45000ms
14/12/11 11:46:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(86, ams04-001.ff.company.com, 42281) with no recent heart
beats: 57690ms exceeds 45000ms
14/12/11 11:46:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(199, ams06-007.ff.company.com, 42191) with no recent heart
beats: 55775ms exceeds 45000ms
14/12/11 11:46:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(231, ams07-002.ff.company.com, 43118) with no recent heart
beats: 60260ms exceeds 45000ms
14/12/11 11:46:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(222, ams05-001.ff.company.com, 43298) with no recent heart
beats: 61485ms exceeds 45000ms
14/12/11 11:46:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(377, ams02-018.ff.company.com, 37743) with no recent heart
beats: 79057ms exceeds 45000ms
14/12/11 11:46:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(311, ams02-011.ff.company.com, 45750) with no recent heart
beats: 51776ms exceeds 45000ms
14/12/11 11:46:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(278, ams04-015.ff.company.com, 44067) with no recent heart
beats: 50376ms exceeds 45000ms
14/12/11 11:46:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(29, ams02-007.ff.company.com, 41374) with no recent heart
beats: 60999ms exceeds 45000ms
14/12/11 11:46:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(334, ams02-015.ff.company.com, 41753) with no recent heart
beats: 47364ms exceeds 45000ms
14/12/11 11:46:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(259, ams03-007.ff.company.com, 42693) with no recent heart
beats: 79956ms exceeds 45000ms
14/12/11 11:46:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(372, ams03-014.ff.company.com, 34671) with no recent heart
beats: 65436ms exceeds 45000ms
14/12/11 11:46:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(347, ams05-004.ff.company.com, 43788) with no recent heart
beats: 97571ms exceeds 45000ms
14/12/11 11:46:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(48, ams06-014.ff.company.com, 39829) with no recent heart
beats: 54585ms exceeds 45000ms
14/12/11 11:46:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(18, ams05-001.ff.company.com, 47585) with no recent heart
beats: 92109ms exceeds 45000ms
14/12/11 11:46:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(328, ams05-014.ff.company.com, 46641) with no recent heart
beats: 50316ms exceeds 45000ms
14/12/11 11:46:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(367, ams05-008.ff.company.com, 46649) with no recent heart
beats: 103934ms exceeds 45000ms
14/12/11 11:46:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(171, ams07-013.ff.company.com, 34033) with no recent heart
beats: 57573ms exceeds 45000ms
14/12/11 11:46:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(285, ams02-012.ff.company.com, 47738) with no recent heart
beats: 77673ms exceeds 45000ms
14/12/11 11:46:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(64, ams02-005.ff.company.com, 37446) with no recent heart
beats: 54808ms exceeds 45000ms
14/12/11 11:46:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(247, ams02-006.ff.company.com, 34425) with no recent heart
beats: 94065ms exceeds 45000ms
14/12/11 11:46:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(326, ams04-009.ff.company.com, 42986) with no recent heart
beats: 59846ms exceeds 45000ms
14/12/11 11:46:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(34, ams04-004.ff.company.com, 38722) with no recent heart
beats: 102073ms exceeds 45000ms
14/12/11 11:46:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(250, ams02-014.ff.company.com, 49267) with no recent heart
beats: 58329ms exceeds 45000ms
14/12/11 11:46:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(220, ams07-008.ff.company.com, 34192) with no recent heart
beats: 73227ms exceeds 45000ms
14/12/11 11:46:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(210, ams05-006.ff.company.com, 45834) with no recent heart
beats: 78363ms exceeds 45000ms
14/12/11 11:46:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(408, ams06-011.ff.company.com, 38264) with no recent heart
beats: 46099ms exceeds 45000ms
14/12/11 11:46:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(297, ams07-009.ff.company.com, 42141) with no recent heart
beats: 93427ms exceeds 45000ms
14/12/11 11:46:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(391, ams07-004.ff.company.com, 45193) with no recent heart
beats: 56851ms exceeds 45000ms
14/12/11 11:46:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(253, ams05-017.ff.company.com, 34291) with no recent heart
beats: 68842ms exceeds 45000ms
14/12/11 11:46:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(242, ams04-008.ff.company.com, 34453) with no recent heart
beats: 82566ms exceeds 45000ms
14/12/11 11:46:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(23, ams03-003.ff.company.com, 36277) with no recent heart
beats: 65167ms exceeds 45000ms
14/12/11 11:46:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(252, ams06-014.ff.company.com, 35864) with no recent heart
beats: 79931ms exceeds 45000ms
14/12/11 11:46:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(215, ams07-003.ff.company.com, 45138) with no recent heart
beats: 59335ms exceeds 45000ms
14/12/11 11:46:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(83, ams03-005.ff.company.com, 49567) with no recent heart
beats: 65344ms exceeds 45000ms
14/12/11 11:46:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(368, ams05-016.ff.company.com, 37847) with no recent heart
beats: 61355ms exceeds 45000ms
14/12/11 11:46:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(15, ams03-017.ff.company.com, 34182) with no recent heart
beats: 77764ms exceeds 45000ms
14/12/11 11:46:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(113, ams07-003.ff.company.com, 35568) with no recent heart
beats: 66124ms exceeds 45000ms
14/12/11 11:46:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(206, ams06-002.ff.company.com, 49212) with no recent heart
beats: 51212ms exceeds 45000ms
14/12/11 11:46:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(136, ams04-004.ff.company.com, 39548) with no recent heart
beats: 77925ms exceeds 45000ms
14/12/11 11:46:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(333, ams07-002.ff.company.com, 43942) with no recent heart
beats: 51098ms exceeds 45000ms
14/12/11 11:46:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(358, ams07-014.ff.company.com, 36087) with no recent heart
beats: 79695ms exceeds 45000ms
14/12/11 11:46:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(244, ams04-017.ff.company.com, 40014) with no recent heart
beats: 72141ms exceeds 45000ms
14/12/11 11:46:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(383, ams05-003.ff.company.com, 33745) with no recent heart
beats: 63753ms exceeds 45000ms
14/12/11 11:46:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(192, ams05-010.ff.company.com, 48977) with no recent heart
beats: 55525ms exceeds 45000ms
14/12/11 11:46:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(35, ams06-001.ff.company.com, 38520) with no recent heart
beats: 103732ms exceeds 45000ms
14/12/11 11:46:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(234, ams04-012.ff.company.com, 35147) with no recent heart
beats: 50933ms exceeds 45000ms
14/12/11 11:46:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(353, ams07-010.ff.company.com, 43811) with no recent heart
beats: 53951ms exceeds 45000ms
14/12/11 11:46:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(394, ams02-017.ff.company.com, 35148) with no recent heart
beats: 67442ms exceeds 45000ms
14/12/11 11:46:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(369, ams02-013.ff.company.com, 45611) with no recent heart
beats: 99398ms exceeds 45000ms
14/12/11 11:46:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(349, ams02-006.ff.company.com, 49386) with no recent heart
beats: 56665ms exceeds 45000ms
14/12/11 11:46:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(281, ams05-003.ff.company.com, 38694) with no recent heart
beats: 54604ms exceeds 45000ms
14/12/11 11:46:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(343, ams02-010.ff.company.com, 37899) with no recent heart
beats: 77142ms exceeds 45000ms
14/12/11 11:46:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(224, ams04-009.ff.company.com, 41920) with no recent heart
beats: 100230ms exceeds 45000ms
14/12/11 11:46:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(160, ams04-007.ff.company.com, 33817) with no recent heart
beats: 69036ms exceeds 45000ms
14/12/11 11:46:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(51, ams07-006.ff.company.com, 48603) with no recent heart
beats: 49702ms exceeds 45000ms
14/12/11 11:46:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(184, ams06-003.ff.company.com, 36051) with no recent heart
beats: 65510ms exceeds 45000ms
14/12/11 11:46:13 ERROR SendingConnection: Exception while reading
SendingConnection to ConnectionManagerId(ams05-013.ff.company.com,46759)
java.nio.channels.ClosedChannelException
	at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
	at
org.apache.spark.network.nio.SendingConnection.read(Connection.scala:419)
	at
org.apache.spark.network.nio.ConnectionManager$$anon$10.run(ConnectionManager.scala:238)
	at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
14/12/11 11:46:13 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams05-013.ff.company.com:41707] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:46:13 ERROR ConnectionManager: Exception while sending message.
java.nio.channels.ClosedChannelException
	at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
	at
org.apache.spark.network.nio.SendingConnection.read(Connection.scala:419)
	at
org.apache.spark.network.nio.ConnectionManager$$anon$10.run(ConnectionManager.scala:238)
	at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
14/12/11 11:46:13 ERROR YarnClientClusterScheduler: Lost executor 342 on
ams05-013.ff.company.com: remote Akka client disassociated
14/12/11 11:46:13 ERROR ConnectionManager: no messageStatus for failed
message id: 457
14/12/11 11:46:13 WARN TaskSetManager: Lost task 647.1 in stage 0.0 (TID
5944, ams05-013.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:46:13 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 342
14/12/11 11:46:13 WARN TaskSetManager: Lost task 851.0 in stage 1.0 (TID
5840, ams05-015.ff.company.com): java.io.IOException: sendMessageReliably
failed because ack was not received within 60 sec
       
org.apache.spark.network.nio.ConnectionManager$$anon$8$$anonfun$run$19.apply(ConnectionManager.scala:903)
       
org.apache.spark.network.nio.ConnectionManager$$anon$8$$anonfun$run$19.apply(ConnectionManager.scala:902)
        scala.Option.foreach(Option.scala:236)
       
org.apache.spark.network.nio.ConnectionManager$$anon$8.run(ConnectionManager.scala:902)
        java.util.TimerThread.mainLoop(Timer.java:555)
        java.util.TimerThread.run(Timer.java:505)
14/12/11 11:46:13 ERROR ConnectionManager: Exception while sending message.
java.nio.channels.CancelledKeyException
	at
org.apache.spark.network.nio.ConnectionManager.run(ConnectionManager.scala:437)
	at
org.apache.spark.network.nio.ConnectionManager$$anon$7.run(ConnectionManager.scala:170)
14/12/11 11:46:13 ERROR ConnectionManager: no messageStatus for failed
message id: 235
14/12/11 11:46:13 ERROR YarnClientClusterScheduler: Lost executor 166 on
ams02-005.ff.company.com: remote Akka client disassociated
14/12/11 11:46:13 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams02-005.ff.company.com:41749] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:46:13 WARN TaskSetManager: Lost task 5821.0 in stage 0.0 (TID
5945, ams02-005.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:46:13 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 166
14/12/11 11:46:17 ERROR YarnClientClusterScheduler: Lost executor 384 on
ams06-012.ff.company.com: remote Akka client disassociated
14/12/11 11:46:17 WARN TaskSetManager: Lost task 5121.0 in stage 0.0 (TID
5355, ams06-012.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:46:18 WARN TaskSetManager: Lost task 1156.3 in stage 0.0 (TID
5843, ams05-005.ff.company.com): java.io.IOException: sendMessageReliably
failed because ack was not received within 60 sec
       
org.apache.spark.network.nio.ConnectionManager$$anon$8$$anonfun$run$19.apply(ConnectionManager.scala:903)
       
org.apache.spark.network.nio.ConnectionManager$$anon$8$$anonfun$run$19.apply(ConnectionManager.scala:902)
        scala.Option.foreach(Option.scala:236)
       
org.apache.spark.network.nio.ConnectionManager$$anon$8.run(ConnectionManager.scala:902)
        java.util.TimerThread.mainLoop(Timer.java:555)
        java.util.TimerThread.run(Timer.java:505)
14/12/11 11:46:18 WARN TaskSetManager: Lost task 7453.0 in stage 0.0 (TID
5099, ams05-015.ff.company.com): java.lang.OutOfMemoryError: GC overhead
limit exceeded
        java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:322)
       
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1133)
       
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
       
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
       
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
       
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
       
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
       
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
       
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
       
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
        java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347)
       
org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:42)
       
org.apache.spark.storage.DiskBlockObjectWriter.write(BlockObjectWriter.scala:195)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4$$anonfun$apply$2.apply(ExternalSorter.scala:744)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4$$anonfun$apply$2.apply(ExternalSorter.scala:743)
        scala.collection.Iterator$class.foreach(Iterator.scala:727)
       
org.apache.spark.util.collection.ExternalSorter$IteratorForPartition.foreach(ExternalSorter.scala:801)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4.apply(ExternalSorter.scala:743)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4.apply(ExternalSorter.scala:739)
        scala.collection.Iterator$class.foreach(Iterator.scala:727)
        scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
       
org.apache.spark.util.collection.ExternalSorter.writePartitionedFile(ExternalSorter.scala:739)
       
org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:70)
       
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)
       
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
        org.apache.spark.scheduler.Task.run(Task.scala:56)
       
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:186)
       
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
       
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        java.lang.Thread.run(Thread.java:744)
14/12/11 11:46:23 ERROR YarnClientClusterScheduler: Lost executor 207 on
ams05-015.ff.company.com: remote Akka client disassociated
14/12/11 11:46:23 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams05-015.ff.company.com:39236] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:46:23 ERROR SendingConnection: Exception while reading
SendingConnection to ConnectionManagerId(ams05-015.ff.company.com,44621)
java.nio.channels.ClosedChannelException
	at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
	at
org.apache.spark.network.nio.SendingConnection.read(Connection.scala:419)
	at
org.apache.spark.network.nio.ConnectionManager$$anon$10.run(ConnectionManager.scala:238)
	at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
14/12/11 11:46:23 ERROR ConnectionManager: Exception while sending message.
java.nio.channels.ClosedChannelException
	at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
	at
org.apache.spark.network.nio.SendingConnection.read(Connection.scala:419)
	at
org.apache.spark.network.nio.ConnectionManager$$anon$10.run(ConnectionManager.scala:238)
	at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
14/12/11 11:46:23 ERROR ConnectionManager: no messageStatus for failed
message id: 307
14/12/11 11:46:23 WARN TaskSetManager: Lost task 851.1 in stage 1.0 (TID
5964, ams05-015.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:46:23 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 207
14/12/11 11:46:23 ERROR ConnectionManager: Corresponding SendingConnection
to ConnectionManagerId(ams05-011.ff.company.com,38205) not found
14/12/11 11:46:23 ERROR YarnClientClusterScheduler: Lost executor 382 on
ams05-011.ff.company.com: remote Akka client disassociated
14/12/11 11:46:23 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams05-011.ff.company.com:47874] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:46:23 WARN TaskSetManager: Lost task 92.1 in stage 0.0 (TID
5967, ams05-011.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:46:23 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 382
14/12/11 11:46:23 ERROR ConnectionManager: Corresponding SendingConnection
to ConnectionManagerId(ams02-018.ff.company.com,36368) not found
14/12/11 11:46:23 ERROR ConnectionManager: Exception while sending message.
java.nio.channels.CancelledKeyException
	at
org.apache.spark.network.nio.ConnectionManager.run(ConnectionManager.scala:437)
	at
org.apache.spark.network.nio.ConnectionManager$$anon$7.run(ConnectionManager.scala:170)
14/12/11 11:46:23 ERROR ConnectionManager: no messageStatus for failed
message id: 63
14/12/11 11:46:23 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams02-018.ff.company.com:33468] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:46:23 ERROR YarnClientClusterScheduler: Lost executor 173 on
ams02-018.ff.company.com: remote Akka client disassociated
14/12/11 11:46:23 WARN TaskSetManager: Lost task 2779.1 in stage 0.0 (TID
5970, ams02-018.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:46:23 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 173
14/12/11 11:46:29 ERROR YarnClientClusterScheduler: Lost executor 226 on
ams05-014.ff.company.com: remote Akka client disassociated
14/12/11 11:46:29 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams05-014.ff.company.com:44344] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:46:29 WARN TaskSetManager: Lost task 1931.1 in stage 0.0 (TID
5980, ams05-014.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:46:29 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 226
14/12/11 11:46:31 WARN TaskSetManager: Lost task 166.2 in stage 0.0 (TID
5865, ams02-015.ff.company.com): java.io.IOException: sendMessageReliably
failed because ack was not received within 60 sec
       
org.apache.spark.network.nio.ConnectionManager$$anon$8$$anonfun$run$19.apply(ConnectionManager.scala:903)
       
org.apache.spark.network.nio.ConnectionManager$$anon$8$$anonfun$run$19.apply(ConnectionManager.scala:902)
        scala.Option.foreach(Option.scala:236)
       
org.apache.spark.network.nio.ConnectionManager$$anon$8.run(ConnectionManager.scala:902)
        java.util.TimerThread.mainLoop(Timer.java:555)
        java.util.TimerThread.run(Timer.java:505)
14/12/11 11:46:33 WARN TaskSetManager: Lost task 5182.0 in stage 0.0 (TID
5432, ams03-012.ff.company.com): java.lang.OutOfMemoryError: GC overhead
limit exceeded
       
java.io.ObjectOutputStream$HandleTable.growSpine(ObjectOutputStream.java:2333)
       
java.io.ObjectOutputStream$HandleTable.assign(ObjectOutputStream.java:2278)
        java.io.ObjectOutputStream.writeString(ObjectOutputStream.java:1301)
       
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1171)
       
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
       
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
       
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
       
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
       
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
       
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
       
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
       
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
        java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347)
       
org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:42)
       
org.apache.spark.storage.DiskBlockObjectWriter.write(BlockObjectWriter.scala:195)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4$$anonfun$apply$2.apply(ExternalSorter.scala:744)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4$$anonfun$apply$2.apply(ExternalSorter.scala:743)
        scala.collection.Iterator$class.foreach(Iterator.scala:727)
       
org.apache.spark.util.collection.ExternalSorter$IteratorForPartition.foreach(ExternalSorter.scala:801)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4.apply(ExternalSorter.scala:743)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4.apply(ExternalSorter.scala:739)
        scala.collection.Iterator$class.foreach(Iterator.scala:727)
        scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
       
org.apache.spark.util.collection.ExternalSorter.writePartitionedFile(ExternalSorter.scala:739)
       
org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:70)
       
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)
       
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
        org.apache.spark.scheduler.Task.run(Task.scala:56)
       
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:186)
       
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
       
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        java.lang.Thread.run(Thread.java:744)
14/12/11 11:46:35 ERROR YarnClientClusterScheduler: Lost executor 10 on
ams03-012.ff.company.com: remote Akka client disassociated
14/12/11 11:46:35 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams03-012.ff.company.com:33000] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:46:35 WARN TaskSetManager: Lost task 5692.0 in stage 0.0 (TID
5995, ams03-012.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:46:35 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 10
14/12/11 11:46:36 ERROR TaskSetManager: Task 755 in stage 0.0 failed 4
times; aborting job
14/12/11 11:46:36 WARN TaskSetManager: Lost task 1518.1 in stage 0.0 (TID
5987, ams06-004.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:46:36 WARN TaskSetManager: Lost task 5765.1 in stage 0.0 (TID
5988, ams02-007.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:46:36 WARN TaskSetManager: Lost task 5821.1 in stage 0.0 (TID
5996, ams03-015.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:46:36 WARN TaskSetManager: Lost task 1156.4 in stage 0.0 (TID
5984, ams05-005.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:46:36 WARN TaskSetManager: Lost task 806.1 in stage 0.0 (TID
5966, ams03-003.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:46:36 WARN TaskSetManager: Lost task 2101.1 in stage 0.0 (TID
5930, ams02-014.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:46:36 WARN TaskSetManager: Lost task 5302.3 in stage 0.0 (TID
5973, ams03-004.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:46:36 WARN TaskSetManager: Lost task 144.1 in stage 0.0 (TID
5969, ams04-009.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:46:36 WARN TaskSetManager: Lost task 3299.2 in stage 0.0 (TID
5947, ams03-012.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:46:36 WARN TaskSetManager: Lost task 3614.1 in stage 0.0 (TID
5963, ams06-010.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:46:36 WARN TaskSetManager: Lost task 7718.0 in stage 0.0 (TID
5972, ams07-002.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:46:36 WARN TaskSetManager: Lost task 5342.3 in stage 0.0 (TID
5960, ams06-014.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:46:36 WARN TaskSetManager: Lost task 4909.0 in stage 0.0 (TID
5424, ams04-009.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:46:36 WARN TaskSetManager: Lost task 2544.3 in stage 0.0 (TID
5959, ams02-003.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:46:36 WARN TaskSetManager: Lost task 1728.0 in stage 1.0 (TID
5955, ams05-015.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:46:36 WARN TaskSetManager: Lost task 4607.2 in stage 0.0 (TID
5962, ams05-005.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:46:36 WARN TaskSetManager: Lost task 5731.0 in stage 0.0 (TID
5934, ams05-002.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:46:36 WARN TaskSetManager: Lost task 2040.1 in stage 0.0 (TID
5989, ams06-007.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:46:36 WARN TaskSetManager: Lost task 750.1 in stage 0.0 (TID
5999, ams07-008.ff.company.com): org.apache.spark.TaskKilledException: 
       
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:177)
       
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
       
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        java.lang.Thread.run(Thread.java:744)
14/12/11 11:46:36 WARN TaskSetManager: Lost task 3095.1 in stage 0.0 (TID
5917, ams06-007.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:46:36 WARN TaskSetManager: Lost task 647.2 in stage 0.0 (TID
5975, ams05-013.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:46:36 WARN TaskSetManager: Lost task 3503.1 in stage 0.0 (TID
5993, ams05-005.ff.company.com): TaskKilled (killed intentionally)
org.apache.spark.SparkException: Job aborted due to stage failure: Task 755
in stage 0.0 failed 4 times, most recent failure: Lost task 755.4 in stage
0.0 (TID 5875, ams07-008.ff.company.com): java.io.IOException:
sendMessageReliably failed because ack was not received within 60 sec
       
org.apache.spark.network.nio.ConnectionManager$$anon$8$$anonfun$run$19.apply(ConnectionManager.scala:903)
       
org.apache.spark.network.nio.ConnectionManager$$anon$8$$anonfun$run$19.apply(ConnectionManager.scala:902)
        scala.Option.foreach(Option.scala:236)
       
org.apache.spark.network.nio.ConnectionManager$$anon$8.run(ConnectionManager.scala:902)
        java.util.TimerThread.mainLoop(Timer.java:555)
        java.util.TimerThread.run(Timer.java:505)
Driver stacktrace:
	at
org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1191)
	at
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1180)
	at
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1179)
	at
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
	at
org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1179)
	at
org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:694)
	at
org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:694)
	at scala.Option.foreach(Option.scala:236)
	at
org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:694)
	at
org.apache.spark.scheduler.DAGSchedulerEventProcessActor$$anonfun$receive$2.applyOrElse(DAGScheduler.scala:1397)
	at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
	at
org.apache.spark.scheduler.DAGSchedulerEventProcessActor.aroundReceive(DAGScheduler.scala:1352)
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
	at akka.actor.ActorCell.invoke(ActorCell.scala:487)
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
	at akka.dispatch.Mailbox.run(Mailbox.scala:220)
	at
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)
	at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)


scala>
1;2B1;2B1;2B1;2B1;2B1;2B1;2B1;2B1;2B1;2B1;2B1;2B1;2B1;2B1;2B1;2B14/12/11
11:46:36 WARN TaskSetManager: Lost task 3519.4 in stage 0.0 (TID 5788,
ams02-010.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:46:36 WARN TaskSetManager: Lost task 5590.0 in stage 0.0 (TID
5961, ams03-012.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:46:36 WARN TaskSetManager: Lost task 1676.3 in stage 0.0 (TID
5927, ams04-013.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:46:36 WARN TaskSetManager: Lost task 3323.1 in stage 0.0 (TID
5994, ams04-014.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:46:36 WARN TaskSetManager: Lost task 6602.0 in stage 0.0 (TID
5943, ams07-015.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:46:36 WARN TaskSetManager: Lost task 2196.2 in stage 0.0 (TID
5992, ams05-018.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:46:37 WARN TaskSetManager: Lost task 1610.0 in stage 1.0 (TID
5939, ams05-015.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:46:37 WARN TaskSetManager: Lost task 101.1 in stage 0.0 (TID
5903, ams04-005.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:46:37 WARN TaskSetManager: Lost task 5434.1 in stage 0.0 (TID
5956, ams07-001.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:46:37 WARN TaskSetManager: Lost task 1446.1 in stage 0.0 (TID
5938, ams07-001.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:46:38 WARN TaskSetManager: Lost task 5452.0 in stage 0.0 (TID
5459, ams03-015.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:46:38 WARN TaskSetManager: Lost task 5025.0 in stage 0.0 (TID
5635, ams04-011.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:46:39 WARN TaskSetManager: Lost task 4172.0 in stage 0.0 (TID
4530, ams05-014.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:46:39 WARN TaskSetManager: Lost task 3368.1 in stage 0.0 (TID
5951, ams05-010.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:46:40 WARN TaskSetManager: Lost task 5052.1 in stage 0.0 (TID
5953, ams04-005.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:46:40 ERROR YarnClientClusterScheduler: Lost executor 388 on
ams06-003.ff.company.com: remote Akka client disassociated
14/12/11 11:46:40 WARN TaskSetManager: Lost task 4336.0 in stage 0.0 (TID
4774, ams06-003.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:46:40 WARN TaskSetManager: Lost task 977.1 in stage 0.0 (TID
5933, ams07-001.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:46:41 WARN TaskSetManager: Lost task 6604.0 in stage 0.0 (TID
5982, ams07-015.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:46:41 ERROR SendingConnection: Exception while reading
SendingConnection to ConnectionManagerId(ams04-014.ff.company.com,49255)
java.nio.channels.AsynchronousCloseException
	at
java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:205)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:412)
	at
org.apache.spark.network.nio.SendingConnection.read(Connection.scala:419)
	at
org.apache.spark.network.nio.ConnectionManager$$anon$10.run(ConnectionManager.scala:238)
	at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
14/12/11 11:46:41 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams04-014.ff.company.com:48952] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:46:41 ERROR ConnectionManager: Exception while sending message.
java.nio.channels.AsynchronousCloseException
	at
java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:205)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:412)
	at
org.apache.spark.network.nio.SendingConnection.read(Connection.scala:419)
	at
org.apache.spark.network.nio.ConnectionManager$$anon$10.run(ConnectionManager.scala:238)
	at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
14/12/11 11:46:41 ERROR YarnClientClusterScheduler: Lost executor 194 on
ams04-014.ff.company.com: remote Akka client disassociated
14/12/11 11:46:41 ERROR ConnectionManager: no messageStatus for failed
message id: 823
14/12/11 11:46:41 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 194
14/12/11 11:46:41 WARN TaskSetManager: Lost task 5397.0 in stage 0.0 (TID
5881, ams03-012.ff.company.com): java.io.IOException: sendMessageReliably
failed because ack was not received within 60 sec
       
org.apache.spark.network.nio.ConnectionManager$$anon$8$$anonfun$run$19.apply(ConnectionManager.scala:903)
       
org.apache.spark.network.nio.ConnectionManager$$anon$8$$anonfun$run$19.apply(ConnectionManager.scala:902)
        scala.Option.foreach(Option.scala:236)
       
org.apache.spark.network.nio.ConnectionManager$$anon$8.run(ConnectionManager.scala:902)
        java.util.TimerThread.mainLoop(Timer.java:555)
        java.util.TimerThread.run(Timer.java:505)
14/12/11 11:46:42 WARN TaskSetManager: Lost task 5162.0 in stage 0.0 (TID
5487, ams05-001.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:46:42 ERROR YarnClientClusterScheduler: Lost executor 245 on
ams05-004.ff.company.com: remote Akka client disassociated
14/12/11 11:46:42 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams05-004.ff.company.com:39878] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:46:42 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 245
14/12/11 11:46:42 WARN TaskSetManager: Lost task 1908.2 in stage 0.0 (TID
5954, ams06-007.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:46:43 WARN TaskSetManager: Lost task 1661.0 in stage 1.0 (TID
5948, ams05-015.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:46:43 WARN TaskSetManager: Lost task 3720.1 in stage 0.0 (TID
5892, ams03-011.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:46:44 WARN TaskSetManager: Lost task 3935.1 in stage 0.0 (TID
5925, ams07-015.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:46:44 WARN TaskSetManager: Lost task 1341.1 in stage 0.0 (TID
5949, ams04-013.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:46:45 WARN TaskSetManager: Lost task 4962.0 in stage 0.0 (TID
5317, ams04-001.ff.company.com): java.lang.OutOfMemoryError: GC overhead
limit exceeded
        java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:322)
       
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1133)
       
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
       
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
       
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
       
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
       
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
       
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
       
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
       
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
        java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347)
       
org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:42)
       
org.apache.spark.storage.DiskBlockObjectWriter.write(BlockObjectWriter.scala:195)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4$$anonfun$apply$2.apply(ExternalSorter.scala:744)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4$$anonfun$apply$2.apply(ExternalSorter.scala:743)
        scala.collection.Iterator$class.foreach(Iterator.scala:727)
       
org.apache.spark.util.collection.ExternalSorter$IteratorForPartition.foreach(ExternalSorter.scala:801)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4.apply(ExternalSorter.scala:743)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4.apply(ExternalSorter.scala:739)
        scala.collection.Iterator$class.foreach(Iterator.scala:727)
        scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
       
org.apache.spark.util.collection.ExternalSorter.writePartitionedFile(ExternalSorter.scala:739)
       
org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:70)
       
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)
       
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
        org.apache.spark.scheduler.Task.run(Task.scala:56)
       
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:186)
       
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
       
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        java.lang.Thread.run(Thread.java:744)
14/12/11 11:46:45 WARN TaskSetManager: Lost task 5614.0 in stage 0.0 (TID
5255, ams05-009.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:46:46 WARN TaskSetManager: Lost task 2211.1 in stage 0.0 (TID
5910, ams03-010.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:46:46 WARN TaskSetManager: Lost task 4518.3 in stage 0.0 (TID
5844, ams04-005.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:46:47 WARN TaskSetManager: Lost task 5124.0 in stage 0.0 (TID
5595, ams02-014.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:46:47 WARN TaskSetManager: Lost task 92.2 in stage 0.0 (TID
5998, ams06-006.ff.company.com): org.apache.spark.TaskKilledException: 
       
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:177)
       
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
       
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        java.lang.Thread.run(Thread.java:744)
14/12/11 11:46:47 ERROR ConnectionManager: Corresponding SendingConnection
to ConnectionManagerId(ams04-001.ff.company.com,42281) not found
14/12/11 11:46:47 ERROR YarnClientClusterScheduler: Lost executor 86 on
ams04-001.ff.company.com: remote Akka client disassociated
14/12/11 11:46:47 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams04-001.ff.company.com:34562] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:46:47 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 86
14/12/11 11:46:48 WARN TaskSetManager: Lost task 6046.2 in stage 0.0 (TID
5662, ams06-011.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:46:48 WARN TaskSetManager: Lost task 5006.0 in stage 0.0 (TID
5226, ams07-006.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:46:49 ERROR YarnClientClusterScheduler: Lost executor 285 on
ams02-012.ff.company.com: remote Akka client disassociated
14/12/11 11:46:49 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams02-012.ff.company.com:46407] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:46:49 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 285
14/12/11 11:46:49 ERROR YarnClientClusterScheduler: Lost executor 52 on
ams07-014.ff.company.com: remote Akka client disassociated
14/12/11 11:46:49 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams07-014.ff.company.com:34933] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:46:49 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 52
14/12/11 11:46:50 WARN TaskSetManager: Lost task 4246.0 in stage 0.0 (TID
4250, ams06-003.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:46:50 ERROR SendingConnection: Exception while reading
SendingConnection to ConnectionManagerId(ams06-008.ff.company.com,44251)
java.nio.channels.AsynchronousCloseException
	at
java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:205)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:412)
	at
org.apache.spark.network.nio.SendingConnection.read(Connection.scala:419)
	at
org.apache.spark.network.nio.ConnectionManager$$anon$10.run(ConnectionManager.scala:238)
	at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
14/12/11 11:46:50 ERROR ConnectionManager: Exception while sending message.
java.nio.channels.AsynchronousCloseException
	at
java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:205)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:412)
	at
org.apache.spark.network.nio.SendingConnection.read(Connection.scala:419)
	at
org.apache.spark.network.nio.ConnectionManager$$anon$10.run(ConnectionManager.scala:238)
	at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
14/12/11 11:46:50 ERROR YarnClientClusterScheduler: Lost executor 291 on
ams06-008.ff.company.com: remote Akka client disassociated
14/12/11 11:46:50 ERROR ConnectionManager: no messageStatus for failed
message id: 618
14/12/11 11:46:50 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams06-008.ff.company.com:39054] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:46:50 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 291
14/12/11 11:46:51 WARN TaskSetManager: Lost task 4542.0 in stage 0.0 (TID
5003, ams03-010.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:46:52 WARN TaskSetManager: Lost task 790.1 in stage 0.0 (TID
5937, ams04-014.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:46:52 WARN TaskSetManager: Lost task 4010.1 in stage 0.0 (TID
5924, ams05-010.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:46:52 WARN TaskSetManager: Lost task 5768.0 in stage 0.0 (TID
5891, ams02-016.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:46:53 WARN TaskSetManager: Lost task 6151.1 in stage 0.0 (TID
5901, ams03-017.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:46:53 WARN TaskSetManager: Lost task 1683.1 in stage 0.0 (TID
5914, ams05-004.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:46:53 ERROR ConnectionManager: Exception while sending message.
java.nio.channels.CancelledKeyException
	at
org.apache.spark.network.nio.ConnectionManager.run(ConnectionManager.scala:437)
	at
org.apache.spark.network.nio.ConnectionManager$$anon$7.run(ConnectionManager.scala:170)
14/12/11 11:46:53 ERROR ConnectionManager: no messageStatus for failed
message id: 132
14/12/11 11:46:53 ERROR ConnectionManager: Corresponding SendingConnection
to ConnectionManagerId(ams06-006.ff.company.com,42611) not found
14/12/11 11:46:53 ERROR YarnClientClusterScheduler: Lost executor 332 on
ams06-006.ff.company.com: remote Akka client disassociated
14/12/11 11:46:53 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams06-006.ff.company.com:40836] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:46:53 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 332
14/12/11 11:46:54 WARN TaskSetManager: Lost task 5267.0 in stage 0.0 (TID
5460, ams04-006.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:46:54 WARN TaskSetManager: Lost task 3903.1 in stage 0.0 (TID
5904, ams04-018.ff.company.com): java.io.IOException: sendMessageReliably
failed because ack was not received within 60 sec
       
org.apache.spark.network.nio.ConnectionManager$$anon$8$$anonfun$run$19.apply(ConnectionManager.scala:903)
       
org.apache.spark.network.nio.ConnectionManager$$anon$8$$anonfun$run$19.apply(ConnectionManager.scala:902)
        scala.Option.foreach(Option.scala:236)
       
org.apache.spark.network.nio.ConnectionManager$$anon$8.run(ConnectionManager.scala:902)
        java.util.TimerThread.mainLoop(Timer.java:555)
        java.util.TimerThread.run(Timer.java:505)
14/12/11 11:46:55 WARN TaskSetManager: Lost task 4611.0 in stage 0.0 (TID
5084, ams02-002.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:46:55 WARN TaskSetManager: Lost task 3888.1 in stage 0.0 (TID
5887, ams07-012.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:46:55 WARN TaskSetManager: Lost task 148.1 in stage 0.0 (TID
5936, ams04-009.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:46:55 ERROR YarnClientClusterScheduler: Lost executor 127 on
ams05-018.ff.company.com: remote Akka client disassociated
14/12/11 11:46:55 WARN TaskSetManager: Lost task 5243.0 in stage 0.0 (TID
5417, ams05-018.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:46:55 WARN TaskSetManager: Lost task 228.0 in stage 1.0 (TID
5530, ams05-015.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:46:56 WARN TaskSetManager: Lost task 1749.1 in stage 0.0 (TID
5952, ams04-015.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:46:56 ERROR ConnectionManager: Exception while sending message.
java.nio.channels.CancelledKeyException
	at
org.apache.spark.network.nio.ConnectionManager.run(ConnectionManager.scala:437)
	at
org.apache.spark.network.nio.ConnectionManager$$anon$7.run(ConnectionManager.scala:170)
14/12/11 11:46:56 ERROR ConnectionManager: no messageStatus for failed
message id: 141
14/12/11 11:46:56 ERROR YarnClientClusterScheduler: Lost executor 315 on
ams07-012.ff.company.com: remote Akka client disassociated
14/12/11 11:46:56 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams07-012.ff.company.com:42075] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:46:56 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 315
14/12/11 11:46:56 WARN TaskSetManager: Lost task 5385.0 in stage 0.0 (TID
5450, ams02-015.ff.company.com): java.lang.OutOfMemoryError: GC overhead
limit exceeded
       
org.apache.spark.util.collection.SizeTrackingPairBuffer$$anon$1.next(SizeTrackingPairBuffer.scala:61)
       
org.apache.spark.util.collection.SizeTrackingPairBuffer$$anon$1.next(SizeTrackingPairBuffer.scala:52)
        scala.collection.Iterator$$anon$1.next(Iterator.scala:853)
        scala.collection.Iterator$$anon$1.head(Iterator.scala:840)
       
org.apache.spark.util.collection.ExternalSorter$IteratorForPartition.hasNext(ExternalSorter.scala:804)
        scala.collection.Iterator$class.foreach(Iterator.scala:727)
       
org.apache.spark.util.collection.ExternalSorter$IteratorForPartition.foreach(ExternalSorter.scala:801)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4.apply(ExternalSorter.scala:743)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4.apply(ExternalSorter.scala:739)
        scala.collection.Iterator$class.foreach(Iterator.scala:727)
        scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
       
org.apache.spark.util.collection.ExternalSorter.writePartitionedFile(ExternalSorter.scala:739)
       
org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:70)
       
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)
       
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
        org.apache.spark.scheduler.Task.run(Task.scala:56)
       
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:186)
       
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
       
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        java.lang.Thread.run(Thread.java:744)
14/12/11 11:46:57 ERROR SendingConnection: Exception while reading
SendingConnection to ConnectionManagerId(ams05-018.ff.company.com,49580)
java.nio.channels.ClosedChannelException
	at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
	at
org.apache.spark.network.nio.SendingConnection.read(Connection.scala:419)
	at
org.apache.spark.network.nio.ConnectionManager$$anon$10.run(ConnectionManager.scala:238)
	at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
14/12/11 11:46:57 ERROR ConnectionManager: Exception while sending message.
java.nio.channels.ClosedChannelException
	at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
	at
org.apache.spark.network.nio.SendingConnection.read(Connection.scala:419)
	at
org.apache.spark.network.nio.ConnectionManager$$anon$10.run(ConnectionManager.scala:238)
	at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
14/12/11 11:46:57 ERROR ConnectionManager: no messageStatus for failed
message id: 61
14/12/11 11:46:58 ERROR ConnectionManager: Corresponding SendingConnection
to ConnectionManagerId(ams02-015.ff.company.com,41753) not found
14/12/11 11:46:58 ERROR YarnClientClusterScheduler: Lost executor 334 on
ams02-015.ff.company.com: remote Akka client disassociated
14/12/11 11:46:58 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams02-015.ff.company.com:40226] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:46:58 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 334
14/12/11 11:46:59 WARN TaskSetManager: Lost task 1117.1 in stage 0.0 (TID
5919, ams05-003.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:47:03 WARN TaskSetManager: Lost task 2166.3 in stage 0.0 (TID
5932, ams02-010.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:47:03 WARN TaskSetManager: Lost task 2716.1 in stage 0.0 (TID
5915, ams07-007.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:47:03 WARN TaskSetManager: Lost task 4265.1 in stage 0.0 (TID
5605, ams03-013.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:47:06 WARN TaskSetManager: Lost task 756.1 in stage 0.0 (TID
5931, ams07-008.ff.company.com): java.io.IOException: sendMessageReliably
failed because ack was not received within 60 sec
       
org.apache.spark.network.nio.ConnectionManager$$anon$8$$anonfun$run$19.apply(ConnectionManager.scala:903)
       
org.apache.spark.network.nio.ConnectionManager$$anon$8$$anonfun$run$19.apply(ConnectionManager.scala:902)
        scala.Option.foreach(Option.scala:236)
       
org.apache.spark.network.nio.ConnectionManager$$anon$8.run(ConnectionManager.scala:902)
        java.util.TimerThread.mainLoop(Timer.java:555)
        java.util.TimerThread.run(Timer.java:505)
14/12/11 11:47:06 WARN TaskSetManager: Lost task 5700.0 in stage 0.0 (TID
5559, ams05-010.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:47:07 WARN TaskSetManager: Lost task 6601.0 in stage 0.0 (TID
5900, ams07-015.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:47:08 ERROR YarnClientClusterScheduler: Lost executor 149 on
ams07-010.ff.company.com: remote Akka client disassociated
14/12/11 11:47:08 WARN TaskSetManager: Lost task 5264.0 in stage 0.0 (TID
5248, ams07-010.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:47:08 WARN TaskSetManager: Lost task 5008.0 in stage 0.0 (TID
5366, ams02-014.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:47:09 WARN TaskSetManager: Lost task 6389.0 in stage 0.0 (TID
5570, ams04-013.ff.company.com): java.lang.OutOfMemoryError: GC overhead
limit exceeded
       
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1536)
       
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
       
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
       
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
       
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
       
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
       
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
       
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
        java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347)
       
org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:42)
       
org.apache.spark.storage.DiskBlockObjectWriter.write(BlockObjectWriter.scala:195)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4$$anonfun$apply$2.apply(ExternalSorter.scala:744)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4$$anonfun$apply$2.apply(ExternalSorter.scala:743)
        scala.collection.Iterator$class.foreach(Iterator.scala:727)
       
org.apache.spark.util.collection.ExternalSorter$IteratorForPartition.foreach(ExternalSorter.scala:801)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4.apply(ExternalSorter.scala:743)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4.apply(ExternalSorter.scala:739)
        scala.collection.Iterator$class.foreach(Iterator.scala:727)
        scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
       
org.apache.spark.util.collection.ExternalSorter.writePartitionedFile(ExternalSorter.scala:739)
       
org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:70)
       
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)
       
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
        org.apache.spark.scheduler.Task.run(Task.scala:56)
       
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:186)
       
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
       
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        java.lang.Thread.run(Thread.java:744)
14/12/11 11:47:10 WARN TaskSetManager: Lost task 5776.0 in stage 0.0 (TID
5090, ams02-011.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:47:11 ERROR ConnectionManager: Corresponding SendingConnection
to ConnectionManagerId(ams07-010.ff.company.com,49636) not found
14/12/11 11:47:11 WARN TaskSetManager: Lost task 5334.0 in stage 0.0 (TID
5534, ams04-002.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:47:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(129, ams07-002.ff.company.com, 34170) with no recent heart
beats: 75608ms exceeds 45000ms
14/12/11 11:47:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(158, ams03-015.ff.company.com, 37146) with no recent heart
beats: 100154ms exceeds 45000ms
14/12/11 11:47:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(363, ams05-012.ff.company.com, 34128) with no recent heart
beats: 61360ms exceeds 45000ms
14/12/11 11:47:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(246, ams05-009.ff.company.com, 33650) with no recent heart
beats: 82606ms exceeds 45000ms
14/12/11 11:47:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(273, ams07-013.ff.company.com, 49100) with no recent heart
beats: 96213ms exceeds 45000ms
14/12/11 11:47:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(33, ams07-007.ff.company.com, 40594) with no recent heart
beats: 56418ms exceeds 45000ms
14/12/11 11:47:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(101, ams04-016.ff.company.com, 48586) with no recent heart
beats: 85592ms exceeds 45000ms
14/12/11 11:47:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(21, ams06-010.ff.company.com, 46411) with no recent heart
beats: 55893ms exceeds 45000ms
14/12/11 11:47:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(307, ams06-011.ff.company.com, 38294) with no recent heart
beats: 75612ms exceeds 45000ms
14/12/11 11:47:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(386, ams02-004.ff.company.com, 36042) with no recent heart
beats: 99673ms exceeds 45000ms
14/12/11 11:47:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(116, ams03-008.ff.company.com, 37316) with no recent heart
beats: 66570ms exceeds 45000ms
14/12/11 11:47:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(263, ams03-001.ff.company.com, 34549) with no recent heart
beats: 69450ms exceeds 45000ms
14/12/11 11:47:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(205, ams06-011.ff.company.com, 48638) with no recent heart
beats: 48802ms exceeds 45000ms
14/12/11 11:47:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(178, ams05-011.ff.company.com, 32915) with no recent heart
beats: 67465ms exceeds 45000ms
14/12/11 11:47:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(308, ams06-002.ff.company.com, 42143) with no recent heart
beats: 58887ms exceeds 45000ms
14/12/11 11:47:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(71, ams02-018.ff.company.com, 49975) with no recent heart
beats: 75764ms exceeds 45000ms
14/12/11 11:47:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(99, ams03-011.ff.company.com, 35480) with no recent heart
beats: 73751ms exceeds 45000ms
14/12/11 11:47:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(88, ams02-017.ff.company.com, 39274) with no recent heart
beats: 72645ms exceeds 45000ms
14/12/11 11:47:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(310, ams04-018.ff.company.com, 38799) with no recent heart
beats: 49011ms exceeds 45000ms
14/12/11 11:47:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(268, ams02-005.ff.company.com, 44139) with no recent heart
beats: 66058ms exceeds 45000ms
14/12/11 11:47:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(251, ams07-010.ff.company.com, 34127) with no recent heart
beats: 64806ms exceeds 45000ms
14/12/11 11:47:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(104, ams06-002.ff.company.com, 35894) with no recent heart
beats: 75610ms exceeds 45000ms
14/12/11 11:47:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(325, ams02-002.ff.company.com, 40604) with no recent heart
beats: 53058ms exceeds 45000ms
14/12/11 11:47:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(5, ams02-011.ff.company.com, 38211) with no recent heart
beats: 53720ms exceeds 45000ms
14/12/11 11:47:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(238, ams04-004.ff.company.com, 33847) with no recent heart
beats: 65351ms exceeds 45000ms
14/12/11 11:47:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(177, ams04-006.ff.company.com, 35811) with no recent heart
beats: 101166ms exceeds 45000ms
14/12/11 11:47:11 ERROR ConnectionManager: Exception while sending message.
java.nio.channels.CancelledKeyException
	at
org.apache.spark.network.nio.ConnectionManager.run(ConnectionManager.scala:437)
	at
org.apache.spark.network.nio.ConnectionManager$$anon$7.run(ConnectionManager.scala:170)
14/12/11 11:47:11 ERROR YarnClientClusterScheduler: Lost executor 217 on
ams04-013.ff.company.com: remote Akka client disassociated
14/12/11 11:47:11 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams04-013.ff.company.com:45282] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:47:11 ERROR ConnectionManager: no messageStatus for failed
message id: 697
14/12/11 11:47:11 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 217
14/12/11 11:47:13 WARN TaskSetManager: Lost task 5167.0 in stage 0.0 (TID
5456, ams05-014.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:47:14 WARN TaskSetManager: Lost task 6261.0 in stage 0.0 (TID
5517, ams02-011.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:47:18 ERROR SendingConnection: Exception while reading
SendingConnection to ConnectionManagerId(ams04-012.ff.company.com,41255)
java.nio.channels.ClosedChannelException
	at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
	at
org.apache.spark.network.nio.SendingConnection.read(Connection.scala:419)
	at
org.apache.spark.network.nio.ConnectionManager$$anon$10.run(ConnectionManager.scala:238)
	at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
14/12/11 11:47:18 ERROR YarnClientClusterScheduler: Lost executor 336 on
ams04-012.ff.company.com: remote Akka client disassociated
14/12/11 11:47:18 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams04-012.ff.company.com:44567] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:47:18 ERROR ConnectionManager: Exception while sending message.
java.nio.channels.ClosedChannelException
	at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
	at
org.apache.spark.network.nio.SendingConnection.read(Connection.scala:419)
	at
org.apache.spark.network.nio.ConnectionManager$$anon$10.run(ConnectionManager.scala:238)
	at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
14/12/11 11:47:18 ERROR ConnectionManager: no messageStatus for failed
message id: 231
14/12/11 11:47:18 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 336
14/12/11 11:47:18 ERROR SendingConnection: Exception while reading
SendingConnection to ConnectionManagerId(ams07-003.ff.company.com,42665)
java.nio.channels.AsynchronousCloseException
	at
java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:205)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:412)
	at
org.apache.spark.network.nio.SendingConnection.read(Connection.scala:419)
	at
org.apache.spark.network.nio.ConnectionManager$$anon$10.run(ConnectionManager.scala:238)
	at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
14/12/11 11:47:18 ERROR ConnectionManager: Exception while sending message.
java.nio.channels.AsynchronousCloseException
	at
java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:205)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:412)
	at
org.apache.spark.network.nio.SendingConnection.read(Connection.scala:419)
	at
org.apache.spark.network.nio.ConnectionManager$$anon$10.run(ConnectionManager.scala:238)
	at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
14/12/11 11:47:18 ERROR YarnClientClusterScheduler: Lost executor 317 on
ams07-003.ff.company.com: remote Akka client disassociated
14/12/11 11:47:18 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams07-003.ff.company.com:37654] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:47:18 ERROR ConnectionManager: no messageStatus for failed
message id: 674
14/12/11 11:47:18 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 317
14/12/11 11:47:19 WARN TaskSetManager: Lost task 681.1 in stage 0.0 (TID
5941, ams05-018.ff.company.com): java.io.IOException: sendMessageReliably
failed because ack was not received within 60 sec
       
org.apache.spark.network.nio.ConnectionManager$$anon$8$$anonfun$run$19.apply(ConnectionManager.scala:903)
       
org.apache.spark.network.nio.ConnectionManager$$anon$8$$anonfun$run$19.apply(ConnectionManager.scala:902)
        scala.Option.foreach(Option.scala:236)
       
org.apache.spark.network.nio.ConnectionManager$$anon$8.run(ConnectionManager.scala:902)
        java.util.TimerThread.mainLoop(Timer.java:555)
        java.util.TimerThread.run(Timer.java:505)
14/12/11 11:47:19 ERROR SendingConnection: Exception while reading
SendingConnection to ConnectionManagerId(ams02-016.ff.company.com,46711)
java.nio.channels.ClosedChannelException
	at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
	at
org.apache.spark.network.nio.SendingConnection.read(Connection.scala:419)
	at
org.apache.spark.network.nio.ConnectionManager$$anon$10.run(ConnectionManager.scala:238)
	at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
14/12/11 11:47:19 ERROR ConnectionManager: Exception while sending message.
java.nio.channels.ClosedChannelException
	at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
	at
org.apache.spark.network.nio.SendingConnection.read(Connection.scala:419)
	at
org.apache.spark.network.nio.ConnectionManager$$anon$10.run(ConnectionManager.scala:238)
	at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
14/12/11 11:47:19 ERROR YarnClientClusterScheduler: Lost executor 318 on
ams02-016.ff.company.com: remote Akka client disassociated
14/12/11 11:47:19 ERROR ConnectionManager: no messageStatus for failed
message id: 757
14/12/11 11:47:19 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 318
14/12/11 11:47:19 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams02-016.ff.company.com:41950] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:47:19 ERROR YarnClientClusterScheduler: Lost executor 66 on
ams03-014.ff.company.com: remote Akka client disassociated
14/12/11 11:47:19 WARN TaskSetManager: Lost task 5485.0 in stage 0.0 (TID
5280, ams03-014.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:47:20 WARN TaskSetManager: Lost task 4758.0 in stage 0.0 (TID
5358, ams05-013.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:47:21 WARN TaskSetManager: Lost task 661.2 in stage 0.0 (TID
5911, ams03-002.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:47:23 WARN TaskSetManager: Lost task 5175.0 in stage 0.0 (TID
5525, ams03-003.ff.company.com): java.lang.OutOfMemoryError: GC overhead
limit exceeded
        java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:322)
       
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1133)
       
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
       
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
       
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
       
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
       
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
       
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
       
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
       
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
        java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347)
       
org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:42)
       
org.apache.spark.storage.DiskBlockObjectWriter.write(BlockObjectWriter.scala:195)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4$$anonfun$apply$2.apply(ExternalSorter.scala:744)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4$$anonfun$apply$2.apply(ExternalSorter.scala:743)
        scala.collection.Iterator$class.foreach(Iterator.scala:727)
       
org.apache.spark.util.collection.ExternalSorter$IteratorForPartition.foreach(ExternalSorter.scala:801)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4.apply(ExternalSorter.scala:743)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4.apply(ExternalSorter.scala:739)
        scala.collection.Iterator$class.foreach(Iterator.scala:727)
        scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
       
org.apache.spark.util.collection.ExternalSorter.writePartitionedFile(ExternalSorter.scala:739)
       
org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:70)
       
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)
       
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
        org.apache.spark.scheduler.Task.run(Task.scala:56)
       
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:186)
       
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
       
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        java.lang.Thread.run(Thread.java:744)
14/12/11 11:47:26 ERROR ConnectionManager: Exception while sending message.
java.nio.channels.CancelledKeyException
	at
org.apache.spark.network.nio.ConnectionManager.run(ConnectionManager.scala:437)
	at
org.apache.spark.network.nio.ConnectionManager$$anon$7.run(ConnectionManager.scala:170)
14/12/11 11:47:26 ERROR ConnectionManager: no messageStatus for failed
message id: 11
14/12/11 11:47:26 ERROR YarnClientClusterScheduler: Lost executor 23 on
ams03-003.ff.company.com: remote Akka client disassociated
14/12/11 11:47:26 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams03-003.ff.company.com:48733] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:47:26 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 23
14/12/11 11:47:26 ERROR YarnClientClusterScheduler: Lost executor 174 on
ams04-010.ff.company.com: remote Akka client disassociated
14/12/11 11:47:26 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams04-010.ff.company.com:33262] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:47:26 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 174
14/12/11 11:47:27 WARN TaskSetManager: Lost task 4018.0 in stage 0.0 (TID
4044, ams06-015.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:47:28 WARN TaskSetManager: Lost task 5489.0 in stage 0.0 (TID
5421, ams05-007.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:47:30 WARN TaskSetManager: Lost task 2321.1 in stage 0.0 (TID
5985, ams04-008.ff.company.com): java.io.IOException: sendMessageReliably
failed because ack was not received within 60 sec
       
org.apache.spark.network.nio.ConnectionManager$$anon$8$$anonfun$run$19.apply(ConnectionManager.scala:903)
       
org.apache.spark.network.nio.ConnectionManager$$anon$8$$anonfun$run$19.apply(ConnectionManager.scala:902)
        scala.Option.foreach(Option.scala:236)
       
org.apache.spark.network.nio.ConnectionManager$$anon$8.run(ConnectionManager.scala:902)
        java.util.TimerThread.mainLoop(Timer.java:555)
        java.util.TimerThread.run(Timer.java:505)
14/12/11 11:47:30 ERROR YarnClientClusterScheduler: Lost executor 138 on
ams05-013.ff.company.com: remote Akka client disassociated
14/12/11 11:47:30 WARN TaskSetManager: Lost task 4696.0 in stage 0.0 (TID
5070, ams05-013.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:47:33 ERROR ConnectionManager: Corresponding SendingConnection
to ConnectionManagerId(ams05-013.ff.company.com,43105) not found
14/12/11 11:47:33 ERROR YarnClientClusterScheduler: Lost executor 231 on
ams07-002.ff.company.com: remote Akka client disassociated
14/12/11 11:47:33 WARN TaskSetManager: Lost task 7535.0 in stage 0.0 (TID
5498, ams07-002.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:47:34 WARN TaskSetManager: Lost task 4813.0 in stage 0.0 (TID
5395, ams03-013.ff.company.com): java.lang.OutOfMemoryError: GC overhead
limit exceeded
       
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1536)
       
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
       
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
       
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
       
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
       
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
       
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
       
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
        java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347)
       
org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:42)
       
org.apache.spark.storage.DiskBlockObjectWriter.write(BlockObjectWriter.scala:195)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4$$anonfun$apply$2.apply(ExternalSorter.scala:744)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4$$anonfun$apply$2.apply(ExternalSorter.scala:743)
        scala.collection.Iterator$class.foreach(Iterator.scala:727)
       
org.apache.spark.util.collection.ExternalSorter$IteratorForPartition.foreach(ExternalSorter.scala:801)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4.apply(ExternalSorter.scala:743)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4.apply(ExternalSorter.scala:739)
        scala.collection.Iterator$class.foreach(Iterator.scala:727)
        scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
       
org.apache.spark.util.collection.ExternalSorter.writePartitionedFile(ExternalSorter.scala:739)
       
org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:70)
       
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)
       
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
        org.apache.spark.scheduler.Task.run(Task.scala:56)
       
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:186)
       
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
       
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        java.lang.Thread.run(Thread.java:744)
14/12/11 11:47:37 ERROR ConnectionManager: Exception while sending message.
java.nio.channels.CancelledKeyException
	at
org.apache.spark.network.nio.ConnectionManager.run(ConnectionManager.scala:437)
	at
org.apache.spark.network.nio.ConnectionManager$$anon$7.run(ConnectionManager.scala:170)
14/12/11 11:47:37 ERROR ConnectionManager: no messageStatus for failed
message id: 603
14/12/11 11:47:37 ERROR YarnClientClusterScheduler: Lost executor 395 on
ams03-013.ff.company.com: remote Akka client disassociated
14/12/11 11:47:37 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams03-013.ff.company.com:44260] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:47:37 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 395
14/12/11 11:47:38 ERROR ConnectionManager: Corresponding SendingConnection
to ConnectionManagerId(ams06-013.ff.company.com,45968) not found
14/12/11 11:47:38 ERROR YarnClientClusterScheduler: Lost executor 65 on
ams06-013.ff.company.com: remote Akka client disassociated
14/12/11 11:47:38 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams06-013.ff.company.com:37397] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:47:38 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 65
14/12/11 11:47:39 WARN TaskSetManager: Lost task 4890.0 in stage 0.0 (TID
4789, ams06-011.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:47:41 WARN TaskSetManager: Lost task 6407.1 in stage 0.0 (TID
5565, ams02-011.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:47:41 WARN TaskSetManager: Lost task 4074.0 in stage 0.0 (TID
4064, ams02-004.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:47:44 WARN TaskSetManager: Lost task 4185.0 in stage 0.0 (TID
4300, ams06-009.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:47:48 WARN TaskSetManager: Lost task 6323.0 in stage 0.0 (TID
5386, ams04-018.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:47:51 WARN TaskSetManager: Lost task 5584.0 in stage 0.0 (TID
5499, ams07-010.ff.company.com): java.lang.OutOfMemoryError: GC overhead
limit exceeded
        java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:322)
       
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1133)
       
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
       
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
       
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
       
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
       
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
       
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
       
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
       
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
        java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347)
       
org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:42)
       
org.apache.spark.storage.DiskBlockObjectWriter.write(BlockObjectWriter.scala:195)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4$$anonfun$apply$2.apply(ExternalSorter.scala:744)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4$$anonfun$apply$2.apply(ExternalSorter.scala:743)
        scala.collection.Iterator$class.foreach(Iterator.scala:727)
       
org.apache.spark.util.collection.ExternalSorter$IteratorForPartition.foreach(ExternalSorter.scala:801)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4.apply(ExternalSorter.scala:743)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4.apply(ExternalSorter.scala:739)
        scala.collection.Iterator$class.foreach(Iterator.scala:727)
        scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
       
org.apache.spark.util.collection.ExternalSorter.writePartitionedFile(ExternalSorter.scala:739)
       
org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:70)
       
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)
       
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
        org.apache.spark.scheduler.Task.run(Task.scala:56)
       
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:186)
       
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
       
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        java.lang.Thread.run(Thread.java:744)
14/12/11 11:47:51 WARN TaskSetManager: Lost task 5061.0 in stage 0.0 (TID
5347, ams03-009.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:47:53 ERROR ConnectionManager: Corresponding SendingConnection
to ConnectionManagerId(ams07-010.ff.company.com,34127) not found
14/12/11 11:47:53 ERROR YarnClientClusterScheduler: Lost executor 251 on
ams07-010.ff.company.com: remote Akka client disassociated
14/12/11 11:47:53 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams07-010.ff.company.com:33020] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:47:53 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 251
14/12/11 11:47:57 ERROR ConnectionManager: Corresponding SendingConnection
to ConnectionManagerId(ams06-002.ff.company.com,42143) not found
14/12/11 11:47:57 ERROR YarnClientClusterScheduler: Lost executor 308 on
ams06-002.ff.company.com: remote Akka client disassociated
14/12/11 11:47:57 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams06-002.ff.company.com:46767] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:47:57 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 308
14/12/11 11:48:00 ERROR YarnClientClusterScheduler: Lost executor 156 on
ams04-003.ff.company.com: remote Akka client disassociated
14/12/11 11:48:00 WARN TaskSetManager: Lost task 4534.0 in stage 0.0 (TID
5065, ams04-003.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:48:00 ERROR YarnClientClusterScheduler: Lost executor 93 on
ams07-009.ff.company.com: remote Akka client disassociated
14/12/11 11:48:00 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams07-009.ff.company.com:35991] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:48:00 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 93
14/12/11 11:48:00 ERROR YarnClientClusterScheduler: Lost executor 316 on
ams03-012.ff.company.com: remote Akka client disassociated
14/12/11 11:48:00 WARN TaskSetManager: Lost task 4825.0 in stage 0.0 (TID
5195, ams03-012.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:48:02 ERROR ConnectionManager: Corresponding SendingConnection
to ConnectionManagerId(ams03-012.ff.company.com,37243) not found
14/12/11 11:48:05 WARN TaskSetManager: Lost task 4820.0 in stage 0.0 (TID
4938, ams02-001.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:48:09 WARN TaskSetManager: Lost task 3929.0 in stage 0.0 (TID
4028, ams05-017.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:48:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(390, ams02-009.ff.company.com, 48941) with no recent heart
beats: 90586ms exceeds 45000ms
14/12/11 11:48:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(379, ams06-009.ff.company.com, 49439) with no recent heart
beats: 92531ms exceeds 45000ms
14/12/11 11:48:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(109, ams02-003.ff.company.com, 42196) with no recent heart
beats: 89173ms exceeds 45000ms
14/12/11 11:48:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(218, ams03-008.ff.company.com, 42311) with no recent heart
beats: 45908ms exceeds 45000ms
14/12/11 11:48:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(144, ams05-009.ff.company.com, 35453) with no recent heart
beats: 45129ms exceeds 45000ms
14/12/11 11:48:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(267, ams02-013.ff.company.com, 46235) with no recent heart
beats: 63600ms exceeds 45000ms
14/12/11 11:48:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(137, ams06-001.ff.company.com, 41014) with no recent heart
beats: 65872ms exceeds 45000ms
14/12/11 11:48:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(114, ams02-016.ff.company.com, 36456) with no recent heart
beats: 70611ms exceeds 45000ms
14/12/11 11:48:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(264, ams06-004.ff.company.com, 46639) with no recent heart
beats: 66311ms exceeds 45000ms
14/12/11 11:48:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(387, ams02-012.ff.company.com, 47245) with no recent heart
beats: 45895ms exceeds 45000ms
14/12/11 11:48:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(95, ams06-015.ff.company.com, 34770) with no recent heart
beats: 47489ms exceeds 45000ms
14/12/11 11:48:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(330, ams07-011.ff.company.com, 35252) with no recent heart
beats: 47715ms exceeds 45000ms
14/12/11 11:48:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(266, ams05-016.ff.company.com, 38478) with no recent heart
beats: 45660ms exceeds 45000ms
14/12/11 11:48:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(373, ams04-002.ff.company.com, 39055) with no recent heart
beats: 48836ms exceeds 45000ms
14/12/11 11:48:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(186, ams02-009.ff.company.com, 37437) with no recent heart
beats: 71734ms exceeds 45000ms
14/12/11 11:48:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(282, ams06-012.ff.company.com, 44208) with no recent heart
beats: 97426ms exceeds 45000ms
14/12/11 11:48:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(279, ams04-006.ff.company.com, 43193) with no recent heart
beats: 96678ms exceeds 45000ms
14/12/11 11:48:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(216, ams02-016.ff.company.com, 35582) with no recent heart
beats: 71280ms exceeds 45000ms
14/12/11 11:48:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(60, ams06-004.ff.company.com, 33110) with no recent heart
beats: 56817ms exceeds 45000ms
14/12/11 11:48:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(360, ams04-003.ff.company.com, 48594) with no recent heart
beats: 101780ms exceeds 45000ms
14/12/11 11:48:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(76, ams05-011.ff.company.com, 40542) with no recent heart
beats: 82130ms exceeds 45000ms
14/12/11 11:48:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(337, ams03-016.ff.company.com, 43080) with no recent heart
beats: 99092ms exceeds 45000ms
14/12/11 11:48:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(211, ams02-003.ff.company.com, 34326) with no recent heart
beats: 81753ms exceeds 45000ms
14/12/11 11:48:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(321, ams03-017.ff.company.com, 35759) with no recent heart
beats: 53886ms exceeds 45000ms
14/12/11 11:48:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(385, ams05-007.ff.company.com, 39296) with no recent heart
beats: 90754ms exceeds 45000ms
14/12/11 11:48:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(123, ams06-010.ff.company.com, 49691) with no recent heart
beats: 45700ms exceeds 45000ms
14/12/11 11:48:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(265, ams05-008.ff.company.com, 33695) with no recent heart
beats: 79357ms exceeds 45000ms
14/12/11 11:48:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(143, ams05-004.ff.company.com, 34168) with no recent heart
beats: 52304ms exceeds 45000ms
14/12/11 11:48:11 WARN BlockManagerMasterActor: Removing BlockManager
BlockManagerId(110, ams02-008.ff.company.com, 34143) with no recent heart
beats: 66193ms exceeds 45000ms
14/12/11 11:48:18 WARN TaskSetManager: Lost task 4930.0 in stage 0.0 (TID
5420, ams06-009.ff.company.com): java.lang.OutOfMemoryError: GC overhead
limit exceeded
       
org.apache.spark.util.collection.ExternalSorter$IteratorForPartition.next(ExternalSorter.scala:811)
       
org.apache.spark.util.collection.ExternalSorter$IteratorForPartition.next(ExternalSorter.scala:801)
        scala.collection.Iterator$class.foreach(Iterator.scala:727)
       
org.apache.spark.util.collection.ExternalSorter$IteratorForPartition.foreach(ExternalSorter.scala:801)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4.apply(ExternalSorter.scala:743)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4.apply(ExternalSorter.scala:739)
        scala.collection.Iterator$class.foreach(Iterator.scala:727)
        scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
       
org.apache.spark.util.collection.ExternalSorter.writePartitionedFile(ExternalSorter.scala:739)
       
org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:70)
       
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)
       
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
        org.apache.spark.scheduler.Task.run(Task.scala:56)
       
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:186)
       
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
       
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        java.lang.Thread.run(Thread.java:744)
14/12/11 11:48:20 ERROR YarnClientClusterScheduler: Lost executor 333 on
ams07-002.ff.company.com: remote Akka client disassociated
14/12/11 11:48:20 ERROR YarnClientClusterScheduler: Lost executor 379 on
ams06-009.ff.company.com: remote Akka client disassociated
14/12/11 11:48:20 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams06-009.ff.company.com:33060] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:48:20 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 379
14/12/11 11:48:21 WARN TaskSetManager: Lost task 5799.0 in stage 0.0 (TID
5284, ams03-008.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:48:22 ERROR SendingConnection: Exception while reading
SendingConnection to ConnectionManagerId(ams04-010.ff.company.com,49712)
java.nio.channels.ClosedChannelException
	at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
	at
org.apache.spark.network.nio.SendingConnection.read(Connection.scala:419)
	at
org.apache.spark.network.nio.ConnectionManager$$anon$10.run(ConnectionManager.scala:238)
	at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
14/12/11 11:48:22 ERROR YarnClientClusterScheduler: Lost executor 72 on
ams04-010.ff.company.com: remote Akka client disassociated
14/12/11 11:48:22 ERROR ConnectionManager: Exception while sending message.
java.nio.channels.ClosedChannelException
	at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
	at
org.apache.spark.network.nio.SendingConnection.read(Connection.scala:419)
	at
org.apache.spark.network.nio.ConnectionManager$$anon$10.run(ConnectionManager.scala:238)
	at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
14/12/11 11:48:22 ERROR ConnectionManager: no messageStatus for failed
message id: 415
14/12/11 11:48:22 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams04-010.ff.company.com:36571] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:48:22 WARN TaskSetManager: Lost task 4955.0 in stage 0.0 (TID
5136, ams04-010.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:48:22 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 72
14/12/11 11:48:28 WARN TaskSetManager: Lost task 5637.0 in stage 0.0 (TID
5522, ams02-005.ff.company.com): java.lang.OutOfMemoryError: GC overhead
limit exceeded
        java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:322)
       
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1133)
       
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
       
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
       
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
       
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
       
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
       
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
       
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
       
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
        java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347)
       
org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:42)
       
org.apache.spark.storage.DiskBlockObjectWriter.write(BlockObjectWriter.scala:195)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4$$anonfun$apply$2.apply(ExternalSorter.scala:744)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4$$anonfun$apply$2.apply(ExternalSorter.scala:743)
        scala.collection.Iterator$class.foreach(Iterator.scala:727)
       
org.apache.spark.util.collection.ExternalSorter$IteratorForPartition.foreach(ExternalSorter.scala:801)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4.apply(ExternalSorter.scala:743)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4.apply(ExternalSorter.scala:739)
        scala.collection.Iterator$class.foreach(Iterator.scala:727)
        scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
       
org.apache.spark.util.collection.ExternalSorter.writePartitionedFile(ExternalSorter.scala:739)
       
org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:70)
       
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)
       
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
        org.apache.spark.scheduler.Task.run(Task.scala:56)
       
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:186)
       
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
       
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        java.lang.Thread.run(Thread.java:744)
14/12/11 11:48:29 ERROR ConnectionManager: Corresponding SendingConnection
to ConnectionManagerId(ams02-012.ff.company.com,41746) not found
14/12/11 11:48:29 ERROR YarnClientClusterScheduler: Lost executor 81 on
ams02-012.ff.company.com: remote Akka client disassociated
14/12/11 11:48:29 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams02-012.ff.company.com:38881] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:48:29 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 81
14/12/11 11:48:29 ERROR YarnClientClusterScheduler: Lost executor 220 on
ams07-008.ff.company.com: remote Akka client disassociated
14/12/11 11:48:29 WARN TaskSetManager: Lost task 5158.0 in stage 0.0 (TID
5497, ams07-008.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:48:29 ERROR YarnClientClusterScheduler: Lost executor 329 on
ams03-003.ff.company.com: remote Akka client disassociated
14/12/11 11:48:29 WARN TaskSetManager: Lost task 4888.0 in stage 0.0 (TID
5340, ams03-003.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:48:30 WARN TaskSetManager: Lost task 5569.0 in stage 0.0 (TID
5467, ams04-003.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:48:30 ERROR ConnectionManager: Exception while sending message.
java.nio.channels.CancelledKeyException
	at
org.apache.spark.network.nio.ConnectionManager.run(ConnectionManager.scala:437)
	at
org.apache.spark.network.nio.ConnectionManager$$anon$7.run(ConnectionManager.scala:170)
14/12/11 11:48:30 ERROR ConnectionManager: no messageStatus for failed
message id: 221
14/12/11 11:48:30 ERROR YarnClientClusterScheduler: Lost executor 370 on
ams02-005.ff.company.com: remote Akka client disassociated
14/12/11 11:48:30 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams02-005.ff.company.com:44368] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:48:30 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 370
14/12/11 11:48:31 ERROR ConnectionManager: Corresponding SendingConnection
to ConnectionManagerId(ams07-005.ff.company.com,41036) not found
14/12/11 11:48:31 ERROR YarnClientClusterScheduler: Lost executor 172 on
ams07-005.ff.company.com: remote Akka client disassociated
14/12/11 11:48:31 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams07-005.ff.company.com:37670] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:48:31 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 172
14/12/11 11:48:33 ERROR YarnClientClusterScheduler: Lost executor 15 on
ams03-017.ff.company.com: remote Akka client disassociated
14/12/11 11:48:34 WARN TaskSetManager: Lost task 4895.0 in stage 0.0 (TID
5261, ams03-017.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:48:36 ERROR YarnClientClusterScheduler: Lost executor 113 on
ams07-003.ff.company.com: remote Akka client disassociated
14/12/11 11:48:36 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams07-003.ff.company.com:48927] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:48:36 WARN TaskSetManager: Lost task 5237.0 in stage 0.0 (TID
5339, ams07-003.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:48:36 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 113
14/12/11 11:48:36 ERROR YarnClientClusterScheduler: Lost executor 224 on
ams04-009.ff.company.com: remote Akka client disassociated
14/12/11 11:48:36 WARN TaskSetManager: Lost task 5072.0 in stage 0.0 (TID
5553, ams04-009.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:48:37 ERROR ConnectionManager: Corresponding SendingConnection
to ConnectionManagerId(ams03-017.ff.company.com,34182) not found
14/12/11 11:48:37 WARN TaskSetManager: Lost task 4867.0 in stage 0.0 (TID
5435, ams02-004.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:48:37 ERROR ConnectionManager: Corresponding SendingConnection
to ConnectionManagerId(ams04-009.ff.company.com,41920) not found
14/12/11 11:48:38 ERROR YarnClientClusterScheduler: Lost executor 51 on
ams07-006.ff.company.com: remote Akka client disassociated
14/12/11 11:48:38 WARN TaskSetManager: Lost task 5080.0 in stage 0.0 (TID
5462, ams07-006.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:48:40 WARN TaskSetManager: Lost task 5654.0 in stage 0.0 (TID
5538, ams02-018.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:48:41 ERROR YarnClientClusterScheduler: Lost executor 188 on
ams04-001.ff.company.com: remote Akka client disassociated
14/12/11 11:48:41 WARN TaskSetManager: Lost task 4709.0 in stage 0.0 (TID
5088, ams04-001.ff.company.com): ExecutorLostFailure (executor lost)
14/12/11 11:48:42 WARN TaskSetManager: Lost task 5426.0 in stage 0.0 (TID
5541, ams03-017.ff.company.com): java.lang.OutOfMemoryError: GC overhead
limit exceeded
       
java.io.ObjectOutputStream$HandleTable.growEntries(ObjectOutputStream.java:2350)
       
java.io.ObjectOutputStream$HandleTable.assign(ObjectOutputStream.java:2275)
        java.io.ObjectOutputStream.writeString(ObjectOutputStream.java:1301)
       
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1171)
       
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
       
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
       
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
       
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
       
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
       
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
       
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
       
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
        java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347)
       
org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:42)
       
org.apache.spark.storage.DiskBlockObjectWriter.write(BlockObjectWriter.scala:195)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4$$anonfun$apply$2.apply(ExternalSorter.scala:744)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4$$anonfun$apply$2.apply(ExternalSorter.scala:743)
        scala.collection.Iterator$class.foreach(Iterator.scala:727)
       
org.apache.spark.util.collection.ExternalSorter$IteratorForPartition.foreach(ExternalSorter.scala:801)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4.apply(ExternalSorter.scala:743)
       
org.apache.spark.util.collection.ExternalSorter$$anonfun$writePartitionedFile$4.apply(ExternalSorter.scala:739)
        scala.collection.Iterator$class.foreach(Iterator.scala:727)
        scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
       
org.apache.spark.util.collection.ExternalSorter.writePartitionedFile(ExternalSorter.scala:739)
       
org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:70)
       
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)
       
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
        org.apache.spark.scheduler.Task.run(Task.scala:56)
       
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:186)
       
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
       
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        java.lang.Thread.run(Thread.java:744)
14/12/11 11:48:43 ERROR ConnectionManager: Exception while sending message.
java.nio.channels.CancelledKeyException
	at
org.apache.spark.network.nio.ConnectionManager.run(ConnectionManager.scala:437)
	at
org.apache.spark.network.nio.ConnectionManager$$anon$7.run(ConnectionManager.scala:170)
14/12/11 11:48:43 ERROR ConnectionManager: no messageStatus for failed
message id: 815
14/12/11 11:48:44 ERROR YarnClientClusterScheduler: Lost executor 321 on
ams03-017.ff.company.com: remote Akka client disassociated
14/12/11 11:48:44 WARN ReliableDeliverySupervisor: Association with remote
system [akka.tcp://sparkExecutor@ams03-017.ff.company.com:44414] has failed,
address is now gated for [5000] ms. Reason is: [Disassociated].
14/12/11 11:48:44 ERROR YarnClientSchedulerBackend: Asked to remove non
existant executor 321
14/12/11 11:48:48 WARN TaskSetManager: Lost task 115.1 in stage 0.0 (TID
5863, ams06-011.ff.company.com): TaskKilled (killed intentionally)
14/12/11 11:48:48 WARN TaskSetManager: Lost task 4835.0 in stage 0.0 (TID
5213, ams06-006.ff.company.com): TaskKilled (killed intentionally)




Failed Stages (2)

Stage Id	Description	Submitted	Duration	Tasks: Succeeded/Total	Input	Shuffle
Read	Shuffle Write	Failure Reason
1	
map at <console>:25 +details
org.apache.spark.rdd.RDD.map(RDD.scala:270)
$line34.$read$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:25)
$line34.$read$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:43)
$line34.$read$$iwC$$iwC$$iwC$$iwC.<init>(<console>:45)
$line34.$read$$iwC$$iwC$$iwC.<init>(<console>:47)
$line34.$read$$iwC$$iwC.<init>(<console>:49)
$line34.$read$$iwC.<init>(<console>:51)
$line34.$read.<init>(<console>:53)
$line34.$read$.<init>(<console>:57)
$line34.$read$.<clinit>(<console>)
$line34.$eval$.<init>(<console>:7)
$line34.$eval$.<clinit>(<console>)
$line34.$eval.$print(<console>)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:606)
org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:852)
org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1125)
org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:674)
2014/12/11 11:24:00	23 min	
12/8834 (3 failed)	2.2 GB		3.3 GB	
Job aborted due to stage failure: Task 755 in stage 0.0 failed 4 times, most
recent failure: Lost task 755.4 in stage 0.0 (TID 5875,
ams07-008.ff.avast.com): java.io.IOException: sendMessageReliably failed
because ack was not received within 60 sec
       
org.apache.spark.network.nio.ConnectionManager$$anon$8$$anonfun$run$19.apply(ConnectionManager.scala:903)
       
org.apache.spark.network.nio.ConnectionManager$$anon$8$$anonfun$run$19.apply(ConnectionManager.scala:902)
        scala.Option.foreach(Option.scala:236)
       
org.apache.spark.network.nio.ConnectionManager$$anon$8.run(ConnectionManager.scala:902)
        java.util.TimerThread.mainLoop(Timer.java:555)
        java.util.TimerThread.run(Timer.java:505)
Driver stacktrace:
0	
map at <console>:25 +details
2014/12/11 11:23:59	23 min	
5180/8834 (1124 failed)	713.1 GB		1321.4 GB	
Job aborted due to stage failure: Task 755 in stage 0.0 failed 4 times, most
recent failure: Lost task 755.4 in stage 0.0 (TID 5875,
ams07-008.ff.avast.com): java.io.IOException: sendMessageReliably failed
because ack was not received within 60 sec
       
org.apache.spark.network.nio.ConnectionManager$$anon$8$$anonfun$run$19.apply(ConnectionManager.scala:903)
       
org.apache.spark.network.nio.ConnectionManager$$anon$8$$anonfun$run$19.apply(ConnectionManager.scala:902)
        scala.Option.foreach(Option.scala:236)
       
org.apache.spark.network.nio.ConnectionManager$$anon$8.run(ConnectionManager.scala:902)
        java.util.TimerThread.mainLoop(Timer.java:555)
        java.util.TimerThread.run(Timer.java:505)
Driver stacktrace:




--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/ERROR-YarnClientClusterScheduler-Lost-executor-Akka-client-disassociated-tp20625.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Re: ERROR YarnClientClusterScheduler: Lost executor Akka client disassociated

Posted by DB Tsai <db...@dbtsai.com>.
Hi Muhammad,

Maybe next time you can use http://pastebin.com/ to format and paste
the cleaner scala code snippet so other can help you easier. Also,
please only paste the significant portion of stack-trace which causes
the issue instead of giant logs.

First of all, In your log, it seems that you run out of memory, and I
guess the problem is you are trying to cache the whole
`clickstreamRDD`. Since you are not necessary using it so many time,
you may not need to cache it for better performance. Or at least, you
storage persistence should be `disk and memory` to avoid out of
memory.

Secondly, `groupByKey` is very expensive here. It's probably not the
root cause why the job is not finished, but `groupByKey` will shuffle
all the data to the reducer. In your case, you can do filter first
which will be executed in parallel in mapper side, and then do
`groupByKey`. You can specify higher num of task when you do
`groupByKey`. I'll recommend you to find a way to write your logic
using `reduceByKey` or `combineByKey` to yield much better performance
since those two operations can reduce or combine the data in mapper
side which will lead to much less shuffle traffic.

Finally, you may want to break down which part of your code causes the
issue to make debugging easier.

Sincerely,

DB Tsai
-------------------------------------------------------
My Blog: https://www.dbtsai.com
LinkedIn: https://www.linkedin.com/in/dbtsai


On Thu, Dec 11, 2014 at 4:48 AM, Muhammad Ahsan
<mu...@gmail.com> wrote:
> ----------------------------------------------
> Code
> ----------------------------------------------
> scala> import org.apache.spark.SparkContext._
> import org.apache.spark.SparkContext._
>
> scala> import org.apache.spark.rdd.RDD
> import org.apache.spark.rdd.RDD
>
> scala> import org.apache.spark.sql.SchemaRDD
> import org.apache.spark.sql.SchemaRDD
>
> scala> import org.apache.spark.sql.hive.HiveContext
> import org.apache.spark.sql.hive.HiveContext
>
> scala> import org.apache.spark.{SparkConf, SparkContext}
> import org.apache.spark.{SparkConf, SparkContext}
>
> scala> val hiveContext: HiveContext = new HiveContext(sc)
> hiveContext: org.apache.spark.sql.hive.HiveContext =
> org.apache.spark.sql.hive.HiveContext@2de76244
>
> scala> val numDays = 2
> numDays: Int = 2
>
> scala> case class Click(
> /* about 20 fields of type STRING */
> )
> defined class Click
>
> scala> val inputRDD = new Array[SchemaRDD](numDays)
> inputRDD: Array[org.apache.spark.sql.SchemaRDD] = Array(null, null)
>
> scala> for (i <- 1 to numDays) {
>      |             if (i < 10) {
>      |                 inputRDD(i - 1) =
> hiveContext.parquetFile("hdfs://................" + i)
>      |             } else {
>      |                 inputRDD(i - 1) =
> hiveContext.parquetFile("hdfs://................" + i)
>      |             }
>      |
>      |         }
> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
> SLF4J: Defaulting to no-operation (NOP) logger implementation
> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
> details.
>
> scala>         var unionRDD = inputRDD(1)
> unionRDD: org.apache.spark.sql.SchemaRDD =
> SchemaRDD[1] at RDD at SchemaRDD.scala:104
>
> scala>         for (i <- 1 to inputRDD.length - 1) {
>      |             unionRDD = unionRDD.unionAll(inputRDD(i))
>      |         }
>
> scala> val inputRDD = unionRDD
> inputRDD: org.apache.spark.sql.SchemaRDD =
> SchemaRDD[2] at RDD at SchemaRDD.scala:104
> scala>
>
> scala> inputRDD.registerTempTable("urlInfo")
>
> scala>         val clickstreamRDD = hiveContext.sql("select * from urlInfo "
> +
>      |             "where guid regexp '^[0-9a-f-]{36}$' " +
>      |             "AND ((callerid  > 3 AND callerid <10000) OR callerid >
> 100000 " +
>      |             "OR (callerid    =3 AND browsertype = 'IE')) " +
>      |             "AND countrycode regexp '^[A-Z]{2}$'")
> clickstreamRDD: org.apache.spark.sql.SchemaRDD =
> SchemaRDD[3] at RDD at SchemaRDD.scala:104
> scala>
>
> scala>         clickstreamRDD.registerTempTable("clickstream")
>
> scala>         clickstreamRDD.cache()
> res4: clickstreamRDD.type =
> SchemaRDD[3] at RDD at SchemaRDD.scala:104
>
> scala>     val guidClickRDD = clickstreamRDD.map(row =>
> (row(7).asInstanceOf[String], {
>      |             val value = Click(row(0).asInstanceOf[String],
>      |                 row(1).asInstanceOf[String],
> row(2).asInstanceOf[String],
>      |                 row(3).asInstanceOf[String],
> row(4).asInstanceOf[String],
>      |                 row(5).asInstanceOf[String],
> row(6).asInstanceOf[String],
>      |                 row(7).asInstanceOf[String],
> row(8).asInstanceOf[String],
>      |                 row(9).asInstanceOf[String],
> row(10).asInstanceOf[String],
>      |                 row(11).asInstanceOf[String],
> row(12).asInstanceOf[String],
>      |                 row(13).asInstanceOf[String],
> row(14).asInstanceOf[String],
>      |                 row(15).asInstanceOf[String],
> row(16).asInstanceOf[String],
>      |                 row(17).asInstanceOf[String],
> row(18).asInstanceOf[String],
>      |                 row(19).asInstanceOf[String])
>      |             value
>      |         }))
> guidClickRDD: org.apache.spark.rdd.RDD[(String, Click)] = MappedRDD[14] at
> map at <console>:25
>
> scala> val blackList: RDD[(String, Click)] =
> guidClickRDD.groupByKey().filter(row => row._2.size == 1).map(row =>
>      |             (row._1.asInstanceOf[String], Click("", "", "", "", "",
> "", "", "", "", "", "", "", "", "", "", "", "", "", "", "")))
> blackList: org.apache.spark.rdd.RDD[(String, Click)] = MappedRDD[27] at map
> at <console>:27
>
> scala>         val guidClickFRDD = guidClickRDD.subtractByKey(blackList)
> guidClickFRDD: org.apache.spark.rdd.RDD[(String, Click)] = SubtractedRDD[28]
> at subtractByKey at <console>:29
>
> scala> guidClickFRDD.reduceByKey((x, y) => {
>      |             /* commutative and associative function */
>      |             Click("US", "US", "US", "US", "US", "US", "US", "US",
> "US", "US", "US", "US", "US", "US", "US", "US", "US", "US", "US", "US")
>      | }).take(200).foreach(println)
>
> ----------------------------------------------
> EXPECTED OUTPUT
> ----------------------------------------------
>
> (Key_A,Click(US,US,US,US,US,US,US,US,US,US,US,US,US,US,US,US,US,US,US,US,US))
> (Key_B,Click(US,US,US,US,US,US,US,US,US,US,US,US,US,US,US,US,US,US,US,US,US))
> (Key_C,Click(US,US,US,US,US,US,US,US,US,US,US,US,US,US,US,US,US,US,US,US,US))
> (Key_D,Click(US,US,US,US,US,US,US,US,US,US,US,US,US,US,US,US,US,US,US,US,US))
>
>
>
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/ERROR-YarnClientClusterScheduler-Lost-executor-Akka-client-disassociated-tp20625p20626.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Re: ERROR YarnClientClusterScheduler: Lost executor Akka client disassociated

Posted by Muhammad Ahsan <mu...@gmail.com>.
----------------------------------------------
Code
----------------------------------------------
scala> import org.apache.spark.SparkContext._
import org.apache.spark.SparkContext._

scala> import org.apache.spark.rdd.RDD
import org.apache.spark.rdd.RDD

scala> import org.apache.spark.sql.SchemaRDD
import org.apache.spark.sql.SchemaRDD

scala> import org.apache.spark.sql.hive.HiveContext
import org.apache.spark.sql.hive.HiveContext

scala> import org.apache.spark.{SparkConf, SparkContext}
import org.apache.spark.{SparkConf, SparkContext}

scala> val hiveContext: HiveContext = new HiveContext(sc)
hiveContext: org.apache.spark.sql.hive.HiveContext =
org.apache.spark.sql.hive.HiveContext@2de76244

scala> val numDays = 2
numDays: Int = 2

scala> case class Click(
/* about 20 fields of type STRING */
)
defined class Click

scala> val inputRDD = new Array[SchemaRDD](numDays)
inputRDD: Array[org.apache.spark.sql.SchemaRDD] = Array(null, null)

scala> for (i <- 1 to numDays) {
     |             if (i < 10) {
     |                 inputRDD(i - 1) =
hiveContext.parquetFile("hdfs://................" + i)
     |             } else {
     |                 inputRDD(i - 1) =
hiveContext.parquetFile("hdfs://................" + i)
     |             }
     | 
     |         }
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
details.

scala>         var unionRDD = inputRDD(1)
unionRDD: org.apache.spark.sql.SchemaRDD = 
SchemaRDD[1] at RDD at SchemaRDD.scala:104

scala>         for (i <- 1 to inputRDD.length - 1) {
     |             unionRDD = unionRDD.unionAll(inputRDD(i))
     |         }

scala> val inputRDD = unionRDD
inputRDD: org.apache.spark.sql.SchemaRDD = 
SchemaRDD[2] at RDD at SchemaRDD.scala:104
scala> 

scala> inputRDD.registerTempTable("urlInfo")

scala>         val clickstreamRDD = hiveContext.sql("select * from urlInfo "
+
     |             "where guid regexp '^[0-9a-f-]{36}$' " +
     |             "AND ((callerid  > 3 AND callerid <10000) OR callerid >
100000 " +
     |             "OR (callerid    =3 AND browsertype = 'IE')) " +
     |             "AND countrycode regexp '^[A-Z]{2}$'")
clickstreamRDD: org.apache.spark.sql.SchemaRDD = 
SchemaRDD[3] at RDD at SchemaRDD.scala:104
scala> 

scala>         clickstreamRDD.registerTempTable("clickstream")

scala>         clickstreamRDD.cache()
res4: clickstreamRDD.type = 
SchemaRDD[3] at RDD at SchemaRDD.scala:104

scala>     val guidClickRDD = clickstreamRDD.map(row =>
(row(7).asInstanceOf[String], {
     |             val value = Click(row(0).asInstanceOf[String],
     |                 row(1).asInstanceOf[String],
row(2).asInstanceOf[String],
     |                 row(3).asInstanceOf[String],
row(4).asInstanceOf[String],
     |                 row(5).asInstanceOf[String],
row(6).asInstanceOf[String],
     |                 row(7).asInstanceOf[String],
row(8).asInstanceOf[String],
     |                 row(9).asInstanceOf[String],
row(10).asInstanceOf[String],
     |                 row(11).asInstanceOf[String],
row(12).asInstanceOf[String],
     |                 row(13).asInstanceOf[String],
row(14).asInstanceOf[String],
     |                 row(15).asInstanceOf[String],
row(16).asInstanceOf[String],
     |                 row(17).asInstanceOf[String],
row(18).asInstanceOf[String],
     |                 row(19).asInstanceOf[String])
     |             value
     |         }))
guidClickRDD: org.apache.spark.rdd.RDD[(String, Click)] = MappedRDD[14] at
map at <console>:25

scala> val blackList: RDD[(String, Click)] =
guidClickRDD.groupByKey().filter(row => row._2.size == 1).map(row =>
     |             (row._1.asInstanceOf[String], Click("", "", "", "", "",
"", "", "", "", "", "", "", "", "", "", "", "", "", "", "")))
blackList: org.apache.spark.rdd.RDD[(String, Click)] = MappedRDD[27] at map
at <console>:27

scala>         val guidClickFRDD = guidClickRDD.subtractByKey(blackList)
guidClickFRDD: org.apache.spark.rdd.RDD[(String, Click)] = SubtractedRDD[28]
at subtractByKey at <console>:29

scala> guidClickFRDD.reduceByKey((x, y) => {
     |             /* commutative and associative function */
     |             Click("US", "US", "US", "US", "US", "US", "US", "US",
"US", "US", "US", "US", "US", "US", "US", "US", "US", "US", "US", "US")
     | }).take(200).foreach(println)

----------------------------------------------
EXPECTED OUTPUT
----------------------------------------------

(Key_A,Click(US,US,US,US,US,US,US,US,US,US,US,US,US,US,US,US,US,US,US,US,US))
(Key_B,Click(US,US,US,US,US,US,US,US,US,US,US,US,US,US,US,US,US,US,US,US,US))
(Key_C,Click(US,US,US,US,US,US,US,US,US,US,US,US,US,US,US,US,US,US,US,US,US))
(Key_D,Click(US,US,US,US,US,US,US,US,US,US,US,US,US,US,US,US,US,US,US,US,US))



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/ERROR-YarnClientClusterScheduler-Lost-executor-Akka-client-disassociated-tp20625p20626.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org